Auto deploy script hangs

[Danp2]

Describe the bug XVA download doesn't occur and import is hung at 0%

To Reproduce Steps to reproduce the behavior:

bash -c "$(curl -s https://raw.githubusercontent.com/Jarli01/xenorchestra_installer/master/xoce-import.sh)"

Expected behavior Script to complete successfully

Screenshots This is the SSH console --

[08:43 xenserver-slnqfzrh ~]# bash -c "$(curl -s https://raw.githubusercontent.com/Jarli01/xenorchestra_installer/master/xoce-import.sh)"

Welcome to the XOCE auto-deploy script!

Network settings:
IP address? [dhcp]

Your XOCE will be started using DHCP

Importing XOCE VM…
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    82  100    82    0     0     75      0  0:00:01  0:00:01 --:--:--    75

Here's the task in XO --

Additional context I believe this is the same issue discussed earlier there the download source is doing some sort of redirect

[Jarli01]

Do you have a non-running VM with the same name under your VM list?

[Jarli01]

If you do, please delete it or rename it and try the import again. At this time I'm unable to reproduce, see the attached.

[Danp2]

No, a duplicate VM name wouldn't cause this issue. See this for proof of the redirect issue --

[08:54 xenserver-slnqfzrh ~]# curl https://srv-file8.gofile.io/download/CYaI0l/XOCE.xva --output test.file
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    82  100    82    0     0     74      0  0:00:01  0:00:01 --:--:--    74

[08:57 xenserver-slnqfzrh ~]# cat test.file
To download this file, please visit the download page : https://gofile.io/d/CYaI0l

[Jarli01]

Showing progress albeit slow (can't control the hosting provider)

[Jarli01]

Are you able to try and directly download the file with the direct xva link? I'm unable to produce this same issue in my lab across multiple test systems.

[Danp2]

Yes.... here are my steps --

1. From the browser, navigate to https://srv-file8.gofile.io/download/CYaI0l/XOCE.xva

Receive message To download this file, please visit the download page : https://gofile.io/d/CYaI0l

2. Navigate to https://gofile.io/d/CYaI0l and receive this webpage --

3. Navigate to original URL (https://srv-file8.gofile.io/download/CYaI0l/XOCE.xva) and now the download occurs

[Danp2]

Seems like they track IP addresses -- https://news.ycombinator.com/item?id=11982662

[Jarli01]

Hrm... I didn't encounter that splash page. What's really needed is a dedicated web host to serve up the file if that's the issue

[Danp2]

Makes sense because you were the one to upload the file. ;-)

[Danp2]

IMO, you are asking for trouble with the pre-built XVA because you are going to have too many uneducated people trying to deploy it, asking basic questions, wanting support, etc.

[Jarli01]

These people already exist, #79 as an example. I think the value outweighs the possible risk of additional trouble.

[Danp2]

Can you upload a small test file to this same site and provide the link? I want to test something on this end

[Jarli01]

One moment

[Jarli01]

Direct: https://srv-file16.gofile.io/download/Yl3tcr/sample.txt

Source: https://gofile.io/d/Yl3tcr

[Danp2]

Was trying to see if I could "trick" the site into automatically downloading the file without the redirection issue. No luck so far.

[Jarli01]

Hrm. . . this wouldn't be an issue if I had a website that could host these files without issue, sadly I don't. I wonder if there are any public hosting platforms like gofile that aren't stupid. . .

[Danp2]

Might have figured something out. Can you upload a small text file for testing?

[Jarli01]

Yeah,

Source: https://gofile.io/d/dtW1P8

Direct: https://srv-file10.gofile.io/download/dtW1P8/sample2.txt

[Danp2]

Ok... If I go here first https://srv-file10.gofile.io/getUpload?c=dtW1P8

then the download from https://srv-file10.gofile.io/download/dtW1P8/sample2.txt proceeds as expected

[Jarli01]

How about this link, https://nc.nl.tab.digital/s/cdBzESzStRkGFoF/download ?

[Danp2]

That appears to work without any issues. Seemed faster too.

[Jarli01]

Okay, I may get that to work, but I might need to gzip the file and decompress it on the fly as it as a file size limitation.

Let me test right quick, that link is going to stop working.

[Danp2]

FWIW, I have a few sites that I host on ASO. I could probably host the file there. Or what about hosting it on Github?

[Jarli01]

GH has a limit of ~125MB iirc for free accounts, if you'd be willing to host it that would be great. I'd do it myself if I had anything I was operating.

I'm also working up a smaller install to see if I can't get the xva any smaller.

[Danp2]

Just checked and I don't think ASO would work due to storage / bandwidth limitations.

[Jarli01]

Gotcha, well let me see what I can do to shrink this xva down, we may be able to get it to fit and work there. Alternatively maybe there is a community member here who would be willing to host for this

[Danp2]

Maybe someone over on ML would be willing to host it for you

[Jarli01]

Yeah that is what I was thinking about checking in on. Still would want to get the file size down as much as possible though.

[Danp2]

Alternatively, we could stick with gofile.io and just add an additional curl command to trick it into performing the download

[Jarli01]

I too was thinking about that ... Lol

[Danp2]

I don't think it would be too difficult. Break the unique info (server and file code) into variables and then used them in the subsequent curl statements

[Jarli01]

Just adding an extra curl at the top of the script would fix the issue (temporarily) and using the direct download from there on. The pain comes from making any updates to the xva (os changes, updates etc) that the file would need to get uploaded again and multiple links updated.

I think going the NC route may work out better in the long run.

[Danp2]

Right... that's why I was suggesting to break out the portions that change into variables so that you can update them and have the changes flow through to all associated links.

Yes... Moving to a different host would be a better long term fix.

[Jarli01]

I'm making a post on ML to see if anyone would be willing to offer up some space on their NC installs for this. Maybe

[Danp2]

I also have an O365 account. What about using OneDrive to host it?

[Jarli01]

That would be another option for sure too, so long as you're good with it. The file would simply need to be publicly available.

[Jarli01]

I'll setup a free account and see if I can get an XVA that'll fit/work.

[Danp2]

I've got 15GB on my personal OneDrive account. Then I've got my business account as well.

FWIW, I think you only get 5GB on the free account.

[Jarli01]

Yeah it is 5GB on the free account, not sure how small the new XVA is going to be. Will find out in a bit.

[Danp2]

You can get the file size down to around 3GB if you export with gzip compression

[Jarli01]

Yeah I have it smaller without the compression already, just trying to get it even smaller. It's at about 5.5GB without any compression, but I think I can get it to around 3 with a custom install (which I'm working on currently).

[Jarli01]

Please test with the updated XVA, it's not the smallest one I've made yet (that's still in progress) but it should at least work. Use the direct import option and tell me if it stalls out.

[Jarli01]

Closing this as this appears to be an issue with the web host I have available and this is outside of my control.