Large file uploads fail on BioAcoustica

[edwbaker]

Seem to stall after around 70MB.

[jrdh]

Seems to be ok to me, perhaps it was an intermittent network issue? Feel free to reopen if you encounter it again.

[edwbaker]

Nope - still an issue on the Recording node type.

[jrdh]

Strange. I just tried this a few times and it worked ok. I'll try again later off-site to see that changes anything.

[edwbaker]

Quite possibly. I have had success within the NHM but not from outside. I can only get a few (usually about 2-3) Mbps upload to the server which also seems odd.

[jrdh]

Just tried this again on my phone's connection and it worked fine again. The connection was slow but it worked. When I have a decent connection it zips along quite happily so there doesn't seem to be any issues with connection speed generally. What browser are you using?

[edwbaker]

I've tried Safari, Firefox and Chrome. All fail.

How big a file are you trying?

[jrdh]

I just downloaded the file associated with this recording: http://bio.acousti.ca/content/p-msd2 and then reuploaded with a different name into a new test recording, so about ~280MB.

[edwbaker]

Yeah, I can't do that. Previously I could get up to 2GB. (We don't have many files that size, but it's the standard length of some handheld recorders so is a standard length for some eco acoustics research).

[jrdh]

I'll contact TS to see if there have been any changes recently that would be causing a speed drop off. We haven't done anything with the server or docker images so any issues are probably in the network or somewhere outside of the machine that runs bioacoustica. In terms of the rate limiting, are you getting a 429 status code result when it fails?

[jrdh]

Additionally, I just tried a curl-ing a file from within the museum and I got the max rate on our wired base 100 network of ~10MB/s whilst attempting the 2GB download. It did fail though so I'll see if I can find any log information about why it got chopped.

➜  ~ curl http://bio.acousti.ca/sites/default/files/DR0000_0028.wav > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 36 2046M   36  748M    0     0  10.9M      0  0:03:06  0:01:08  0:01:58 10.9M
curl: (18) transfer closed with 1362022443 bytes remaining to read

[jrdh]

Furthermore, I just checked the VCL for bioacoustica and the rate-limiting block that exists on the normal scratchpads sites doesn't exist so I don't think there's any rate-limiting deployed on bioacoustica.

[jrdh]

I think the curl: (18) transfer closed with 1362022443 bytes remaining to read issue I saw is due to the Varnish cache running out of space, this is from the varnish logs:

ExpKill        LRU x=3906794\n","stream":"stdout","time":"2020-03-12T11:01:26.751203588Z
ExpKill        LRU_Exhausted\n","stream":"stdout","time":"2020-03-12T11:01:26.751206896Z
FetchError     Could not get storage\n","stream":"stdout","time":"2020-03-12T11:01:26.751210206Z
BackendClose   31 boot.default\n","stream":"stdout","time":"2020-03-12T11:01:26.751213636Z

Could not get storage is the key bit, seems to be something that happens when varnish runs out of space in its LRU cache. I thought we had something in place to avoid caching large files but perhaps that is not working or not working in all cases. I'm looking into it now.

[edwbaker]

I've just tried to upload the standard 2GB file on the NHM wired network with Firefox - runs at full speed but fails with "The connection was reset".

[edwbaker]

That curl response is very familiar.

[therobyouknow]

Can you try this again please @edwbaker ? Thank you.

[edwbaker]

Two issues here have become conflated - one is a problem uploading large files, and the second is downloading them.

I have downloaded circa 10GB in five files successfully onto a NHM VM - so the download seems to be working. Although the initial issue involved downloading the entire dataset for analysis, which is considerably larger. I can run this process, omitting the analysis to check (if that would be useful).

[therobyouknow]

Good news on the success with the downloading. And uploading - how's that working for you?

[edwbaker]

Still testing - the site is very slow outside the NHM network

[therobyouknow]

Let me know if you'd like me to also try how your testing. My uplink is 150Mb/s which may avoid any standard timeouts that could mask how the site is working (or not working).