search

clemente-lab / mmeds-meta

A database for storing and analyzing omics data
https://mmeds.org
2 stars 1 forks source link

Long uploads may error out #375

Open adamcantor22 opened 2 years ago

adamcantor22 commented 2 years ago

Describe the bug I attempted to complete a 2-hour long upload, at the end of which I received an error. The error stated that there was no 'subject' file available in cp.session. This likely means that over the course of the upload, the data from the previous steps was cleared, causing the upload to not be complete once it had finally finished. In a MMEDS production scenario, someone with low access to internet could find themselves in a situation where they are unable to use MMEDS uploads due to internet speeds.

To Reproduce Difficult. The reason for my long upload was that I was uploading ~55 GB over the Mount Sinai VPN, with an upload rate of ~9 MB/s. Uploading a large amount of data over that VPN would be the most reliable way to reproduce.

Expected behavior Data should not time out while upload is commencing.

Additional context Not easy to get other information on this, and this is based on only one occurrence. Consider this tentative, and liable to change.

adamcantor22 commented 2 years ago

Happened again, with the same ~55 GBs, in a much shorter timespan, ~25 minutes. Error:

Screenshot from 2022-01-25 13-04-30

adamcantor22 commented 2 years ago

More Additional Context Up until now, it seems that most uploads have maxed out at 5 or 6 GBs. It is understandable that 55 GB, an order of magnitude larger, presents difficulty for us.

adamcantor22 commented 2 years ago

Upload success when brought down to ~28 gigs, ~14 minutes.

adamcantor22 commented 2 years ago

Linking with issue #219