Large FASTQs causer server disconnect

[jecorn]

I'm using the docker container of CRISPRAnalyzeR with the overlayfs filesystem (so no limit on devicesize) and a huge upload limit (max_upload=12096). I can upload fastq.gz's ~ 1 GB in size successfully, but files ~ 4 GB and greater (e.g. background samples) often go through upload, the progress bar maxes out, but then the server hangs and disconnects (message "disconnected from server"). The server can be reconnected, but the data is gone. So far I haven't been able to upload a full set of fastqs, since I'm trying to compare several samples of sizes 4.3 GB, 3.9 GB, 1.3 GB, 1.2 GB.

[jwinter6]

Hi Jacob, I remember that you used larger files in the past, did it work back then? Could you please tell me also which version you are running?

Moreover, when you start the docker file, could you also please run it with verbose_logfiles=TRUE, upload the data and once it gets disconnect, just open the page again and then go to About CRISPRAnalyzeR and send me the log files to jan.winter@dkfz-heidelberg.de?

The largest file sizes I was able to check from our data was 1.5 GB - and this went through. I am really eager to solve this, and I will prepare some "test" sample data of 4.3 GB to check the upload process the upcoming weekend.

Thanks for your help,

Jan

[jwinter6]

hi Jacob,

I think I found the reason, though for me it is not crashing but showing an error within the upload progress bar stating Bad Gateway. As I have introduced load-balancing to circumvent the shiny-server open source restriction in the last version, I need a couple of days to rule this out.

In the meantime I would be happy to get some additional logs if possible.

best Jan

[jwinter6]

Hi jacob,

I still try to figure out how to get shinys HTML5 engine to accept more than 4GB individual file size and will coem back asap.

Best Jan