search

RNAcentral / rnacentral-webcode

RNAcentral website source code
https://rnacentral.org
Apache License 2.0
31 stars 8 forks source link

Problem downloading large files with text search results #558

Open AntonPetrov opened 3 years ago

AntonPetrov commented 3 years ago

One of the users experienced a problem downloading a ~200Mb fasta file (389,619 sequences) with text search results:

RNA* AND TAXONOMY:"9606" AND so_rna_type_name:"LncRNA" AND qc_warning_found:"False" AND has_genomic_coordinates:"True"

https://rnacentral.org/export/results?job=6bf68ffd-e38f-403d-bccb-d5e0eef45dd3

The connection would either drop or the file would appear to be downloaded but would not uncompress because of an unexpected end of file (similar to this issue).

I ran the following command to download the file locally but the result was the same:

kubectl cp prod/rnacentral-<pod-id>:/srv/rnacentral/export/6bf68ffd-e38f-403d-bccb-d5e0eef45dd3.fasta.gz lncrnas.fa.gz

Then I ran the same command on the EBI cluster. It worked almost instantly and the file was fine.

This problem must have something to do with network timing out when the connection is not very fast or with the way the files are streamed by the django export app. Alternatively, it could have something to do with the load balancer or some k8s configs.

@carlosribas please don't look at this until you are back from 🌴!