search

virusseq / portal-ui

Canadian VirusSeq Data Portal
https://virusseq-dataportal.ca/
GNU Affero General Public License v3.0
8 stars 8 forks source link

Data Release Page not Updating #254

Closed nithujohn closed 2 years ago

nithujohn commented 2 years ago

Hi @scottcain, it looks like the data release page is not updating for a while (since May 6th). Please have a look. Also the front page is partially updating. Thanks! image

scottcain commented 2 years ago

I just wrote a comment on https://github.com/cancogen-virus-seq/portal/issues/222 This is a critical bug that is being addressed now but unfortunately is not an easy fix.

b-f-chan commented 2 years ago

Assign to @UmmulkiramR to investigate - There is currently a limit to the download size due to one of the libraries we are using in the Singularity code, I think @joneubank first suggestion was to investigate that library and see if there is something else out there we can use (please connect with him for details)

ghost commented 2 years ago

Debugging guidelines by Jaser - https://wiki.oicr.on.ca/display/OV/Fix+latest+release+data+not+being+created

joneubank commented 2 years ago

Our dataset has grown to now be too large to build a single archive with the tools in our code base. We have theorized that the issue is based on the Java language limit on Array length (MAX_INTEGER which is 2^32 elements, approx 2Gb size limit) that is being overcome by our archive library when it converts the data to a byte array. This means that we will need to either find a new library that is not limited in this way, or we will need to stop building an archive of the full data set.

ghost commented 2 years ago

We have a temporary fix for this Provide isolated fasta downloading functionality #279 and Change Tar library header properties for unstuck bundling issues #39

We will move this to prod in day or two. Currently placing this ticket in backlog as we wait for a long term solution to this.

UmmulkiramR commented 2 years ago

@sifavahora Just to be clear - as we have discussed, only the isolated fasta downloading functionality https://github.com/cancogen-virus-seq/portal/issues/279 would be moved to prod.

Code changes to resolve https://github.com/cancogen-virus-seq/singularity/issues/39 would be moved to testing once you give a go-ahead.

ghost commented 2 years ago

We decided to use solution https://github.com/cancogen-virus-seq/singularity/issues/39 as that was the best. This bug is resolved. However we are planning to redesign the page, Will capture requirements in subsequent tickets. Closing this ticket.