search

midas-network / covid19-scenario-modeling-hub

COVID-19 Scenario Modeling Hub
63 stars 46 forks source link

File size issue when greater than 25MB #2

Closed JohnLevander closed 3 years ago

JohnLevander commented 3 years ago

scc-usc commented 14 hours ago Hi,

I tried to upload my forecasts today via the web but realized that there is a limit of 25MB per file. My file size is 75MB, and as I recall from the last week, someone else had also mentioned that their file size was 75MB. Please let me know if a decision has been made on file size reduction. In case you are considering smaller files, here are some quick ideas to consider:

Reduce the number of quantiles to at most one-third Split the file by scenario Thanks, Ajitesh

mattkinsey commented 3 years ago

Just a quick suggestion, gzipping the files greatly reduces their size b/c there's a lot of redundant text in every column but 'value' (and even then the ascii itself is compressible). I got slightly better than a 5x reduction on the submission we did to the forecast-hub this week at the default compression level:

[matt@krang]$ gzip -k 2021*; du -sh 2021*
25M     2021-01-04-JHUAPL-Bucky.csv
4.3M    2021-01-04-JHUAPL-Bucky.csv.gz

And as an added bonus pandas.read_csv will read the gzipped csv just fine without any extra flags; if that's being used to postprocess these.

scc-usc commented 3 years ago

For the benefit of others -- instead of trying to upload via web, I ended up cloning my forked repo, adding the file to it, and then submitting the pull request from there. If this is the intended way, perhaps it will be helpful to add the details in the read me for those who are new to Git.