pinskylab / OceanAdapt

Scripts and data for OceanAdapt website to visualize shifts in marine animal distributions
http://oceanadapt.rutgers.edu
MIT License
13 stars 6 forks source link

Dat_exploded csv files are too large for GitHub #147

Open df511 opened 3 years ago

df511 commented 3 years ago

dat_exploded gained a considerable amount of data this year, as we are using unfiltered data for map-making and added three new regions and a new year's data. For this reason, the files needed to be parsed to by region to be exported. However, even these regional files are too large as gzipped CSVs (csv.gz) (many of these files are >100 MB, GitHub's file size limit).

For this reason, this year's data_exploded files were exported as .rds

The Gulf of Alaska dat_exploded (dat_explodedGulf of Alaska.rds.zip) needed to be additionally compressed to be just under 100 MB.

We need to consider other compression and/or parsing options for next year.