search

iobis / env-data

ENV-DATA related issues and documentation
2 stars 0 forks source link

eMoF files getting too large? #7

Open albenson-usgs opened 7 years ago

albenson-usgs commented 7 years ago

Has there been any discussion about eMoF files becoming too large to process? I'm working on a dataset now that I'm only including three measurements (temperature where gear was deployed, fish length frequency, and weight) and already has 1,858,402 rows. I could add in weather, wind speed, water transparency, sea condition, wind direction, surface temperature but I'm afraid the file will get too big to load into IPT and/or for my computer to be able to create it. Is it appropriate for me to pick the measurements that I think would be most useful? Create two eMoF files?

Daphnisd commented 7 years ago

You can zip the eMoF file before uploading it to IPT, which will reduce it's size. A emof txt file of 1,4 million records (=250 mb), becomes zipped only 9 mb. IPT allows uploading zipped files. I think it will be rare if it's still too big after that, but in those cases splitting into 2 emof files before uploading would help.

pieterprovoost commented 7 years ago

@albenson-usgs Are you repeating environmental measurements for every occurrence? If so, you may want to create parent events and link the environmental data there.

Let me know if you have any technical issues related to file size. I would not leave any measurements out if they are available.

albenson-usgs commented 7 years ago

@pieterprovoost No, I'm not repeating per occurrence. Only per event.