Large CSV tables result in "Out of Memory" exceptions on server

PASTAplus / dex

Explore and subset CSV tables using associated EML metadata

Apache License 2.0

3 stars 0 forks source link

Large CSV tables result in "Out of Memory" exceptions on server #40

Closed servilla closed 2 years ago

servilla commented 2 years ago

Large CSV tables result in "Out of Memory" exceptions on the server. Specifically, this table in DeX loads into Pandas Profile, Subset, but results in an exception when attempting to view the plot.

The corresponding data package in the EDI Data Portal (staging) is: https://portal-s.edirepository.org/nis/mapbrowse?packageid=edi.143.11

rogerdahl commented 2 years ago

We've added a limit to the number of rows that can be processed in a single CSV to prevent out of memory exceptions. Now set to 1m. Can be adjusted as needed.

rogerdahl commented 2 years ago

Current limit works on number of cells and is set to 50m.