cal-adapt / climakitae

A Python toolkit for retrieving, visualizing, and performing scientific analyses with data from the Cal-Adapt Analytics Engine.
https://climakitae.readthedocs.io
BSD 3-Clause "New" or "Revised" License
19 stars 2 forks source link

CSV file export size estimate and disk space protection #350

Closed elehmer closed 2 months ago

elehmer commented 3 months ago

Description of PR

This PR adds a CSV file size estimate and helps protect file system from filling up. The file estimate is just a 1st approximation using 150 characters per line as a guess. It also removes the 0.85GB CSV file limit (let users decide if the file is too big for their use). We still warn of larger than Excel can handle and warning that large CSV export will take awhile.

Summary of changes and related issue

https://trello.com/c/5J8YQ2HI/569-ability-to-estimate-manage-file-sizes-csv

Relevant motivation and context

Protect HUB file system from filling up from CSV export.

Dependencies required for this change?

Fixes # (issue), delete if not necessary

Type of change

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.

Checklist:

mdkoenig commented 3 months ago

Ability to ~estimate~ manage file sizes (CSV)

claalmve commented 2 months ago

Looks good to me! I tested on NetCDF and CSV exports of data and they all work well. Warnings are definitely helpful for users to consider subsetting datasets before exporting.

I made a couple comments but nothing critical.