U/jrbogart/minimize mem

LSSTDESC / skyCatalogs

Create sky catalogs and provide access via API

BSD 3-Clause "New" or "Revised" License

6 stars 5 forks source link

U/jrbogart/minimize mem #32

Closed JoanneBogart closed 1 year ago

JoanneBogart commented 1 year ago

Read data from parquet a row group at a time to avoid memory spikes

JoanneBogart commented 1 year ago

The sort of test you describe is what I would have done next if the issue of different numbers hadn't been resolved. I agree it's worth having around and will add it in a future PR.