search

zooniverse / decals

Morphological classifications for the Galaxy Zoo-DECaLS collaboration
MIT License
7 stars 3 forks source link

Chunk for performance on 300k+ size dataset #27

Closed mwalmsley closed 6 years ago

mwalmsley commented 6 years ago

Download is much reduced when acting on the full catalog. Manual chunking could fix this. This has been implemented but not yet investigated.

mwalmsley commented 6 years ago

This has been done, default chunk size 20k. Much faster - almost impossible to run without chunking, in fact.