search

SCIInstitute / dSpaceX

dSpaceX Visualization Library and Tool
5 stars 2 forks source link

Is there a better way to save models #189

Open kylinmb opened 4 years ago

kylinmb commented 4 years ago

PCA models for a single field for meshes are really large (8+ gb) it may be difficult for a user to have multiple of these models on their system in the current storage format. Is there some other way to save the models that doesn't take so much storage?

cchriste commented 4 years ago

Models can now be read/written in binary, so they're a fraction of the size of the csv text files and 5-10x faster to load.

These files could be compressed to be even smaller on disk using Python zipfile or gzip. Then we can read them using zlib or libzip.

The models we currently provide are available as zip files that can be downloaded locally from /usr/sci/data/dSpaceX (soon to be moved to /usr/sci/projects/dSpaceX/data).