NCAS-CMS / cfstore

A CF compliant data store with interfaces to local and remote storage.
MIT License
2 stars 1 forks source link

Restore the vertical table #110

Open bnlawrence opened 1 year ago

bnlawrence commented 1 year ago

With the move from sqlalchemy to django orm we lost the use of vertical tables and replaced them with json dictionaries (for variable properties). This is leading to scalability problems, and the obvious solution, using standard tables, will be very costly in terms of memory insofar as what should be a sparse matrix will be stored as full matrix but with most elements empty.

We need to work out how to nicely implement via the django ORM, vertical tables.

bnlawrence commented 1 year ago

(Possible easy option: store all the properties in another database and continue to use sqlalchemy for that database alone.)