search

nel-lab / mesmerize-core

High level pandas-based API for batch analysis of Calcium Imaging data using CaImAn
Other
60 stars 15 forks source link

consider offering a MongoDB backend #187

Closed kushalkolar closed 12 months ago

kushalkolar commented 1 year ago

Still allow the pandas DataFrame backend, but also allow using a MongoDB backend to store everything except binary files.

Have a way to import and export between mesmerize pandas DataFrame and Mongo.

For caiman extensions we can just use pandas as a front end and query the mongodb. This should be possible, needs more thought.

Advantages: can work with or run multiple items simultaneously, don't have to worry about the DataFrame file being constantly overwritten. Can add multiple items simultaneously and keep working on then concurrently from different processes, such as realtime gridsearch during live experiments.

kushalkolar commented 1 year ago

Something that tracks changes, maybe using Change Streams: https://www.mongodb.com/basics/change-streams

kushalkolar commented 12 months ago

we're not doing this