search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Massive refactor to remove session dependency #274

Open rknop opened 1 month ago

rknop commented 1 month ago

Dealing with SQLAlchmey sessions is an ongoing nightmare. It keeps coming back to bite us, in various ways.

Since it's not an option to keep session open for a long time, we can't use SQLAlchemy the way it wants to be used. We will make life easier on ourselves if we use it less.

The paradigm: objects will no longer be attached to sessions. When they need to be saved to the database, we will explicitly save them, attaching them to a session just long enough to save them to the database. (The session will then be closed after the saving is done.)

One consequence: we will no longer be able to rely on SQLAlchemy lazy loading. So, for instance, the "Image.source_list" attribute will have to be modified, as it won't work as an SQLAlchmey relationship. (There are lots of cases like this.) One thing we could do is implement manual lazy loading ourselves, but I don't think that's a good idea; I think we should elimiate the image.source_list (and image.psf, and image.wcs, and image.upstream_images, etc.) fields altogether. Reason: if we don't have some concept like a session, we will multiply load objects, which will lead to a lot of gratuitous memory use. The solution is to have one place to go to look for all those things, and that is the DataStore.

This will require a lot of refactoring, but I will probably start it later this week, because I"m finding it extremely painful and difficult to work with this code base as it is right now.

guynir42 commented 1 month ago

Please don't start on this until we've talked it through. I may have ideas on how to do this exactly.