transientskp / tkp

A transients-discovery pipeline for astronomical image-based surveys
http://docs.transientskp.org/
BSD 2-Clause "Simplified" License
19 stars 15 forks source link

TraP slowdown when image store is enabled #538

Open gijzelaerr opened 8 years ago

gijzelaerr commented 8 years ago

Alexander his team indicated that TraP 4.0rc1 becomes quite slow when image store is enabled. When disabled TraP is super fast. This is because the images are stored in the database in serial (that is how SQL works) and not in parallel anymore (MongoDB was filesystem based). So question now is, is the slowdown too much or not. @AntoniaR indicated she doesn't need the image store, so for her this shouldn't be a problem.

If the slowdown is too much, we have various options:

I prefer to leave it just the way it is, since it makes it much easier to manage the data and it requires less services so simplifies TraP.

AntoniaR commented 8 years ago

Clarification, I said that I stored images in other ways. Some kind of intelligent image store will be needed for multiple datasets, not just AARTFAAC, and I had basically proposed your first option when we discussed this - i.e. an image store separate to TraP but that Banana accesses.

So I think working round it for now is ok but we will need to quantify the slowdown and work round it better in the longer term.

gijzelaerr commented 8 years ago

point one is a logistics issue, not a software issue. I think how that turns out the AARTFAAC team has to decide when things get more a final form. I propose we just leave it for now as is, storing images this way is for debugging or experimenting.

AntoniaR commented 2 months ago

Image storage is not scalable for the long term use of the TraP. This option should be completely removed for R7.