Closed jimmymathews closed 2 months ago
As part of this issue, research the most sensible method of delivering this signal. Of course, it should not be triggered by an exposed API endpoint, since the API server is public. Probably there is some relatively simple mechanism that is available for kubernetes pods.
This is obsolete.
The ondemand containers (counts service, squidpy metrics service, etc.) currently load up their in-memory data stores on initialization. It takes about 5 minutes for our current list of ~6 datasets.
However, during those 5 minutes, these services are not available. Moreover, if datasets are added to the database incrementally over time, a lot of computational effort is wasted re-loading all datasets when only one dataset (the new one) needs to be loaded.
For this issue, implement a signal that can be received by the ondemand container TCP server that causes it to intelligently update its data store. That is, it should fork a background task to add to its cache any new datasets it finds in the database, while leaving the other datasets in memory as they are. To force an update of a specific already-existing dataset, perhaps a study name parameter can be supplied.