Initialize `MongoDownloader` only once to limit unnecessary connection to DB

dachengx commented 2 weeks ago

Before you submit this PR: make sure to put all operations-related information in a wiki-note, a PR should be about code and is publicly accessible

What does the code in this PR do / what does it improve?

Started from slack thread, we need to find a way to limit connections to DB.

This line controls the connection to DB: https://github.com/XENONnT/straxen/blob/b8fe520deca55b8c5ac0cb53249b38639f0ae5de/straxen/common.py#L225. Since in fuse we used a lot of resource protocol to download maps: https://github.com/XENONnT/straxen/blob/b8fe520deca55b8c5ac0cb53249b38639f0ae5de/straxen/config/protocols.py#L47, each calling of resource protocol will trigger a connection.

This PR checks if the cached file exist BEFORE build the connection to DB: caching downloader.list_files() in line https://github.com/XENONnT/straxen/blob/b8fe520deca55b8c5ac0cb53249b38639f0ae5de/straxen/common.py#L226C17-L226C40 for future reference so that you will not need to initialize straxen.MongoDownloader each time you call resource protocol.

Can you briefly describe how it works?

Put mongo_downloader = mongo_storage.MongoDownloader() and mongo_downloader_files = mongo_downloader.list_files() in straxen/storage/__init__.py and replace downloader = straxen.MongoDownloader() with mongo_downloader.

Can you give a minimal working example (or illustrate with a figure)?

Please include the following if applicable:

[ ] Update the docstring(s)
[ ] Update the documentation
[ ] Tests to check the (new) code is working as desired.
[ ] Does it solve one of the open issues on github?

Notes on testing

Until the automated tests pass, please mark the PR as a draft.
On the XENONnT fork we test with database access, on private forks there is no database access for security considerations.

All italic comments can be removed from this template.

dachengx commented 1 week ago

Not a good approach as the failed test suggested.

dachengx commented 1 week ago

Replaced by https://github.com/XENONnT/straxen/pull/1398

XENONnT / straxen