search

XENONnT / outsource

Job submission of reprocessing
4 stars 2 forks source link

Upload cached corrections and to-download info #151

Open FaroutYLq opened 5 months ago

FaroutYLq commented 5 months ago

Triggered by discussion here. The concern is currently outsource is occupying too many DB connection, endangering DB service. We need to deeply understand where DB is called first.

FaroutYLq commented 5 months ago

Assuming we have to live with 5k cap, we basically can only have 30% at most for our full computational power.

FaroutYLq commented 5 months ago

We observed >8 connection for a typical already before making. Something need to be improved here, which is basically initialized and not really used until we call related DB operation.

rsriya commented 1 month ago

Isn't this issue solved by this PR - https://github.com/XENONnT/utilix/pull/122?

FaroutYLq commented 1 month ago

Isn't this issue solved by this PR - XENONnT/utilix#122?

@rsriya I am not sure it fully solves the story. I would say at least it helps.

dachengx commented 1 month ago

Isn't this issue solved by this PR - XENONnT/utilix#122?

It is not totally solved by that. But it can help a bit.

dachengx commented 1 month ago

We observed >8 connection for a typical already before making. Something need to be improved here, which is basically initialized and not really used until we call related DB operation.

This PR will reduce the connection: https://github.com/XENONnT/admix/pull/59.

But I am worrying about the too-large discrepancy between the master and stable branch of admix, which might result in another round of cleaning after we merge them.

dachengx commented 3 weeks ago

I marked this PR as wontfix because the less DB connection after https://github.com/XENONnT/admix/pull/59 is expected.

And it could be dangerous if we download and upload the corrections maps.