search

icecube / flarestack

Unbinned likelihood analysis code for astroparticle physics datasets
https://flarestack.readthedocs.io/en/latest/?badge=latest
MIT License
8 stars 7 forks source link

Use `HTCondor` python bindings #304

Open JannisNe opened 1 year ago

JannisNe commented 1 year ago

There are actually python bindings for HTCondor clusters: link. That means a lot of the stuff in the HTCondorSubmitter class could use that, which would make everything more robust, probably. Just leaving this here as an idea!

mlincett commented 1 year ago

I have seen this used in some other projects but I have mixed feelings about it.

I like having a static and declarative job description in the form of a plaintext submit file, that gives you an outline of the job(s) at a glance.

Replacing this with something procedural could make the whole thing a bit opaque and harder to debug, but at the same time allows for more sanity checks (i.e. does $INPUT_FILE exist or not) at runtime.

Still, I am in favour of it :)

JannisNe commented 12 months ago

In general I think flarestack's behaviour of submitting jobs automatically is not optimal. I think the submission to the cluster should always happen manually, i.e. flarestack should only create the submission file. That would also remove the need for the flarestack process to hang around until all jobs have completed.