search

cms-gem-daq-project / gem-plotting-tools

Repository for GEM commissioning plotting tools
GNU General Public License v3.0
1 stars 26 forks source link

Update clusterAnaScurve.py for new computing environment. #95

Closed bdorney closed 6 years ago

bdorney commented 6 years ago

Brief summary of issue

Since gem-plotting-tools has been updated to be more of a package that is installed or downloaded as a standalone release the clusterAnaScurve.py tool no longer properly sets up the computing env on the worker node of lxplus.

Additionally CERN IT is moving lxplus away from using the scheduler bsub to moving HTCondor:

http://batchdocs.web.cern.ch/batchdocs/

Now might be a good time to migrate to HTCondor.

Expected Behavior

The worker node on lxplus should:

  1. Download a given release and all supporting software,
  2. Unpack it and install it,
  3. Set the env
  4. Launch the analysis
  5. Return the results

You may have to adapt this procedure to instead download all tarballs/zip files, and then send them to the worker node instead of having the worker node do this.

Presently right now the worker node is using AFS directories this typically is "frowned" upon and angry emails may be sent from cluster administers. Perhaps we should alter this.

Current Behavior

The computing env setup by clusterAnaScurve.py is no longer compatible with how the package is installed as either a rpm, zip, or tarball. Specifically the following lines need to be updated:

https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/dd3c00f47844ac620279ac57e6dd1339d188583b/macros/clusterAnaScurve.py#L150-L163

Additionally the example shown in the README.txt needs to be updated: https://github.com/cms-gem-daq-project/gem-plotting-tools/#full-example-for-p5-s-curve-data

Context (for feature requests)

Need cluster computing abilities to re-analyze all s-curve data taken in P5.

bdorney commented 6 years ago

Partially related to:

https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/94

In that the documentation for using clusterAnaScurve.py may need to be updated.

bdorney commented 6 years ago

Support for the lxplus batch submission system was added by #105. Given limited resources let's close this for now; it can be reopened later if HTCondor goes live.