alignment workflow

[tomeichlersmith]

This resolves #342

I have written up a new component PEDE which writes a pede steering file and runs it.

To Do

• [x] document how to define location of pede if it isn't in PATH
• [x] document optional parameters in job.json
• [x] check that MillepedeII cmake script works
• [x] add a Apply component which applies the millepede.res to a detector compact.xml

[tomeichlersmith]

@cbravo135 @sarahgaiser @pbutti Could y'all look at this and see if it makes sense? I haven't tested it much but I'm curious if I'm using hps-mc as intended.

[tknelson]

@cbravo135 @sarahgaiser @pbutti Could y'all look at this and see if it makes sense? I haven't tested it much but I'm curious if I'm using hps-mc as intended.

Oh... did @pbutti actually find this? 😁

[tomeichlersmith]

Setup at SDF

• Environment script: https://github.com/tomeichlersmith/hps-env/blob/main/sdf-bare-metal.sh
• hpsmc config: https://github.com/tomeichlersmith/hps/blob/main/hpsmc
• java on HEAD of master and built separately from hps-mc

Configure hps-mc:

mkdir build
cmake -DHPSMC_ENABLE_PEDE=ON ..
make install

Only need to enable pede if pede is not already in your PATH.

Running at SDF

Get list of files we want to use for alignment. Here I'm just using ideal condition, MC FEEs as an example.

ls /sdf/group/hps/mc/4pt55GeV/fee/idealCond/fee_recon_20um120nA*slcio > 2019-fee-mc-ideal-conditions.list

Copy over the example from this branch of hps-mc to where you are working

cp -r /sdf/group/hps/mc/examples/alignment/* .

Generate the full list of jobs from the template and the list of files

hps-mc-job-template -j 1 -a tracking/vars.json \
  -i events 2019-fee-mc-ideal-conditions.list 1 \
  tracking/job.json.templ accum_jobs.json

Run the tracking+accumulation step with hps-java. Here I am just running one job, but you could run all 200 with hps-mc-batch slurm.

hps-mc-job run -d /scratch/eichl008/hps-mc java accum_jobs.json -i 1

Edit the pede/job.json file to include the output .bin file from the previous command. Run the alignment minimizer job which includes applying the deduced parameter, constructing the new detector iteration, and merging the GBL histogram files it was able to find automatically.

hps-mc-job run \
  -c pede/.hpsmc \ # extra configuration for pede job
  -d /scratch/eichl008/hps-mc \ # choose scratch directory
  pede pede/job.json

[tomeichlersmith]

Batch Running Update

I think I got batch running over tracking to work. Needed to provide several options to hps-mc-batch:

hps-mc-batch slurm \
  --log-dir $PWD/batch/log/
  --sh-dir batch/sh/ \
  --env /sdf/group/hps/users/eichl008/hps/env/sdf-bare-metal.sh \
  --memory 5000 \
  --run-dir /scratch/eichl008/hps-mc-batch \
  java accum_jobs.json

[cbravo135]

It has been requested that we keep track of the history of a detector in a README file in the detector directory. Can you add a component that updates/appends (or creates if it doesn't exist) the history of the detector names along with the to_float list and the first bin file name? We can add some extra notes too, but this info is most of what we need to make the detectors reproducible.

[tomeichlersmith]

Alright, I've started those updated you've mentioned @cbravo135 - still need to test that they work as intended.

[tomeichlersmith]

Alright, I've tested the defining the parameter name and appending a log of the detector name and the parameters that were floated to the file README.md in the detector directory. Both are working on a simple pede run so that's good :heavy_check_mark:

[tomeichlersmith]

Make example run out of the box by putting example files at JLab and then have hps-mc wget them.

[tomeichlersmith]

More To Do

• [x] Make example run out of the box by putting example files at JLab so that hps-mc can wget them
• [x] Update README.md writer to dump parameters in JSON format so they are easier to read if there are a lot of them

[cbravo135]

Apply is currently not updating the detector name in the compact.xml file when generating the new detector. I am seeing compact.xml.prev being made, and a diff with the new compact.xml shows no changes to compact.xml.

[tomeichlersmith]

@cbravo135 I updated the _apply.py module to write the detector name as well as the updated parameters.

https://github.com/JeffersonLab/hps-mc/pull/344/commits/7ae07a87be54a774fa3a853610e8054b6c0869df

Please pull and try again.

[cbravo135]

Pulled and tried again. The detector names are now written correctly. The old lcdd file still exists in the directory though, better to remove the old one. Also, the new compact.xml does not have updated values for the floated parameters, only line in a diff of compact.xml and compact.xml.prev is the detector name line.

[tomeichlersmith]

Ah! Found the bug, I was not testing correctly. Basically a small typo leading to using the bound method of a class rather than what that method would return - please pull and try again.

[tomeichlersmith]

Alright, I have patched the issues with the templating I was doing (TLDR: the variable-substituion in hps-java does not allow variable names to have numerical digits in them). Both the steering files in the examples directory use the templating as intended. It does not include all of the parameters of SimpleGblAliDriver but it is a good start and would be able to easily be expanded.

Known Outstanding Issues (still)

• [x] rebase/merge so that I can update this branch to use the logging from #354
• [ ] discuss with @cbravo135 and @pbutti about what the steering file should be (i.e. how should the ali driver be configured? what cuts should be defaults? how should the plots be filled?)
  • The examples/alignment/tracking/tracking_kf_alignment.lcsim is the steering file I will be working with.