DQM Data Pipeline

[JeremyMcCormick]

Fully automate creation of the DQM files.

Data Pipeline

• Scan for new files (either use timestamp file or a database to tell which files have already been processed)
• For new files, run the recon on the EVIO and generate the DQM plots by submitting an Auger batch job (or do we have other machines that we can dedicate to this and run locally?)
• Combine the DQM outputs into a single ROOT file (this may need to be done multiple times as new files show up)
• Copy the ROOT output to the dqm-web/files drop dir

Database

• run number
• file sequence number
• EVIO file name/path
• recon file path (update after generating LCIO)
• number of events sampled
• DQM file path (update after generating ROOT)
• DQM file aggregated ? (update after generating combined ROOT)

We can use the existing DQM database for this table.

[JeremyMcCormick]

Python tools are being developed in this project.

https://github.com/jeremymccormick/hps-python-tools

Job submission to run the recon and DQM in the same job is working.

[JeremyMcCormick]

This is basically done (code is not in hps-java).