search

XENON1T / processing

Processing scripts for XENON data
Apache License 2.0
0 stars 4 forks source link

Fix for use of tensor flow #54

Open jpienaar13 opened 6 years ago

jpienaar13 commented 6 years ago

Changes to OSG-sites in order for MC processing to work with TensorFlow

briedel commented 6 years ago

This does decrease the pool significantly. I don't recall: How is the tensorflow step separate from the rest? We should be able to run with a different requirements expression for that step. @sthapa ?

sthapa commented 6 years ago

It might be worth it to separate out the steps a bit. Run the MC stuff with one set of requirements and then to do the tensorflow with other requirements. I'll check the pegasus docs to see if this is possible.

The biggest change would be to update mc_process to specify the outputs from the steps prior to tensorflow and then the tensorflow input steps. I can modify the mc_process script to do this.

sthapa commented 6 years ago

Checked and it looks like we can use per job requirements and have different steps for the tensorflow jobs. Just need to determine files that need to be copied as input for that and update the mc_process.py to transfer the files and set the job requirements.

pdeperio commented 6 years ago

This is not so urgent now that we've merged https://github.com/XENON1T/hax/pull/205 (we're not actually using TF yet in the main analysis).

Also, the stage (data reduction) that uses TF is quite fast, so still not sure it's worth copying the whole (processed/reconstructed) file per job.

So I think we can wait for sites to upgrade or some fix for TF compatibility.