search

XENON1T / processing

Processing scripts for XENON data
Apache License 2.0
0 stars 4 forks source link

change fax_waveform to all python codes #40

Closed zhut19 closed 6 years ago

zhut19 commented 7 years ago

Translate old .sh codes all into .py so that every fax simulation step can be sub classes of a generic ProductionProcess class. Would require less work for changing individual features in each step.

Write all configuration parameters into .ini file, instead of passing them one by one as system arguments.

Dividing jobs submission up in a different style, instead of submitting each list of events and each task to a single job, here I combine all tasks of processing a list of events into one job and put those job into groups with equal number of jobs and submit each group to one batch job. By doing so, 1) Reduce number of batch job and prolong the run time for each one. 2) Reduce total run time by only import and initiate pax and hax once in each group 3) One can quickly see the result of the first few lists of event.

Issues: 1) Haven't add array processing feature from old fax_waveform in here. 2) The grouping method makes log file write into two parts, one from batch process one from each job. 3) Some issues needs hardcode fix, that importing local scripts pose a problem as running at remote nodes change {cwd} to a {tmp} folder 3) Having trouble batch produce using pax 6.6.5 while there is no problem doing it at login node, this should be a general problem.

JosephJHowlett commented 7 years ago

Looks like a huge improvement! Could you add some more details in the usage part of the readme? I see that the code itself is very well documented - but not sure what steps to take between cloning and running.

JosephJHowlett commented 7 years ago

@zhut19 status? If you can update the readme some I can check it out and test it - I'd love to use this for the bias/smearing study.

Also as I think Patrick suggested maybe better to leave current framework alone as default and make this one adjacent.

JosephJHowlett commented 6 years ago

After testing this, I can say it is certainly a big improvement for producing fax-only data. A couple things that I think could make it easier to use:

  1. Can pax version be added as a config item?
  2. Would it be simple to put all of the user-configured things in the same place? Maybe fax_production_main.py is the best place for this. Its nice in this framework that its easy to hack around depending on what you want to do, but we should try to make the simplest usage case as easy as possible (considering many people can benefit from fax production, and even undergrads have previously needed to struggle through my awful code).
  3. The readme is now very nice. We should try to add some overview of what the framework does, explain that it uses its own config (distinct from a pax config), and explain a bit more or simplify the grouping of sbatch submissions (which is useful for iterating over config parameters, but was a bit confusing for my usage). I'm happy to help with this since I'm your first customer.

Then, maybe we can make this a separate directory and leave the previous edition of fax_waveform alone as the default. Then maybe @pdeperio can merge and we can continue to work on it.

I have high hopes for this - if it's simple enough for anyone to use then more people can tune fax parameters and we can have better MC/data matching.

zhut19 commented 6 years ago

I did another pull request, and leaving the original fax_waveform intact.