Expand python modules with more helpful/robust functions

[tomeichlersmith]

Our python modules currently are very bare-boned, and I think we can expand them to make the use of ldmx-sw much easier and robust. I am planning to add more interfaces that will make the python configuration less likely to make small mistakes.

Here is a list of planned improvements:

• [x] Add a directory of event files to the process as input files
• [ ] A few pre-written drop/keep rules that can be added
  • e.g. Only Reco, Sim and Reco (no digi), everything
• [ ] Functions to add different chains to the process (e.g. digi pipelines or cutflows)
• [x] Derive python classes with member functions to help set the parameters
  • Only have the parameter name hardcoded once in python and once in C++
  • Biggest Change: Each processor, primary generator, user action could have its own derived python class with helper functions. Probably won't need to do the classes with a smaller number of parameters or are less complicated.
  • This may be the change the forces us to abandon python2 support (derived class syntax changes from python2 to python3)
  • Put all defaults in these derived classes so printing the python configuration is more transparent
• [ ] Try to maintain backwards compatibility for the user's run scripts, but no promises.

Isn't this just more code nonsense? Yes, but these changes will make ldmx-sw easier to use for newbies and help it be more robust for all of us. Hopefully will drastically reduce the number of small errors like mispelling parameter names.

[tomeichlersmith]

Two updates on my thought process:

1. Boost.Python looks like a good wrapper for the python-C interface that I am going to use.
2. I am going to try to rid ourselves of the parameters python members. Instead, each object that needs parameters will have a "mirror" python object that has configuration parameters. Then the ConfigurePython class can import thess members into a C++ map that the class will have access to. This way, the python configuration object will check for misspellings at runtime.

[omar-moreno]

Before deciding on a library please make sure to consider all alternatives e.g. https://github.com/pybind/pybind11.

Also, this is a big change so it would be good to see a presentation on this before moving forward.

On Tue, Jun 16, 2020, 1:48 PM Tom Eichlersmith notifications@github.com wrote:

Two updates on my thought process:

1. Boost.Python https://www.boost.org/doc/libs/1_69_0/libs/python/doc/html/tutorial/tutorial/embedding.html looks like a good wrapper for the python-C interface that I am going to use.
2. I am going to try to rid ourselves of the parameters python members. Instead, each object that needs parameters will have a "mirror" python object that has configuration parameters. Then the ConfigurePython class can import thess members into a C++ map that the class will have access to. This way, the python configuration object will check for misspellings at runtime.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LDMX-Software/ldmx-sw/issues/780#issuecomment-645003822, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4JMXFPZRXTJ7LBH7DEROLRW7LDHANCNFSM4NKUX5FA .

[tomeichlersmith]

@omar-moreno Two things:

1. I am going to break Python2 compatibility with these changes for the same reason I reference in PR #806 .
2. I am going to break configuration backwards compatibility. This means that you can still read in event files before these changes, but you will need to update your run scripts. I know this is a hassle, but this is the way we should be going and honestly it's a lot of work to try to maintain backwards compatibility.

[omar-moreno]

@omar-moreno Two things:

1. I am going to break Python2 compatibility with these changes for the same reason I reference in PR #806 .

Sounds good.

2. I am going to break _configuration_ backwards compatibility. This means that you can still read in event files before these changes, but you will need to update your run scripts. I know this is a hassle, but this is the way we should be going and honestly it's a lot of work to try to maintain backwards compatibility.

Just to be clear, this should only impact the python configs right? This has (or should) have nothing to do with the event model?

[tomeichlersmith]

Correct, I am not touching the event model at all. The only C++ class that I am changing is ConfigurePython.