C++ Code Generator Service

[BenGalewsky]

Story

As an analyzer I want ServiceX to generate C++ code from my submitted analysis description language so I can use the experiment approved framework without writing C++ code

Assumptions

1. This will be a new Microservice with a REST interface
2. The service will generate C++ and python code to represent the submitted func_adl selection
3. The generated files will be encoded into a zip file which will be returned to the caller in the response body
4. Based on the simplified LISPy language

Steps to Implement

1. Create REST server in Code Generator Repo
2. Model Dockerfiles on the one in ServiceX_App
3. Add config setting for Code Generator endpoint to app.conf
4. Modify the transform request handler to include a select property which will be a func_adl select statement
5. In Preflight Check Handler invoke the service to translate Select statement to columns
6. Save the resulting columns in the TransformRequest columns property

Acceptance Criteria

1. Given I have a valid ADL select when I run the transformer then the generated code will be mounted into the transformers as a configMap
2. Given I have an ADL select AND there is a syntax error in the select WHEN I submit the transform THEN I should receive an error

[gordonwatts]

Your criteria 2 seem to imply we will build a library of these C++ things and keep them around. Is that what you intended here? The key is the word "re-run", which implies re-running three days later or something.

BTW. For something simple, a zipped C++ file is about 5 KB. For something larger it is about 10KB (40 columns, with delta-R comparisons between objects).

[BenGalewsky]

I guess now that we are passing the code in as part of the message there is no need for caching. I'll remove that acceptance criteria

[masonproffitt]

I think I'm now clearer on the details of the text AST language than I am on the context of this issue. What does the term "columns" actually mean here? The C++ transformer runs on xAODs, which are of course not flat ntuples, so I don't think of them as having proper columns. Are we indeed talking about xAOD files and referring to things like object collections as "columns"? Or are we talking about columns in the output of the transformer, which is something like a flat ROOT ntuple?

Edit: After some discussion, it's clear that "columns" refer to the intended output of the transformer.

[masonproffitt]

I've started putting together the parser and transformer to go from the text AST to a Python AST, using Lark. The basic grammar is now defined here. The initial parser and transformer are also in the repository. So far it can handle literals and names. The next step is to get a basic Select working.

[BenGalewsky]

Also, I've created an initial Microservice to host this: https://github.com/ssl-hep/ServiceX_Code_Generator

[masonproffitt]

I've managed to get the column output working, which you can see a very bare example of at https://github.com/iris-hep/ast-language/blob/master/round_trip_to_columns_example.ipynb. This is going from the originally parsed Python AST, transformed to the new text AST language, parsed back to Python AST, and then to list a columns.

[gordonwatts]

Where is this now? From what Mason said it sounds like a basic version is now working? The next job is to get my stuff integrated in?

[masonproffitt]

Okay, now this is really getting somewhere. I've made a much more complete example here. All the basic nodes are now covered. There are some more complex Python nodes that aren't implemented (like IfExp), but this is pretty close to being fully functional.

[gordonwatts]

Progress!! The front end python language can now generate @masonproffitt 's ast-language!!

The work for this is progressing in the following work items: https://github.com/iris-hep/func_adl/issues/10, https://github.com/iris-hep/func_adl.xAOD/issues/7, and https://github.com/iris-hep/func_adl.xAOD.backend/issues/3. Of those three, the first two are now complete. Next step is to drive the changes from the first two into the 3rd (the bit that translates the code into C++ and runs it). Once that is done and tests pass, the last step is an integration test (https://github.com/gordonwatts/func_adl_server/issues/32). When that all works, then our end should be ready to move the code into the real ServiceX.

I will update this issue as I make progress. If you are frustrated with the lack up updates here, feel free to click through on the 3rd issue above to see progress.

[gordonwatts]

Big progress! func_adl_xaod now passes all its tests (85% code coverage, not as good as it should be, but enough to move forward). Next steps:

1. Re-integrate with my func_adl_server. This is to make sure I've not forgotten some corner case with this new ast-language
2. Integration with ServiceX

[gordonwatts]

Another major bit of background work finished: the back-end .sh scripts now allow you to process multiple files in a container, and compile only once. See https://github.com/iris-hep/func_adl.xAOD/issues/20 for details of what had to happen for this step.

I think this is the last step that can happen independently of pre-existing ServiceX code.

Note: This actually belongs in #28. I copied this comment there.

[gordonwatts]

With this last update to ServiceX, the helm chart will deploy this, and it appears to work. Tho because the transformer (#28) isn't in yet, this isn't ready for an actual deployment.