EventDisplay re-design

[tomeichlersmith]

CMS uses a specific event display workflow that I think is really nice. Basically, their actual event display is a JavaScript-based web application that takes json files as inputs. Then they have analyzers that can translate the event bus objects into the json files in a specific way that is readable by their web app.

I like this process for three reasons:

1. With the event display being a web-sever, it could be hosted within the container and visited by the user at a local IP address.
2. All of the event object to display object translation is done separately from the drawing of the display. This is helpful because then the event display will be faster and it will be easier to debug (just open up the json to make sure the translation was done correctly).
3. If we want to change how an object is displayed, we don't touch the event display itself. Only the analyzer that does the translation.

The main downside of this process is that the display of the detector geometry is hard coded into the event display (like it currently is for us), but this issue may be rectified if I can find a method of translating gdml into the json files that the display reads in for the geometry.

[tomeichlersmith]

I think we can use a more general, open-source project Phoenix to base our event display on. It looks like it is pretty active and takes .obj files as input for the geometry and .json files as input for the event objects.

[tomeichlersmith]

It looks like we could convert our gdml into .root or .json.gz files that are importable by Phoenix:

• https://github.com/HSF/phoenix/issues/122
• https://github.com/HSF/phoenix/pull/117

[tomeichlersmith]

Translation to json files from our objects in ROOT is necessary. I think an easy way to do this is with a python module. https://chadrick-kwag.net/json-dumpingserializing-custom-python-classes/

[omar-moreno]

Why not just build on what DD4hep has already done? Their geo display is pretty nice.

On Wed, Feb 10, 2021 at 10:34 AM Tom Eichlersmith notifications@github.com wrote:

Translation to json files from our objects in ROOT is necessary. I think an easy way to do this is with a python module. https://chadrick-kwag.net/json-dumpingserializing-custom-python-classes/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LDMX-Software/ldmx-sw/issues/888#issuecomment-776921289, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4JMXANZQ7KHGC6TKX2WKLS6LGRTANCNFSM4QD4EDMQ .

[tomeichlersmith]

I haven't looked into it...good idea

[tomeichlersmith]

The "documentation" makes me worried, but I will play around with it when I get the chance.

[omar-moreno]

Documentation is pretty bad ... but still might be worth over building from scratch. They seem to answer questions readily.

On Wed, Feb 10, 2021 at 10:41 AM Tom Eichlersmith notifications@github.com wrote:

The "documentation" https://dd4hep.web.cern.ch/dd4hep/usermanuals/DDEveManual/DDEveManualse1.html#x2-10001 makes me worried, but I will play around with it when I get the chance.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/LDMX-Software/ldmx-sw/issues/888#issuecomment-776925622, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4JMXH6OXQRWAX3GTFIUULS6LHNDANCNFSM4QD4EDMQ .

[tomeichlersmith]

I guess, I'm happy to go either way and we need to just pick a path. Both paths have their pros and cons, so I want to lay them out here. I like number 2 better because the GUI doesn't use ROOT and actually looks cool, but the translation layer is a big con.

1. ROOT-based GUI

Pros

• Similar to our current approach
• Can be integrated with our geometry description and (potentially) use DD4hep directly
• Reads data files directly

Cons

• Slow
• Struggle to container-ize the application
  • This is because ROOT::Eve libraries don't compile without access to a display during compilation.

2. NPM-based GUI with Translation Layer

Pros

• GUI is fast and looks nice (not cluttered)
• Translation of ROOT objects to displayed objects is output to a file so it can be easily debugged
• Easily containerizable, the actual application is just viewed in a browser and the GUI handles importing the event file.

Cons

• Both the detector description and the event objects have to be translated to objects readable by Phoenix.

[tomeichlersmith]

copying in my notes from slack chatting with @cmantill and @mrsolt

---

The biggest frustration with developing a event visualization is the detector. How can we access the detector information so that it can be drawn and the visualization knows which detector cell to highlight for a specific hit. The "easy" way to get a visualized detector is to use G4 and the full GDML description but as you point out, this is too detailed. For the event display, we don't want to see all the individual material layers in the ECal, for example, since that just clutters the view. Other experiments have switched to a two-stage model where the processing framework (ldmx-sw in our case) writes out a new file (e.g. JSON) that only contains the objects that we wish to draw and then have a different program based on a language better-suited for visualization (e.g. javascript) load this information. This issue links to a project we could use called Phoenix that is this second program. Then the work for us would be two stages (1) getting our detector geometry read-able by this program and (2) writing processors that can convert our hits/tracks/particles into the objects to be drawn and write them to a JSON file.

1. I think the best idea would be to have a "condenser" program which writes a simplified geometry that can be loaded by the event display using G4's GDML reader to parse the full geometry and then re-write only the parts we care about. This would be hopefully fast enough to run that geometry changes (like v13->v14) would only require a quick re-run of this program.
2. This is probably simpler in terms of understanding the API but there are a lot of bugs lurking in order to convert our objects into the objects to be drawn. For example, the ecal hits would best be understood as tiny little hexagons but the hcal hits would be long bars oriented properly. This is where the geometry conditions would come into play.

how bad of an idea is to make [a summer student] develop a base for event visualization for ldmx-sw?

I think this project is only suited to someone already familiar with C++. The complexities here would require them to be not-confused by C++ syntactical issues so that they can focus on the geometry complexities. Understanding JS and NPM so they could more easily interface with Phoenix would be a plus but I don't think is necessary since Phoenix is already a finished product (more or less, we just need to plug in our geometry and produce the JSON event files).

Do you know if there has been any event visualization done before?

Yes. There was a ROOT TEve event display which attempted to be able to run LDMX event ROOT files (you can still see the ancient ruins of this display on trunk under the EventDisplay directory). The reason I abandoned it is because there was no automatic loading of the detector geometry and so the geometry needed to be manually updated (and I hate ROOT). After I abandoned this (and even slightly before since it was so awful to use - crashing after too many events regularly), many folks started using other visualization techniques which display various options - the process I describe above is the one I think would be easiest and longest-lasting.

1. G4 - with some time spent with the visualization manual, we could coax G4 into reading our ROOT files and drawing our objects. I fear the amount of work this would take is high; however, it may be worthwhile since then we would get the automatic geometry conversion "for free".
2. Matplotlib - basically requires the user to manually draw the detector elements they wish to be viewed. Not horrendous in our case since our detector is relatively simple, but still bothersome. I believe this would be hard to maintain since there is still no automatic geometry conversion.
3. NPM/Phoenix - what I described above. Erik Walin used this for the 8GeV paper event vis, although he mentioned he just hard-coded the geometry (and so there is still no conversion/condenser tool) but he could be a potential resource to help get an understanding on how to use Phoenix (or a Phoenix-like program).

[tomeichlersmith]

Potential program for converting geometry: https://bitbucket.org/jairhul/pyg4ometry/src/develop/

[AnmolS1Z]

Plan to create Event Display:

1. Convert the gdml files to root - matrices and parameterizations aren't easily converted so we'll need to do some extra work with them
2. Convert those root files to gltf files
3. Insert the gltf files into Phoenix to create display
4. Once everything works, can start writing event data.

[AnmolS1Z]

You can skip steps 1 and 2 if you already have a root file. Converting from gdml to gltf: 1) Remove any loops, array brackets, or references to other gdml files in the file you want to convert. This means flattening the loops, replacing the array bracket with the exact element from the array; So for the array "[1,2,3,4]", replace array[1] with 1, since gdml uses 1 based indexing, and replacing entity calls with the contents of the file.

2) Convert gdml to root with gdml_to_root_export.cxx using the following command: "ldmx root -l gdml_to_root_export.cxx". Be sure the filename inside gdml_to_root_export.cxx is the correct gdml file

3) Use the following repo: https://github.com/HSF/root_cern-To_gltf-Exporter/tree/main, make an html file based off of export_LHCb.html, replacing filenames and arrays as needed. A working html file for the whole detector will be included as export.html.

4) The export.html file has 7 array for the parts it needs to generate. If you don't want to include a part, just comment out that line. This has been done for the magnet

5) Put the new gltf file into phoenix

[AnmolS1Z]

Code has been pushed to 888-eventdisplay-re-design branch under EventDisplay directory

[pbutti]

Very nice! @AnmolS1Z Perhaps would make sense instead to copy code from HSF (nominally this code https://github.com/HSF/root_cern-To_gltf-Exporter) to add it as submodule or just point to the original fork.

I've actually already written the app to support the phoenix event display with a similar gltf file obtained from dumping the LDMX geometry, see screenshot attached. Additionally, there is no need to modify the GDML code provided that we produce the .root geometry that uses the G4 Parser instead. I can provide the code to do that if needed.

The package that runs that is here: https://github.com/pbutti/ldmx-event-display.git

Probably a good time to join forces =)

[AnmolS1Z]

Thank you. I've changed the root to gltf to a submodule. Could you provide the gdml to root code?

[pbutti]

Sure! If you don't need it urgently I'll just put it on github tomorrow with instructions.

[pbutti]

We probably just need our own export.html to parse the root into gltf. For the node.js application I posted above I can provide instructions, but basically it's just necessary to install the node_modules via npm install and run the app. Make sure you have the v15.2.8 of angularjs (by npm i install -g @angular/cli@15.2.8) I'll post instructions tomorrow.