Output parquet file has no 'photons' field

[AnthonySandhar-Cruz]

I am using the docker image trying to run the PPC example (and similar code with different parameters), and each time, the parquet file generated does not have the 'photons' field when trying to analyze the output. The array 'test_arr' is always None in the 'construct_output' method in 'prometheus.py'.

[MeighenBergerS]

Hi Anthony,

There are a couple of things that could be going on here. My suspicion is that PPC/PPC_CUDA is not linked, and/or the CUDA version is not compiled for your system.

Do you have the outputs from the simulation run? Could you add it here? (not the final parquet file, but the terminal outputs)

Are you running it on your own computer or a cluster?

Cheers, Stephan

[AnthonySandhar-Cruz]

I am running it from my own computer. I do not have a GPU on this machine so I am utilizing the non-CUDA version of PPC. I am running everything in the pre-built docker image, and I am utilizing the newest version of the 'example_PPC.py' example from the repository (just changing from PPC_CUDA to PPC). The simulation output of that code is

--------------------------------------------------------------
--------------------------------------------------------------
Launching simulation
CURRENT SET 1
Importing LeptonInjector
Trying default pythonpath
Trying custom path set in config
The path is /opt/LI/install/lib/python3.9/site-packages
<frozen importlib._bootstrap>:228: RuntimeWarning: to-Python converter for earthmodel::LIEarthModelCalculator already registered; second conversion method ignored.
Creating Controller
Initializing Earth
Using...
Freezing ice
Executing Injectors
Note: LIC file already exists, appending to end.
Starting up VolumeInjector
Propagating Hadrons: 100%|█████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  7.49it/s]
I am melting.... AHHHHHH!!!!
RAM memory % used: 38.53
Finished call
--------------------------------------------------------------
--------------------------------------------------------------

If I print 'self.injection[0]' before and after the 'sim' method in 'prometheus.py', there is a clear difference, meaning that propagation is occurring but 'hits' is always empty.

[MeighenBergerS]

Hi! Thanks for sending the output!

Okay, so filling the parquet file may be the problem.

Just to be sure that nothing strange is happening (due to the container), could you re-run the simulation after setting some options?

(PATH_TO_PROMETHEUS = # Wherever the prometheus folder is)

1. First, before running the code, ensure you have the most recent version of Prometheus.
2. Go to the "/PATH_TO_PROMETHEUS/prometheus/resources/PPC_executables/PPC/" directory and compile PPC (make cpu)
3. In the PPC example add these options:

config['photon propagator']["PPC"]["paths"]['ppc_tmpfile'] = '.event_hits_test.ppc.tmp'
config['photon propagator']["PPC"]["paths"]['f2k_tmpfile'] = '.event_losses_test.f2k.tmp'
config['photon propagator']["PPC"]["paths"]['ppctables'] = '/PATH_TO_PROMETHEUS/prometheus/resources/PPC_tables/south_pole/'
config['photon propagator']["PPC"]["paths"]['ppc_exe'] = '/PATH_TO_PROMETHEUS/prometheus/resources/PPC_executables/PPC/ppc'
config['photon propagator']["PPC"]["paths"]["ppc_tmpdir"] = "./.ppc_tmp_test"
config['photon propagator']["PPC"]["simulation"]['supress_output'] = True

What you should see while running is something along the lines of:

Processing f2k muons from stdin on device 0
photons: 524288  hits: 0
photons: 524288  hits: 0
photons: 524288  hits: 0
photons: 524288  hits: 0
...

Which shows that PPC is properly running.

[AnthonySandhar-Cruz]

I made sure to get the latest version of Prometheus from the repository and copied it to my docker container. Running 'make cpu' in the PPC directory gave this as the output

ln -s ppc.cu ppc.cxx || true
ln: failed to create symbolic link 'ppc.cxx': File exists
ln -s pro.cu pro.cxx || true
ln: failed to create symbolic link 'pro.cxx': File exists
g++ ppc.cxx -O2 --fast-math -o ppc
In file included from ppc.cxx:499:0:
f2k.cxx: In function ‘void xppc::print()’:
f2k.cxx:311:73: error: ‘struct xppc::hit’ has no member named ‘dth’
   printf("HIT %d %d %f %f %f %f %f %f\n", n.str, n.dom, h.t, h.z, h.dth, h.dph, h.pth, h.pph);
                                                                     ^~~
f2k.cxx:311:80: error: ‘struct xppc::hit’ has no member named ‘dph’
 f("HIT %d %d %f %f %f %f %f %f\n", n.str, n.dom, h.t, h.z, h.dth, h.dph, h.pth, h.pph);
                                                                     ^~~
f2k.cxx:311:87: error: ‘struct xppc::hit’ has no member named ‘pth’
 %d %d %f %f %f %f %f %f\n", n.str, n.dom, h.t, h.z, h.dth, h.dph, h.pth, h.pph);
                                                                     ^~~
f2k.cxx:311:94: error: ‘struct xppc::hit’ has no member named ‘pph’
 d %d %f %f %f %f %f %f\n", n.str, n.dom, h.t, h.z, h.dth, h.dph, h.pth, h.pph);
                                                                           ^~~
Makefile:38: recipe for target 'cpu' failed
make: *** [cpu] Error 1

If I ignore this and add the suggested lines to the PPC example, when I run it, I get the exact same output as before and I do not get anything like what you have shown.

[MeighenBergerS]

Thank you.

That is a bug, and we have to fix it.

In the meantime, do you have the option to use PPC_CUDA? I.e. Can you run your docker container with GPU support? The PPC_CUDA version of the propagator should fix this problem.

[AnthonySandhar-Cruz]

Yes, I do have a machine that can run the docker container with GPU support. I will do that then. Thank you!

[MeighenBergerS]

Great! As an added perk, the CUDA version is also much faster than the PPC baseline version.

Concerning the bug though: @jlazar17 pushed a fix for the issue you were having if you want to go back to using PPC.

[AnthonySandhar-Cruz]

Thank you for the help! It is working now. Part of the issue was that the suppress output option being true was stopping an error message saying that it could not find the PPC executable. So, the combination of make cpu as well as fixing the path to the executable fixed the issue.

[MeighenBergerS]

Great to hear! Hope everything else goes smoothly!