Open tueda opened 1 year ago
@irinaespejo @Sinclert If I fix this can I bump up the release number to 0.3.6
or will that not get picked up by workflows? I guess another way of saying this is what needs to get updated anytime there's a new release of an image?
Hey @tueda, thanks for your report.
@matthewfeickert Feel free to do so. IIRC this Docker image is only used within the madminer-tutorial repository. You would need to update every reference to the new tag, similar to what this commit did.
I am facing the same issue and now I cannot downgrade numpy
, some common errors happen when using the function delphes.analyse_delphes_samples()
, any suggestions?
Here is my error message:
Traceback (most recent call last): File "/Users/younes/Desktop/madminer/examples/tutorial_particle_physics/delphes_level.py", line 77, in <module> _ = plot_distributions( ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/plotting/distributions.py", line 134, in plot_distributions sa = SampleAugmenter(filename, include_nuisance_parameters=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/sampling/sampleaugmenter.py", line 74, in __init__ super().__init__(filename, disable_morphing, include_nuisance_parameters) File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/analysis/dataanalyzer.py", line 53, in __init__ ) = load_madminer_settings(filename, include_nuisance_benchmarks=include_nuisance_parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/utils/interfaces/hdf5.py", line 63, in load_madminer_settings analysis_params = _load_analysis_params(file_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/utils/interfaces/hdf5.py", line 872, in _load_analysis_params with h5py.File(file_name, "r") as file: ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/h5py/_hl/files.py", line 567, in __init__ fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/h5py/_hl/files.py", line 231, in make_fid fid = h5f.open(name, flags, fapl=fapl) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 106, in h5py.h5f.open OSError: Unable to open file (file signature not found)
👋🏻 Apologies for the delay on the resolution.
I decided to forward fix the h5py - numpy compatibility issue, by removing the h5py requirement from the madminer-jupyter-env
requirements.txt file in commit https://github.com/madminer-tool/madminer-jupyter-env/commit/2db21db3022c5b5fd289b4d66f3943191dc06417 (it must be exclusively installed as a madminer dependency).
Please, @tueda , could you try madminer-jupyter-env
v0.3.6 image and let us know if your issue is solved?
Thank you for the fix. I have tried the 0.3.6 image and it worked: at least import madminer
does not give any errors.
Recently, my collaborator told me that the Docker image does not work properly with the physics tutorial (part 2A). It turned out that miner.run()
fails silently because MG5_aMC_v2.9.4 is not compatible with NumPy 1.24.4.
Log file: mg_processes/signal1/run_01_tag_1_debug.log
).
See also this bug report and the corresponding fix commit. Upgrading the MG5_aMC to 2.9.16+ could be a potential fix.
Thanks for the bug report. I just pushed image version 0.3.7
containing the described MadGraph version upgrade.
Thanks for uploading 0.3.7. Unfortunately, the image still has some problems.
When you run /madminer/software/MG5_aMC_v2_9_16/bin/mg5_aMC
, you see:
A Pythia8 path is specified via the option 'pythia8_path' but no path for option
'mg5amc_py8_interface_path' is specified. This means that Pythia8 cannot be used
leading order simulations with MadEvent.
Consider installing the MG5_aMC-PY8 interface with the following command:
MG5_aMC>install mg5amc_py8_interface
This causes the physics tutorial (part 2B) to fail because miner.run()
silently fails (log).
Actually, it seems that installing mg5amc_py8_interface
failed during the build of the image (RUN echo "install pythia8" | python3 ${MG_BINARY_PATH}
). When you look at /madminer/software/MG5_aMC_v2_9_16/HEPTools/MG5aMC_PY8_interface/mg5amc_py8_interface_install.log
, you find:
MG5aMC_PY8_interface compilation output log:
b"g++ MG5aMC_PY8_interface.cc -o MG5aMC_PY8_interface -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/include -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//include -ldl -fPIC -lstdc++ -std=c++11 -O2 -DHEPMC2HACK -DGZIP -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/include -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -lz -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -lpythia8 -ldl -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -lHepMC\n/usr/bin/ld: /madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib/libpythia8.so: undefined reference to `pthread_create'\ncollect2: error: ld returned 1 exit status\nmake: *** [Makefile_mg5amc_py8_interface_static:8: MG5aMC_PY8_interface] Error 1\n"
------------------------------------------------
Error during the compilation of MG5aMC_PY8_interface:
b"g++ MG5aMC_PY8_interface.cc -o MG5aMC_PY8_interface -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/include -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//include -ldl -fPIC -lstdc++ -std=c++11 -O2 -DHEPMC2HACK -DGZIP -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/include -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -lz -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -lpythia8 -ldl -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -lHepMC\n/usr/bin/ld: /madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib/libpythia8.so: undefined reference to `pthread_create'\ncollect2: error: ld returned 1 exit status\nmake: *** [Makefile_mg5amc_py8_interface_static:8: MG5aMC_PY8_interface] Error 1\n"
This appear to be the pthread
issue raised in here, here or here.
I'm not sure why this pthread issue occurred in your build; currently, if I build the image locally, then the compilation of MG5aMC_PY8_interface successfully completes. (But I had the same in the past with Ubuntu images; perhaps due to the layer cache??)
If this pthread
issue can be fixed, then the next problem is in the MadMiner physics tutorial (part 4A), where limits.expected_limits()
eventually calls np.histogramdd()
with normed
in Histo._fit()
. This parameter has been removed since NumPy 1.24.0
.
Thanks again for the detailed bug report.
At this stage, I would say you know more about what is going wrong with the tutorial Docker image, than myself, or any other person associated to this project (which has been unmaintained for a few years now).
Seems like the easiest path forward would be for you to open a PR, and for us to review it. Feel free to tag me once you do.
Maybe the first step would be adding CI, which checks if changes cause further problems: #10.
The current image has
h5py
2.10.0 (specified inrequirements.txt
), which is incompatible withnumpy
1.24.1 installed in the image:Maybe
numpy
version could be fixed to an older one, as in https://stackoverflow.com/q/74852225.