Trying hawc_hal with py3

hayalaso commented 3 years ago

I started a py3 environment to also move the HAWC software to python 3. I installed astromodels, threeml and hawc_hal. I tried this at the UMD cluster. (Linux 3.10.0-957.el7.x86_6)

When trying to run hawc_hal, the program stops. There's no concise error for a test analysis I try to run on the crab.

terminate called without an active exception
Aborted (core dumped)

I cloned hawc_hal and installed it so that I could run tests. Running pytest:

Passed:

test_serialization.py
test_bilinear_interpolation.py
test_psf.py
test_copy.py
test_roi.py
test_on_point_source.py

Failed:

test_root_to_hdf.py
test_plugin_creation.py
test_complete_analysis.py
test_model_residual_maps.py
test_maptree.py
test_geminga_paper.py

Here's an example of one output

====================== test session starts ======================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /data/disk01/home/hayalaso/Software/hawc_software_py3/externals/hawc_hal
collected 2 items                                               

test_plugin_creation.py Fatal Python error: Aborted

Thread 0x00007fac8a11a700 (most recent call first):
  File "/data/disk01/home/hayalaso/Software/hawc_software_py3/externals/ape/External/root/6.16.00/lib/root/ROOT.py", line 484 in _processRootEvents
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/threading.py", line 870 in run
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007facd2e2a740 (most recent call first):
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/hawc_hal-1.0-py3.8.egg/hawc_hal/maptree/from_root_file.py", line 208 in _read_partial_tree
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/hawc_hal-1.0-py3.8.egg/hawc_hal/maptree/from_root_file.py", line 140 in from_root_file
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/hawc_hal-1.0-py3.8.egg/hawc_hal/maptree/map_tree.py", line 60 in from_root_file
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/hawc_hal-1.0-py3.8.egg/hawc_hal/maptree/map_tree.py", line 29 in map_tree_factory
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/hawc_hal-1.0-py3.8.egg/hawc_hal/HAL.py", line 62 in __init__
  File "/data/disk01/home/hayalaso/Software/hawc_software_py3/externals/hawc_hal/tests/test_plugin_creation.py", line 17 in test_plugin_from_root
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/python.py", line 1641 in runtest
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 311 in from_call
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/data/disk01/home/hayalaso/Software/miniconda3/envs/aerie_py3/bin/pytest", line 8 in <module>
Aborted (core dumped)

henrikef commented 3 years ago

How did you install hawc_hal/astromodels/threeML? I see it's referencing ROOT installed via ape, I wonder if that's a problem.

Does python -c "import hawc_hal" work? How about python -c "import threeML"?

henrikef commented 3 years ago

Oh, can you also try python -c "import ROOT"?

henrikef commented 3 years ago

ALso, I've only tested this with python 3.7 so far. I just added another test for python 3.8.

henrikef commented 3 years ago

(There's a partial outage for github actions right now though, so the automated tests aren't running ☹️ )

hayalaso commented 3 years ago

Importing things in the python terminal work fine. I installed the threeml package using the pip install git+https://stuff for threeml. I did install ROOT with ape, since I have things running now with py3 and aerie. It's root 6. I don't know if threeml is based on that?

hayalaso commented 3 years ago

Looks like the way aerie is installed by default is using root 5 which is then used when building threeml. Is the conda version using root 5?

henrikef commented 3 years ago

I have root 6.22/06, apparently.

Is there a good reason you need to have aerie and threeML/HAL in the same environment? As I recall, there were other issues with that previously (not sure if they've been fixed now).

Here's how I've been installing HAL:

conda create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" "threeml>=2" root
conda activate new_hal
pip install --no-binary :all: root_numpy
pip install git+https://github.com/threeml/hawc_hal.git

(usual way, see README ), followed by

pip uninstall astromodels threeML -y
pip install git+https://github.com/threeml/astromodels.git
pip install git+https://github.com/threeml/threeML.git

(this should only be necessary until there's a new release of threeML on conda).

hayalaso commented 3 years ago

Well, since I'm trying to move things of aerie into py3, I figure it would make sense to have aerie and threeML/HAL in the same environment.

henrikef commented 3 years ago

Closing this as everything seems to work in python3 now. Please re-open if there are any remaining issues.

threeML / hawc_hal

Trying hawc_hal with py3 #54