installation problems about python3 environment

[xiaojieww]

Describe the bug

I can't install the environment on cluster with recommended command lineconda create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" "threeml>=2" root, it will stay forever at the step solving environment. So I use the command shown below to do the install:

conda create --name new_hal
conda install -c conda-forge numpy scipy matplotlib
conda install -c conda-forge ipython numba reproject
conda install -c conda-forge root
pip install git+https://github.com/threeml/astromodels.git
pip install git+https://github.com/threeml/threeML.git
pip install git+https://github.com/threeml/hawc_hal.git
pip install --no-binary :all: root_numpy

It seems work, but then I met some problems while running the crab example use python3_quick branch (file path:threeml-analysis-scripts/fitModel/crab_example.sh), it will crash. Also the crashing will happen with the test example provided by https://github.com/threeML/hawc_hal

I'll put the log files below to help with the troubleshooting procedure.

Log files

install_new_hal.log pytest_hawc_hal.log pytest3ML.log pytestAstromodels.log testMr421.log testCrab.log

[henrikef]

Hi @xiaojieww!

The pytest_hawc_hal.log and pytestAstromodels.log seem incomplete, can you please re-upload them?

The error in pytest3ML.log (AttributeError: module 'astropy.utils.data' has no attribute '_find_pkg_data_path') is unrelated to HAL issues and I believe @grburgess has already submitted a fix to astropy.

We're currently only testing hawc_hal with python 3.7. Could you please try installing it with that instead of python 3.9? HAL relies on root_numpy, which hasn't been updated in a long time and might not work with current versions of python.

Last: If the hawc_hal tests pass but you're still having issues with the fit, please provide exact instructions to reproduce the issue. This is easiest if you provide an example script using publicly available data/response files.

[cbrisboi]

After looking this over, I have a one idea that @henrikef didn't type faster than did. Some of those might reveal a path forward first.

Can you try using mamba(https://github.com/mamba-org/mamba) to solve the environment? I discussed this recently with @grburgess but have not had a chance to attempt it. You would modify this step (replacing conda with mamba) from the README.md file: conda create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" "threeml>=2" root

[xiaojieww]

Hi @henrikef and @cbrisboi

Thanks for the quick response! When I run the pytest for astromodelsand hawc_hal they didn't complete and just like the logfile I uploaded above. I don't know if the unsuccessful installation of the environment can cause the problem? It shouldn't be the file itself, the log file only honestly recorded all the outputs I got.

I can try to install with python 3.7 first with the procedure I did above, then try to using the command linemamba create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" "threeml>=2" root. I think maybe it is better to change one thing at the same time to locate the problem?

[cbrisboi]

Please try only one thing at a time so that we know exactly what the solution was, in case others encounter this issue in the future.

[xiaojieww]

Hi @cbrisboi and @henrikef,

Use the mamba command line mamba create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" as the first step can set up the new environment successfully and also very quickly, the analysis also can run very smoothly.

I'am still working on useconda (python 3.7) to step by step set up the environment, I'll keep you posted about the results.

[xiaojieww]

Hi @henrikef,

I tried several times yesterday, and found that if I use the python 3.7 to install environment with command line:

conda create --name new_hal2
conda install -c conda-forge python=3.7 numpy scipy matplotlib
conda install -c conda-forge root
pip install git+https://github.com/threeml/astromodels.git
pip install git+https://github.com/threeml/threeML.git
pip install git+https://github.com/threeml/hawc_hal.git
pip install --no-binary :all: root_numpy
conda install -c conda-forge pytest

there will be errors happen while installing astromodels. And all the pytest aren't pass. pip install --upgrade git+https://github.com/threeml/astromodels.git command shown no error but didn't help. After run pytest after the upgrade, the pytest are still the same with previous one.

installNewhalLogPython3_7_2.log pytest_astromodels_beforeUpdate.log pytest_hawc_hal_beforeUpdate.log pytest_3ML_beforeUpedate.log updateAstromodelsOutput.log pytest_astromodels_afterUpdate.log

[henrikef]

I have no idea what's going on with tempita there, but it doesn't seem related to HAL in any way. Anyway, if it works with mamba, then why not just keep using that?

[xiaojieww]

I already use the mamba-created environment to do the analysis, just curious about the issues now. Maybe it's related to the way we create the environment? I guess we can stick to using mamba to set up the environment in the future.

[grburgess]

@xiaojieww I am not sure, but mixing pip and conda is generally a bad idea. This is usually due to the fact that versioning and tooling between the two installation systems can be inconsistent. If you use conda, I would go all conda for the 3ML packages.

[xiaojieww]

@grburgess Thanks! The reason why I didn't use conda to install all the packages is it's too slow to do the installation. I'll try to install all the package use conda to find out whether the mixing pip and conda caused the problems. Thankfully we have mamba can help with set up environment quickly and successfully.

[henrikef]

Is this still an ongoing issue?

[xiaojieww]

I believe this issue has already been solved by using mamba to install, I will close this issue.

[henrikef]

I don't see any changes, neither on README.md nor in the automated CI test (which have been failing consistently due to installation issues). Where did you fix it?

[xiaojieww]

I changed it in the README.md. The old installation command line is
conda create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" "threeml>=2" root which is caused the problem. While the new installation command line is already the mamba create --name new_hal -c conda-forge -c threeml numpy scipy matplotlib ipython numba reproject "astromodels>=2" "threeml>=2" root