search

threeML / hawc_hal

HAWC Accelerated Likelihood - python-only framework for HAWC data analysis
BSD 3-Clause "New" or "Revised" License
11 stars 21 forks source link

multiprocessing problem in from_root_file.py #97

Closed Catbohe closed 4 months ago

Catbohe commented 4 months ago

Hello, I encountered a problem when I fit some HAWC private data on my laptop. I have successfully installed threeML and hawc_hal. When I did the fit using fitModel.py, It crashed at line 249 in from_root_file.py (https://github.com/threeML/hawc_hal/blob/cd2b4b5071c2e8e574c3938b942ade52ee364036/hawc_hal/maptree/from_root_file.py#L249). It seems that the problem is produced by the multiprocessing module. Also, It seems that that module starts more than one process("Srarting 3ML" occurred 3 times). If I run the same script on a HAWC server, it can work smoothly.

My laptop information: MacBook Pro Apple M2 Max macOS Ventura version 13.5 Xcode Version 15.0.1

miniconda https://repo.anaconda.com/miniconda/Miniconda3-py311_23.11.0-2-MacOS-x86_64.sh (I alse tried arm64, it also has the same problem )

I also set environment when installing threeML: export OMP_NUM_THREADS=1 export MKL_NUM_THREADS=1 export NUMEXPR_NUM_THREADS=1

19:20:33 INFO Starting 3ML! init.py:35 (Some warning )
19:20:44 WARNING No fermitools installed lat_transient_builder.py:44 …………………………

Performing likelihood analysis...

Specifying ntransits currently does not work in hal, this is all @chad's fault. Loading as MODEL file: new_source.model

……………………………………

Using analyis bins [……………………] 19:20:37 INFO Using transits contained in maptree HAL.py:79 INFO Reading Maptree! from_root_file.py:214 () …………………………………………

19:20:42 INFO Starting 3ML! init.py:35 (Some warning information)……………………
19:20:44 WARNING No fermitools installed lat_transient_builder.py:44

Performing likelihood analysis...

Specifying ntransits currently does not work in hal, this is all @chad's fault. Loading as MODEL file: new_source.model (Some warning information)……………………
Using analyis bins [……………………] 19:20:46 INFO Using transits contained in maptree HAL.py:79 INFO Reading Maptree! from_root_file.py:214 Traceback (most recent call last): File "", line 1, in File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main exitcode = _main(fd, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/spawn.py", line 131, in _main prepare(preparation_data) File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path main_content = runpy.run_path(main_path, ^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 291, in run_path File "", line 98, in _run_module_code File "", line 88, in _run_code File "/Users/zwang/work/source_ana/new_source/fitModel.py", line 515, in like = HAL("HAWC", map_tree, det_res, roi, options.pixelwidth) #,n_transits) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/site-packages/hawc_hal/HAL.py", line 88, in init self._maptree = map_tree_factory( ^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/site-packages/hawc_hal/maptree/map_tree.py", line 26, in map_tree_factory return MapTree.from_root_file(map_tree_file, roi, n_transits, n_workers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/site-packages/hawc_hal/maptree/map_tree.py", line 64, in from_root_file data_analysis_bins, transits = from_root_file( ^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/site-packages/hawc_hal/maptree/from_root_file.py", line 249, in from_root_file with multiprocessing.Pool(processes=n_workers) as executor: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/pool.py", line 215, in init self._repopulate_pool() File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/pool.py", line 306, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/pool.py", line 329, in _repopulate_pool_static w.start() File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) ^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/context.py", line 288, in _Popen return Popen(process_obj) ^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch prep_data = spawn.get_preparation_data(process_obj._name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/spawn.py", line 164, in get_preparation_data _check_not_importing_main() File "/Users/zwang/software/hawc_software_test/miniconda3/envs/hal_analysis/lib/python3.11/multiprocessing/spawn.py", line 140, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

    To fix this issue, refer to the "Safe importing of main module"
    section in https://docs.python.org/3/library/multiprocessing.html

19:20:52 INFO Starting 3ML! init.py:35 WARNING WARNINGs here are NOT errors init.py:36 WARNING but are inform you about optional packages that can be installed init.py:37 WARNING to disable these messages, turn off start_warning in your config file init.py:40 WARNING Multinest minimizer not available
……………………

sergiohcdna commented 4 months ago

Hello @Catbohe Seems that the problem is related to try to install package that are only available for x86 arch, and not for arm64 arc.

I installed threeml and HAL in a Macbook pro with M3 chip and read the HAWC maptree using multiprocessing and did not find any errors.

As for now, astromodels is not available for arm64 arch, and when trying to install using conda, the system will complaint that there is not match between the astromodels and threeml available versions. And trying to install using x86-specific packages do not help as they are optimized for another arch.

Please try the following to check if you can solve the problem:

Let me know if you find any other potential errors.

Catbohe commented 4 months ago

Hi

Hello @Catbohe Seems that the problem is related to try to install package that are only available for x86 arch, and not for arm64 arc.

I installed threeml and HAL in a Macbook pro with M3 chip and read the HAWC maptree using multiprocessing and did not find any errors.

As for now, astromodels is not available for arm64 arch, and when trying to install using conda, the system will complaint that there is not match between the astromodels and threeml available versions. And trying to install using x86-specific packages do not help as they are optimized for another arch.

Please try the following to check if you can solve the problem:

  • Erase your current virtual environment
  • Reinstall conda (miniconda) for arm64 arch
  • Use the following instruction to create the virtual environment:
conda create -y --name hal_env -c conda-forge numpy scipy matplotlib ipython "numba<0.57" reproject root python=3.10 "uproot<5.2" jupyter notebook

Please note that the python version is fixed to 3.10. The same goes for uproot (uproot<5.2)

  • Then, activate the virtual environment and install additional packages if needed (ebltable, naima, etc)
  • Install HAL using pip:
python -m pip install git+https://github.com/threeml/astromodels
python -m pip install git+https://github.com/threeml/threeml
python -m pip install git+https://github.com/threeml/hawc_hal

Let me know if you find any other potential errors.

Hi,

Thanks for your reply. I followed your instructions but it still does not work.

sergiohcdna commented 4 months ago

Hi @Catbohe Can you provide text file with the error? BTW, I think the only thing I have different now, it is my configuration file where I set to use parallel by default, but I remember to ran the example without that.

sergiohcdna commented 4 months ago

About the multiple init messages, that's the expected output, as the script is running in n number of cores you specified to read the map tree.