threeML / hawc_hal

HAWC Accelerated Likelihood - python-only framework for HAWC data analysis
BSD 3-Clause "New" or "Revised" License
13 stars 22 forks source link

Crashing when computing errors #77

Open igherzog opened 2 years ago

igherzog commented 2 years ago

Describe the bug

I am attempting to perform a 3ML likelihood analysis using the updated python3 version of hawc_hal and on the python3_quick branch. I am using the newest iteration of the neural network energy reconstructed data map (maptree and response file linked below) and I'm encountering an odd error. After the fit initially converges, I get this error:

Traceback (most recent call last): File "./fitModel.py", line 584, in errAll=jl.get_errors() File "/data/disk01/home/igherzog/hawc_software/miniconda3/envs/new_hal/lib/python3.7/site-packages/threeML/classicMLE/joint_likelihood.py", line 430, in get_errors errors = self._minimizer.get_errors() File "/data/disk01/home/igherzog/hawc_software/miniconda3/envs/new_hal/lib/python3.7/site-packages/threeML/minimizer/minuit_minimizer.py", line 268, in get_errors "MIGRAD results not valid, cannot compute errors." threeML.minimizer.minimization.CannotComputeErrors: MIGRAD results not valid, cannot compute errors.

It throws this error when performing the error calculations in the fitModel.py script (see below in the Reproduce section). I do not get this error if I use an older iteration of the neural network energy energy estimator or if I use a simpler model. I am not able to attach my model file as a yml file (here it is as a .txt file NN_DBE_final_model.txt) but I can fit the Crab (single source, log-parabola) and, moving to my region, each of my sources individually but when I fit them together I get the above error.

To Reproduce To get the error shown above I run this script on the computing cluster at UMD that HAWC uses. I'm using my hawc_hal environment (path:/data/disk01/home/igherzog/hawc_software/miniconda3/envs/new_hal) and am using a fresh install of the environment.

python /data/disk01/home/igherzog/Grad_stuff/J2031_415/threeml-analysis-scripts/fitModel/fitModel.py --use-bins B2C0Ea B2C0Eb B2C0Ec B2C0Ed B2C0Ee B3C0Eb B3C0Ec B3C0Ed B3C0Ee B3C0Ef B4C0Ec B4C0Ed B4C0Ee B4C0Ef B4C0Eg B5C0Ec B5C0Ed B5C0Ee B5C0Ef B5C0Eg B5C0Eh B6C0Ed B6C0Ee B6C0Ef B6C0Eg B6C0Eh B6C0Ei B7C0Ee B7C0Ef B7C0Eg B7C0Eh B7C0Ei B8C0Eg B8C0Eh B8C0Ei B8C0Ej B9C0Eg B9C0Eh B9C0Ei B9C0Ej B9C0Ek B10C0Eh B10C0Ei B10C0Ej B10C0Ek B10C0El --ROI-radius 5 -o . --map-tree /lustre/hawcz01/scratch/userspace/dezhih/service/nn_estimator/nn_sky_map/skymap/pass5_nn_version4_chunk1031_chunk1143/combined/combined-chunk103-chunk1142/rejiggered-chunk103-chunk1142/Pass5-Version4-NN-maptree-ch103-ch1143.root --model /data/disk01/home/igherzog/Grad_stuff/J2031_415/threeml-analysis-scripts/fitModel/Models/NHIT_systematic_studies/final_models/NN_DBE_final_model.yml --ROI-template /data/disk01/home/igherzog/Grad_stuff/J2031_415/threeml-analysis-scripts/fitModel/fits_files/2_degree_mask.fits --ROI-center 306.456 40.731 --like --Name NN-trial --det-res /lustre/hawcz01/scratch/userspace/dezhih/service/nn_estimator/nn_sky_map/skymap/pass5_nn_version4/combined/rejiggered/maptree/detRes-NN-allDecs-aligned.root --SaveModel --estimator P5_NN_2D --hawcPlugin hal

Expected behavior I expected the fit to converge normally (see "good" log file attached below) and finish without crashing. It displays two sets of resultant fit parameters separated by the error calculation step my analysis is crashing on.

Log files successful-fit-result.txt failed-fit-result.txt

Desktop (please complete the following information): CentOS 7 on the UMD cluster that HAWC uses for part of its analysis.

Additional context Please let me know if this is the wrong place for this, HAWC also has a gitlab repository but I think this is an issue with 3ML and not the fitting script used.