search

threeML / hawc_hal

HAWC Accelerated Likelihood - python-only framework for HAWC data analysis
BSD 3-Clause "New" or "Revised" License
11 stars 22 forks source link

Fit broken #34

Closed cbrisboi closed 4 years ago

cbrisboi commented 4 years ago

Currently running a test fit using the public dataset on (https://data.hawc-observatory.org/datasets/crab_data/index.php) is broken, after some discussion internally it appears more complicated scripts are breaking in a similar way.

I set up an environment using these lines:

ENV_NAME=hal_test
# create the environment
conda create --name $ENV_NAME -c conda-forge python=3.7 numpy scipy matplotlib numba

# pip install some things
#pip install --no-binary :all: root_numpy

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1

conda activate $ENV_NAME

# Right now we have numpy 1.17.5, this causes an error with the interpolation package (required in astromodels), so we update numpy
# this gives numpy 1.19.0
pip install numpy 

# When running real example, tables is not optional apparently
#Traceback (most recent call last):
#  File "crab_fit_logparabola.py", line 26, in <module>
#    flat_sky_pixels_size=0.1)
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/hawc_hal/HAL.py", line 59, in __init__
#    self._maptree = map_tree_factory(maptree, roi=roi)
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/hawc_hal/maptree/map_tree.py", line 31, in map_tree_factory
#    return MapTree.from_hdf5(map_tree_file, roi)
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/hawc_hal/maptree/map_tree.py", line 44, in from_hdf5
#    data_analysis_bins = from_hdf5_file(map_tree_file, roi)
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/hawc_hal/maptree/from_hdf5_file.py", line 23, in from_hdf5_file
#    with Serialization(map_tree_file) as serializer:
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/hawc_hal/serialize.py", line 19, in __enter__
#    self._store = HDFStore(self._filename, complib='blosc:lz4', complevel=9, mode=self._mode)
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/pandas/io/pytables.py", line 518, in __init__
#    tables = import_optional_dependency("tables")
#  File "/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/pandas/compat/_optional.py", line 92, in import_optional_dependency
#    raise ImportError(msg) from None
#ImportError: Missing optional dependency 'tables'.  Use pip or conda to install tables.
pip install tables

pip install git+https://github.com/threeML/astromodels.git

pip install git+https://github.com/threeML/threeML.git

pip install git+https://github.com/threeML/hawc_hal.git

Using the data on the public page linked previously, I stripped down that fit example to the following: minimal_fail.py

Changing between minimizers (minuit and scipy) do not appear to affect the outcome, as shown in the logs below. minuit: failure_minimizer_minuit.log

scipy: failure_minimizer_scipy.log

cbrisboi commented 4 years ago

Starting from commit 862df321d4353c2affed5b0fbcba6b517fec5617 I addressed each error encountered in the master branch for hawc_hal and put it in this branch. https://github.com/cbrisboi/hawc_hal/tree/py3_back_to_future

This does not solve everything, but it does minimally solve the issue for this environment and minimal_fail.py. I will now switch to testing on the script provided on the hawc public page to examine the remaining things not updated a bit more widely. I was inspired to this approach by @ndilalla's statement that

I tried to check out the HAL master before the merge of the py3 pull request and I am able to reproduce the correct results (with the last version of threeML and astromodels). So this confirms us it is something we introduced with the last pull request

My assumption/hope is that by carefully updating errors as they are encountered we can overcome/identify where problems were introduced.

cbrisboi commented 4 years ago

This is the output of minimal_fail.py on this branch

/Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/astromodels/core/parameter.py:555: UserWarning: We have set the min_value of xc to 1e-99 because there was a postive transform
  warnings.warn('We have set the min_value of %s to 1e-99 because there was a postive transform' % self.path)

WARNING UserWarning: Using default configuration from /Users/chad/miniconda3/envs/hal_test/lib/python3.7/site-packages/threeML/data/threeML_config.yml. You might want to copy it to /Users/chad/.threeML/threeML_config.yml to customize it and avoid this warning.

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

Creating singleton for /Users/chad/science/hal_tests/HAWC_9bin_507days_crab_response.hd5

WARNING UserWarning: We have set the min_value of crab.spectrum.main.Powerlaw.K to 1e-99 because there was a postive transform

WARNING RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

trial values: -23,-2.5 -> logL = -23541.653
trial values: -22.951,-2.5 -> logL = -23453.068
trial values: -23.049,-2.5 -> logL = -23623.529
trial values: -22.995,-2.5 -> logL = -23533.102
trial values: -23.005,-2.5 -> logL = -23550.137
trial values: -23,-2.4947 -> logL = -23539.912
trial values: -23,-2.5053 -> logL = -23543.351
trial values: -23,-2.4973 -> logL = -23540.757
trial values: -23,-2.5027 -> logL = -23542.537
trial values: -22.995,-2.5 -> logL = -23533.102
trial values: -22.976,-2.5 -> logL = -23498.220
trial values: -22.927,-2.5 -> logL = -23406.155
trial values: -22.783,-2.5 -> logL = -23085.517
trial values: -22.364,-2.5 -> logL = -21676.379
trial values: -21.235,-2.5 -> logL = -25312.888
trial values: -21.774,-2.5 -> logL = -18997.740
trial values: -21.236,-2.5 -> logL = -25304.889
trial values: -21.499,-2.5 -> logL = -19436.746
trial values: -21.773,-2.5 -> logL = -18993.470
trial values: -21.776,-2.5 -> logL = -19002.054
trial values: -21.774,-2.5 -> logL = -18995.816
trial values: -21.775,-2.5 -> logL = -18999.673
trial values: -21.774,-2.4975 -> logL = -18999.332
trial values: -21.774,-2.5025 -> logL = -18996.161
trial values: -21.629,-2.7871 -> logL = -19213.789
trial values: -21.721,-2.6081 -> logL = -18833.180
trial values: -21.72,-2.6081 -> logL = -18832.284
trial values: -21.721,-2.6081 -> logL = -18834.091
trial values: -21.721,-2.6062 -> logL = -18833.452
trial values: -21.721,-2.6101 -> logL = -18832.933
trial values: -21.721,-2.6069 -> logL = -18833.344
trial values: -21.721,-2.6094 -> logL = -18833.025
trial values: -21.662,-2.6025 -> logL = -18808.553
trial values: -21.677,-2.604 -> logL = -18804.106
trial values: -21.677,-2.604 -> logL = -18804.124
trial values: -21.678,-2.604 -> logL = -18804.100
trial values: -21.677,-2.6028 -> logL = -18803.980
trial values: -21.677,-2.6053 -> logL = -18804.245
trial values: -21.672,-2.5737 -> logL = -18804.071
trial values: -21.675,-2.5887 -> logL = -18803.305
trial values: -21.674,-2.5887 -> logL = -18803.312
trial values: -21.675,-2.5887 -> logL = -18803.306
trial values: -21.675,-2.5877 -> logL = -18803.308
trial values: -21.675,-2.5898 -> logL = -18803.310
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.309
trial values: -21.676,-2.5888 -> logL = -18803.309
trial values: -21.675,-2.5877 -> logL = -18803.309
trial values: -21.675,-2.5898 -> logL = -18803.309
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.309
trial values: -21.676,-2.5888 -> logL = -18803.309
trial values: -21.675,-2.5877 -> logL = -18803.309
trial values: -21.675,-2.5898 -> logL = -18803.309
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5886 -> logL = -18803.304
trial values: -21.675,-2.589 -> logL = -18803.304
trial values: -21.675,-2.5877 -> logL = -18803.310
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5886 -> logL = -18803.304
trial values: -21.675,-2.589 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5887 -> logL = -18803.304
trial values: -21.675,-2.5888 -> logL = -18803.304
trial values: -21.675,-2.5886 -> logL = -18803.304
Best fit values:

                                                       result             unit
parameter                                                                     
crab.spectrum.main.Powerlaw.K      (2.113 +/- 0.028) x 10^-22  1 / (cm2 keV s)
crab.spectrum.main.Powerlaw.index            -2.589 +/- 0.012                 

Correlation matrix:

col0 col1
---- ----
1.00 0.36
0.36 1.00

Values of -log(likelihood) at the minimum:

       -log(likelihood)
HAWC       18803.304206
total      18803.304206

Values of statistical measures:

     statistical measures
AIC          37610.608552
BIC          37629.339351
ndilalla commented 4 years ago

Thank you @cbrisboi! I am more or less doing the same thing. I am working on this branch here: https://github.com/threeML/hawc_hal/tree/fix_py3_pull_request We carefully need to understand which changes have affected the outcome of the analysis.

ndilalla commented 4 years ago

This issue is fixed in the pull request #35 that is almost ready to ship to master.