Issues with v1.2 validation

[registerrier]

In the preparation of Gammapy v1.2 release, an issue appeared in the validation hess-dl3-dr1. The memory usage goes to very large values and the action hangs. The issue appears during the msh1552 fit.

[registerrier]

The output is here:

INFO:__main__:Running 3d analysis, msh1552
INFO:__main__:Reading config
INFO:gammapy.analysis.config:Setting logging config: {'level': 'INFO', 'filename': None, 'filemode': None, 'format': None, 'datefmt': None}
INFO:__main__:Running observations selection
INFO:gammapy.analysis.core:Fetching observations.
INFO:gammapy.data.data_store:Observations selected: 20 out of 20.
INFO:gammapy.analysis.core:Number of selected observations: 20
INFO:__main__:Running data reduction
INFO:gammapy.analysis.core:Creating reference dataset and makers.
INFO:gammapy.analysis.core:Creating the background Maker.
WARNING:gammapy.analysis.core:No background maker set. Check configuration.
INFO:gammapy.analysis.core:Start the data reduction loop.
INFO:gammapy.makers.reduce:Computing dataset for observation 20136
INFO:gammapy.makers.reduce:Running MapDatasetMaker
[...]  removed for brevity
INFO:gammapy.makers.reduce:Running MapDatasetMaker
INFO:gammapy.makers.reduce:Running SafeMaskMaker
INFO:__main__:Setting the model
INFO:gammapy.analysis.core:Reading model.
INFO:gammapy.analysis.core:Models

Component 0: SkyModel

  Name                      : msh1552
  Datasets names            : None
  Spectral model type       : PowerLawSpectralModel
  Spatial  model type       : DiskSpatialModel
  Temporal model type       : 
  Parameters:
    index                         :      2.000   +/-    0.00             
    amplitude                     :   1.00e-11   +/- 0.0e+00 1 / (TeV s cm2)
    reference             (frozen):      1.400       TeV         
    lon_0                         :    228.530   +/-    0.00 deg         
    lat_0                         :    -59.160   +/-    0.00 deg         
    r_0                   (frozen):      0.200       deg         
    e                     (frozen):      0.000                   
    phi                   (frozen):      0.000       deg         
    edge_width            (frozen):      0.010                   

Component 1: FoVBackgroundModel

  Name                      : stacked-bkg
  Datasets names            : ['stacked']
  Spectral model type       : PowerLawNormSpectralModel
  Parameters:
    norm                          :      1.000   +/-    0.00             
    tilt                  (frozen):      0.000                   
    reference             (frozen):      1.000       TeV         

INFO:__main__:Running fit ...
INFO:gammapy.analysis.core:Fitting datasets.
WARNING:gammapy.modeling.parameter:Value 0.0 is outside bounds [0.0, 1.0] for parameter 'e'
^C
Aborted!

The memory usage reaches 40 Gb!!!

[registerrier]

The issue appears when the minmizer puts all parameter values to nan after a weird choice of parameter values.

type    name      value         unit        error      min        max     frozen is_norm link prior
---- --------- ----------- -------------- --------- ---------- ---------- ------ ------- ---- -----
         index  2.4217e+00                0.000e+00        nan        nan  False   False           
     amplitude  3.1032e-12 TeV-1 s-1 cm-2 0.000e+00  0.000e+00        nan  False    True           
         lon_0  2.2854e+02            deg 0.000e+00  2.270e+02  2.300e+02  False   False           
         lat_0 -5.9162e+01            deg 0.000e+00 -6.066e+01 -5.766e+01  False   False           
           r_0  1.9433e-01            deg 0.000e+00  1.000e-01  5.000e-01  False   False           
             e  5.7091e-01                0.000e+00  0.000e+00  1.000e+00  False   False           
           phi -6.3301e-07            deg 0.000e+00        nan        nan  False   False           
          norm  1.0067e+00                0.000e+00        nan        nan  False    True           
          tilt  1.2337e-01                0.000e+00        nan        nan  False   False           
type    name      value         unit        error      min        max     frozen is_norm link prior
---- --------- ----------- -------------- --------- ---------- ---------- ------ ------- ---- -----
         index -5.6539e+01                0.000e+00        nan        nan  False   False           
     amplitude  5.2373e-10 TeV-1 s-1 cm-2 0.000e+00  0.000e+00        nan  False    True           
         lon_0  2.2834e+02            deg 0.000e+00  2.270e+02  2.300e+02  False   False           
         lat_0 -5.8025e+01            deg 0.000e+00 -6.066e+01 -5.766e+01  False   False           
           r_0  4.9999e-01            deg 0.000e+00  1.000e-01  5.000e-01  False   False           
             e  7.9507e-04                0.000e+00  0.000e+00  1.000e+00  False   False           
           phi -4.0325e-01            deg 0.000e+00        nan        nan  False   False           
          norm  3.8538e+02                0.000e+00        nan        nan  False    True           
          tilt -1.8011e+02                0.000e+00        nan        nan  False   False           
type    name   value      unit        error      min        max     frozen is_norm link prior
---- --------- ----- -------------- --------- ---------- ---------- ------ ------- ---- -----
         index   nan                0.000e+00        nan        nan  False   False           
     amplitude   nan TeV-1 s-1 cm-2 0.000e+00  0.000e+00        nan  False    True           
         lon_0   nan            deg 0.000e+00  2.270e+02  2.300e+02  False   False           
         lat_0   nan            deg 0.000e+00 -6.066e+01 -5.766e+01  False   False           
           r_0   nan            deg 0.000e+00  1.000e-01  5.000e-01  False   False           
             e   nan                0.000e+00  0.000e+00  1.000e+00  False   False           
           phi   nan            deg 0.000e+00        nan        nan  False   False           
          norm   nan                0.000e+00        nan        nan  False    True           
          tilt   nan                0.000e+00        nan        nan  False   False   

With nan parameters, the MapEvaluator goes haywire.

[registerrier]

This is solved. CLosing