Load fit res

[EdLeming]

Fixed a bug in echidna.output.store.load_fit_results - files can now be dumped and loaded sucessfully

[EdLeming]

@ashleyrback, could you quickly have a look at / merge this for me? It's only a two line change

[ashleyrback]

Looks fine.

One thing though, on an a FitResults hdf5 created previously, I just tested load, then dump - which was fine, but then when I tried load again, I got this error:

/home/ashley/snoplus/software/echidna/echidna-pr/echidna/core/config.pyc in load(cls, global_config, spectral_config, name)

    469         if not spectral_config.get(main_key):
    470             logging.getLogger("extra").debug("\n\n%s\n" % str(spectral_config))
--> 471             raise KeyError("Cannot read spectra fit config dictionary. "
    472                            "Please check it has the correct form")
    473         for syst in spectral_config[main_key]:

KeyError: 'Cannot read spectra fit config dictionary. Please check it has the correct form'

and looking in the log file:

config.py:470 [load()] DEBUG   : 

{u'spectral_fit_parameters': {}}

It appears to have deleted the spectral fit parameters entries.

Do you get this problem as well with a FitResults produced purely through the latest version?

[EdLeming]

Hmmmm, yeah I do stil see this.

Interestingly, if I call fit_results._fit_config.get_name() in your script before you dump, the error disappears...

I added a commit to sort out an issue with dumping the arrays without retaining the ordering - I had to do this for all the other load / dump methods earlier in the week

[ashleyrback]

@EdLeming, something still doesn't seem quite right here, see ipython session:

In [45]: fit_results.get_fit_config().dump()
Out[45]: 
({'global_fit_parameters': {}},
 {'spectral_fit_parameters': {'test_spectrum_rate': {'base': None,
    'best_fit': None,
    'bins': 101,
    'current_value': None,
    'high': 13.0,
    'logscale': None,
    'logscale_deviation': None,
    'low': 7.0,
    'penalty_term': None,
    'prior': 10.0,
    'sigma': 0.1}}})

In [46]: store.dump_fit_results("fit_results.hdf5", fit_results)

In [47]: fit_results = store.load_fit_results("fit_results.hdf5")

In [48]: fit_results.get_fit_config().dump()
Out[48]: ({'global_fit_parameters': {}}, {'spectral_fit_parameters': {}})

In [49]: store.dump_fit_results("fit_results.hdf5", fit_results)

In [50]: fit_results = store.load_fit_results("fit_results.hdf5")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-50-999ce4e624d0> in <module>()
----> 1 fit_results = store.load_fit_results("fit_results.hdf5")

/home/ashley/snoplus/software/echidna/echidna-pr/echidna/output/store.pyc in load_fit_results(file_path, group_name)
    466             spectral_config=json.loads(group.attrs["fit_config"],
    467                                        object_pairs_hook=OrderedDict)[1],
--> 468             name=fit_config_name)
    469         fit_results = FitResults(fit_config=fit_config,
    470                                  spectra_config=spectra_config, name=name)

/home/ashley/snoplus/software/echidna/echidna-pr/echidna/core/config.pyc in load(cls, global_config, spectral_config, name)
    469         if not spectral_config.get(main_key):
    470             logging.getLogger("extra").debug("\n\n%s\n" % str(spectral_config))
--> 471             raise KeyError("Cannot read spectra fit config dictionary. "
    472                            "Please check it has the correct form")
    473         for syst in spectral_config[main_key]:

KeyError: 'Cannot read spectra fit config dictionary. Please check it has the correct form'

The contents of the fit config seems to be deleted at some point between dump and load.

[EdLeming]

OK, we'll look at this later

[ashleyrback]

@EdLeming I've created a unittest to help us debug this, see my copy of this branch. Feel free to merge this in to your branch to help with debugging. I'll try an continue having a look too.

[EdLeming]

Thanks, I'll try to have a look today

[EdLeming]

It also looks like the penalty terms aren't being saved correctly. My penalty terms are equal to my best fit results

[EdLeming]

@ashleyrback, I've merged with your branch and set @jwaterfield to review.

[jwaterfield]

All good and makes sense. Tests now pass. Merging