Sidpy Fitter save Fitting Function

[ramav87]

In the sippy fitter class, we should save the fitting function. We can do this by serializing the function with dill, before saving it to hdf5. Assuming the we got the fit_data from the fitter class, we can do:

import dill

# Create your dictionary
my_dictionary = {'my_function': SHO_fit_flattened}

# Serialize the functions in the dictionary using dill and encode as base64
encoded_dict = {}
for key, value in my_dictionary.items():
    serialized_value = dill.dumps(value)
    encoded_value = base64.b64encode(serialized_value).decode('utf-8')
    encoded_dict[key] = encoded_value

fit_data.metadata['fitting_function'] = encoded_dict #save to the metadata

Then to access the function again, we can do

fit_fn_packed = fit_data.metadata['fitting_function']
loaded_dict = {}
encoded_value = fit_fn_packed['my_function']
serialized_value = base64.b64decode(encoded_value)
loaded_dict[key] = dill.loads(serialized_value)
fit_fn = loaded_dict['my_function'] #retrieve the function

[ramav87]

This functionality has been added to the fitter_fn branch:

https://github.com/pycroscopy/sidpy/tree/fitter_fn

For reading the functions, I modified pyNSID:

https://github.com/pycroscopy/pyNSID/tree/decode_functions

Though pyNSID passes tests, sippy fails almost all of them with this update. Will need to look into it.

[utkarshp1161]

I will try to find out more about the failing tests.

[utkarshp1161]

HI @ramav87 ,

Assertion failing at line 63(validate_dataset_properties function) in sidpy/tests/sid/test_dataset.py when executing test_fitter.py.

Reason: there is an additional key "fitting_functions" within the dataset.metadata that is not present in the standalone metadata dictionary.

metadata.keys(): dict_keys(['info_1', 'instrument', 'fit_parms_dict'])

dataset.metadata.keys(): dict_keys(['info_1', 'instrument', 'fit_parms_dict', 'fitting_functions'])

QHow do you suggest to handle this?

[ramav87]

The best way to proceed is that you should modify the test so it can now also look for the fitting function if available, and if available, it should be able to read it properly (so the test should be modified to reflect that)

On Tue, Mar 5, 2024 at 12:18 AM utkarshp1161 @.***> wrote:

Assertion failing at line 63(validate_dataset_properties function) in /Users/utkarshpratiush/project/sidpy_rama_branch/sidpy/tests/sid/test_dataset.py when executing test_fitter.py:

Reason: there is an additional key "fitting_functions" within the dataset.metadata that is not present in the standalone metadata dictionary.

metadata.keys(): dict_keys(['info_1', 'instrument', 'fit_parms_dict'])

dataset.metadata.keys(): dict_keys(['info_1', 'instrument', 'fit_parms_dict', 'fitting_functions'])

QHow do you suggest to handle this?

— Reply to this email directly, view it on GitHub https://github.com/pycroscopy/sidpy/issues/200#issuecomment-1977985297, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNDQFK4TL2GGX5WOTHSBRTYWVIQTAVCNFSM6AAAAABDCEHAYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZXHE4DKMRZG4 . You are receiving this because you authored the thread.Message ID: @.***>

[utkarshp1161]

Addressed in :

https://github.com/pycroscopy/sidpy/commit/fbdc67bb5b18d4efa6803937056b1a05dc9c6ab3