Setting a variance model in the lumispy object after Jacobian transform

[jordiferrero]

After some discussion with @LMSC-NTappy, we realised that the variance model stored in the LumiSpectrum.metadata.Signal.Noise_properties.variance should no longer be homoscedastic.

Instead, the noise variance should also be converted to eV, but with the intensities multiplied by the square of the jacobian (according to Var(aX) = a**2*Var(X)).

I am not sure what the best way to implement this subtlety is in our luminescence object. I understand that, once you have your signal, then you can compute the variance manually and then apply the Jacobian transformation. But is there a way to define this noise in a generic way (to the class rather than to the data), so when the cl.to_eV() function is called, such transformation is also applied to the variance metadata (should there be any).

This is the Hyperspy guideline on noise metadata.

Let's use this chat to discuss further @LMSC-NTappy. Any thoughts?

[LMSC-NTappy]

Dear @jordiferrero,

thank for putting a comment on this. Indeed, it would be great to implement this! I think it is much of a problem, the noise properties of hyperspy admit a signal object of the same dimensions as the data object in the sig.metadata.Signal.Noise_properties.variance properties of the object. There's even a method dedicated to setting it up: data_ev.estimate_poissonian_noise_variance(sev_varmap)

Based on my workflow, I would do something like this:

variance_map_nm = data_nm.get_variance_map() #Assuming that a variance map already exists
data_eV = data_nm.to_eV() #Considering that the jacobian conversion is performed

EnergyAxis = data_eV.axes_manager.signal_axes[0] #To get axis points of the energy axis

variance_map_eV = sev._deepcopy_with_new_data(\
            varmap.isig[::-1].data*(hc/EnergyAxis**2)**2) #Jacobian-square renormalised energy map

data_eV.estimate_poissonian_noise_variance(variance_map_eV )

What do you think?

[jlaehne]

Thanks for the initiative.

In principle, I would add a function var2eV similar to https://github.com/LumiSpy/lumispy/blob/34a247bfe37e9b74967908f5524f0319eae6bfd7/lumispy/utils/axes.py#L108

It would differ only by the additional squaring (your 4th line). Then in https://github.com/LumiSpy/lumispy/blob/34a247bfe37e9b74967908f5524f0319eae6bfd7/lumispy/signals/luminescence_spectrum.py#L58 one would check whether the variance in the metadata is set and if yes call the conversion. Thus the transformation would be integrated directly in to_eV().

We would have to do the same for to_invcm.

I could draft a PR proposal later today, unless you want to do it yourself.

[LMSC-NTappy]

Thanks for this! Sounds just what we need.

My only remaining question is, is there a method get the variance map from the linear parameters set in Noise Properties? I have never tried to set these and have always computed the variance map directly (bc I estimate it from PCA)

Cheers

Nicolas

[jordiferrero]

Just to document this discussion for the future. @jlaehne has set as default the s.estimate_poissonian_noise_variance() for estimating the noise in any general noise signal. However, if you pass in your own noise signal, as @LMSC-NTappy suggests, then the Jacobian transformation also works.