Starfish-develop / Starfish

Tools for Flexible Spectroscopic Inference
https://starfish.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
68 stars 22 forks source link

global_cov:log_amp reasonable values for the sigma-contours plot? #148

Open AustenWallis opened 1 year ago

AustenWallis commented 1 year ago

Hi,

We are trialling starfish to see if we can implement some of your ideas here into our groups existing radiative transfer work.

We have your example files (setup.ipynb/single.ipynb) and it all runs smoothly apart from the sigma-contours are enormous (1e8) on the residuals plot. Example below.

Screenshot 2022-12-13 at 10 57 43

We have managed to reduce this error through changing the log_amp parameter in the SpectrumModel(...) function to -11 (instead of 38) and changing the priors "global_cov:log_amp" value to (-11, 3) instead of (38,1). For us, this produces a similar looking plot to the readthedocs example. Numbers and plot below

Screenshot 2022-12-13 at 11 06 45

Is a log_amp value of -11 considered to be okay/reasonable and normal behaviour?

As this seems significantly different to the original. I'm not sure if you have changed something in the code since the readthedocs was written that might account for this difference?

The only other difference we had noted was the numbers outputted from training the emulator was different, such as the log likelihood. Does this all sound okay? Numbers from initial and secondary training below.

1st

Screenshot 2022-12-13 at 11 18 12

2nd

Screenshot 2022-12-13 at 11 18 18

This was verified by my supervisor on a separate machine. We both produced identical log-likelihood numbers of the emulator training session and in turn, identical log_amp numbers. Starfish version 0.4.2

Many thanks, Austen

mileslucas commented 1 year ago

Hi Austen,

It's been a while since I've used Starfish to fit spectra (~3-4 years) and I only used it on a couple stars from the same low resolution instrument (IRTF/SPeX) so I'm actually not as familiar with reasonable outputs. That being said, if the fitting looks good (it does) and you're not having troubles with convergence (looks okay since two separate computers got same result) then I wouldn't be too wary of it.

ITDebicki commented 1 year ago

To add to this thread, the issue with the disparity seems to be coming from this commit, where planck scaling was removed from the log_amp parameter: https://github.com/Starfish-develop/Starfish/commit/3f058df47c9bfd5634cda3d2bb8f4ad16bb2da4e

AustenWallis commented 1 year ago

Okay sure thing, thanks for the replies!