continuum fitting fails with complex numbers

[londumas]

Sometime for some low S/N spectrum I get:

/home/gpfs/manip/mnt0607/bao/hdumasde/Program/Igm_hub/pyLyA/py/pylya/data.py:145: ComplexWarning: Casting complex values to real discards the imaginary part mig.migrad()

To investigate, but seems to be linked to the quasar continuum going negative, zero or from negative to positive.

I get that with: /home/gpfs/manip/mnt0607/bao/hdumasde/Program/Igm_hub/pyLyA/bin/do_deltas.py --in-dir /home/gpfs/manip/mnt0607/bao/Spectra/SpectraV5_10_0 --drq /home/gpfs/manip/mnt0607/bao/Spectra/DR14Q_v1_1.fits --out-dir /home/gpfs/manip/mnt0607/bao/hdumasde/Code/Igm_Hub/Test_pyLya/ --rebin 1 --mode spec --zqso-min 6.0 --zqso-max 10.0 --lambda-min 3600.0 --lambda-max 7235.0 --lambda-rest-min 0.001 --lambda-rest-max 10000.0

[londumas]

Seems to be linked also to unmasked skylines

[ngbusca]

You are pushing the code into realms where it wasn't built to work... I'm not sure I wan't to fix this

[andreufont]

Now that you are looking at it, Hélion, it would be quite interesting to look at what fraction of spectra do not get a valid continuum from the code, as a function of quasar redshift and magnitude (r).

Andreu

On 25 January 2017 at 17:05, Nicolas Busca notifications@github.com wrote:

You are pushing the code into realms were it wasn't built to work... I'm not sure I wan't to fix this

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/igmhub/pyLyA/issues/17#issuecomment-275167544, or mute the thread https://github.com/notifications/unsubscribe-auth/ADscRzGajemP-3d2Lgv21HikCW-XlQHQks5rV4DXgaJpZM4LtsGc .

[londumas]

I kind of disagree Nicolas, We want the code to perform everywhere, not only in the Lyman-alpha forest region. So we should track such fails, see the reason why and remove them from our forest list.

Maybe we should add a self.fit_is_correct bool in the forest to see if the fit is good, at least test the "is_valid" value of iminiut.

[ngbusca]

Sorry, but these flags:

--lambda-rest-min 0.001 --lambda-rest-max 10000.0

don't make sense.

@andreufont the continuum fit doesn't fail on any spectrum.

[andreufont]

Wow! That's impressive. I thought you had a minimum SNR for which you were able to get a good continuum fit. Are there no quasars in DR12Q that are pathological, or that have all pixels in the forest masked?

Andreu

On 25 January 2017 at 17:27, Nicolas Busca notifications@github.com wrote:

Sorry, but these flags:

--lambda-rest-min 0.001 --lambda-rest-max 10000.0

don't make sense.

@andreufont https://github.com/andreufont the continuum fit doesn't fail on any spectrum.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/igmhub/pyLyA/issues/17#issuecomment-275174079, or mute the thread https://github.com/notifications/unsubscribe-auth/ADscRxzXhxjD8bJOu8_uVuiHk2K-rueLks5rV4YAgaJpZM4LtsGc .

[londumas]

@ngbusca I don't say that the continuum only fails when taking --lambda-rest-min 0.001 --lambda-rest-max 10000.0 I just put the command to keep track, what quasars does that is yet to understand and find.

The only things I know now is that it happens for some cases.

[ngbusca]

@andreufont I preselect spectra with 50 or more unmasked pixels

[ngbusca]

@londumas then please give an example of more standard choices that result in the same problem

[londumas]

This happens with "forest.var_lss" giving negative numbers through interpolation. Then the code takes sp.log(we), where we=iv/(ivvar_lssm**2+1). Apparently the log value of a negative number is a complex.

This happens when the "var_lss" is not well measured or extrapolated way out if its range.

For example at the blue of the lyman-alpha forest, I don't know if there are other cases, maybe we can neglect this effect.

python /home/gpfs/manip/mnt0607/bao/hdumasde/Program/Igm_hub/pyLyA/bin/do_deltas.py --in-dir /home/gpfs/manip/mnt0607/bao/Spectra/SpectraV5_10_0 --drq /home/gpfs/manip/mnt0607/bao/Spectra/DR14Q_v1_1.fits --out-dir /home/gpfs/manip/mnt0607/bao/hdumasde/Code/Igm_Hub/Test_pyLya/ --rebin 1 --mode spec --zqso-min 5.0 --zqso-max 6.0 --lambda-min 3600.0 --lambda-max 7235.0 --lambda-rest-min 100.0 --lambda-rest-max 800.0

[londumas]

An easy fix is to add var_lss[(var_lss<0.)] = 0. after var_lss = forest.var_lss(self.ll)

[ngbusca]

can I close this issue?

[londumas]

won't fix since seems to append with unrealistic configuration