search

gully / ynot

Astronomical échellogram digital twins with pixel-perfect machine learning: rehabilitating archival data and pathfinding for EPRV
https://ynot.readthedocs.io
MIT License
4 stars 0 forks source link

Implement cosmic ray pre-processing in the dataset init #1

Closed gully closed 3 years ago

gully commented 3 years ago

The cosmic rays tend to bloat the genuine spectral traces. Consider an optional cosmic-ray pre-processing step embedded into the dataset __init__ stage.

gully commented 3 years ago

Ok, we have added this functionality as an init: FPADataset( inpaint_cosmic_rays = True)

This seems to work fine! It's a bit slow, adding about 10 seconds to load the dataset for only two AB nods, but the performance is pretty good. Ultimately what we should do is retain the masks and then apply those when comparing the model to data. For now we will leave them as inpainted.

inpainting_dataset_demo

gully commented 3 years ago

Note: a side effect of this step is that it multiples the raw signal by the GAIN. We should multiply by the same scalar to data that is not inpainted. Alternatively we could call the gain 1.0, but the cosmic ray algorithm uses the gain when determining the SNR, so best to do things in terms of electrons if possible.