search

DeepIGM / dla_cnn

Pipeline to detect Ly-alpha emission lines in the SDSS3 catalog sightlines. Uses convolutional neural networks to classify sightlines, locate DLAs, and measure column density.
MIT License
9 stars 7 forks source link

Update preprocess.py for normalization #34

Closed ZechangSun closed 4 years ago

ZechangSun commented 4 years ago

Dear X, this code is about the normalization, hope for your advice. Thanks!

profxj commented 4 years ago

Ok, I will review this once the PR #33 is done. Thanks.

profxj commented 4 years ago

This is a little more challenging then what we have tried so far. In particular the quasars will have a wide range of redshifts.

I've suggested 2 options above. Please explore these for a set of ~10 quasars with z=2-4. Thanks.

ZechangSun commented 4 years ago

I finally chose the 1st option since [1070,1170] is out of some spectrum's wavelength range. I have also squashed the bug and updated the previous work which may have problems because of that bug. I have updated a notebook which shows how to normalize the spectrum and 19 normalized spectrum redshift from 2 to 4. Hope for your advice. Thanks!

profxj commented 4 years ago

Ok, a few things:

  1. Which spectra do not cover [1070, 1170]? Or better put, if they don't then we can't search for DLAs, right?
  2. I'd like to see the normalized data that cover the Lya forest. This is what we will be analyzing.

Thanks!

ZechangSun commented 4 years ago

Thanks for your advice above. I discussed with @caiz12, and his advice is:

  1. If (1+z_qso)1216-3000/c 1216 *(1+z_qso) is bluer than 3800A, the sightline is not needed, because of the low s/n in the part lower than 3800A;
  2. when do normalizing, decide which is bluer, 3800A or rest-frame 1070A. If latter, use 3800A, that is due to the low s/n in the part lower than 3800A. I have corrected the other aspects as you have suggested. Thanks!
profxj commented 4 years ago

Ok, this is almost good. Address the bugs and move the normalization code to its own method in preprocess.py.

We'll have to see how this works on low S/N data, but hopefully it will be ok. Thanks @ZechangSun

ZechangSun commented 4 years ago

Dear X, I noticed that you have reminded me that some comments needed to be added to make my code more readable, I have updated my code and added those comments. Thanks for your advice. Besides, I have also noticed that you have changed the code slightly for its readability, I really appreciate your advice, my previous work is confusing sometimes, and I have learnt from your suggestions and will make sure its readability next time. I have updated the other parts of the code to be consistent with the changes you have made. Wish your good healthy and thanks for your help!

profxj commented 4 years ago

Great!

I am merging now.

profxj commented 4 years ago

merging.