implement data quality / bad pixels

[zonca]

stpipe has a dq_init step that initializes the masks/flags for bad pixels, see https://jwst-pipeline.readthedocs.io/en/latest/jwst/dq_init/description.html

the structure of the fits file is:

PIXELDQ is common to all readouts, GROUPDQ instead has the same shape as the science extension.

dq_init connects to the CRDS and gets a static mask based on file metadata and prepares the PIXELDQ extension based on it, then initializes a zero array for the GROUPDQ.

See the implementation at https://github.com/spacetelescope/jwst/tree/master/jwst/dq_init

The definition of all flags used by JWST is at https://jwst-pipeline.readthedocs.io/en/latest/jwst/references_general/references_general.html#data-quality-flags

[zonca]

@arunsurya77 can you prepare an example reference mask I can use for testing? do we have our own definition for flags or we can borrow the one from JWST?

[arunsurya77]

Hi Andrea, Sorry for the delay. I talked to Shelley about this last group meeting, since this applies at the detector level ROP Ramps and not the Level-2 image models I was not sure how to proceed. She suggested we talk about it sometime. I can have another look today and update you. For the flags, I think most of what is in JWST can be reused. Let me check once more today and get back to you on this. Cheers Arun

On Tue, 2 Apr 2019 at 18:50, Andrea Zonca notifications@github.com wrote:

@arunsurya77 https://github.com/arunsurya77 can you prepare an example reference mask I can use for testing? do we have our own definition for flags or we can borrow the one from JWST?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479293635, or mute the thread https://github.com/notifications/unsubscribe-auth/AoO_ri4BL2JEqvq5W5QbS3nPICd2vPPVks5vdAjVgaJpZM4cZbps .

[arunsurya77]

Hi Andrea, I am attaching a mask reference file used for the static badpixel mapping. I just have 100 random hot pixels and 100 dead pixels. The flag values are per the following. It follows the MaskModel schema used for the initdq steps. mask_test.fits https://drive.google.com/file/d/196Sj89ziqH8KXxozaw2wrgzjqNqVhqWh/view?usp=drive_web

https://jwst-pipeline.readthedocs.io/en/latest/jwst/references_general/references_general.html#data-quality-flags

Cheers Arun

On Wed, 3 Apr 2019 at 09:57, Arun Surya arunsuryaoffice@gmail.com wrote:

Hi Andrea, Sorry for the delay. I talked to Shelley about this last group meeting, since this applies at the detector level ROP Ramps and not the Level-2 image models I was not sure how to proceed. She suggested we talk about it sometime. I can have another look today and update you. For the flags, I think most of what is in JWST can be reused. Let me check once more today and get back to you on this. Cheers Arun

On Tue, 2 Apr 2019 at 18:50, Andrea Zonca notifications@github.com wrote:

@arunsurya77 https://github.com/arunsurya77 can you prepare an example reference mask I can use for testing? do we have our own definition for flags or we can borrow the one from JWST?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479293635, or mute the thread https://github.com/notifications/unsubscribe-auth/AoO_ri4BL2JEqvq5W5QbS3nPICd2vPPVks5vdAjVgaJpZM4cZbps .

[zonca]

thanks @arunsurya77 I need to create a RampModel to test the bad pixel map, have you done that already for your work on the ROP? in any case it would be very useful if you can create a pull request even with a temporary version of your code, so I don't risk of duplicating your work.

[arunsurya77]

Hi Andrea, I had created a RampModel of 10x10 pixel with in x-y. So I think we will need 4096x4096 window size to test bad pixel stuff. I will create it and send it to you. I have a script to make those.

Cheers Arun

On Wed, 3 Apr 2019 at 19:54, Andrea Zonca notifications@github.com wrote:

thanks @arunsurya77 https://github.com/arunsurya77 I need to create a RampModel to test the bad pixel map, have you done that already for your work on the ROP? in any case it would be very useful if you can create a pull request even with a temporary version of your code, so I don't risk of duplicating your work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479730465, or mute the thread https://github.com/notifications/unsubscribe-auth/AoO_roC8DG5cfFaK8_MBnciq0GFhFE5Lks5vdWlbgaJpZM4cZbps .

[zonca]

thanks, can you please point to the location on Galactica?

On Thu, Apr 4, 2019 at 9:55 AM Arun Surya notifications@github.com wrote:

Hi Andrea, I had created a RampModel of 10x10 pixel with in x-y. So I think we will need 4096x4096 window size to test bad pixel stuff. I will create it and send it to you. I have a script to make those.

Cheers Arun

On Wed, 3 Apr 2019 at 19:54, Andrea Zonca notifications@github.com wrote:

thanks @arunsurya77 https://github.com/arunsurya77 I need to create a RampModel to test the bad pixel map, have you done that already for your work on the ROP? in any case it would be very useful if you can create a pull request even with a temporary version of your code, so I don't risk of duplicating your work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479730465 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AoO_roC8DG5cfFaK8_MBnciq0GFhFE5Lks5vdWlbgaJpZM4cZbps

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479978884, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXYcvGZeDLO0HFhvudJ3AuYHHzBYdGIks5vdi6CgaJpZM4cZbps .

[arunsurya77]

Hi, It is saved in /data/group/packages/idlpro/iris/readout_iris/test_ramp.fits Cheers Arun

On Thu, 4 Apr 2019 at 10:12, Andrea Zonca notifications@github.com wrote:

thanks, can you please point to the location on Galactica?

On Thu, Apr 4, 2019 at 9:55 AM Arun Surya notifications@github.com wrote:

Hi Andrea, I had created a RampModel of 10x10 pixel with in x-y. So I think we will need 4096x4096 window size to test bad pixel stuff. I will create it and send it to you. I have a script to make those.

Cheers Arun

On Wed, 3 Apr 2019 at 19:54, Andrea Zonca notifications@github.com wrote:

thanks @arunsurya77 https://github.com/arunsurya77 I need to create a RampModel to test the bad pixel map, have you done that already for your work on the ROP? in any case it would be very useful if you can create a pull request even with a temporary version of your code, so I don't risk of duplicating your work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479730465 , or mute the thread <

https://github.com/notifications/unsubscribe-auth/AoO_roC8DG5cfFaK8_MBnciq0GFhFE5Lks5vdWlbgaJpZM4cZbps

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479978884 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AAXYcvGZeDLO0HFhvudJ3AuYHHzBYdGIks5vdi6CgaJpZM4cZbps

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/4#issuecomment-479984958, or mute the thread https://github.com/notifications/unsubscribe-auth/AoO_ruOuyXFOlOXk9Gld98CkfxsF5Ujcks5vdjKLgaJpZM4cZbps .

[zonca]

thanks @arunsurya77, I added a schema for a TMT ramp: https://github.com/zonca/jwst/commit/48ec2e49046de68f0a16db74959685ed3b44690e

[zonca]

see the attached file where I changed the metadata so that it opens up as a TMTRampModel:

ramp.zip

[zonca]

ok, I have ported the dq_init code to iris_pipeline, see example usage: https://gist.github.com/zonca/e15620ff5d26652bc201b180ec00cdce

documentation is available in https://iris-pipeline.readthedocs.io/en/latest/algorithms.html#remove-detector-artifacts