ceers / ceers-miri

the code for CEERS/MIRI data reduction
1 stars 1 forks source link

kernel crash with background subtraction #1

Open cracraft opened 9 months ago

cracraft commented 9 months ago

I'm working on a set of MIRI data, and was trying to use the background subtraction routine(s) shown here for my program. I ran a set of data through the jwst calibration pipeline through calwebb_image2, and then fed those cal.fits files into the super_bg code, which also uses det_src code block. I copied those two functions and their necessary import statements into my notebook. I commented out the part about cleaning the edges, as that seemed to be calling code I didn't have installed, but otherwise left the functions as-is. I can read in the files fine, and it seems to start detecting sources with no problem, but then partway through, the kernel crashes. I have tried it multiple times, different days, and every time I try it, the code fails to complete and I have to start the kernel over again (jupyter kernel for my processing notebook). Other than reading out the filenames, this is all the output I get as the code is progressing: Building master background from a 11 files 32 src detected! 37 src detected! 34 src detected! 35 src detected! 36 src detected!

Do you have any ideas about what is going wrong? Do I need to be using other parts of the code that I didn't include? Related question, I'm trying this in order to compare how ceers handled errors compared to how a different type of background subtraction at the same stage does. (They don't actually touch the error arrays.) But if I can't get output, I can't compare anything.

Guang91 commented 9 months ago

Dear Misty,

Thanks for trying out the super_bg code!

I never saw the kernel crash error. May I ask you to share your cal.fits files with me? (for debugging purposes only, I will delete them once this is done) I will try to see what happens if the same error is reproduced on my machine.

Cheers, Guang

On Mon, Nov 20, 2023 at 4:24 PM Misty Cracraft @.***> wrote:

I'm working on a set of MIRI data, and was trying to use the background subtraction routine(s) shown here for my program. I ran a set of data through the jwst calibration pipeline through calwebb_image2, and then fed those cal.fits files into the super_bg code, which also uses det_src code block. I copied those two functions and their necessary import statements into my notebook. I commented out the part about cleaning the edges, as that seemed to be calling code I didn't have installed, but otherwise left the functions as-is. I can read in the files fine, and it seems to start detecting sources with no problem, but then partway through, the kernel crashes. I have tried it multiple times, different days, and every time I try it, the code fails to complete and I have to start the kernel over again (jupyter kernel for my processing notebook). Other than reading out the filenames, this is all the output I get as the code is progressing: Building master background from a 11 files 32 src detected! 37 src detected! 34 src detected! 35 src detected! 36 src detected!

Do you have any ideas about what is going wrong? Do I need to be using other parts of the code that I didn't include? Related question, I'm trying this in order to compare how ceers handled errors compared to how a different type of background subtraction at the same stage does. (They don't actually touch the error arrays.) But if I can't get output, I can't compare anything.

— Reply to this email directly, view it on GitHub https://github.com/ceers/ceers-miri/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4OFGGZ5HTOCLR23VZPJ53YFNY3VAVCNFSM6AAAAAA7TDBDFOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAYDENBUHA2TKNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

cracraft commented 9 months ago

The data I was using is proprietary. I'm going to test this with some public data and see if it reproduces the error and I'll get back to you with that.

cracraft commented 9 months ago

Ok, I did test it on a separate dataset, and I see the same behavior. I tried to attach the test notebook here, so you can see what code I tried to run and where it crashed, but .ipynb and .html file types don't seem to be able to be attached here. I started with a set of cal files from program 2736. The filenames are in the notebook, and they were run through the default pipeline steps in calwebb_detector1 and calwebb_image2 up to that point. I figure the problem is in how I'm trying to run just a portion of the code, so maybe you can spot something. Just let me know how or what format I should make the notebook and any data available.

Guang91 commented 9 months ago

Could you email me (gyang206265@gmail.com) your notebook? I can run on my files to reproduce the error.