search

astropy / ccdproc

Astropy affiliated package for reducing optical/IR CCD data
https://ccdproc.readthedocs.io
BSD 3-Clause "New" or "Revised" License
89 stars 87 forks source link

Counts are too High Compared to Linux IRAF and 3 Images in a Single Fits File #806

Open kjkoeller opened 1 year ago

kjkoeller commented 1 year ago

When combining bias, flats, or darks together into a master image, the resulting counts per pixel is on average a lot higher using ccdproc than the Linux counter part.

It's actually only like 10 counts higher between the master bias images, but for the master flats there is a 12k-20k count difference (more counts in the Python version vs. the Linux version). I have uploaded some example master files here (GitHub page) and test calibration images here. The other weird thing is, is that the file size between the Linux version and Python. The Linux version is about 4x smaller than the Python corresponding versions.

I am not sure if this is supposed to happen either, but when I open the fits files that are outputted from the Python version, it shows three separate images within the master flat file (same type of thing for bias, darks, and science images).

The first image is the master flat, the second image is an image of all 0 count pixel values, and the third is not even a single flat image but maybe a mask?

Screenshot 2023-05-08 100510 Screenshot 2023-05-08 100532 Screenshot 2023-05-08 100552

Versions

Python version: 3.9 Astropy version: 5.1.1 have also tried 5.2.2 Numpy version: 1.22.3 Pyerfa version: 2.0.0.1 Scipy version: 1.9.3 Matplotlib version: 3.5.1

mwcraig commented 1 year ago

Thanks for the detailed report @kjkoeller. One question I can answer right away. I think that ccd_process creates, by default, an error image and a mask image for each output image. That is causing most of the size difference, too.

The other factor that affects size is what precision the results are saved with. Python defaults to 64-bit float which much more resolution than any CCD images actually need. I'm pretty sure IRAF defaults to 32-bit floats.

What settings did you use when you combined the images in IRAF? Was it a median combine, or mean, did you do any clipping, etc?

kjkoeller commented 1 year ago

That is fine @mwcraig , but should that mark and error image be outputted with the new image? It might make it hard to do other photometry analysis on the images if there is 3 images imbedded in a single file. Also the file size is now enormous so having hundreds of images where each image triples or quadruples could be an issue as well.

The combination is a median and there is a high sigma clipping of 5 or higher.

kjkoeller commented 1 year ago

I am a little confused as to why the error image and pixel mask are even created in the ccd_process to begin with, as they are never used again. From what I can tell it would be better to either comment these lines out or have them be a separate return.

https://github.com/astropy/ccdproc/blob/main/ccdproc/core.py#L217

kjkoeller commented 1 year ago

Any updates?