Problem with SEP and Swuft-UVOT frames

[stefanocovino]

HI all,

I am reporting a bug when running sep with Swift-UVOT FITS files.

This is the operations I do:

from astropy.io.fits import get data import sep

dt = getdata('sw00033001022um2_sk_1.fits') data = dt.astype(numpy.float32) bkg = sep.Background(data) bkg.subfrom(data) objs = sep.extract(data, 5. * bkg.globalrms)

and this is what I get:

Exception Traceback (most recent call last)

in () ----> 1 objs = sep.extract(data, 5. \* bkg.globalrms) /Users/covino/.python-eggs/sep-0.1.0-py2.7-macosx-10.5-x86_64.egg-tmp/sep.so in sep.extract (sep.c:4697)() /Users/covino/.python-eggs/sep-0.1.0-py2.7-macosx-10.5-x86_64.egg-tmp/sep.so in sep._assert_ok (sep.c:2828)() Exception: SEP internal error Any idea? I can send the FITS file if useful. Thanks, Stefano

[kbarbary]

I don't see anything wrong with your script. Please do send or point me to the FITS file and I'll have a look.

[stefanocovino]

I don't see anything wrong with your script. Please do send or point me to the FITS file and I'll have a look.

Dear Kyle,

please donwload the FITS file from here: https://www.dropbox.com/s/24tbak9t7cbwkrw/sw00033001022um2_sk_1.fits.zip?dl=0

Bye, Stefano

[kbarbary]

The root problem is that your detection threshold is extremely low: 5. * bkg.globalrms is about 0.019, only slightly higher than the lowest non-zero pixel: 0.01. This means that pretty much every non-zero pixel is being counted as part of an object, and you get one giant object. One giant object triggers the problem described in #9.

If you set the detection threshold to something higher, it works: sep.extract(data, 1.)

Why is the background RMS so low? Part of the problem is that the image includes corners where there is no data and the pixel values are all zero. bkg.globalrms is the statistics of all pixels in the image, so those pixels are biasing the result. However, the RMS is still anomalously low when you look at the spatially variable RMS: bkg.rms() in regions where there is data. This seems to be due to the strange noise properties of this image. There are many zero valued pixels, even where there is data. There is also high pixel-pixel correlation. My guess is that the clipped median algorithm used for the background is clipping most of the high pixel values, leaving mostly zeros, resulting in a very small RMS.

[stefanocovino]

Hi Kyle,

thanks for your detailed explanation. Indeed you're right, the noise pattern of this frame is rather weird.

Stefano

Mobilis in Mobile

Il giorno 05/dic/2014, alle ore 22:45, Kyle Barbary notifications@github.com ha scritto:

The root problem is that your detection threshold is extremely low: 5. * bkg.globalrms is about 0.019, only slightly higher than the lowest non-zero pixel: 0.01. This means that pretty much every non-zero pixel is being counted as part of an object, and you get one giant object. One giant object triggers the problem described in #9.

If you set the detection threshold to something higher, it works: sep.extract(data, 1.)

Why is the background RMS so low? Part of the problem is that the image includes corners where there is no data and the pixel values are all zero. bkg.globalrms is the statistics of all pixels in the image, so those pixels are biasing the result. However, the RMS is still anomalously low when you look at the spatially variable RMS: bkg.rms() in regions where there is data. This seems to be due to the strange noise properties of this image. There are many zero valued pixels, even where there is data. There is also high pixel-pixel correlation. My guess is that the clipped median algorithm used for the background is clipping most of the high pixel values, leaving mostly zeros, resulting in a very small RMS.

— Reply to this email directly or view it on GitHub.

[kbarbary]

In the master branch, I've made it so that the following error message is shown:

Exception: internal pixel buffer full: The limit of 300000 active object pixels over the
detection threshold was reached. Check that the image is background subtracted or
increase the detection threshold (limit reached at pixel 1360040 at x=1085, y=505).

Hopefully that should be a more useful message for future similar problems.

[stefanocovino]

In the master branch, I've made it so that the following error message is shown:

Exception: internal pixel buffer full: The limit of 300000 active object pixels over the detection threshold was reached. Check that the image is background subtracted or increase the detection threshold (limit reached at pixel 1360040 at x=1085, y=505). Hopefully that should be a more useful message for future similar problems.

Hi Kyle,

just one more minor question. Is this specific exception identifiable with a proper try: exception: call?

Bye, Stefano=

[kbarbary]

Not really: the type of the exception is just Exception. There's another reason that an Exception might be thrown during extract() and there's also MemoryError. If you do try: ... except Exception: ... you'll catch all of these.

If there's some good motivation for catching this specific exception, I can define a new exception type. I figured that this is nearly always due to bad inputs, which the programmer would want to fix rather than continuing.

[stefanocovino]

Not really: the type of the exception is just Exception. There's another reason that an Exception might be thrown during extract() and there's also MemoryError. If you do try: ... except Exception: ... you'll catch all of these.

If there's some good motivation for catching this specific exception, I can define a new exception type. I figured that this is nearly always due to bad inputs, which the programmer would want to fix rather than continuing.

Hi Kyle,

well, what you write is of course correct. However, I was thinking to a situation where the analysis is blind, and basing on the output (number of sources) you can change the threshold accordingly to somebody's goal. If, however, the code crashes because a "guessed" threshold, good in most cases, is instead completely wrong there is no way to intercept the event (too big object) and react properly.

However, I do agree it is not a main problem.

Bye, Stefano

[RobinThomas95]

Hi everyone, I'm working with UVIT FITS files. I tried using SEP for background subtraction. After completion of the same, I save the background-subtracted image as a new FITS file with header same as the original file. On opening this newly saved file in SAODS9, I find white patches near areas that have bright knots. I'm attaching the original image and the background-subtracted image. The first image is the background-subtracted image while the second is the original image. Does the problem lie in my saving the new FITS file?