Are you interested in DQ flags plugin?

[pllim]

I am writing a local plugin that spits out DQ flags for a given pixel when user uses it on a DQ array. This is mainly for use at STScI, but I am wondering if there is a wider interest here. Example of DQ flags: http://www.stsci.edu/hst/acs/analysis/reference_files/data_quality_flags.html

[ejeschke]

Are you planning to do an RGB overlay like shown in the the URL you posted?

[pllim]

Not at the moment. User click on a pixel with the DQ value, the plugin displays the various DQ flags that went into it. Very simple.

The tricky part is support for multi-instrument DQ definitions -- Can Ginga .cfg file store nested dictionary? I answered my own question: Yes.

[pllim]

Below is a screenshot of my plugin. The image is a real HST Advanced Camera for Surveys product.

[ejeschke]

Hey, looks good! I think you need to add the colored overlay as an option. If you look at the Overlays plugin it pretty much will tell you exactly how to do that. If you could make the table of values (with colors) user definable then I think it would definitely be useful for a lot of people. I can think of one use case for one of our instruments right now.

[pllim]

Overlay might be tricky, but we can brain storm here.

The DQ extension is 16-bit integer. The DQ flags are stored as 2**n, so there are 16 possible values. I can't see how overlaying 16 different colors is going to be informative to anyone.

Currently, the parsing code can take several seconds to run for an entire 4k x 2k array; It needs to find all possible DQ flags for every pixel in the array. To do overlay, we will need to do this in advance, not when user clicks on a pixel. In addition to optimization (we can try multi-threading), there also needs to be a way to cache the results, because you don't want to recalculate it when user switch to another image and back. Solved.

Ideas welcome!

[pllim]

Leonardo Ubeda (STScI) suggested a different approach. He would like to see SCI extension being displayed, while the plugin lists DQ flags for the selected pixel. What do you think, @ejeschke ?

[leonardoubeda]

Just one additional suggestion (which will be an excellent tool to provide within DrizzlePac): How about you display a drizzled product and the selected pixel shows the properties of the contributing pixels from the FLT files ? For example the number of counts, the used weight, etc. This would be a useful tool to verify what happens to cosmic rays, flux conservation, etc. I know this is a different topic, but since you have already designed the code, I thought it might be worth the effort. Thanks!

[ejeschke]

Hmm, I agree that if you try to mix the colors with alpha blending it's not going to be all that helpful. How about coloring the pixel overlay only according to the flag that the user clicked on (just one flag at a time)? You could have a combobox of flag names and whichever one the user selects it colors those pixels.

[pllim]

@leonardoubeda , incorporating drizzled product is too specific to HST. It would be something that STScI distribute separately, not as part of Ginga's core distribution. We will have to discuss that one offline. It is a good idea nonetheless. Thanks!

@ejeschke , yes, that might be useful. But there is still the issue of performance and caching. Is the caching to be done within the plugin, or is there a way to piggyback on DataSrc?

[ejeschke]

The other ideas also sound excellent. If the overlay is not useful for STScI purposes, don't bother with it.

[ejeschke]

How about caching the tables in the AstroImage objects (as metadata)? Are they quite large? If attached as metadata then they can be garbage collected when items drop out of the Datasrc (unless the user has set the size to be unlimited). Any way to make them sparse tables (if the user is only interested in certain flags, say)?

[pllim]

The overlay actually sounds pretty useful. But I would probably limit the number of comboboxes you can check simultaneously for any meaningful representation (3-5?).

Here's a way to incorporate both in the same plugin:

• When user does a single click, it lists the different DQ flags for that pixel (as shown above).
• If the displayed image is SCI extension, Ginga force-loads the accompanying DQ extension automatically and display the flags while SCI image is still displayed. This requires the image to conform to a standard EXTNAME and EXTVER convention in multi-extension FITS image, which might not work for some people. Also, force-loading might mess user's expectation of Datasrc unless the size is set to unlimited. Yeah, I am not sure if this feature should be in the core distribution. Perhaps this is something STScI can implement later by sub-classing.
• When user check comboboxes and click a button, the overlay is shown. It can be cleared with a separate button.

As for caching:

• When DQ flags are separated for a single pixel, all of them have to be extracted. There is no way to just extract some of them. Maybe @sosey can confirm, since she wrote the initial algorithm.
• The "table" (dict might more useful here) is only as sparse as the number of non-good pixels present in a given image. So in the worst case scenario, where all the pixels are flagged, it will be a dict containing all the pixels for the image. A 4k x 2k image will have over 8 million pixels. But then again, if there are no good pixels at all, no one would find it useful anyway.

[sosey]

You might try mihai's math exclusion, that's probably faster that the loop I had dismantling the DQs.

my_dq_flag=4+16+32 match = [0] if (my_dq_flag&0xffff)==0 else [1<<x for x in range(16) if (1<<x & my_dq_flag)]

Then you just need the dictionary for each instrument which relates the flag to the string description. Since it's returning a list, you can just pull the flags you want from the list for caching.

If this was the dictionary:
dq={4:'one',16:'two',32:'three'}

[dq[i] for i in match] will return the list of description strings for the pixel

[mcara]

@pllim @sosey I do not see the code here, but based on what @sosey wrote + code from e-mail, the computation of 1<<x could be removed. For example, if the dictionary 'dqval' is defined as it was in the e-mailed code:

dqval = {
...................
    'ACS': {
        0: 'OK',
        1: 'Lost during compression',
        2: 'Replaced by fill value',
        4: 'Bad detector pixel or beyond aperture or HRC upper-right defect',
        8: 'Masked by aperture feature or HRC occulting finger',
        16: 'Hot pixel',
        32: 'CTE tail',
        64: ('Warm pixel (since Oct 8, 2004) or permanent hot pixel '
             '(before Oct 8, 2004)'),
        128: 'Bad column',
        256: 'Full-well or A-to-D saturated pixel',
        512: 'Bad pixel in reference file (FLAT, polarizer, or dust mote)',
        1024: 'Charge traps',
        2048: 'A-to-D saturated pixels',
        4096: 'Cosmic rays and detector artifacts (AstroDrizzle, CR-SPLIT)',
        8192: 'Cosmic rays (ACSREJ)',
        16384: 'Manually flagged by user',
        32768: 'Not used'},
...................
}

then one could do the following:

valid_dq_flags = dqval[instrument].keys()
match = [0] if (my_dq_flag&0xffff)==0 else [x for x in valid_dq_flags if (x & my_dq_flag)]

and valid_dq_flags = dqval[instrument].keys() could be computed only once. Alternatively, even my original code can be made more efficient if:

valid_dq_flags = list(1<<x for x in range(16))

is computed only once when dqval is computed.

The first approach of getting the keys from dqval has a serious drawback as it does not allow (easily) to detect the case when my_dq_flag may contain bits that are not allowed for a given instrument. For example, imagine that for ACS 5th bit (flag=32) was not defined at all in dqval. The first version (with dqval[instrument].keys()) will return a match that will not contain flag 32 when my_dq_flag has it while the second approach (with list(1<<x for x in range(16))) will return a match that contains flag 32. Then this can be caught and an appropriate error message about weird DQ flags may be displayed. So, it depends on what you want to do.

[mcara]

Also, what are you going to do about WFPC2 images whose DQ array is in a separate file...

[mcara]

I would like to suggest that you also consider displaying very short messages as an option, especially if this information is going to be displayed in tooltips. For example, if you have a parameter such as long_msg (which could be True or False) then the dqval dictionary could be modified as follows (for example):

dqval = {
...................
    'ACS': {
        0: ['GOODPIXEL', 'OK'],
        1: ['SOFTERR', 'Lost during compression'],
        2: ['DATALOST', 'Replaced by fill value'],
        4: ['DETECTORPROB', 'Bad detector pixel or beyond aperture or HRC upper-right defect'],
...................
}

See Table 3.10 for suggestions of short names for COS in COS data handbook and other instruments have similar abbreviations.

Then,

for m in match:
    print(dqval[instrument][m][long_msg])

[pllim]

Thanks, @sosey and @mcara! I'll take a look next week.

[pllim]

@mcara

what are you going to do about WFPC2 images

1. Ginga cannot read GEIS image and we don't have WFPC2 budget to worry about that anymore.
2. User can request alternate multi-extension FITS format for WFPC2 anyway.

[pllim]

Here is the updated DQ plugin:

It has accompanying configuration file and optional data tables (in order to implement customizations as suggested by @mcara). Would be nice to have #166 and #175 resolved first, if possible.

If you don't want this in core distribution, that's fine too. I don't know how useful this is outside STScI. Let me know either way!

@leonardoubeda , it can now display associated DQ values even when the SCI image is being displayed, as shown in the attached screenshot.

[pllim]

I have officially included this as part of stginga package prototype (https://github.com/spacetelescope/stginga). Feel free to re-open if you want to revisit this.

[mcara]

@pllim

what are you going to do about WFPC2 images

Ginga cannot read GEIS image and we don't have WFPC2 budget to worry about that anymore. User can request alternate multi-extension FITS format for WFPC2 anyway.

The issue is not in capability to read GEIS or WAIVER FITS but rather the fact that WFPC2 (and other instruments) have DQ flags in a separate file (e.g., "_c1m.fits"). Even multi-extension WFPC2 files suffer from this. Thus, dealing with these instruments would require searching for presence of a DQ file named similarly to the image file but with a different suffix.

[pllim]

@mcara , let's move this particular discussion over to spacetelescope/stginga#6 since the plugin currently resides there.