How to get Thumbs not to reset modified data in buffer?

[pllim]

This is similar to what I do to ContentsManager subclass in #72 . What would be the equivalent of Contents.switch_image() in Thumbs?

[ejeschke]

@pllim, the method you are interested in is called load_file() and it is in .../ginga/misc/plugins/ThumbsBase.py. When the thumb is clicked on this method is called. You could override it to get custom behavior.

[ejeschke]

BTW, if you are not concerned about memory usage spiraling out of control, you can simply set the preference for numImages=1000 or some large number. Then images will not be expired out of the Datasrc, and should not have to be reloaded.

There are ways to do this programmatically, but maybe the simplest way to experiment with it is to switch to the default channel "Image". Start the Preferences plugin (or press the gear on the toolbar). Save the preferences using the button at the bottom of the preferences.

Now open $HOME/.ginga/channel_Image.cfg with a text editor. Change the value of numImages and save the file. Restart the reference viewer. Unless the preferences are saved for other channels independently, the preferences for the Image channel will be inherited to any new channels created.

[pllim]

I tried setting numImage=50 in the config file but it does not appear to work. I only have 4 images opened. Theoretically, numImages=4 should have sufficed, right?

[ejeschke]

This is the case where you altered the numpy data array of the image in memory, correct? It shouldn't be reloading the image from the file. Cut levels, etc. may be set to current settings. If you turn on the debug log and click on a thumbnail, does it show a message of the form:

2015-02-12 13:33:26,866 | D | Control.py:778 (switch_name) | Image 'SUPA01118768 .fits' is no longer in memory; attempting to load from /home/eric/testdata/SPCAM /SUPA01118768.fits

The bit about "is no longer in memory" is important. If you don't see that message, it shouldn't be reloading from a file.

[pllim]

No, I cannot find the mention of "memory" in the log file in debug mode. Control.py reported "large image update" at one point, but there are too many messages for me to decide which ones are relevant to the issue here.

Am I not setting the data correctly? It is done as follow, as you suggested. I am hoping that set_data() would propagate the updated image data to the rest of Ginga.

    def sub_bg(self):
        """Subtract background, and update contents manager and display."""
        if self.bgval == 0:
            return True

        image = self.fitsimage.get_image()
        new_data = image.get_data() - self.bgval

        s = '{0} subtracted from {1}'.format(self.bgval, image.metadata['name'])
        add_history(image.metadata['header'], s)
        self.logger.info(s)

        self.fitsimage.set_data(new_data, metadata=image.metadata)
        self.fitsimage.redraw()
        self.redo()

        chname = self.fv.get_channelName(self.fitsimage)
        image = self.fitsimage.get_image()

        # Update file listing
        list_plugin_obj = self.fv.gpmon.getPlugin('ContentsManager')
        list_plugin_obj.set_modified_status(chname, image, 'yes')

        return True

[ejeschke]

I see. I think that using set_data() on the VIEWER may actually construct a new AstroImage object to contain it. Try:

    image.set_data(new_data, metadata=image.metadata)

(with this method it should be unnecessary to call viewer.redraw(), as the viewer will be notified that the image data has changed).

[pllim]

That's it; That fixed the problem with Contents and Thumbs. Thank you!

However, MultiDim seems to reset the values even with this change. When I click the modified image name on Contents or Thumbs again afterwards, then it goes back to the correct (subtracted) values. So, it appears that MultiDim reads from file regardless? How do I get around this?

[ejeschke]

That's because MultiDim operates by using astropy/pyfits to iterate through the HDUs, so it always reloads from the file. Probably needs a subclass on MultiDim with some method overrides to cache HDU loads, or maybe even a rethink/refactor on the way it is implemented. I'll take a look,

[pllim]

Well, the active extension info is in the image.metadata['header'], right? Maybe Ginga could check if the selected extension matches the one in metadata; and if it matches, loads from get_data() instead of from file?

[ejeschke]

I'm testing out a version of MultiDim that uses the extant machinery to keep track of images that are already loaded. But, this raises the question: should separate HDUs show up as separate Thumbs and Contents entries? I'm thinking of a scheme where we append a [1], [2], etc. to the name. Wondering if this is desirable behavior by default or not.

Do we want to extend this all the way out to data cubes; i.e. slices? Seems like we'd want to draw the line at HDUs.

[pllim]

Thanks for looking into this!

I cannot speak for data cubes, as my usage does not currently involve them. I agree that getting this to also work on cubes might over-complicate things.

Appending [1] or [2] to names sounds fine, as that is also how DS9 and IRAF represent multi-extension FITS. However, I do not have a good solution on how to nicely show them in Contents and Thumbs. To show them individually and cache them all will quickly clutter up the display and eat up resources. But perhaps that is the only practical solution, as DS9 also displays each extension as a separate frame. I definitely do not want the feature where all the extensions are displayed automatically, but rather only display the extensions that I specifically ask for, except for the default [SCI,1] that is displayed when an image is opened.

[ejeschke]

I'm thinking that Contents could probably start another level of nesting. This might even be used as a kind of quick way to access HDUs without opening MultiDim. In other words, if you clicked on the entry, it would expand to show the HDUs under it and then you could click individual ones to open them. This would definitely not be difficult to implement.

For Thumbs, as you say, it could get messy quickly. I suppose it would be possible to have some kind of collapsing/expanding thumbnails.

Both of these need to be approached carefully. Just from my own observatory I know we have ramp frames that have hundreds of HDUs in them!

[pllim]

I agree that listing all extensions at once (even if not pre-loaded) won't work for images with hundreds of them. Usually users just want to look at the extensions that interest them. For example, I might just want to look at SCI,1 and DQ,1 to make sure my data quality flags are where they should be for EXTVER=1 and I don't care about all the other extensions. I am still in favor of on-demand listing/thumbs.

[ejeschke]

Could you try commit c4410d72c1244b4507374574ea78da8dc3671ceb thoroughly ? This now tracks which HDU was loaded across Thumbs and Contents. There is still some work to finish--when visiting an old image by clicking in thumbnails (for example) the MultiDim plugin gui is not yet updated to show which HDU was loaded in (also, the data cube controls need to be re-instantiated).

[pllim]

I am trying your improved MultiDim but I get the traceback below. Is it trying to compute WCS from the primary header although it has no image data displayed?

WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
.../numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
.../numpy/core/_methods.py:69: RuntimeWarning: invalid value encountered in true_divide
  ret, rcount, out=ret, casting='unsafe', subok=False)
.../numpy/core/_methods.py:83: RuntimeWarning: Degrees of freedom <= 0 for slice
  warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)
.../numpy/core/_methods.py:95: RuntimeWarning: invalid value encountered in true_divide
  arrmean, rcount, out=arrmean, casting='unsafe', subok=False)
.../numpy/core/_methods.py:117: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)

Also this in the log file:

2015-02-25 15:00:50,240 | W | MultiDim.py:260 (set_hdu) | Empty data part in HDU #0
2015-02-25 15:00:50,463 | I | Contents.py:149 (add_image) | name=image002[0]
2015-02-25 15:00:50,467 | I | Control.py:872 (_switch_image) | Update image start
2015-02-25 15:00:50,467 | I | Control.py:877 (_switch_image) | Setting image...
2015-02-25 15:00:50,491 | E | ImageView.py:493 (set_image) | Failed to initialize image: num_points not in range 0-0
2015-02-25 15:00:50,492 | E | ImageView.py:498 (set_image) | Traceback:
  File ".../lib/python2.7/site-packages/ginga-2.2.20150212221817-py2.7.egg/ginga/ImageView.py", line 490, in set_image
    self.auto_levels(redraw=False)
  File ".../lib/python2.7/site-packages/ginga-2.2.20150212221817-py2.7.egg/ginga/ImageView.py", line 1423, in auto_levels
    loval, hival = autocuts.calc_cut_levels(image)
  File ".../lib/python2.7/site-packages/ginga-2.2.20150212221817-py2.7.egg/ginga/AutoCuts.py", line 392, in calc_cut_levels
    AutoCutsError("num_points not in range 0-%d" % (total_points))

2015-02-25 15:00:50,691 | E | Control.py:890 (_switch_image) | Failed to continue operation: ufunc 'bitwise_or' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

[pllim]

I think this works well enough for now. In one instance, it gives me empty drop-down box for HDU selection, but it was on a modified buffer, so not something Ginga supports natively. I was able to reset that by switching to another image and back. You can close this if you wish.