Image annotating from dm3 and dm4

[magnunor]

When opening dm3 or dm4 files in Digital Micrograph, the various annotating done when acquiring the data is shown. For example in a HAADF-STEM image one can see where a line scan was done, as shown in the picture.

Looking around s.original_metadata I find AnnotationGroupList, which I guess are these annotations. It would be nice if these could be shown when using s.plot, with some argument to enable or disable showing them.

One possibility would be "linking" to the line scan or spectrum image, using the discussed ROI feature https://github.com/hyperspy/hyperspy/issues/44

[magnunor]

I added some very basic functionality for this in https://github.com/magnunor/hyperspy/tree/NEW_load_roi_annotations

Currently it will only work for dm3 (and maybe dm4) files.

[magnunor]

I added support for loading more ROI types from dm3 files: lines, text, ellipses.

The current problem is knowing what the different "AnnotationType" numbers refer to. Some of them can be found here: http://www.dmscripting.com/example_controlling_annotations_with_component_commands.html

2 - line annotation 3 - arrow annotation 4 - double arrow annotation 5 - box annotation 6 - oval annotation 8 - spot mask annotation 9 - array mask annotation 13 - text annotation 15 - bandpass mask annotation 17 - group annotation 19 - wedge mask annotation

Future work: Add support for other image formats. Use more of the metadata (color, font, ...). Further extend the ROI class as described in https://github.com/hyperspy/hyperspy/issues/44

[francisco-dlp]

This is great. Are you aware of releated efforts in hyperspy/hyperspy#295? Do you (@magnunor and @pburdet) think that the drawing portion of this code could be merged into the new Markers class introduced in hyperspy/hyperspy#295? If that can be done, then the DM specific code could be moved to io_plugins/digital_micrograph.py where it'll simply translate the DM tags into HyperSpy's metadata. What do you think?

[magnunor]

Ooooops, I deleted pburdets comment by mistake.

[magnunor]

But I agree: Markers should be merged first, then I can make a new branch with the digital_micrograph specific code and afterwards integrate my plotting part into the Markers framework.

[magnunor]

Since the ROI functionality in https://github.com/hyperspy/hyperspy/pull/425 is still not fully complete, I guess I'll wait a little bit with implementing it into that framework.

For the implementation, I'm guessing the different ROIs can inherit RectangularROI or some other more general "template" ROI.

[francisco-dlp]

That sounds reasonable.

ROIs (including RectangularROI) should inherit from a BaseROI class that doesn't exist yet. Once we polish RectangularROI, we can then move the general parts to BaseROI.

[magnunor]

I had a quick look through the Markers implementation, and as far as I can tell there is no way to permanently add a marker to a signal? Ergo there is (currently) no way to add a marker to a signal and have the marker show up every time you plot the signal?

In https://github.com/magnunor/hyperspy/tree/NEW_load_roi_annotations I added a s.roi_list, which plotted all the roi when using s.plot(plot_annotations=True). Would something similar be a good way to implement the features discussed in this issue?

[vidartf]

In general I think it is best if fewer thing in hyperspy recreate the plot (many thing simply need the plot to be updated). That would at least help alleviate the need for permanence. However, I can see there would be use cases where you would want to persist artists on the Signal, so maybe the best way would be to add a collection on Signal that accepts "artists" that have a common interface like e.g a plot() and an update() function. This could then be used by both markers, ROIs, and anything else needed in the future. Signal.plot() should then have a keyword argument on whether to plot these.

[to266]

I agree that plotting by default should be kept uncluttered for performance reasons.

As for the collection of the artists, I agree with @vidartf. Particularly for markers, the dictionary-like metadata structure should be more than enough (as most of the parameters of the markers are for matplotlib, which uses dictionaries to pass the plotting parameters anyway).

[vidartf]

Storing persistent artists in metadata sounds good, as that is then saveable and loadable out of the box. The remaining issue then is to solve how to store the artists in dict, and how and when to translate to classes when plotting. Storing and parsing the actual data should be left to the class I think, but a general parser that translates to classes is needed. Maybe a storage format could be like this:

s.metadata.artists = {
    <index>: {   # <index> should just be an enumeration
        'type': 'marker.text',   # For example
        'kwargs': {   # This is passed to marker via marker_instance.from_dict(kwargs)
            'data': <numpy array as text>
            'marker_properties': <marker.marker_properties dict>
}}}

Used in an example:

import hyperspy.hspy as hs
s = hs.signals.Signal(np.zeros((10, 10)))
s.axes_manager.set_signal_dimension(2)

m = hs.utils.markers.text(3.0, 5.4, "My marker text", color='red')
s.add_marker(m)

# Add to metadata. This should ideally happen in a handler that figures out the index
# Also, 'kwargs' should come from a marker.to_dict() function
s.metadata.artists = {
    '0': { 
        'type': 'marker.text',
        'kwargs': dict(
            x=m.data['x1'],
            y=m.data['y1'],
            text=m.data['text'],
            **m.marker_properties
)}}

# Read from metadata
artists= []
for d_artist in s.metadata.artists.as_dictionary().itervalues():
    type_name = d_artist['type']
    # Somehow resolve type to class, here simply by simple if-tree matching
    if type_name == 'marker.text':
        atype = hs.utils.markers.text
    elif type_name == 'marker.horizontal_line':
        atype = hs.utils.markers.horizontal_line
    # ... etc. for other types
    a = atype(**d_artist['kwargs'])
    artists.append(a)

[jat255]

I agree that keeping this information in the metadata seems best.

Might I suggest that there be some sort of easily accessible 'switch' for the user to turn the markers on or off, when the signal is plotted (perhaps this already exists and I'm just unaware). I could then picture if you are updating a plot, turning the markers off first, doing your updating, and then turning them back on (if desired) could improve performance quite a bit.

[to266]

A switch should definitely be there, yes. However which one would you prefer? Removing the markers altogether when "off", or just not updating them when iterating, etc.? The "fast" iteration from utils.plot.plot_images seems to do a good job already (with deepcopy of the metadata, if I am not mistaken), and stops the plot updates when iterating.

Also, I think there is no need for the s.metadata.artists (if that's where we decide to put them, I think @francisco-dlp might have suggestions for that) to be a dictionary to begin with - now the metadata supports saving/loading lists, etc. Also no need to save the numpy array as text.

The general artist parser I think should be a class that all of them inherit from. Then a nice __repr__ can be added for markers, maybe even some sort of "meta-metadata", where comments / names for artists can be added

[vidartf]

@jat255 I'm not sure I understood why you'd want to turn toggle the markers off and on during plotting, could you elaborate (maybe with an example)?

@to266 I haven't really been following the bit with saving lists, but I'm assuming this would then be a list of dictionaries just to remove the need for indexing etc? s.metadata.artists was just something I used as I had to have a name for the example, so I'm not set on that. If a well defined standard exists for saving numpy arrays directly (in binary form) within the metadata then I'm all for it, I just didn't know it was possible :)

[jat255]

@vidartf, perhaps the thought isn't fully flushed out, but given the slow performance of certain functions that iterate through signals (like the background subtraction going through an EELS spectrum image) when the plot is visible, I was thinking that any way to reduce the amount of "work" that must be done when updating a plot could be useful. Like @to266 suggested, perhaps just not updating the markers until the end of whatever process could be sufficient, although it might appear messy to the user to have irrelevant markers shown during an update.

On another train of thought, I could see this marker functionality being capable of reproducing the "Mirror extraction ROI" function that is available in DigitalMicrograph (see below). Namely, this would be able to show where on a survey image from which a spectrum originates. I guess this information could be held in some sort of data structure in the metadata as well?

[vidartf]

@jat255 Thanks, I get what you mean now. Although I don't think it won't be necessary for that purpose as soon as hyperspy/hyperspy#413 is working, I agree that it would be a useful feature to have.

On another train of thought, I could see this marker functionality being capable of reproducing the "Mirror extraction ROI" function that is available in DigitalMicrograph (see below).

That is being worked on in hyperspy/hyperspy#425 (and quite a bit on mine and @francisco-dlp's forks), and in the current implementation in my fork, ROIs can be used for navigation and easily mirrored across several signals. That implementation relies upon widgets instead of markers though, as widgets are differentiated as being user-interactive (e.g. draggable). To totally mimic the behavior of DM, the "Spectrum image" line and "Beam" indicators would need to be read out and added as markers, which is under way here.

[jat255]

I'm not sure what the most recent status on this is, but I implemented a simple plotter for the "survey images" stored by DM when collecting EELS or EDS. I put it into my external "tools" module here.

I just wanted to leave it here in case someone came looking for this functionality, and perhaps it can be a workaround for right now before this issue is fully resolved.

For reference, here's what it plots:

[vidartf]

Looks nice! Do you know if the spectrum image etc. is loaded with an offset? Ideally, it would have an offset equal to the location of the subregion in the survey image (or the survey could have a negative offset, if that is easier). Then, mirroring navigation ROIs across the different signals should be trivial.

[jat255]

I'm not exactly sure what you mean, but it looks like the offset is contained in the spectrum image file as well as in the survey image. In the spectrum image, the position of the box within the survey image is located at:

>>> dm.file_reader('EELS_SI_4-Si_L-edge.dm3')[0]['original_metadata']['ImageList']['TagGroup0']['ImageTags']['SI']['Acquisition']['Survey Image']['Spectrum Image Rect']
(81, 122, 154, 171)

The same information is in the survey image, which I use in my plotting method on line 140.

[vidartf]

Sorry if I went a bit quick, what I meant was this: If you take the coordinates of the SI rect, and add those as a offsets to the signal axes in AxesManager of the SI etc. , then the coordinates in the different signals will line up (if the files are not already saved like that). That will make it a lot easier to mirror navigation and ROIs across the different signals.

I did quick and dirty example (this one was a little bit more work since DM saved one with nm units, and the other two with µm, so I had to correct that...):

[jat255]

Oh okay, yes, I think you're right, then. The coordinates in the SI start at (0,0) in the top left (so without any offset loaded in by default). I think you're right then, that if you add the coordinates in the "Rectangle" tag, you'll end up at the right place. The coordinates in the rectangle tuple are stored in the order (top, left, bottom, right).

[thomasaarholt]

@jat255, could you give an example of how your plot_dm3_survey_with_markers() should work? I installed your hyperspy_tools, but I can't get it to work.

[jat255]

@thomasaarholt sure thing! I haven't used this code in a while, but it still seems to work just fine with the latest version of hyperspy. It does require seaborn to be installed, just because I like how it looks. You could fork the repo to remove that dependency, though.

Anyway, here is how I use it. Hopefully that helps you some.

[magnunor]

I'm guessing with all the ROI/markers functionality having been implemented, we could take a look at implementing this for 1.0.0? I guess it shouldn't be too hard, since we've already got code which does this, even if it is not currently in HyperSpy directly.

[thomasaarholt]

@jat255 and I were talking a few days ago about saving the ROI/marker information form the SI Survey Image in the metadata. Currently if you save the survey image as hdf5, the ROI is lost. Thoughts on this?

[magnunor]

Not too familiar with the ROI code, but is the ROI stored in the metadata as a class? If yes, we would have to implement a ROI_to_dict or something like that, so we can save it in the HDF5-files.

[francisco-dlp]

The ROIs have a nice property: their __repr__ method prints the code to re-create them, making it trivial to store them as a simple string e.g.:

>>> rroi = hs.roi.RectangularROI(0, 0, 1, 1)
>>> rroi
RectangularROI(left=0, top=0, right=1, bottom=1)