profiling a ROI instead of only single point with cursor?

[rjlopez2]

Hi @ch-n

Thanks for your plugin, it is really fantastic and helping me a lot with my application!

I may have missed in the documentation but I wander if there is an easy way to do a time profile plotting of a ROI (or multiples) created by the user, let's say a square or some others using the shape layers? shall this be include it as a feature?

Ideally the user would add ROI(s) and move them around to visualize the averaged profile of such region in a similar way as is now done with a single point by the cursor.

Additionally, if you could guide me how to do it I could potentially give it a try since this feature would be quite useful for our application.

cheers

Ruben

[Chris-N-K]

Good Morning,

at the moment there is no such functionality, but it certainly would be nice. There is the napari-plot-profile plugin that does use the shapes layer to plot the values along a line shape, but it does not account for time resolved data. I guess it should be not too hard to make something similar for time series data.

If you would use the plugin with both functionality, what would be the best way to switch between the two modes in your opinion? I think two separate widgets selectable in the plugins menu or the napari-toolsmenu would be a good solution. In this case we would need two different Plotter classes (different draw() methods). Do you think it should be possible to dock both at the same time? Could be problematic. Another solution would be to place the other plotter in a additional tab and allow just the active tab to extract and plot data.

In the plot-profile plugin you have to spawn the shapes layer your self, I think it would be a better solution to add a button to the widget to spawn a specific shapes layer directly connected to the widget.

Best regards Chris

[rjlopez2]

If you would use the plugin with both functionality, what would be the best way to switch between the two modes in your opinion?

I would probably set a radio button (or something similar) to toggle between options.

I think two separate widgets selectable in the plugins menu or the napari-toolsmenu would be a good solution.

yes, this could be a good way to go

Do you think it should be possible to dock both at the same time? Could be problematic.

Probably not, since you still have to hold shift to activate the cursor functionality. However, it would be interesting to know in which scenario this behavior could be conflictive.

Another solution would be to place the other plotter in a additional tab and allow just the active tab to extract and plot data.

yes, but might be would be also interesting for the user to explore in the same plot the ROIs "average" profiles along with the single-point cursor profile?

In the plot-profile plugin you have to spawn the shapes layer your self, I think it would be a better solution to add a button to the widget to spawn a specific shapes layer directly connected to the widget.

yes, this should do the job. Another alternative would be to detect the shapes layers (may be in a similar way it detect the image layers), list them and allow the user to select what shapes layer shall be plotted?

I am new to napari plugin developing but I will try to give it a try using the example as you mention from the napari-plot-profile plugin and see how far I can get something to work.

and thanks lot for taking your time on this :)

Best,

Ruben

[Chris-N-K]

Hi @rjlopez2,

a widget to toggle the cuntionality under the options tab or directly in the plotter tab might be an option as well yes.

The conflicts I were thinking about are more regarding visualization. If we use two different widgets it could be quite cluttered to dock them both at the same time.

If you want them in one canvas we can not use different classes but need to give the Plotter draw() method functionality to toggle the different visualisation protocols. In this context the best solution would probably be to make a check box or similar to activate roi based plotting and / or voxel based plotting, executed by a flexible draw method of a single Plotter class.

The LayerSelector can be easily extended to select shapes Layers.

You can fork the repo and give it a try if you like. As soon as you have a working solution you can make a pull request. If you like some help with the integration in the current classes just ask.

Thanks for the contribution, Chris

[rjlopez2]

Hi Chris,

thanks for your message and sorry for the lack of reaction lately. Indeed I agree that displaying two plots would be a bit too cluttered. Ideally one should be able to select from one to another "modus" e.g (as you have mentioned: roi based plotting and / or voxel based plotting) and display the plot. That should be sufficient for a range of applications I guess...

I am still going through the logic of your classes/methods/callbacks and as soon as I am ready I may come back at some point to you for more specific questions and/or possible ask for support.

cheers

Ruben

[rjlopez2]

Hi Chris,

I manage to create a logic that work for plotting the mean of a given square ROI in a shape layer. While this can certainly be polished and improved, I think it could work as a good starting point.

However, I am having some issues to obtain the information from the toggle definition I created which will ultimately allow the user to switch between the two plotting modalities. May be you can help me or provide some suggestions to overcome this. Here is a brief description of what I did:

1. I create the self.toggle method in the VoxelPlotter class here using the same logic as you did for the additional plotting parameters (e.g self.autoscale , self.max_label_len, self.autoscale)
2. I also defined it in the OptionsManager method (see here) and connected it few lines later
3. I add it as well in the update_options method here
4. and finally I included in the plotter_options here

After all this changes I still do not manage to read the self.toggle method when I call from within the draw method (line 129)

I have to hardcoded the toggle element that would ideally shift from voxel to ROI plotting and this seem to work as expected, which is defined here. I used this, only to test the functionality and it seems to do the job for different ROIs (must be square) and different Shape layers .

There, however, it supposed to be the self.toggle method that switch between the two logics, but the issue here is that I get an error saying that the self.toggle method is not defined in the VoxelPlotter class, when I replace the test variable by the self.toggle.

Appreciate any support here since I am a newbie with python and particularly with OOP.

I f you have time and want to give it a go use my cloned repo the branch shape_layer_plot_functionality, or let me know if it is fine to make a pull request.

Cheers.

[Chris-N-K]

Hi Ruben, a big thanks that you are willing to participate to the plug in, really appreciated.

On a first glance I can not find any possible reasons for your problems with the toggle attribute. Should be set and accessible if the class is properly initiated. I have to take a deeper look on running code. Can do that maybe at the weekend.

You did not implement any shape layer specificity yet right?

At the moment the code is not in a state ready for a push. Let's figure out the error first. Afterwards, I will set up a branch you can then push to. The code cleanup we can do there.

Greetings Chris

[rjlopez2]

Hi Chris,

a big thanks that you are willing to participate to the plug in, really appreciated.

yes, I am also very glad to do it :). This is my first contribution to FOOS so please bare with my lack of experience sometimes.

On a first glance I can not find any possible reasons for your problems with the toggle attribute. Should be set and accessible if the class is properly initiated. I have to take a deeper look on running code. Can do that maybe at the weekend.

alright, at this point I exhausted all my ideas so I will wait until you give it a try.

You did not implement any shape layer specificity yet right?

if by shape specification you mean the shape form in the shape layer (e.g. ellipse, line , square... ), No. The current version is "tested" only in square/rectangle shapes, but I can try to check if can be extended to other 2d shapes forms.

At the moment the code is not in a state ready for a push. Let's figure out the error first. Afterwards, I will set up a branch you can then push to. The code cleanup we can do there.

alright, I wait then for yours instruction to proceed after we figure out the problem.

Thanks a lot again.

Ruben

[rjlopez2]

Hi Chris,

I just pushed a change where now any 2d shape is supported for plotting. From now I won't make any more push so just use the last version from the corresponding branch in my clone if you want to give a try.

Cheers.

Ruben

[Chris-N-K]

Hi Ruben,

sadly I encountered some issues with your current code. It won't plot the shape areas. I get an error message, regarding a wrong sized mask. The reason for the error is that the to_labels method of a shapes layer does only accept a 2d shape or a shape matching the highest dimensionality in the viewer. Therefore, we must either us the complete or 2d shape.

I think the to_labels method is not optimal, I'm already talking to the devs for there opinion on the topic of extracting from images with shapes.

I found the reason your toggle did not work, you moved the plotting part of the draw method one tab too far therefore it did only trigger when you moved into the shape part.

I'm currently reworking your code. I already fixed the toggle part and reworked it a little bit. During the week I will try to further work on the code.

Cheers, Chris

[rjlopez2]

Hi Chris,

thanks for taking the time to have a look at the code and spotting the mistake. Great that you managed to find the problem with the toggle button! As for the new error occurring when using the mask generated with the to_labels method I found it surprising since for me seems to be working as expected in xy-t images. may be the problem is when using higher dimension images? I have to have a look at it... Are you also using the last version that I mention in my last comment of my 86607fd last push ?

What are the image dimension you are using for testing?

The steps I do for testing the functionality are:

1. launch napari and attach the plugin
2. I load an image (xy-t)
3. create a new shape layer
4. from the new shape layer, create a shape of type rectangle, polygon or ellipse (even line works but I ma not sure if this is meaningful or has any user-case application)
5. on the layer selector, pick the image layer AND the shape layer.
6. usually you see the plot right the way but helps if you press shift.
7. this works if you add more ROIs within a single shape layer or multiples shapes layers.

you can see from the screenshot attached file.

This functionality however is missing the reactivity that you have in single voxel plotting with the mouse cursor and automatically plotting/updating as your cursor across the image.

At the moment, if I want to update the plot as I change the current position the ROIs I need to press shift along or de-select and re-select the shape layer in the shape selector box. Ideally would be great to plot the shapes right the way, and as you reposition the shape across your image the plot automatically updates without need to have to tick the layer shape box every time or pressing shift. This functionality would be very nice to implement it but at the moment I am don't know how to do it. I am looking also to the napari-plot-profile plugin from Robert Hasse who implement something similar but on shape type lines in XY images.

Anyway, thanks again and let me know if you managed to reproduce the plotting in this way.

Ruben

[Chris-N-K]

Hi, yes it is a problem with higher dimensional images like 4d in 3d is all fine as it full fills the conditions of the function to be either 2d or the dim of the highest dim layer --> your 3d but with 4d you skip only time and end with 3d what is neither 2d nor like the highest dim layer. We can prevent that if we always take only 2d and just inflate it afterwards.

I'm pretty sure there should be a call back for shape position updates and we can hook up the draw method to it like with shift+move.

Do you think we should allow multiple shapes layers for the ROI selection? I think it might be better to automatically add an empty and specifically named (ROI selection) shapes layer when switching to the shapes / ROI mode. This way the user could still use multiple ROIs at once but we do not need to add the shapes layer to the selector widget.

I made a working example. The function for data extraction:

def extract_mean_ROI_shape_time_series(current_step, layer, labels, idx_shape):
    """Extract the array element values inside a ROI along the first axis of a napari viewer layer.

    :param current_step: napari viewer current step
    :param layer: a simgle image layer
    :param labels: the label for the given shape
    :param idx_shape: the index value for a given shape
    """

    ndim = layer.ndim
    dshape = layer.data.shape

    # convert ROI label to mask --> added support for 4D layers, as well as slight rework of the input params
    if ndim == 3:
        mask = np.tile(labels == (idx_shape + 1), dshape)
    else:  # 4d
        # respect the current step --> 2D ROI on 3D volume
        raw_mask = np.zeros((1, *dshape[1:]), dtype=bool)
        raw_mask[0, current_step[1], ...] = labels == (idx_shape + 1)
        mask = np.repeat(raw_mask, dshape[0], axis=0)

    # extract mean and append to the list of ROIS    
    return layer.data[mask].reshape(dshape[0], -1).mean(axis=1)

The Plotter class edits:

def draw(self):
      """
      Draw a value over time line plot for the voxels of all layers in the layer attribute at the position stored
      in the cursor_pos attribute. The first dimension is handled as time.
      """
      handles = []

      if self.layers: # --> changed logic for slimmer code
          for layer in self.layers:
              # get layer data
              if self.max_label_len:
                  lname = layer.name[slice(self.max_label_len)]
              else:
                  lname = layer.name

              # plot voxel / pixel time series
              if self.mode == 'Voxel' and self.cursor_pos.size != 0:
                  # extract voxel time series
                  vts = extract_voxel_time_series(
                      self.cursor_pos,
                      layer
                  )
                  # add graph
                  if not isinstance(vts, type(None)):
                      handles.extend(self.axes.plot(vts, label=lname))

              # plot mean value from square ROI(s) in shape layers 
              # --> switched to only one shapes layer so no more shape layer iteration
              elif self.mode == 'Shapes' and self.selection_layer and len(self.selection_layer.data) > 0:
                  # convert shape to 2d labels to be used later for the mask
                  labels = self.selection_layer.to_labels(layer.data.shape[-2:]) # --> now we always take 2d labels
                  # iterate over ROIs in shapes layer
                  for idx_shape in range(self.selection_layer.nshapes):
                      # calculate finally the mean value
                      roi_ts = extract_mean_ROI_shape_time_series(
                          self.viewer.dims.current_step,  # get current step from viewer --> we need this for 4d support
                          layer,
                          labels,
                          idx_shape
                      )
                      # dropped  the shape type we could use it but I ear it will clutter the legend
                      if not isinstance(roi_ts, type(None)):
                          # add graph
                          handles.extend(self.axes.plot(roi_ts, label=f'{lname}_ROI-{idx_shape}'))

      if handles:
          self.axes.set_title(f'Position: {self.cursor_pos}')
          self.axes.tick_params(
              axis='both',  # changes apply to the x-axis
              which='both',  # both major and minor ticks are affected
              bottom=True,  # ticks along the bottom edge are off
              top=False,  # ticks along the top edge are off
              labelbottom=True,
              left=True,
              right=False,
              labelleft=True,
          )
          self.axes.set_xlabel('Time')
          self.axes.set_ylabel('Pixel / Voxel Values')
          if not self.autoscale:
              self.axes.set_xlim(self.x_lim)
              self.axes.set_ylim(self.y_lim)
          self.axes.legend(loc=1)
      else:
          self.axes.annotate(
              'Hold "Shift" while moving the cursor\nover a selected layer\nto plot pixel / voxel time series',
              (0.5, 0.5),
              ha='center',
              va='center',
              size=15,
          )
          self.axes.tick_params(
              axis='both',  # changes apply to the x-axis
              which='both',  # both major and minor ticks are affected
              bottom=False,  # ticks along the bottom edge are off
              top=False,  # ticks along the top edge are off
              labelbottom=False,
              left=False,
              right=False,
              labelleft=False,
          )

Added self.mode and self.selection_layer as class attributes as well.

def update_options(self, options_dict):
      # print('update options')
      self.autoscale = options_dict['autoscale']
      self.x_lim = options_dict['x_lim']
      self.y_lim = options_dict['y_lim']
      if options_dict['truncate']:
          self.max_label_len = options_dict['trunc_len']
      else:
          self.max_label_len = None
      self.set_mode(options_dict['mode'])
      self._draw()

def set_mode(self, mode):
      self.mode = mode
      if mode == 'Shapes':
          self.viewer.add_shapes(data=None, face_color='transparent', name='ROI selection')
          self.selection_layer = self.viewer.layers['ROI selection']
      elif mode == 'Voxel':
          if 'ROI selection' in self.viewer.layers:
              self.viewer.layers.remove('ROI selection')
          self.selection_layer = None

The OptionsManager changes:

    # Add toggle widget to layout --> switched to a combobox, allows for further modes like points mode ^^
    # Changed the name as well to better represent the actual functionality
    self.mode = QtWidgets.QComboBox()
    self.mode.addItems(['Voxel', 'Shapes'])
    # further down changed all toggle to mode

I do it this way to keep the editing history intact for the branch. I guess I can now put up a branch for the update and you can then make a pull request to the branch. Afterwards I could push my changes or if you have further suggestions implement them first.

Cheers, Chris

[Chris-N-K]

Well I got a little bit carried away and added support for points layers xD

[rjlopez2]

Hey Chris thanks a lot, I see how this is now taking shape :)

I see now the problem with 4D images, I wander if your solution can also catch for even higher dimensional images, etc for images that contain also channel information...? In any case, may be for more complex images probably is tricky to plot and one may need to pick a slide of it to visualize it.

I'm pretty sure there should be a call back for shape position updates and we can hook up the draw method to it like with shift+move.

yes, there must be something around, I am aslo checking at the moment for this.

Do you think we should allow multiple shapes layers for the ROI selection?

I don't think it hurts. At least for me is useful. I think that is nice to have it so the user can compare for cases that are doing different things/analysis/annotations in 2 or more layers. At the end the user have the choice to select/deselect the Shape layer that he/she wants to plot from the layer selector list.

I think it might be better to automatically add an empty and specifically named (ROI selection) shapes layer when switching to the shapes / ROI mode...

I see your point. I am just concern that one may create different shapes layer with ROIs that they belong to different processing steps and yet one may need to plot. But I like the idea of having separated the shapes and images layers.

I look then forward to see and test all the changes you have made once you create the branch.

cheers.

Ruben

[Chris-N-K]

There is a branch: add_ROI_and_points_plotting now please make a pull request to this branch. Afterwards we can finish everything on this branch before merging it into main.

I will close this issue and open a new one specifically for the development on the branch.

Best, Chris