Using measurements via the napari assistant to be a part of a workflow

haesleinhuepf / napari-assistant

BSD 3-Clause "New" or "Revised" License

20 stars 5 forks source link

Using measurements via the napari assistant to be a part of a workflow #27

Open jluethi opened 2 years ago

jluethi commented 2 years ago

Hey @haesleinhuepf & team!

I've been looking at the napari assistant to see whether we can support napari-workflows in the fractal high content screening image analysis platform we're building. Very cool what you have built here, I'm very impressed with how interactive workflows can be built and how much the functionality the napari assistant brings together. I'll open a few issues with questions that have come up. Is there a preferred repository to do this on or should I just match it to where it fits best? (e.g. this issue touches the workflows, the napari-skimage-regionsprops plugin and the assistant)

I would like to build a workflow interactively in napari using the napari assistant and then run it headless for many images on a server. The saving of the workflow to a yaml file is great for this and I can load workflows with the napari-workflows functions.

I was wondering whether the assistant could also be used to build measurement workflows that return data tables as an output in headless mode. I saw that when napari-skimage-regionprops is installed, there is a measurement button in the assistant. But when I used it, set some parameters and clicked run (and got a measurement table), this was not added to the napari workflow according to the workflow inspector or the yaml file I saved.

Is that a bug, something that would be intended as a feature that hasn't been implemented yet or a design decision not to have this as part of the napari workflows & the assistant?

Would be cool if we could include this into the napari workflows, such that we run measurements also via those workflows and add new measurement functionality (though you do seem to already cover a lot!) to napari workflow compatible plugins :)

If this is a bug or something that would be a welcome addition, are there any pointers on where to start looking at this in the code? Happy to have a go at it if it's a feature of interest.

haesleinhuepf commented 2 years ago

Hi Joel @jluethi ,

Is that a bug, something that would be intended as a feature that hasn't been implemented yet or a design decision not to have this as part of the napari workflows & the assistant?

Would be cool if we could include this into the napari workflows, such that we run measurements also via those workflows and add new measurement functionality (though you do seem to already cover a lot!) to napari workflow compatible plugins :)

It's not a bug, it's just not implemented yet in the assistant. I presume, workflows support pandas data frames as input/output. Just at the moment, you cannot setup a workflow that uses dataframes from the assistant.

The reason is that there is no list of DataFrames similar to a list of Layers in napari. If we had a central storage for named dataframes (CC @andy-sweet), it would be easy to make use of it in the assistant for recording dataframe-related workflows.

There is one thing at hand: We could update the assistant code so that it allows outputing python code for generating dataframes using the Measurement button in the assistant. This does not include functions that take a dataframe as input. Would that be helpful? I'm asking because it might just be some hours of work and I've been thinking of it for a while. We would need to insert something here, e.g. storing which function produced layers.features into layers.metadata: https://github.com/haesleinhuepf/napari-assistant/blob/main/napari_assistant/_gui/_category_widget.py#L241

And here something for the code generation from the above stored stuff: https://github.com/haesleinhuepf/napari-workflows/blob/c62dfefabeb0ae6e00519d2295c325e1a9fa97d7/src/napari_workflows/_workflow.py#L625

Let me know what you think and have a great weekend!

Best, Robert

andy-sweet commented 2 years ago

The reason is that there is no list of DataFrames similar to a list of Layers in napari. If we had a central storage for named dataframes (CC @andy-sweet), it would be easy to make use of it in the assistant for recording dataframe-related workflows.

Thanks for calling this out and giving some context to one use case!

I too think it's strange that the only way we can store tabular measurements is with the Layer.features associated with a particular layer instance. Even when your measurement is a combination of one Labels layer and one Image layer, I'm not convinced that those only belong in the Labels layer, so having a store dedicated to tabular measurements provides a suitable alternative.

That being said, I don't think this feature is a high priority on the implied napari roadmap right now - i.e. I'm not aware of any major active contributors that are working on it or any plans to do so in the next 6 months or so. If someone wants to push it forward, I would be enthusiastic to give feedback and help get it through. I'm also happy to mention the feature in any roadmap discussions and will try to reference this issue as a way to prioritize the work.

haesleinhuepf commented 2 years ago

If someone wants to push it forward, I would be enthusiastic to give feedback and help get it through

Feedback is always welcome! :-)

Regarding the time-line: @jluethi how urgent is it for you? In my bubble of the eco-system we effectively have only one napari-plugin that consumes tables and produces tables and/or plots (link to related issue below). Thus, we certainly could improvise an intermediate solution in case it becomes necessary. At the end, the data structure we are talking about is a dictionary. Where it lives is secondary at the moment. Btw - if it's not obvious - I'm big fan of experimenting with new concepts in plugin-space and later seeing how to move thing upstream in the dependency river. Moving upstream is always easier than the other way.

jluethi commented 2 years ago

We could update the assistant code so that it allows outputing python code for generating dataframes using the Measurement button in the assistant.

This sounds great! It's an interesting question whether tables could be used as input in a workflow system, but I can see how that fits less well to what the napari assistant is currently doing. But the main purpose for us is anyway getting tables as an output.

I've started to look through the code you linked to above and your suggested implementation sounds like it would do the trick for our use-case. Is this something you will have a chance to look into soon? Otherwise, I can give it a try and make a PR if I get it working. We're just at the stage of implementing measurements for Fractal, so I want to push to see if we can do the measurements via napari assistant as well :)

Happy to be part of the discussion of how we can save tables in a more general way and how we'd use tables as inputs. Though that isn't a main priority for us at this time.

jluethi commented 2 years ago

After some further exploration of the napari workflows setup, I just realized that we can actually start building the implementation of using napari workflows for both image processing (already working) and making measurements by just building some workflows outside of napari for the moment (e.g. by building workflows in python using your examples from the workflows tutorial). This already supports having functions that take an image and return a dataframe. 🚀

=> Still very much interested in being able to do add measurements to a workflow from the napari assistant, but this issue won't block us from implementing napari-workflow based measurements in Fractal.

I'll be working on that implementation first to see what other topics come up when we use the napari workflows as a basis to run image processing & measurements (+ will open further issues when they come up). Still happy to contribute to assistant improvements to set dataframes as outputs though :)

PS: Big fan of the way the napari workflows are working! I just discovered I can get multiple outputs by providing a list of targets to the get function. This is 👌🏻👌🏻👌🏻

jluethi commented 2 years ago

These napari-workflows are really neat! I defined 7 example cases of different inputs & outputs we will implement in Fractal to start with and it works really nicely to run them on different input images. See examples here: https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/31

Given how nice this works, very interested to see how we can let users directly create workflows that have a table as an output in the napari assistant!

pr4deepr commented 2 years ago

Thanks for posting these examples @jluethi I've been working on something similar as well for this repo: https://github.com/BioimageAnalysisCoreWEHI/napari_lattice

In my use case, I was trying to implement a way a user could return a combination of images and tables from a workflow and and then save each according to filetype. An example for me is cropping an image, applying filtering, thresholding and creating a measurement table. For sanity checks, its good to have a max projection of the image and binary mask as well, in addition to the measurement table.

Its not an efficient way, but in napari-lattice, the output from any custom workflow is scanned:

if its an image, its saved as a tiff (or h5 in future releases)
if its a dict or list, saved as a csv
collection of images, dicts or lists -> loop through this collection and process according to type

Open to suggestions for improvements and feedback on how to even implement this in workflows/assistant.