Project Motivation/Features

[kne42]

Taken from an email sent by @royerloic

The project, in a nutshell, would be to build a minimal — but well built — proof of concept image processing and visualisation tool that would bring the power of python to the masses (of Biologists) who are generally (unfortunately) scared by terminals, coding, and complex installations. We see this as complementary and distinct in use case to other efforts such as Jupyter, and tailored to a large untapped community (Microscopy and Biology).

The tool would consist in:

• stand-alone ’single-file’ app ‘batteries-included’ for easy installation and running (biologists!=hacker). We want a true performant desktop app, so web-based tech is excluded for now.

• PyQT5 based 1D, 2D, and 3D viewers for numpy arrays We already have a library for 3D volumetric rendering available with Spimagine, first very basic prototype of OpenGL+Qt5 2D viewer available too, 1D will just be classic plotting… useful for line-plots in microscopy…

• Semi-automatic population of menus with most existing image processing/analysis algorithms present in numpy, scikit-image, etc …

• Possibility to add additional ‘functions’ as plugins. Idea: plugins are just github repos and can be installed by just giving a github url. versioning, branches, etc is for free through github…

• Automatic script generation based on the sequence of menu calls with undo and redo - similarly of how it is done in Illustrator and Photoshop.

• Usage of the Dask infrastructure for out-of-core images.

• etc…

[ctrueden]

I'd like to ask about the longer-term vision of this project.

My goal for ImageJ & Fiji is to broaden their scope beyond Java, to include also Python and JavaScript, so that we can have a plugin-driven image analysis tool that ships plugins from all three ecosystems. At a CZI hackathon earlier this year, we started a proof-of-concept Electron version of ImageJ that could—with additional effort—offer something like that. I was thinking we'd probably need a JavaScript-centric plugin framework, but one capable of shipping also Python and/or Java code as appropriate.

How do you see Napari fitting in with something like that, if at all?

[kne42]

@ctrueden I was initially pushing for a browser-based (electron) version but we identified a few major issues with a javascript-based app:

• electron is too slow and prone to crashing
• we don’t think that it will properly be able to handle the large amounts of data that we will throw at it; it’s not designed for that
• browser-based apps limit usage of keybindings as there are special reserved combinations
• javascript is just a terrible language

I think if you wanted a true cross-language platform you would just write it all in C, which most languages are based on, and is fast enough to accomodate high-dimensional data. Plugins could be written in a C-derived language and interfaced appropriately.

However, to answer your question, Napari is, at its core, just a Python plugin system. While we plan to provide an official GUI for users, the core can be shipped separately from our GUI and interface with any other process that can communicate with Python, including ImageJ.

[ctrueden]

@kne42 Do you think Napari + its GUI could drive the distribution of JS-based and Java-based plugins, in addition to only Python plugins? If so, it could be a solution worth converging upon by the broader community, including ImageJ and Fiji, as well as the various JavaScript viewer efforts in the works (e.g. imaging-tools, vtk.js, ipyvolume).

we don’t think that it will properly be able to handle the large amounts of data

The main idea would be that data is hosted in shared memory, perhaps primarily on the Python side as numpy arrays, since Java (and I believe JavaScript; see chanzuckerberg/pyrepl; @bkmartinjr would you care to comment?) can access it from there without copying.

electron is too slow and prone to crashing

I think performance-wise, it would come down to the environment for which the plugin was written, no? The crashiness is more concerning to me—but I have used a few Electron apps in recent years and it did not seem particularly egregious to me. Is this a well-accepted phenomenon in the Electron community? Do you expect that it won't be addressed in coming years?

I think if you wanted a true cross-language platform you would just write it all in C

Another promising solution is GraalVM (see here for a nice breakdown of what it can do). If the Python support gets good enough, it might be a candidate for a unifying technology here. Definitely worth keeping an eye on it.

javascript is just a terrible language

The answer I keep hearing to that (very true! 😛) claim is "TypeScript".

Napari is, at its core, just a Python plugin system. While we plan to provide an official GUI for users, the core can be shipped separately from our GUI and interface with any other process that can communicate with Python, including ImageJ.

Awesome! So Napari is really the framework, and napari-gui is the UI? Sounds good. Let me know when you feel the Napari framework is far enough along to attempt integrating into ImageJ. I'd be happy to give it a shot then!

[kne42]

@ctrueden

Do you think Napari + its GUI could drive the distribution of JS-based and Java-based plugins, in addition to only Python plugins?

We have plans for this, but our primary focus right now is getting the Python side up and running.

Java (and I believe JavaScript; see chanzuckerberg/pyrepl; [...]) can access [numpy arrays] without copying

Very cool! This would make interprocess communication and data-passing a lot easier.

Another promising solution is GraalVM

This looks interesting! However, I am hesitant to use non-standard implementations of Python as they are difficult to maintain and may not behave as expected. For this reason, projects such as numba and cython have largely replaced PyPy.

The answer I keep hearing to that (very true! 😛) claim is "TypeScript".

I honestly don't like TypeScript either and I think that one of the major hinderances to the development of Jupyter plugins is how difficult and unintuitive it is to write.

So Napari is really the framework, and napari-gui is the UI?

Yes. While we will ship them together on our end, we want them separable so that any application or other GUI can easily interface with Napari plugins.

[jni]

@ctrueden 👋 ! =)

I feel like @kne42 is projecting a little too much certainty. Napari is an evolving vision. I am excited to keep hacking at it with your input. But I want to second most her comments. Electron in particular I find a leaky abstraction — it's rare to see an Electron app that doesn't have that web-app "smell". And they also tend to be processor hogs without due care.

There's two angles to my interpretation of Napari:

• make scikit-image functions accessible to a non-coding crowd (this is the plugin side)
• provide a Python-native viewer/annotator for >2D ndarrays, which is sorely lacking currently

We are indeed trying to keep the two sides as uncoupled as possible, so that e.g. I myself can use the viewer without having to deal with all the plugin stuff, which is kinda useless for someone like me who lives in the IPython prompt.

Regarding interoperability with other frameworks, I'm not the most qualified to comment on this, but seeing e.g. how hard packaging bioformats with CellProfiler is, I'm not optimistic about a perfectly seamless solution, and am instead inclined to think that efforts in this direction would quickly end up in dependency hell. I would very much like to start with a simple "Open image/array in Fiji" / "Open image/array in IPython kernel ", and so forth. The former is probably just a packaging problem at this point with imglyb, right?

But perhaps I am just being unambitious. =)

[royerloic]

I second both Kira and Juan, right now Napari is aimed at filling a clear gap in the python ecosystem. I am now learning more and more python and numpy for offline processing, and a viewer could really be helpfull. exciting stuff.

Sent from my iPhone

On Sep 25, 2018, at 17:41, Juan Nunez-Iglesias notifications@github.com wrote:

@ctrueden 👋 ! =)

I feel like @kne42 is projecting a little too much certainty. Napari is an evolving vision. I am excited to keep hacking at it with your input. But I want to second most her comments. Electron in particular I find a leaky abstraction — it's rare to see an Electron app that doesn't have that web-app "smell". And they also tend to be processor hogs without due care.

There's two angles to my interpretation of Napari:

make scikit-image functions accessible to a non-coding crowd (this is the plugin side) provide a Python-native viewer/annotator for >2D ndarrays, which is sorely lacking currently We are indeed trying to keep the two sides as uncoupled as possible, so that e.g. I myself can use the viewer without having to deal with all the plugin stuff, which is kinda useless for someone like me who lives in the IPython prompt.

Regarding interoperability with other frameworks, I'm not the most qualified to comment on this, but seeing e.g. how hard packaging bioformats with CellProfiler is, I'm not optimistic about a perfectly seamless solution, and am instead inclined to think that efforts in this direction would quickly end up in dependency hell. I would very much like to start with a simple "Open image/array in Fiji" / "Open image/array in IPython kernel ", and so forth. The former is probably just a packaging problem at this point with imglyb, right?

But perhaps I am just being unambitious. =)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[bkmartinjr]

The main idea would be that data is hosted in shared memory, perhaps primarily on the Python side as numpy arrays, since Java (and I believe JavaScript; see chanzuckerberg/pyrepl; @bkmartinjr would you care to comment?) can access it from there without copying.

Curtis, thanks for the ping. I'm struggling a bit to understand the Napari vision, so my comments should be taken with a grain of salt...

I originally wrote pyrepl as part of an experiment in creating a notebook-like UI that could do interactive speed image transformations based on user inputs (eg, a slider controls a user-defined image filter). It had two goals:

• execute user-defined Python image processing code (or really any code that was suspect) in an isolated Python sandbox, so as to not destabilize the app if the user's code does bad things (I didn't use ipython, but it was a REPL like system).
• move imaging data from the sandbox to the viewer quickly enough to do 30+fps view updates

As I think of the problem, sandboxing is a critical requirement (both to protect the app from the user code, and to isolate hardware resources where the OS doesn't help -- eg, GPU state). To hit both sandboxing and interactive speed goals, I elected to use a shared memory transport (with all the pluses and minuses that implies).

I did base my viewer on electron. While I agree it is a big old battleship, it comes with a lot of capabilities ...which depending your perspective, can be helpful or simply overhead. In my case, it had zero impact on overall performance, and I like the web environment...

@jni @kne42 realize it's early, but is there any thinking yet on how napari would sandbox the python execution? Happy to contribute to the discussion if it is helpful...

[kne42]

@bkmartinjr We have been considering executing all plugin and user-defined code in an IPython kernel. See #22

[bkmartinjr]

@kne42 - that definitely solves for the isolation issue. Wondering about performance - Jupyter (in my experience) has poor round-trip performance when rendering images of any size, largely due to the way they marshal data in/out of the REPL. Is interactive perf a non-goal for Napari, or do you have an alternative solution? (this was the second of my two goals in the pyrepl experiment)

[jni]

@bkmartinjr performance is definitely a goal for Napari. We just don't have experience with this. There are certainly ways to get interactive (rendering) performance out of Jupyter-based workflows — see e.g. IPyVolume, bqplot, etc. However they do result in duplication of data. This might be a worthwhile tradeoff for us, if it avoids us having to build a whole new set of infrastructure (such as PyREPL).

Personally I know exactly 0 units about sandboxing, which I agree with you is necessary, and perhaps as a corollary I don't know what "shared memory transport" means, so I would be hesitant not to outsource all of this to the IPython kernel. Unless you have both the time and inclination to handle all of this for us. =P

[royerloic]

I agree with what has been said. A few points: i) Performance is paramount indeed, that means both UI performance, image processing speed and memory efficiency. First, we want a very fast and smooth GUI which means QT5, VisPy, OpenGL, shaders, etc... Second, we will ship optimised numpy and scipy libraries as default in the app. Third, for local execution we don't want to have memory duplication, unless we gain something by for example running a kernel on another machine or on the cloud (idea: modularity of backend execution).

ii) Sandboxing should not degrade performance in the case of local execution. Also, security is the responsibility of the user that adds a non-built-in plugin, or of the plugin author. We can't possibly be hacker proof here. Having plugins be git repos in plain readable python code would help greatly. @kne42 knows how to possibly intercept imports to libraries that may be dangerous. We might want to have a warning if a plugin seems to import suspicious packages... But there is no way to prevent counter-measures and we should probably rely mostly on a web of trust among Napari devs and plugin authors, and users -- not very different from other ecosystems...

iii) We hope that ipython kernels can be spawned from within the Napari python process and share memory with it seamlessly. remote kernels incur a cost but in exchange of a feature (remote execution on potentially more powerful system)

Hope this clarifies some aspects of the project.

[jni]

@royerloic small point:

security is the responsibility of the user that adds a non-built-in plugin, or of the plugin author. We can't possibly be hacker proof here.

I don't think we are concerned about security here, but about stability: we don't want plugins to bring down the toolbar/viewer.

[royerloic]

Ok, I see. The biggest problem in that case is that computation must happen in a separate thread from the UI event loop. But since python is not a true multi-threaded programming language (GIL), we might have slowdowns, unless we actively mitigate this with noGIL or even Numba to reduce potential interference between the UI thread and the compute thread...

[jni]

Large portions of NumPy and scikit-image are nogil already, so I wouldn't worry too much about threads being a bottleneck.