Neural Networks Branch

[Nicholas-Schaub]

I had an idea for trying to bring convolutional neural networks to ImageJ. However, I see that there is a Neural Networks branch already here in the tutorials section. Is this already in active development? I tried to do some searching and didn't find much for plugins for ImageJ, but maybe I missed something?

[bnorthan]

Hi @Nicholas-Schaub , @kephale and I worked on this for basically one day last year, the main purpose was just to demonstrate how convolutional neural networks work (for learning purposes mainly). We never got a chance to follow up, as we both got busy with many other things.

Realistically we may not want to (and have resources to) write our own deep learning framework, so instead we probably want to take advantage of an existing one like DL4J. If you have some ideas it would be great to hear them. @dietzc and @ctrueden also may have advice to give you.

[kephale]

My interest is still piqued. I still think it would be really cool to make a convnet in ImageJ because (based on the design @bnorthan and I started on) it would allow one to really build the net in a bottom-up way, inspecting each layer/kernels/etc. in much more detail. Doing that in DL4J is (at least when we started this code) quite a bit more awkward, since the architecture has to be specified a priori then you have to bludgeon your way into the neural nets guts. The downside is code repetition, which is a pretty big downside...

[dietzc]

It really depends on what we want to do. For teaching purposes, I fully agree: It would be great to be able to build small and simple convolutional networks in ImageJ-Ops. If one really wants to reuse existing networks (e.g. from any model zoo), I'd rather integrate existing frameworks, which are optimized for speed, memory efficiency, distributed training and scoring etc. Deeplearning4J improves everyday, but also other frameworks (e.g. Tensorflow, MXNet or BigDL) have Java bindings available.

[bnorthan]

@dietzc , @kephale , @Nicholas-Schaub

I mentioned to @ctrueden that it would be useful to start an "experimental" ops repository to test/develop new ideas. One thing I want to do is wrap some FFT and matrix operations from ND4J, wrap JTransform, and run some performance comparisons.

Ofcourse ND4J is the sibling project of DL4J, so I think deep learning experiments could also go in this repository.

What do you guys think??

[ctrueden]

I mentioned to @ctrueden that it would be useful to start an "experimental" ops repository to test/develop new ideas.

Do we need one centralized "experiments" repository? Or would it be OK to simply start an experimental repository in your GitHub space whenever something like this comes up? What are the advantages of putting it in the imagej organization? Increased visibility?

[bnorthan]

@ctrueden

Do we need one centralized "experiments" repository?

My main motivation for a centralized "experiments" repository would be to encourage collaboration. If we had a repository set up, with dependencies to ND4J, DL4J, JTransform, and other 3rd party libraries and examples showing how to convert ImgLib2 structures and wrap libraries with ops, I think it would be a good starting point for others. Deep learning and deconvolution have some common elements (need for fast math library, potential need for GPU) and it would nice to have a place to experiment with others.

Another option is to keep using imagej-tutorials for these kind of examples, but It sounds like Image-tutorials is being phased out (#30).

I am also fine with just putting experiments like this in my own repository when things come up. I guess the main question is, what is the best way to encourage collaboration on experimental ideas??

[ctrueden]

My main motivation for a centralized "experiments" repository would be to encourage collaboration.

That's fair. So here we go! https://github.com/imagej/ops-experiments

[Nicholas-Schaub]

Ah, this really blew up. I wasn't expecting so many responses so quickly.

@bnorthan I was thinking about doing this in DL4J but wrapping it up into an ImageJ plugin. I know Knime has an implementation using DL4J, but I was thinking about something more accessible. I think one of the issues with deep learning, especially CNNs, is that they are built and then left in a language that might make them difficult to access or use for the people that might be truly interested in using them. Just as an example, I just created a CNN for a clinical trial to do cell segmentation that was far more accurate than what they were currently using. The problem was that they had to hand me the data, I would process it, and then hand it back. My thought for an ImageJ plugin would be to create a nice interface between networks generated in toolboxes like Theano and Tensorflow and ImageJ, so someone could open up an image in ImageJ and process it using one of these trained networks. At least, this would be my first step in creating this kind of plugin.

The second phase would be to create a training interface. I think I have a good idea of how to create a general set of simple rules for constructing neural networks that might make it fast and easy to set up and train. The down side is that the average user doesn't have the computing power to train these in a relatively timely manner, so I think the training side of the plugin would have to be a little more specialized.

The main reason for my post was to not start a project someone else has already begun. From the sounds of it, my time might be well spent trying to do something like this since it doesn't exist and no one appears to be working on it. If anyone wants to collaborate, I would love to share my ideas. I have a pretty good outline of what needs to be done to make this type of a plugin easy to use.

[Nicholas-Schaub]

Regarding the point on what is the best way to encourage collaboration on experimental ideas, I think setting up starter projects that have the dependencies set up for things like DL4J and ND4J ready to go would be a good start. It might serve as a good incubator to get over the hurdle of setting things up. That way, if you know you want to do something in DL4J for example, you could clone the DL4J example repository and go from there.

[dietzc]

just out of curiosity, what do you mean by:

..., but I was thinking about something more accessible

? Do you mean the way we integrate DL4J, KNIME itself or DL4J itself? Really, just because I'm curious how we could improve what we are doing.

[ctrueden]

I think setting up starter projects that have the dependencies set up for things like DL4J and ND4J ready to go would be a good start.

@Nicholas-Schaub Let me know if you want access to the ops-experiments repository. You could of course do the fork and pull thing as appropriate too—or just work in your own repository—but maybe it makes sense to hack on some code together here, I don't know.

[Nicholas-Schaub]

@dietzc What I mean by more accessible is something in ImageJ, because the majority of the biologists I work with won't take the time to learn Knime. What I would like is an ImageJ plugin that can take a network that I train and apply it to an image loaded in ImageJ. My collaborators would use something like this because they know ImageJ, and they tend to be intimidated every time you try to teach them a new piece of software. Sorry, it wasn't meant to be offensive. I honestly haven't taken a look at the implementation in Knime yet, but if I were to start this project then the Knime implementation is where I would begin. Does the Knime/DL4J implementation allow you to take pretrained networks from Caffe, MatConvNet, Theano, and Tensorflow and run them on an image?

@ctrueden Sure, I'd like access to the ops-experiments repository.

[ctrueden]

I'd like access to the ops-experiments repository.

I sent you an invite!

[dietzc]

@Nicholas-Schaub no worries. I wasn't thinking that you were offensive. Our KNIME DL4J implementation currently only allows to execute Dl4J networks.

My collaborators would use something like this because they know ImageJ, and they tend to be intimidated every time you try to teach them a new piece of software.

Yup. It also doesn't make sense to convince people to use a tool which slows them down. If they see the benefit, they will use it. If not, not :-)

I honestly haven't taken a look at the implementation in Knime yet, but if I were to start this project then the Knime implementation is where I would begin.

I doubt that this would be very helpful for what you are trying to achieve, but you can check it on github.com/knime/knime-dl4j.

Does the Knime/DL4J implementation allow you to take pre-trained networks from Caffe, MatConvNet, Theano, and Tensorflow and run them on an image?

Nope. However, DL4J in theory can run Keras networks. There is not really any backend which can run them all and most of the formats are not really compatible.

I think if your ultimate goal is to deploy your networks to users as an application / plugin (and I think you don't train more than a network per month?!), then the easiest might be to simply write an ImageJ2 plugin which reaches out to whatever backend and exposes the functionality directly. If you figure, that you always train your networks e.g. in DL4J, Tensorflow or MXNet or anything else with java bindings, then you can generalize this functionality by providing services and converters to ImgLib2 data structures which the plugins have in common.

[ctrueden]

the easiest might be to simply write an ImageJ2 plugin which reaches out to whatever backend and exposes the functionality directly.

For what it's worth, I am currently (this month) working on exposing basically that workflow for TensorFlow networks. Gonna create some kind of "NN chooser" UI for selecting from available/known networks, downloading on demand (since they can be large), and running them with the appropriate NN plugin—just TensorFlow to start, but I'll think about how to keep it general so that other frameworks can be slotted in to the same UI in the future. Sound good?

[Nicholas-Schaub]

@dietzc The idea behind the plugin would be to analyze the data structures that are used to saved trained networks and try to convert them into a DL4J network. I think there is enough in common between the different libraries at this point where it might be easier to just convert the layers and weights into a DL4J network rather than just trying to run them on the native platform. But, maybe that's more work since you'd have to keep up with new features of each library, or you might lose functionality when converting to a DL4J network.

@ctrueden If you're working on that, I'll see what you come up with and maybe try to build on top of that. Maybe I'll still start on trying to convert certain types of networks into DL4J structures since I think some frameworks like MatConvNet might be difficult if not impossible to access from ImageJ without a Matlab license.

[Nicholas-Schaub]

Oh, and I've trained 3 different networks in the last month. I've developed a pretty nice method of scaling sizes up and down for speed versus accuracy for image segmentation. Depending on the images, I can train a network in a couple of days with >95% accuracy using ~10,000,000 parameters. The training is done on a GPU of course, but I've been trying to keep a minimal number of parameters so that my collaborators might be able to use them in test mode without having to wait too long to see the results when they run it on their CPU.

[ctrueden]

The ImageJ-TensorFlow project has seen an initial release, although it is still in incubation. Some folks at MPI-CBG (e.g. @frauzufall) are continuing to hack on that library as needed to support use cases there, and we will do the same at LOCI as needed. PRs welcome from all comers. ^_^

There is also now a Microscope Image Focus Quality Classifier plugin available from the TensorFlow update site which uses ImageJ-TensorFlow and the TensorFlow Java API for the heavy lifting.

We still do not have a NN graphical browser component, but we do support downloading, unpacking and caching TensorFlow models on demand from remote resources.

Please do comment here, or on those respective projects on GitHub or the ImageJ Forum, with thoughts, questions, suggestions, etc.

I'm closing this issue, since it is rather open-ended, and we have a nice start on TensorFlow integration now. But feel free to reopen, and/or open additional issues related to NNs, as needed to drive the progress.