Working together on Leaf

Hey @AtheMathmo, hey everyone,

This issue is an invitation and can be closed, removed or ignored if no interest exists.

Leaf matures slowly into an alternative to Tensorflow, Torch and co. Because of Leaf's API, Layers and Solvers, it is a tool for deep learning, other neural network based algorithms as well as (almost) any machine learning concept. (Refer to me or the Leaf Book for more information on that one).

We followed your progress at rusty-machine. We like what you build and we want to invite you working together with us on Leaf, because I think we have the same goals; provide a compelling tool for Machine Learning in Rust. We believe that Leaf + rusty-machine would provide more value for the rust community than both of them on their own.

Hey @MichaelHirn !

Thanks for reaching out. I have also been watching your progress with leaf and have been very impressed with how much you have achieved.

I definitely want to provide as much value as possible to the community and this may be a great way to achieve that. I'd really like to hear more about how you would like to collaborate/combine the two.

I'm happy to continue this discussion here or we could switch to somewhere else?

Ohh great, thanks for the quick reply. I was unsure how this invite would be construed.

So I was thinking, that some parts already exist with leaf. Like the computation part on multiple machines (the performance), the easy and scaleable abstraction of Layers and Solvers (the interface) and coming with Leaf 0.2.1 the first guides on how to build machine learning applications with leaf, serialization of modules and the leaf-service. But leaf is very focused on visual recognition with deep learning so far.

Rusty-machine has already implementations for the most popular machine learning algorithms that a lot of people would like to use to solve their problems and build applications with.

But right now, people need to make a tradeoff and learn two different tools when they want to build machine learning applications in Rust. Quick prototyping: rusty-machine and when more data and deep learning concepts are needed: leaf. But it would be very easy to give them both in one tool.

If we combine the algorithms of rusty-machine and the interface and backend of leaf, people can go seamlessly from k-means to deep residual networks (or whatever is hot in deep learning). And with the Layer and Solver abstraction of leaf, the entire infrastructure for this maneuver already exists.

Going from k-means to some conv net could be as easy as this:

K-Means

net_cfg.add_layer(LayerConfig::new("k-means", KMeansConfig { k: 2 }));

Some conv net (or whatever)

net_cfg.add_layer(LayerConfig::new("conv", ConvolutionConfig { num_output: 20, filter_shape: vec![5], stride: vec![1], padding: vec![0] }));
net_cfg.add_layer(LayerConfig::new("pooling", PoolingConfig { mode: PoolingMode::Max, filter_shape: vec![2], stride: vec![2], padding: vec![0] }));
net_cfg.add_layer(LayerConfig::new("linear1", LinearConfig { output_size: 500 }));

I believe, that this would give the Rust community

one go-to library for all things machine learning related and therefore, increase collaboration
ease entrance barrier to machine learning as it is easy to switch from simpler algorithms to the latest research
a stronger competitor against the C++ based Tensorflow.

I was unsure how this invite would be construed.

I think it's exciting!

I'd love to see some of these algorithms take advantage of the performance that leaf supplies. And I agree it would definitely make Rust a strong choice for machine learning. I'd definitely like to pursue this but do have a few, for lack of a better word, concerns.

It seems from what you've said that this move would involve rewriting the rusty-machine algorithms to be supported by leaf and (at least partially) depreciating the current library). I think this is probably the only way to achieve the single tool as you suggest and Leaf has great support right now for the back-end heavy lifting.

My worry is that I tried to design rusty-machine to follow in the footsteps of sklearn and similar libraries - this promotes super simple, minimal effort development of machine learning models. I think leaf also provides a very simple, clean, interface but it definitely adds some complication to building these models. To try and illustrate what I mean, in rusty-machine:

use rusty_machine::learning::k_means::KMeansClassifier;
use rusty_machine::learning::UnSupModel;

// Create a model with 3 clusters
let mut model = KMeansClassifier::new(3);

// Train it
model.train(&training_data) // A matrix of training data

// Predict from it
let test_outputs = model.predict(&test_data); // A matrix of test data

You can make this more complicated but in essence it boils down to the 3 lines above. From what I have seen of leaf this would require the user to instantiate a layer, place it into a network, choose a solver (I'm not sure how this last part would work for k-means as the solver is model-specific if you understand what I mean). This is quite a few more hurdles to overcome.

Another concern is that I want to provide native support for rusty-machine. This means being able to use it without needing openblas, lapack, cuda, etc. Of course it will be awesome to support these, but I don't want them to be necessary - similar to numpy.

Finally I'd like to learn more about how we can express all models within the interface you've defined in leaf. I can perhaps see how k-means would work but I think it would need it's own unique Solver? How about models which don't have iterative Solvers - like Naive Bayes, or linear regression (with least-squares fit)? We can discuss this more in the future I suppose :).

With all of that said - this is just my understanding and I would love to learn more about your vision for this. I would also very much like to play around with implementing some of this on top of leaf to see exactly what it could look like. I'd also like to know if you see another path outside of rebuilding rusty-machine? I'm not against that but would definitely like to do some preliminary work before committing to it.

I hope this hasn't put you off! I think leaf is awesome and am excited to explore what we can do together :)

Wow, nice. Thank you so much for the write-up. I had no idea.

I agree, rusty-machine is something, that should exist on its own. I think it is very valuable. Sadly, I don't know too much about how exactly sklearn works, but I feel it has a valid point and so has rusty-machine.

rusty-machine is probably as straightforward as it gets with kmeans and co., with leaf it would be a bit more complicated.

// prepare k-means network (model)
let mut k_means_cfg = SequentialConfig::default();
classifier_cfg.add_input("data", &vec![batch_size, 10]);
k_means_cfg.add_layer(LayerConfig::new("k-means", KMeansConfig { k: 2 }));

// set up solver
let mut solver_cfg = SolverConfig { network: LayerConfig::new("network", k_means_cfg), solver: SolverKind::Iterative(SGDKind::KMeans), .. SolverConfig::default() };
let mut solver = Solver::from_config(backend.clone(), backend.clone(), &solver_cfg);

// Train it
solver.train_minibatch(inp_lock.clone(), label_lock.clone());

// Predict from it
let test_outputs = solver.network().forward(&test_data);

But, you can build leaf without the need to have any 3rd party library like openblas, CUDA and co. installed. This is controlled via the build flags. Something like cargo build --features native should do it. The abstraction of Collenchyma helps a lot here to keep this clean and manageable.

For the Solver part this would be a phase of exploration. Right now the Solver has very likely a bit more SGD bias than it should have because it wasn't really challenged so far. But its concept is similar to that of the layer and it's implementation is very, very flexible (The Leaf book draft has some explanation of the concept in chapter 2. and 3.) From what I understand of the k-means algorithm, I think this should be straightforward to implement into leaf's Solver abstraction.

So, right now I am pretty excited to see if we can put rusty-machine and leaf together and I have two ideas. Neither of them would compromise any of the frameworks I guess (and hope).

A very simple proof of concept could be to fork leaf, requiring rust-machine in the cargo manifest and just use rusty-machine in the few API endpoints of leaf. (There is also a guide in the leaf book on how to create a new layer). Same with the Solver. I think the rust implementation of leaf's Layer and Solver is flexible enough to set this up quick and dirty.

A long-term strategy could be to open a new collenchyma plugin e.g. collenchyma-ml (for the lack of a better name). This would expose algorithm traits like for example trait KMeans which can be implemented for the different computation languages (native | Rust, OpenCL, CUDA, ...). One implementation for native would then be rusty-machine. Leaf would then require collenchyma-ml and use in the KMeans layer (and solvers) the collenchyma-ml function like collenchyma.kmeans which would then call rusty-machine if leaf was build for native.

I feel, if we should do something like this together, it would be pretty cool to announce that rusty-machine and leaf are compatible useable. We would create much more value for the community that way, than if both of us would reinvent the wheel on what the other library has already created. And I think we can develop both of our libraries as we wish without any compromises, as leaf doesn't have any opinion on how a Layer or Solver implementation has to look.

This sounds awesome!

I think I'll need to learn a little more about how leaf works and is structured under the hood. Maybe a good way to do this would be to explore the quick and dirty solution that you described.

Right now there is a pretty strong coupling between the machine learning algorithms and the native linear algebra. It will take a little time to correct this - and it is something I have been planning to do for a while. I'm not sure how far we can get before doing this within rusty-machine. We could probably get something working but it would not be utilizing Collenchyma.

Moving forward I see something pretty similar to your second proposal. I would separate out the native linear algebra support from rusty-machine and this could be supported within collenchyma (behind a native feature flag as you suggest. Then we could write some new layers within leaf which would interface the models in rusty-machine. I'm not entirely sure how the solvers would work in all cases - we can figure this out though!

We need to figure out exactly where the layers of abstraction will fall. And there are certainly still some questions about how we implement solvers for different models - but I think we can tackle this.

Going forward I think we should talk through some of the models and sketch out some implementation details. It will be easier to switch to a new mode of communication - I'm happy to go with the rusty-machine gitter but if you have any other suggestions I'm all ears.

Perfect, I am really thrilled to see if we can make this work.

I must admit that I do not have a huge understanding of how most of the algorithms, which are implemented in rusty-machine, work. But I looked at how k-means and linear regression is implemented and I think it is quite easy to have a quick and dirty version working.

So as described in 2. Layers there are right now five conceptual types of layers (which do not differ at all in functionality, just helps to keep them logically seperate). I think rusty-machine's implementation of k-means and linear regression (which I think would be of the loss layer type) contains both the Layer and the Solver part. Making a k-means or linear regression loss Layer is probably very, very straightforward. Basically just using what is already there in rusty-machine and packing it into the .forward and .from_config methods of leaf. There is actually a step by step guide 2.3 Create a new Layer for how to do this. (Happy to receive feedback if that proved helpful.)

The Solver part will be slightly trickier, as I don't really have a lot of experience implementing a new solver. But conceptually it should not differ from implementing a Layer. Same Rust design pattern. In the Solver you would just make use of .train function.

So I think this would be how the abstraction would play out. Next step could be opening a new branch and just trying the quick and dirty hack. I would be happy to contribute in case some refactoring (maybe for the solver) is needed.

I am happy to switch over to the rusty-machine gitter. Or if there are questions for leaf, we could also go to the leaf gitter. There are more people, who could help out with leaf specifics.

I am really excited to see how the two will work out together.

Closing this issue as leaf is currently lacking maintainers :(.

If development kicks off again I'd be happy to reopen this and try and get some connections between the two. Until then I'll be focusing on just rusty-machine.

AtheMathmo / rusty-machine

Working together on Leaf #37