List of "projects using plum"?

[jlapeyre]

It would be useful to have a list of projects that use plum, or even other Python MD libraries. Best would be OSS so we can see exactly how it is used.

(Maybe there is a way to do a pypi reverse-dependency search?)

Some things that such a list might help with:

• My main motivation: Say I want to introduce MD in a garden-variety existing large project. The response of the other devs will be "no". It would be useful to point to code bases using MD that didn't cause the end of the world, or even helped.
• Even if I think plum is the best MD library for a particular project, but the best large example uses, say multipledispatch, it's worth knowing. If I can convince others (and most importantly myself!) that this is was a good choice, it is a relatively small jump to say plum is better in this case. The harder question is whether MD is useful at all.
• What are the design problems and choices peculiar to Python? For example, integration with Python's OO model.

[wesselb]

Hey @jlapeyre,

This is a good suggestion! I'm fully on board with this. I agree that having examples/answers to the bullet points would be really nice.

I'm going to be away for roughly a week, but will have a shot at composing a list like this after that. If you feel like contributing, then of course a PR would be more than welcome!

[jlapeyre]

Good to hear!

If you feel like contributing, then of course a PR would be more than welcome!

I've looked a bit, but haven't found anything yet. I can look some more. Maybe soliciting from users would help.

[PhilipVinc]

It would be useful to have a list of projects that use plum, or even other Python MD libraries.

NetKet, a library for Jax/Flax Machine Learning targeted at quantum Physics, uses plum extensively to pick the right, efficient implementation for a large combination of objects that interact.

In fact, I got involved in plum's development in order to add a few features that I needed for NetKet.

My main motivation: Say I want to introduce MD in a garden-variety existing large project. The response of the other devs will be "no". It would be useful to point to code bases using MD that didn't cause the end of the world, or even helped.

Multiple Dispatch is undoubtedly very useful. The main problem is that it's not very common and well integrated in Python. If you are asking purely about MD, there is quite a large collection of talks and discussion from within the Julia community about it. I'd like to mention in particular this talk The Unreasonable effectiveness of Multiple Dispatch which explains very well what you gain with it.

I'd also like to mention the fact that PyTorch recently implemented an internal Multiple Dispatch system and publicly stated that They are going towards something that looks a lot like (MD) Julia. Some of their rationale is described in this blog post and in this podcast.

It would be useful to point to code bases using MD that didn't cause the end of the world, or even helped.

(As I see you work on qiskit...) In NetKet we have several 'state' (ket-like) types (you could think of them MPS-like, StateVector-like, in a sense) and several operators type (PauliStrings, FermionStrings, SparseMatrices, Ising...). While we have general implementations, for speed we have many states and operators each geared towards a specific application.

We use plum mainly for the expect(state, operator) and expect_and_grad(state, operator, options) functions in order to dispatch to the best implementation possible. We do have generic ones that work for all types that implement our API, but this allows us to have specialised implementations.

What are the design problems and choices peculiar to Python? For example, integration with Python's OO model.

I'm not sure what you are asking. But one thing that comes to my mind is that the power of Julia's dispatch system is that it works greatly with parametric types. The problem with plum/python is that it's hard to get this right because there was not (until very recently) support for parametric types in the type annotations.

[jlapeyre]

NetKet, a library for Jax/Flax Machine Learning targeted at quantum Physics

Great. This is exactly the kind of thing I am looking for.

Requoting what you quoted:

The response of the other devs will be "no".

Turns out that's not universal. In fact, I brought it up with qiskit devs. Actually several are rather positively disposed towards it. But, naturally they don't want to jump in blindly. And they did point out some concerns, partly in my example code (lack of clarity, performance, etc.)

this talk The Unreasonable effectiveness of Multiple Dispatch

There are a few versions of this talk (a good version is by both Stefan and Jeff at a compiler (I think) conference). I really like them. But these are talks by the people who are heavily invested in this approach. So another dev, especially one not familiar with the concept, would be right to be cautious about accepting the arguments. Seeing actually Python projects that are benefiting is more convincing, I think.

We use plum mainly for the expect(state, operator) and expect_and_grad(state, operator, options)

I think it's only partly coincidence that this is the same thing that my PR introducing multiple dispatch in qiskit targeted.

What are the design problems and choices peculiar to Python? For example, integration with Python's OO model.

I'm not sure what you are asking.

I came to Python from Julia rather than other way around. Julia was designed and grew with MD. But Python already has its popular paradigms or techniques. Most notably that things are organized around classes. But, maybe this is not really an issue.

the power of Julia's dispatch system is that it works greatly with parametric types.

I definitely agree. I sort of assumed that if this exists in Python, it won't (yet) be as solid and useful as it is in Julia.

Another thing. In Julia you often see long dispatch chains (maybe too long... unless you use a debugger to step through.) And heavy use of higher order functions used to inject behavior into code. These are usually devirtualized, inlined, elided and incur no performance penalty. But, in an interpreted language it would be difficult to design code in this style.

[wesselb]

I also know of Geometric Kernels, which uses Plum to provide a backend-agnostic implementation (something that works with PyTorch/TF/JAX/etc).

There is fastdispatch, which I believe is eventually to be used in fastcore.

There is PySSAGES, which I'm not familiar with, but which seems to use Plum. (@InnocentBug)

There is EMLP, which I'm also not familiar with, but which seems to use Plum to some extent.

Perhaps there are a few more projects using it.

I also have a few projects of my own which are quite intensively using Plum:

• LAB: provide types to make things backend-agnostic, so it works with PyTorch/TF/JAX/etc.
• Matrix: provide structured matrix types to accelerate linear algebra, building on LAB
• MLKernels: positive-definite kernels, buidling on LAB and Matrix
• Stheno: probabilistic programming with Gaussian processes, building on LAB, Matrix, and MLKernels
• Varz: backend-agnostic hyperparameter optimisation, buidling on LAB
• GPCM, OILMM, GPAR: implementations of some machine learning models, using all of the above
• neuralprocesses: a compositional framework for constructing neural processes, also using a fair few above packages

Once we decide on a way forward with the documentation (see #56, #57), let's put together this list!

[ice-tong]

Hi @wesselb @jlapeyre ~

MMEval is a unified evaluation library for multiple machine learning libraries. And MMEval also employee plum-dispatch (<2.0.0) for mutiple dispatch. For more see https://github.com/open-mmlab/mmeval/pull/4

[wesselb]

Hey @ice-tong! Thanks for mentioning MMEval. That's great to hear. :) I think it's time to start putting together this list.

[nstarman]

@wesselb, here's some more

• https://docs.kidger.site/quax/
• https://pypi.org/project/unxt/
• https://pypi.org/project/coordinax/
• https://pypi.org/project/galax/

[PhilipVinc]

netket

[wesselb]

I've added all projects mentioned in this issue to a list at the bottom of the README.

Please feel free to open a PR to add to the list. :)