Closed jlapeyre closed 4 months ago
wesselb
commented
2 years ago Hey @jlapeyre,
This is a good suggestion! I'm fully on board with this. I agree that having examples/answers to the bullet points would be really nice.
I'm going to be away for roughly a week, but will have a shot at composing a list like this after that. If you feel like contributing, then of course a PR would be more than welcome!
jlapeyre
commented
2 years ago Good to hear!
If you feel like contributing, then of course a PR would be more than welcome!
I've looked a bit, but haven't found anything yet. I can look some more. Maybe soliciting from users would help.
PhilipVinc
commented
2 years ago It would be useful to have a list of projects that use plum, or even other Python MD libraries.
NetKet, a library for Jax/Flax Machine Learning targeted at quantum Physics, uses plum extensively to pick the right, efficient implementation for a large combination of objects that interact.
In fact, I got involved in plum's development in order to add a few features that I needed for NetKet.
My main motivation: Say I want to introduce MD in a garden-variety existing large project. The response of the other devs will be "no". It would be useful to point to code bases using MD that didn't cause the end of the world, or even helped.
Multiple Dispatch is undoubtedly very useful. The main problem is that it's not very common and well integrated in Python. If you are asking purely about MD, there is quite a large collection of talks and discussion from within the Julia community about it. I'd like to mention in particular this talk The Unreasonable effectiveness of Multiple Dispatch which explains very well what you gain with it.
I'd also like to mention the fact that PyTorch recently implemented an internal Multiple Dispatch system and publicly stated that They are going towards something that looks a lot like (MD) Julia. Some of their rationale is described in this blog post and in this podcast.
It would be useful to point to code bases using MD that didn't cause the end of the world, or even helped.
(As I see you work on qiskit...) In NetKet we have several 'state' (ket-like) types (you could think of them MPS-like, StateVector-like, in a sense) and several operators type (PauliStrings, FermionStrings, SparseMatrices, Ising...).
While we have general implementations, for speed we have many states and operators each geared towards a specific application.
We use plum mainly for the expect(state, operator) and expect_and_grad(state, operator, options) functions in order to dispatch to the best implementation possible. We do have generic ones that work for all types that implement our API, but this allows us to have specialised implementations.
What are the design problems and choices peculiar to Python? For example, integration with Python's OO model.
I'm not sure what you are asking. But one thing that comes to my mind is that the power of Julia's dispatch system is that it works greatly with parametric types. The problem with plum/python is that it's hard to get this right because there was not (until very recently) support for parametric types in the type annotations.
jlapeyre
commented
2 years ago NetKet, a library for Jax/Flax Machine Learning targeted at quantum Physics
Great. This is exactly the kind of thing I am looking for.
Requoting what you quoted:
The response of the other devs will be "no".
Turns out that's not universal. In fact, I brought it up with qiskit devs. Actually several are rather positively disposed towards it. But, naturally they don't want to jump in blindly. And they did point out some concerns, partly in my example code (lack of clarity, performance, etc.)
this talk The Unreasonable effectiveness of Multiple Dispatch
There are a few versions of this talk (a good version is by both Stefan and Jeff at a compiler (I think) conference). I really like them. But these are talks by the people who are heavily invested in this approach. So another dev, especially one not familiar with the concept, would be right to be cautious about accepting the arguments. Seeing actually Python projects that are benefiting is more convincing, I think.
We use plum mainly for the
expect(state, operator)andexpect_and_grad(state, operator, options)
I think it's only partly coincidence that this is the same thing that my PR introducing multiple dispatch in qiskit targeted.
What are the design problems and choices peculiar to Python? For example, integration with Python's OO model.
I'm not sure what you are asking.
I came to Python from Julia rather than other way around. Julia was designed and grew with MD. But Python already has its popular paradigms or techniques. Most notably that things are organized around classes. But, maybe this is not really an issue.
the power of Julia's dispatch system is that it works greatly with parametric types.
I definitely agree. I sort of assumed that if this exists in Python, it won't (yet) be as solid and useful as it is in Julia.
Another thing. In Julia you often see long dispatch chains (maybe too long... unless you use a debugger to step through.) And heavy use of higher order functions used to inject behavior into code. These are usually devirtualized, inlined, elided and incur no performance penalty. But, in an interpreted language it would be difficult to design code in this style.
wesselb
commented
2 years ago I also know of Geometric Kernels, which uses Plum to provide a backend-agnostic implementation (something that works with PyTorch/TF/JAX/etc).
There is fastdispatch, which I believe is eventually to be used in fastcore.
There is PySSAGES, which I'm not familiar with, but which seems to use Plum. (@InnocentBug)
There is EMLP, which I'm also not familiar with, but which seems to use Plum to some extent.
Perhaps there are a few more projects using it.
I also have a few projects of my own which are quite intensively using Plum:
Once we decide on a way forward with the documentation (see #56, #57), let's put together this list!
ice-tong
commented
1 year ago Hi @wesselb @jlapeyre ~
MMEval is a unified evaluation library for multiple machine learning libraries. And MMEval also employee plum-dispatch (<2.0.0) for mutiple dispatch. For more see https://github.com/open-mmlab/mmeval/pull/4
wesselb
commented
1 year ago Hey @ice-tong! Thanks for mentioning MMEval. That's great to hear. :) I think it's time to start putting together this list.
nstarman
commented
4 months ago PhilipVinc
commented
4 months ago wesselb
commented
4 months ago I've added all projects mentioned in this issue to a list at the bottom of the README.
Please feel free to open a PR to add to the list. :)
It would be useful to have a list of projects that use plum, or even other Python MD libraries. Best would be OSS so we can see exactly how it is used.
(Maybe there is a way to do a pypi reverse-dependency search?)
Some things that such a list might help with:
multipledispatch, it's worth knowing. If I can convince others (and most importantly myself!) that this is was a good choice, it is a relatively small jump to say plum is better in this case. The harder question is whether MD is useful at all.