Python bindings with PyO3

alecandido commented 3 years ago

Hi @cschwan, I was reading about PyO3, and after a while I started thinking that it may be not so hard to provide bindings for pineappl using it.

I don't know how much you care, but I decided to make a try, just in my spare time. The goal it's to port a minimal subset of things, just a proof of concept, to expand after to the full one.

It should not take so much, but of course it's not my main business, so it will take its time. Do you mind if I try?

I also thought it was a good way to familiarize a little more with pineappl, since soon it will be useful (both because of yadism, but also because I am going to work on pineko too, and I know you are the expert on pineappl, but it won't hurt to know something more).

If you agree maybe you can suggest me which one could be "a minimal set".

cschwan commented 3 years ago

@AleCandido We need to replace the use of LagrangeSubgridV2 with another type. I can do the replacement, but I'd like to have an integration test for pineappl_py. Could you please add one (add a .yaml file generated by yadism, and the test should just check that the generated grid gives the right predictions)?

felixhekhorn commented 3 years ago

@cschwan Huhu, is this what you need? https://github.com/N3PDF/yadism/commit/967d3ec9e443ba917b26635fe2c2222445815eaf please tell us if you can run it!

cschwan commented 3 years ago

@felixhekhorn How do I install yadmark?

cschwan commented 3 years ago

Hi @felixhekhorn and @AleCandido , the FKTable class is definitely a good idea, and for the time being it's all we need in this direction. Before we merge this branch into master, however, I'd like to make the FKTable class a Rust struct and then thinly wrap it like any other class - in that regard it stands out from the rest now.

In general, we should come up with a list of rules that guide the design of the Python interface. From what we've done already, I'd say they are:

the interface should thinly wrap the Rust functionality so that we can use it in Python. More code is allowed if we need to convert between types that are expected in Python, for instance numpy ndarrays, etc.
the Rust structs should map one-to-one to Python classes, and the same should hold true for their methods, and also for the all of the names of classes/methods. Is there are standard way of naming things in Python? Does it differ from Rust?
more items?

Before we merge, we should also make sure that

it's easily possible to migrate from the old Python interface (written by @scarrazza) to the new one. After a while we should remove the old interface
we have working unit tests for the new Python interface. I think it's quite easy to break it when changing things on the Rust side.

Finally, I propose to rename pineappl_py to pyneappl, what do you think?

alecandido commented 3 years ago

Starting from the top:

perfectly fine for the FKTable object, I understand that is weird to have some features available only through the wrapper, so we can do it (it's a small class at the end)
if ever there will be more need to decouple fktables from grids this struct can be used

About the guidelines: I believe that's enough, especially because we are aiming for something minimal. Your crate is organized by structs, then in python is easy to mirror by classes. Spare functions would just turn into functions, and even modules are available on both sides (even nested), so it is only required to mirror. I would remain a little bit flexible on the amount of things that can be done in python, since pyo3 is not always complete or very easy to use sometimes applying some workaround through a little bit more code on the python side is fine as well. Nevertheless: less is more.

About the merge:

actually what I forgot (or I thought we dropped) is that these bindings are going to replace @scarrazza ones, but then should we provide complete bindings? Or should we limit ourselves to the only things that are currently relevant for yadism and NNPDF?
I would not rely on the unit tests still work after the last batch of updates, I need to have a look. For sure they have to be extended

pyneappl makes perfect sense, I would say it's quite pythonic. But if we are going to replace @scarrazza ones then I wonder, why not pineappl itself? (pyneappl would be more stylish the same, but maybe keeping pineappl is a more direct choice)

cschwan commented 3 years ago

@AleCandido @felixhekhorn The last commit adds the convolute_eko2 method, which is also used in the Python bindings already. If you want to disable the new method simply change eko2 to eko there. The whole thing compiles, but unfortunately the DIS example from Pineko doesn't work yet - the numbers are simply wrong. I'm not sure if I can debug it today, but if you want to play with it in the meantime, please go ahead!

In any case, I implemented the following:

scale variations should work for xir == 1.0, for xir != 1.0 there a FIXME in the code
new terminology: the high-scale grid is now called the source grid, the low-scale grid is the target grid. I think that's easier to remember and more general: the EK operator transforms the source grid to the target grid!
the loops are ordered quite differently from convolute_eko:
- I first loop over all the indices (bin, lumi) of the source grid, and add all orders together in a single grid
- only then the EK operators are applied, and hopefully that's faster for LHC grids

cschwan commented 3 years ago

An update of my last comment:

scale variations should work now, the FIXME has been fixed in commit 0977653; however, it's not been tested yet
DIS, both with the PDF at x1 and with the PDF at x2, and LHC processes seem to work. I've tested the ATLAS_DY_7TEV_49FB_HIMASS.pineappl.lz4 grid, which includes NLO QCD+EW corrections. The grid is evolved in about 20 minutes on my computer, previously 4 hours were needed. The critical section is still the one marked in the code (the core of the evolution):

https://github.com/N3PDF/pineappl/blob/4b9129fd7d58f4c8e9f6188dd7a56abccbf7b3f0/pineappl/src/grid.rs#L1414-L1444

cschwan commented 3 years ago

Before merging into master we have to fix a few things (ordered by importance):

[x] try adding a Grid::convolute method in the Python part; implemented in commit 9a3153e41e867589814b370b15c6cae4ae277f70
[x] wire Rust FkTable into the interface
[x] fix cargo test problem
[x] in Rust add struct for an FK table and synchronise it the Python interface
[x] change create c'tors in Python?
[x] do we need PyBinRemapper? Yes, we do: https://github.com/N3PDF/pineappl/pull/51#issuecomment-912437811
[x] minimise number of methods/structs in the interface
[x] check pub vs. pub(crate) in the Rust-Python bridge
[ ] add tests
[x] add more documentation
[x] check README
[x] strip down .gitignore?