Investigate possible Python interface improvements

[cschwan]

I've just seen the following on reddit:

https://www.reddit.com/r/rust/comments/rk12bg/writing_rust_libraries_for_the_python_scientific/

Can we learn anything from it for our Python bindings? In particular I noticed the following points:

• their wheel publishing action seems much simpler (see also this comment)
• do we want static typing (MyPy)?
• the numpy usage could be relevant for #82

[alecandido]

Thank you very much, I guess one of the reasons why it has been rather difficult to write these bindings actually was lack of examples, so it's good to have some more.

Unfortunately, I don't believe those points are much relevant, but this boils down to opinions in some cases.

Let's provide details:

• workflow: our workflow is already simple enough, the possible improvements are the following:
  • the only complicate branch is linux, but this because it is the only one for which you need to compile inside a container (and this is actually what the maturin action does for linux), but pineappl has many more dependencies than ragged-buffer, and unfortunately this requires us to run inside a container with a C compiler, that unfortunately seems not to be present on the PyPA provided manylinux containers https://github.com/N3PDF/pineappl/blob/cd7bbaf298326fc5edcdcb2b75868784816dfe5e/pineappl_py/package/Containerfile#L8 (if I remember, in particular should be syn or quote fault, i.e. we have some dependency that implements non-trivial macros, and somehow for that a C compiler is needed)
  • I noticed that we're dealing inconsistently with python version between Mac and Windows, we can make it uniform https://github.com/N3PDF/pineappl/blob/cd7bbaf298326fc5edcdcb2b75868784816dfe5e/.github/workflows/wheels.yml#L29-L31 (I guess I didn't before because @felixhekhorn was pressing me not to waste time on Windows)
  • actually, we might want to collapse the jobs for Mac and Windows, since they should be equal (apart from a further --universal2 flag for Mac, but this is simple to add with include matrix sub-item
  • actually in its include he is also defining target, but then is unused, so we get nothing (it should appear as ${{ matrix.target }})
  • the moment the maturin action will provide support to patch manylinux container, we can drop our own and join all the branches (simply providing a bigger matrix)
• static typing is only used for linting and IDE suggestions in python, i.e. more or less the same as TypeScript, but it's much less adopted; nevertheless, it's true that we only depend on Numpy, and it recently adopted types, so we can even do (even though there is not much code in python, so it's even not that useful, and another layer of complication)
• we should need nothing external for #82
  • the issue itself claims that we're doing it, simply not consistently, so we simply need to extend it (but we already have one or more working examples)
  • according to what I found, he only uses Numpy arrays in input, that is much simpler, since you don't have to play with lifetimes, while it's much more complicate to return arrays that have been created inside the function (because if you don't link to an external lifetime, they should expire inside the function, since the scope is there) https://github.com/N3PDF/pineappl/blob/a1fe0350b00a1129e35d680c5dfa4448a59f8fa6/pineappl_py/src/grid.rs#L410

[cschwan]

@AleCandido OK thanks for going over the points I mentioned. Feel free to leave the Issue open (for the points you mentioned) or to close it!

[alecandido]

I guess I can leave it open to improve a bit the wheels.yml workflow, I'll do it.

The last point has already its own dedicated issue, while the second one I'm not sure we want. In case, I would open another issue/PR to implement it.

[cschwan]

Here's another discussion: https://www.reddit.com/r/rust/comments/rtnwd5/python_extensions_in_rust_lessons_learned_from/

[alecandido]

Here's another discussion: https://www.reddit.com/r/rust/comments/rtnwd5/python_extensions_in_rust_lessons_learned_from/

Interesting in general, thanks :)

Do you believe there is anything in particular we should take care in pineappl? (and we're not already doing...)

[cschwan]

I still have to read it, it was also a way to 'bookmark' it :smile:

[alecandido]

In commit 22964c2774fca019e10acc64162b71b988af23db I actually merged the windows + Mac workflows, now they should be more consistent.

As usual, I couldn't test, since it requires running. Nevertheless, the change is tiny, so I don't expect much noise, and all the jobs will run independently in any case (less repetition in the declaration, same repetition in the run, as wanted).

Whenever we like we can even close this issue, or we can keep it open for the second discussion, if you find anything useful for pineappl.

[cschwan]

Let me make another beta release to test it...

[cschwan]

It seems to have worked, there I'll close this Issue.