Question regarding contributing and folder structure (not an issue)

[paxcalpt]

Hi everyone, we've been translating parts of our code from java to python, heavily investing in using cython as much as possible

I passed through a phase where I've been grumpy towards gpus, as we always have a massive backlog of people asking for help to get our code running on their... well... 1998 brick... but even going to C-land via cython isn't always as fast as we'd hope for.

I'll be looking at playing with clesperanto in the next days, I'm trying to make sense how you break your code into tier folders (as we're likely keen to contribute back), is there a place where you discuss this a bit or can you give me some pointers? For example, if I wanted to make some kernels reimplementing some of our interpolators from here (https://github.com/HenriquesLab/NanoPyx/tree/main/src/nanopyx/core/transform) - is there good way to go about it?

I get that most kernels are under the kernel folders, I'm just confused about how you distribute the interfaces across folders

Apologies if this is actually obvious and I've just not found the right page/resource yet

Keep up the good work!

[haesleinhuepf]

Hey Ricardo @paxcalpt ,

great to see you here.

I'm trying to make sense how you break your code into tier folders (as we're likely keen to contribute back), is there a place where you discuss

The tiers are related to complexity. tier0 is low level basic stuff. tier1 uses tier0, and so on. tier9 contains the crazy stuff 🤯

For example, if I wanted to make some kernels reimplementing some of our interpolators from here (https://github.com/HenriquesLab/NanoPyx/tree/main/src/nanopyx/core/transform) - is there good way to go about it?

I collected some functions for affine transforms in a special (high level) tier8. @pr4deepr is quite experienced with special interpolation techniques and @iionichi will soon start in my group working on implementing an interpolation technique for affine transforms. I propose you open another transform-specific issue to discuss feature requests and details.

I get that most kernels are under the kernel folders, I'm just confused about how you distribute the interfaces across folders

The right place to put the _my_new_function.py is in a higher tier than all the functions it uses. E.g. if your function uses functions from tier1 and tier3, it is supposed to be in tier4.

And there is also an open.cl file organizational issue @StRigaud reminds me from time to time: Currently, we have two places where to store open.cl files. The correct way to put them is in the kernels folder, which is a separate repository. However, this requires working with git submodules and is a bit cumbersome. E.g. a pull-request to pyclesperanto-prototype would require a parallel pull-request to clij-opencl-kernels. For developing new kernels, it is much easier to put the whatever_function.cl file next to the corresponding _whatever_function.py file in the same tier folder. For getting started I recommend the second path and later we need to clean up things.

Apologies if this is actually obvious and I've just not found the right page/resource yet

Apologies that the right page/resource was not written yet. In case you think it makes sense to clarify things, I'm happy to schedule zoom meetings. Wednesdays 3 pm CET is for example a good slot in general.

Thanks for reaching out! I'm looking forward to your contribution and I'm happy to assist.

Best, Robert

[paxcalpt]

Hi Robert, thank you for the quick feedback =) Sounds like I can wait a bit for @iionichi

Apologies that the right page/resource was not written yet. In case you think it makes sense to clarify things, I'm happy to schedule zoom meetings. Wednesdays 3 pm CET is for example a good slot in general.

No worries at all, I'll browse the code a bit more to get acquainted. Sometime in the future it might be cool to have a "how to contribute" markdown

[StRigaud]

I'm jumping in on the open.cl folder and organisation.

The kernel repo hold the CLIJ2.5x kernels as they are. Though since we started Clesperanto we added some new, and corrected some more, as well as rewrite some so they are directly nd thus removing the _2d or _3d tag in the kernel files.

This effort is translated in the clesperanto_kernels branch, and overall to clean a bit all the kernels that were developed over time. Although most kernel are already their, some are still missing because a bit harder to update.

In a ideal world, all repos and effort rely on the same kernel version, avoiding to have local development in some repo and not in others. E.g. a new kernel is developed but not shared in the kernel repo leading to a discrepancies between the different implementations of the library (Java, C++, Python).

However, this requires working with git submodules and is a bit cumbersome. E.g. a pull-request to pyclesperanto-prototype would require a parallel pull-request to clij-opencl-kernels.

The pyclesperanto repo (which is the correct pyclesperanto to use btw :laughing: ) as well as the CLIc_prototype and the clesperantoj_prototype are now completely sync with the kernel repo without submodules or github heavy wizardry. So they are ways to not use submodules :wink:

I'm happy to schedule zoom meetings. Wednesdays 3 pm CET is for example a good slot in general.

Happy to tag along!

Sometime in the future it might be cool to have a "how to contribute" markdown

I dit a prototype of "how to contribute" one for CLIc, the C++ version of Clesperanto, not sure it help but who knows. And true, we should start to make some better docs for the project!

[paxcalpt]

The pyclesperanto repo (which is the correct pyclesperanto to use btw 😆 )

Ah, sorry. I followed the links here. Maybe worth redirecting the python link to the pyclesperanto repo?

[haesleinhuepf]

The pyclesperanto repo (which is the correct pyclesperanto to use btw 😆 )

Ah, sorry.

I'm not sure if pyclesperanto (without _prototype) is ready for use in science projects. Its functionality is yet quite limited. Correct me if you think I'm wrong @StRigaud 😉

[StRigaud]

It was a joke, you DL the correct one :wink:

pyclesperanto_prototype is pure python and rely on pyopencl or cupy as a GPU back-end. Its the most developped and advance in functionnality and ready to be use by all.

pyclesperanto is a pythonized version of the C++ clesperanto library CLIc, still in development but should already work. However it is less advanced in functionnalities and operations available, Also it is a bit more funcky to contribute as all the magic is in the C++ layer.

ready for use in science projects

not ready for science stuff, but ready to play with

[pr4deepr]

Thanks for the post @paxcalpt and nice to meet you. Thanks @haesleinhuepf for the mention. Happy to help with the interpolation stuff. For the interpolation, I did play around with interpolation kernels and in the end translated python code to opencl. Was also figuring out how to debug opencl code! :)