Question about pybind11 and linking

[petrasvestartas]

Hi @brgcode ,

Thank you for a help during the visit in Zurich.

Short description what I did: I went through compas_cgal in order to replicate the workflow by my code without installing this repo, simply by learning from the files that are uploaded here. I managed to run the slicer from C++ in compas_wood - new empty environment you helped to create.

My current workflow: 1) I set up a Visual Studio 2019 project that links pybind11, CGAL, boost and eigen. Compiles without issues. 2) Then I added 4 files in compas_wood with a more or less similar code to original repo: slicer.py slicer.pyi types.py test_slicer.py CGAL.zip

3) Lastly I tested tect_slicer.py and I get a result of 338 polylines, which I assume is correct and verifies that C++ and CPython talks to each other:

Question: How do you link C++ Python binding directly in VS Code because I have to write following path which I assume is different on every computer and depending on operating system these bindings must be compiled differently on each machine:

I know that this can be difficult to explain, but I need this to move forward and depending on your time I could come to Zurich if needed.

import sys
folder = "C:/IBOIS57/_Code/Software/Python/Pybind11Example/vsstudio/Release/"
if folder not in sys.path:
    sys.path.append(folder)
import pybind11module

In my case I have a Visual Studio 2019 project that helps to debug the code, link libraries and build pybind11module.cp38-win_amd64.pyd file:

import numpy as np
from compas.plugins import plugin

#Import module from folder instead of using -> from _cgal import slicer

import sys
folder = "C:/IBOIS57/_Code/Software/Python/Pybind11Example/vsstudio/Release/"
if folder not in sys.path:
    sys.path.append(folder)
import pybind11module

@plugin(category='trimesh', pluggable_name='trimesh_slice')
def slice_meshPython(mesh, planes):
    """Slice a mesh by a list of planes.
    Parameters
    ----------
    mesh : tuple of vertices and faces
        The mesh to slice.
    planes : list of (point, normal) tuples or compas.geometry.Plane
        The slicing planes.
    Returns
    -------
    list of arrays
        The points defining the slice polylines.
    """
    vertices, faces = mesh
    points, normals = zip(*planes)
    V = np.asarray(vertices, dtype=np.float64)
    F = np.asarray(faces, dtype=np.int32)
    P = np.array(points, dtype=np.float64)
    N = np.array(normals, dtype=np.float64)

    pybind11module.say_hello()
    pointsets = pybind11module.slicerCGAL.slice_meshCGAL(V, F, P, N)
    return pointsets

[tomvanmele]

information about the pybind11 build systems is available here https://pybind11.readthedocs.io/en/stable/compiling.html#

compas_cgal uses the setuptools version.

you should be able to do the following

conda create -n cgal python=3.8 eigen=3.3 boost-cpp cgal-cpp">=5.0" mpir mpfr pybind11 compas --yes
conda activate cgal
cd compas_cgal
pip install -e .

this should compile the c++ code and install an editable version of the python wrapper in your environment, which is then directly available to Python in VS Code if you activate the cgal environment.

once you want to distribute this to others, ideally you add a conda recipe and push this to conda-forge...

[tomvanmele]

the above will always compile "everything". if you just want to compile one module, you can change the list of extension modules in setup.py

ext_modules = [
    Extension(
        'compas_cgal._cgal',
        sorted([
            'src/compas_cgal.cpp',
            'src/compas.cpp',
            'src/meshing.cpp',
            'src/booleans.cpp',
            'src/slicer.cpp',
            'src/intersections.cpp',
            'src/measure.cpp',
        ]),
        include_dirs=[
            './include',
            get_eigen_include(),
            get_pybind_include()
        ],
        library_dirs=[
            get_library_dirs(),
        ],
        libraries=['mpfr', 'gmp'],
        language='c++'
    ),
]

[petrasvestartas]

Thank you for an answer, I am trying to understand this. To make things less breakable, I just want to gradually modify the current compas_cgal and add parts of new code in my pc.

If I understand correctly, I have to install conda environment, then I can modify the code of C++ directly in VS code?

In Anaconda prompt I typed:

conda create -n cgal python=3.8 eigen=3.3 boost-cpp cgal-cpp">=5.0" mpir mpfr
pybind11 compas --yes
conda activate cgal

But after this breaks. Which directory contains that setup.py file?

cd compas_cgal
pip install -e .

[tomvanmele]

you seem to be working in the cgal environment folder.

the setup.py i am referring to is part of the compas_cgal repo. you should issue the command pip install -e . from there.

once you have managed to build compas_cgal locally as described above, you can try doing the same from compas_wood, or whichever package you're experimenting with...

[petrasvestartas]

Dear @brgcode ,

I am coming back to this issue, since I a bit more confident that my code works and changing from opennurbs +.NET to C++ only with cgal and COMPAS was a bit of a learning curve. All good for now and I am glad I did this, I learnt a lot.

I would like to ask again for a help considering package creation and pushing it outside my pc.

Again I am replicating steps:

I did this:

1. conda create -n compas_cgal python=3.8 eigen=3.3 boost-cpp cgal-cpp">=5.0" mpir mpfr pybind11 compas --yes

2. This time I have compas_cgalinstalled, not cgal. I activate compas_cgal:

3. But I did not manage to go to compas_cgal folder. But I am at least in compas_cgal environment. So I tried command pip install -e . This complains that I am in a wrong directory and setup.py is not here.

4. I searched where the setup.py could be in my pc. Should I run this file?

If yes, I already did this:

[tomvanmele]

it is not clear to me what you are trying to accomplish. i assume you already have a setup that allows you to locally build your code and use it. as a next step, i assume you want to release compas_wood?

if that is the case, i have the feeling you are mixing up a few things, since all of the steps you are performing should be done in your development environment, for example named cgal (but in your case it would make more sense to call it wood or wood-dev), from the compas_wood repo. the folders in the anaconda env directory serve a different purpose and are never involved in this process.

in a nutshell, the proces would be the following

conda create -n wood-dev -c conda-forge python=3.8 mpir mpfr boost-cpp eigen=3.3 cgal-cpp=5.3 pybind11 compas compas_view2 --yes
conda activate wood-dev
conda install cookiecutter
cd path/to/dir/where/you/keep/your/repos
cookiecutter gh:compas-dev/tpl-extension

cookiecutter will ask you a bunch of questions about your package. once this is done, proceed with the following (i am going to assume you named the package compas_wood)

cd compas_wood
git add .
git commit -m "Initial commit."
pip install -r requirements-dev.txt

i think all of the above, we already did together when you were here.

update the repo with you code. and update the setup.py file of the repo similar to the one found in compas_cgal. if there are no bugs in your code and everything is properly linked, every subsequent call to the pip installer will compile your repo making the code available as part of your editable install.

pip install -e .

there is not much more to it than that. as you can see everything happens from within your local repo...

once you have a stable version, you can consider making this available to others via conda-forge. for this you would have to write a conda recipe, but since most of the installation process is handled by setup.py that should be relatively straightforward...

fyi, the recipe for compas_cgal is here https://github.com/brgcode/compas_cgal-feedstock

[petrasvestartas]

Hi @brgcode ,

Thanks for the patience on this issue.

it is not clear to me what you are trying to accomplish. i assume you already have a setup that allows you to locally build your code and use it. as a next step, i assume you want to release compas_wood? I have the same repo as installed with you. I was loading .pyd module that was built using Visual Studio. Now I want to build compas_wood locally.

I followed what you described:

1. added all C++ source files here: https://github.com/ibois-epfl/compas_wood/tree/main/src

2. I changed setup according to compas_cgal https://github.com/ibois-epfl/compas_wood/blob/main/setup.py

3. Afterpip install -e . in directory of C:\Users\petra\compas_wood I get following errors and trying to understand how to make them go away

All the code is already here https://github.com/ibois-epfl/compas_wood. And if I succeed with this, I would like to push to conda-forge.

[tomvanmele]

okay i will have a look at the repo and send a PR

[petrasvestartas]

Thank you

---

The pybind methods are placed in these files to keep it separated from the rest: https://github.com/ibois-epfl/compas_wood/blob/main/src/xxx_interop_python.cpp https://github.com/ibois-epfl/compas_wood/blob/main/src/xxx_interop_python.h

Currently I reference c++ pybind11 modules within lines 15-19, which allowed to develop C++ code via Visual Studio, and this was the main reason to open this issue to know how to build this not via Visual Studio 2019: https://github.com/ibois-epfl/compas_wood/blob/main/src/compas_wood/CGAL/connectionDetection.py

---

If I understand correctly, all header files must be in include folder and .cpp files inside sources files. These must match with this in setup file:

ext_modules = [
    Extension(
        'compas_wood._wood',
        sorted([
            'src/clipper.cpp',
            'src/connection_zones.cpp',
            'src/xxx_interop_python.cpp'           
        ]),
        include_dirs=[
            './include',
            get_eigen_include(),
            get_pybind_include()
        ],
        library_dirs=[
            get_library_dirs(),
        ],
        libraries=['mpfr', 'gmp'],
        language='c++'
    ),
 ]   

After changing the files I get error that eigen is not found C:\Users\petra\compas_wood\include\compas.h(5): fatal error C1083: Cannot open include file: 'Eigen/StdVector': No such file or directory

---

How does python know where are these links to libraries and header files?

 - (Headers) C/C++ -> General -> Additional Include Directions :
    C:\IBOIS57_Code\Software\Python\Pybind11Example\externals\pybind11\include;
    C:\IBOIS57_Code\Software\Python\Pybind11Example\source\module;
    C:\IBOIS57_Code\Software\CPP\CGAL\CGAL-5.3\include;
    C:\IBOIS57_Code\Software\CPP\CGAL\CGAL-5.3\auxiliary\gmp\include;
    C:\IBOIS57_Code\Software\CPP\Eigen\eigen-3.3.9;

- (Libraries) Linker -> Input -> Additional Dependencies :
    C:\IBOIS57_Code\Software\CPP\CGAL\CGAL-5.3\auxiliary\gmp\lib\libgmp-10.lib;
    C:\IBOIS57_Code\Software\CPP\CGAL\CGAL-5.3\auxiliary\gmp\lib\libmpfr-4.lib;

[petrasvestartas]

Hi @brgcode ,

Maybe you had time to look at my code? Or maybe you know what linking issue regarding the last post I have?

[tomvanmele]

sorry, have not had time yet. will try to get to it later this week...

[petrasvestartas]

Dear @brgcode ,

Maybe you have time to look at this issue this week?

If you are too busy, maybe you could explain how linking works when compiling on a local machine? Does a user need to download all the c++ header only libraries or is it somehow done automatically?

[tomvanmele]

if the compiler can't find Eigen it is probably because you have not installed it in your development environment. if Eigen is installed in your environment, the get_eigen_include function in setup.py will find it.

i noticed that your repo is a bit all over the place. for example, you have two calls to setup in setup.py. therefore i would suggest to do a bit of cleaning up first, then create a fresh development env, with all requirements properly installed, and then follow the step-by-step instructions again.

before testing your dev environment on compas_wood, try building compas_cgal first, as a sanity check, since they have the same requirements. if it works for compas_cgal it should work for your lib as well, so then you at least already now that is not the cause of your errors.

the dev instructions for compas_cgal are quite precise. just follow them to the letter...

[petrasvestartas]

compas_cgal_dev compiles

Now I am trying to compile my code. I created a new environment from scratch with eigen, cgal and pybind libraries which we're not installed before.

I have a question about .h files linking. In setup file https://github.com/compas-dev/compas_cgal/blob/main/setup.py line 55 you link .cpp files. In which line do you link headers? I am using inline headers as helper functions that do not have separate .cpp files.

[tomvanmele]

they are in the include dir a few lines further down...

[petrasvestartas]

Thank you

It compiles:

The following question is about functions calls.

Do you wrap function in .pyi files? Like this:

And call this functions using plugins import? Like this:

Are there any other steps needed? I am trying to call simple print function, which is not found currently.

[tomvanmele]

no, the *.pyi files are type stubs. they contain typing information for static type checkers like mypy and pylance. you can ignore those. they are not strictly needed. they are a bit like header files in c++. they don't contain any implementations.

the wrapper functions are in the regular *.py files...

[tomvanmele]

using the plugin functionality only makes sense if there are corresponding pluggables in the core compas framework. so in your case you can leave this out entirely. just make normal wrapper functions that convert the input data to numpy and hand it over to the c++ implementation.

[petrasvestartas]

Thanks.

Sorry to bother again. It seems I still have issues within the building part "pip install -e ." because the .pyd file are 0 byte in size, meaning the building failed. I was mistaken by typing consequently two times pip install -e .

Is there any pip command that deletes previous files similar to visual studio "rebuild" command (clean up the previous build)? Each time, I manually delete _cgal.cp38-win_amd64.pyd file before I write pip install -e .

The building error message does not say much, but it probably means that the compiler does not find .cpp files. And this error happens even if I try to compile one single file such as this one: https://github.com/pybind/python_example/blob/master/src/main.cpp

I am missing something basic.

[tomvanmele]

you should not even have a _cgal....pyd file. this is something created by compas_cgal and has nothing to do with compas_wood...

i am not trying to be mean, but i find the current process completely counter-productive.

i don't understand why you are trying to rush through this. in my opinion, with a step-by-step process this would have been solved ages ago. why not make a clean repo and first get the basics right before adding all your code to it?!

i keep pointing at the compas_cgal repo as a good example to start from because everything is nicely separated and easy to follow. i can't find any of this logic in your current code structure. there are also many parts of the code that are clearly the result of a quick copy-paste and were never properly adapted. other things seem completely missing. for example, and i might be wrong, but as far as i can tell, the pybind11.h headers are never included.

so, the best advice i can give you is to start from scratch with a clean repo and do things step by step so you can figure out where your process goes wrong. starting with a hello function is a good idea, but then do ONLY that first, without having 100 other things going on that could be influencing things. once that works you can start moving stuff back in...

[petrasvestartas]

Finally builds and runs...

Today, I went through this pybind11 example https://github.com/pybind/python_example/blob/master/setup.py and gradually learnt the differences between that simple one and the cgal package. I know they are using different build methods but I found the naming issues I had in .cpp files.

The pybind11 headers were correct, just I do not place them in compas.h but in xxx_interop_python.h

Then I tested basic hello world and eventually the function I wrapped before:

And I the methods works fine, and I get the display as before:

I believe, I was very intimidated by a lot of options in setup file, and yes, rewriting step by step instead of copy-paste helped a lot. Thanks for your help.

I would like also to try my code on my mac to see if it works correctly. And I also have to learn how to use conda-forge.

[tomvanmele]

nice! good for you...

for mac, make sure to use the correct dependencies. there are some differences between windows and mac. again, look at the readme of compas_cgal.

conda-forge is easy if things already work locally...

[petrasvestartas]

It works on Mac:

GCC compiler is a bit different than Windows Visual Studio, I had to adapt a little bit C++ code that it works on both os.

Now I am working on documentation. The repository is located here: https://github.com/petrasvestartas/compas_wood

Could you give an advice/link to tutorials how to upload my repo to conda-forge? I never used it.

[tomvanmele]

this seems solved...