Provide standard build system/script to/and facilitate standard packaging

[WizardUli]

I absolutely hate conda package manager. Not only it does not play well with existing OS packages whether binary or python, it also by default pollutes or even break one's environment in a very perverted way which is exactly what at least one other user stated in https://github.com/CadQuery/cadquery/issues/153 .

Having said that I decided to try to improve the experience for myself and perhaps for other people as well by creating ebuilds for Gentoo and probably also rpms for Fedora etc..

But after examining this repository I'm confused at how to even build it. If it was a standard conda package as far as I understand there would be a build.sh with build steps or calls to another build system.

But all commands which seem to be doing something useful are in azure-pipelines.yml and build-bindings-job.yml? How does this work? In order to build it you send it to some Microsoft machine?! How do you even develop in such repo then?

Would it be possible to use some normal build system like Make, Cmake or some other so one can then easily integrate it into OS specific packaging solutions?

[adam-urbanczyk]

I, on the contrary, like conda a lot and it saved me months of work by now. My advice is to use pure local install and do not do any magic conda init stuff. That way your environment will be untouched.

As stated in the README, if you want to build locally you'll need to understand what is in the azure-piplines.yml. The code of OCP is generated by pywrap (which BTW generates a cmake file too). Assuming you have all the dependencies it boils down to running something along these lines:

python -m bindgen -n $n_cores parse ocp.toml  out.pkl 
python -m bindgen -n $n_cores transform ocp.toml out.pkl out_f.pkl
python -m bindgen -n $n_cores generate ocp.toml out_f.pkl
cmake -B build -S $OCP_src -G Ninja
cmake --build build -j $n_cores -- -k 0

Development in this repo boils down to modifying ocp.toml and rerunning parts of the script above.

[Franck78]

Hello,

I'm running without conda. Not sure i will get something out one day....

• python 3.6.10
• clang 9.0.1
• gcc 7.5

Anyway, it looks like some OCCT headers are not 'compiled' well and it's disturbing. You flagged them Warnings so I think it should produce something in the end.

[D 201022 01:49:55 module:44] ./opencascade/BVH_DistanceField.hxx
[D 201022 01:49:55 module:44] ./opencascade/NCollection_CellFilter.hxx
[D 201022 01:49:55 module:44] ./opencascade/Standard_LicenseError.hxx
[D 201022 01:49:57 module:44] ./opencascade/gp_Parab.hxx
[W 201022 01:49:57 translation_unit:46] ./opencascade/NCollection_CellFilter.hxx
[W 201022 01:49:57 translation_unit:47] dummy.cxx:510:74: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:514:9: error: incomplete result type 'NCollection_CellFilter_InspectorXYZ::Point' (aka 'gp_XYZ') in function definition
[W 201022 01:49:57 translation_unit:47] dummy.cxx:515:25: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:515:46: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:515:67: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:533:74: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXY::Point' (aka 'const gp_XY')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:537:9: error: incomplete result type 'NCollection_CellFilter_InspectorXY::Point' (aka 'gp_XY') in function definition
[W 201022 01:49:57 translation_unit:47] dummy.cxx:538:25: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXY::Point' (aka 'const gp_XY')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:538:46: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXY::Point' (aka 'const gp_XY')
[D 201022 01:49:58 module:44] ./opencascade/Standard_PrimitiveTypes.hxx
[D 201022 01:49:58 module:44] ./opencascade/math_MultipleVarFunction.hxx

D 201022 02:07:31 module:44] ./opencascade/Quantity_Temperature.hxx
[D 201022 02:07:31 module:44] ./opencascade/OSD_Exception_PRIV_INSTRUCTION.hxx
[W 201022 02:07:33 translation_unit:46] ./opencascade/Standard_WarningsRestore.hxx
[W 201022 02:07:33 translation_unit:47] dummy.cxx:33:28: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
[D 201022 02:07:34 module:44] ./opencascade/TColStd_Array1OfExtendedString.hxx
[D 201022 02:07:34 module:44] ./opencascade/TCollection.hxx
[D 201022 02:19:42 module:44] ./opencascade/gce_MakeDir.hxx
[D 201022 02:19:42 module:44] ./opencascade/IntAna_Quadric.hxx
[D 201022 02:19:43 module:44] ./opencascade/Extrema_FuncExtCS.hxx
[D 201022 02:19:44 module:44] ./opencascade/TopoDS_Shell.hxx

[W 201022 02:19:45 translation_unit:46] ./opencascade/IntAna_Quadric.hxx
[W 201022 02:19:45 translation_unit:47] dummy.cxx:41:40: error: unknown type name 'gp_Pln'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:44:40: error: unknown type name 'gp_Sphere'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:47:40: error: unknown type name 'gp_Cylinder'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:50:40: error: unknown type name 'gp_Cone'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:53:42: error: unknown type name 'gp_Pln'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:56:42: error: unknown type name 'gp_Sphere'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:59:42: error: unknown type name 'gp_Cone'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:62:42: error: unknown type name 'gp_Cylinder'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:75:255: error: unknown type name 'gp_Ax3'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:78:26: error: use of undeclared identifier 'gp_Pnt'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:104:20: error: use of undeclared identifier 'gp_Pnt'
[D 201022 02:19:45 module:44] ./opencascade/ProjLib_Projector.hxx

[D 201022 02:29:13 module:44] ./opencascade/IntSurf_Allocator.hxx
[D 201022 02:29:14 module:44] ./opencascade/GeomFill_Tensor.hxx
[W 201022 02:29:16 translation_unit:46] ./opencascade/BRepApprox_SurfaceTool.hxx
[W 201022 02:29:16 translation_unit:47] ./opencascade/BRepApprox_SurfaceTool.lxx:31:108: error: member access into incomplete type 'const BRepAdaptor_Surface'
[W 201022 02:29:16 translation_unit:47] ./opencascade/BRepApprox_SurfaceTool.lxx:32:108: error: member access into incomplete type 'const BRepAdaptor_Surface'

[D 201022 02:31:16 module:44] ./opencascade/IntPatch_PolyhedronTool.hxx
[D 201022 02:31:17 module:44] ./opencascade/BRepBuilderAPI_PipeError.hxx
[W 201022 02:31:19 translation_unit:46] ./opencascade/IntPatch_PolyhedronTool.hxx
[W 201022 02:31:19 translation_unit:47] ./opencascade/IntPatch_PolyhedronTool.lxx:19:18: error: member access into incomplete type 'const IntPatch_Polyhedron'
[W 201022 02:31:19 translation_unit:47] ./opencascade/IntPatch_PolyhedronTool.lxx:24:18: error: member access into incomplete type 'const IntPatch_Polyhedron'

[D 201022 02:34:32 module:41] Processing headers
[D 201022 02:34:32 module:44] ./opencascade/ShapeFix_SequenceOfWireSegment.hxx
[D 201022 02:34:32 module:44] ./opencascade/ShapeCustom_Curve.hxx
[D 201022 02:34:33 module:44] ./opencascade/ShapeConstruct_ProjectCurveOnSurface.hxx
[D 201022 02:34:33 module:44] ./opencascade/MAT2d_Array2OfConnexion.hxx
[D 201022 02:34:33 module:44] ./opencascade/ShapeExtend_Explorer.hxx
[W 201022 02:34:36 translation_unit:46] ./opencascade/ShapeConstruct_ProjectCurveOnSurface.hxx
[W 201022 02:34:36 translation_unit:47] dummy.cxx:210:66: error: use of undeclared identifier 'TColStd_HArray1OfReal'
[W 201022 02:34:36 translation_unit:47] dummy.cxx:213:138: error: use of undeclared identifier 'TColStd_HArray1OfReal'
and more but ttgtb!

[Franck78]

@Grawp , are you working on something about that (avoiding conda) ?

ok, .OCP full of new c++ and the CMakelists.txt

Next step: get something out of Cmake/make

[WizardUli]

@Franck78 Yes, I've started working on that although just very recently so don't expect results tomorrow :D My goals are to:

• Write a proper build script using Make or CMake (since the CMake is already being used for a part of the build).
• Remove all hard-coded paths like https://github.com/CadQuery/pywrap/blob/45cf3240ededaead2e7ea60dcecc9c5cb157e0c6/bindgen/utils.py#L58 (in this case Config.set_library_file(find_library('clang')) seems to be sufficient on Linux systems).
• Write an ebuild for Gentoo and a packaging script for Fedora and test that the newly created build-system integrates well and works on both x64 and amrhf architectures.
• Integrate it with azure-pipelines so it can build conda ocp package like it is doing now so I can offer my changes back to CadQuery community should they choose to consider such generalization of the build system for OCP.

[Franck78]

good.

for now I run basic, run.sh is from pywrap, notice I force 3 time 'python3':

rm -rf ./OCP
rm out.pkl out_f.pkl
mkdir ./OCP
CONDA_PREFIX=/usr python3 -m bindgen -n 7 -v parse ocp.toml out.pkl && python3 -m bindgen -n 7 -v transform ocp.toml out.pkl out_f.pkl && python3 -m bindgen -n 7 -v generate ocp.toml out_f.pkl
cd ./OCP
CONDA_PREFIX=/usr cmake -B build -S . -G Ninja
CONDA_PREFIX=/usr cmake --build build -j2  -- -k 0

with

• copy the 'bindgen' directory from pywrap (why isn't it already here....)
• patch bindgen/utils.py to suit opensuse Leap42:
• add ninja to the list of required packages

@@ -42,7 +42,7 @@
     else:
         rv.append(Path(prefix) / 'lib/clang/8.0.0/include/')
         rv.append(Path(prefix) / 'lib/clang/6.0.1/include/')
-        rv.append(Path(prefix) / 'lib/clang/9.0.1/include/')
+        rv.append(Path(prefix) / 'lib64/clang/9.0.1/include/')
         rv.append(Path(prefix) / 'lib/clang/10.0.1/include/')
         rv.append(Path(prefix) / 'include/c++/v1/')

@@ -58,7 +58,7 @@
         if platform.startswith('win'):
             Config.set_library_file(conda_prefix / 'Library' / 'bin' / 'libclang.dll')
         elif platform.startswith('linux') or platform.startswith('freebsd'):
-            Config.set_library_file(conda_prefix / 'lib' / 'libclang.so')
+            Config.set_library_file(conda_prefix / 'lib64' / 'libclang.so')
         elif platform.startswith('darwin'):
             Config.set_library_file(conda_prefix / 'lib' / 'libclang.dylib')

CMakeLists.j2: force python3. Why is my opensuse still installing python2 and making it 'default'......

if(WIN32)
  set( PYTHON_EXECUTABLE "$ENV{CONDA_PREFIX}/python.exe" )
  add_compile_definitions( _hypot=hypot _AMD64_ _WINDOWS)
  execute_process(COMMAND ${PYTHON_EXECUTABLE} -c "from sys import version_info as vi; print(f'{vi.major}{vi.minor}')"
                  OUTPUT_VARIABLE PYTHON_VER )
  string(STRIP ${PYTHON_VER} PYTHON_VER)
  set( PYTHON_LIBRARY $ENV{CONDA_PREFIX}/libs/python${PYTHON_VER}.lib )
else()
  set( PYTHON_EXECUTABLE "$ENV{CONDA_PREFIX}/bin/python3" )   <===== python3
endif()

FindOpenCascade.cmake: headers are in /usr/local/include/opencascade after a local build of OCCT 7.4

include( FindPackageHandleStandardArgs )

find_path( OPENCASCADE_INCLUDE_DIR Standard.hxx PATHS
           $ENV{CONDA_PREFIX}/local/include/opencascade     <======== +local
           $ENV{CONDA_PREFIX}/Library/include/opencascade )

as you saw in previous report, some warnings during the python3 -m bindgen -n 7 -v transform (I think) Now waiting 4 hours on my slow laptop to see what I get.

[Franck78]

probably consequence of warnings in phase 2 (bindgen transform)

[10/618] Building CXX object CMakeFiles/OCP.dir/AIS.cpp.o
../AIS.cpp: In function ‘void register_AIS(pybind11::module&)’:
../AIS.cpp:1150:501: warning: ‘void AIS_InteractiveContext::Display(const opencascade::handle<AIS_InteractiveObject>&, Standard_Integer, Standard_Integer, Standard_Boolean, Standard_Boolean, AIS_DisplayStatus)’ is deprecated: Deprecated method Display() with obsolete argument theToAllowDecomposition [-Wdeprecated-declarations]
              (void (AIS_InteractiveContext::*)( const opencascade::handle<AIS_InteractiveObject> & ,  const Standard_Integer ,  const Standard_Integer ,  const Standard_Boolean ,  const Standard_Boolean ,  const AIS_DisplayStatus  ) ) static_cast<void (AIS_InteractiveContext::*)( const opencascade::handle<AIS_InteractiveObject> & ,  const Standard_Integer ,  const Standard_Integer ,  const Standard_Boolean ,  const Standard_Boolean ,  const AIS_DisplayStatus  ) >(&AIS_InteractiveContext::Display),
 + a lot more                                                                                                                                                                                              

[27/618] Building CXX object CMakeFiles/OCP.dir/BOPAlgo.cpp.o
FAILED: CMakeFiles/OCP.dir/BOPAlgo.cpp.o 
/usr/bin/c++  -DOCP_EXPORTS -I../ -I/usr/local/include/opencascade -isystem /usr/include/python3.6m -fPIC   -fpermissive -fvisibility=hidden -fvisibility-inlines-hidden -std=c++14 -std=gnu++1z -MD -MT CMakeFiles/OCP.dir/BOPAlgo.cpp.o -MF CMakeFiles/OCP.dir/BOPAlgo.cpp.o.d -o CMakeFiles/OCP.dir/BOPAlgo.cpp.o -c ../BOPAlgo.cpp
../BOPAlgo.cpp: In function ‘void register_BOPAlgo(pybind11::module&)’:
../BOPAlgo.cpp:1568:314: error: ‘TreatCompound’ is not a member of ‘BOPAlgo_Tools’
                     (void (*)( const TopoDS_Shape & ,  NCollection_Map<TopoDS_Shape, TopTools_ShapeMapHasher> & ,  NCollection_List<TopoDS_Shape> &  ) ) static_cast<void (*)( const TopoDS_Shape & ,  NCollection_Map<TopoDS_Shape, TopTools_ShapeMapHasher> & ,  NCollection_List<TopoDS_Shape> &  ) >(&BOPAlgo_Tools::TreatCompound),

[adam-urbanczyk]

@Grawp improvements are welcome. If you want to contribute I'd have some suggestions:

1) Please use cmake, we are targeting 3 platforms here. 2) Indeed pywrap has some relative paths hardcoded based on the conda env layout, I think the best way would be to expose those to the CLI and use the current values as defaults. I don't know how well find_package works in different situations (Windows) 3) I'm quite happy with the current azure setup so I'd be only interested in adding additional jobs.

[adam-urbanczyk]

@Franck78 you can compare the results with the azure CI run (there are warnings being generated there as well). Intermediate results are available as pipeline artifacts.

[Franck78]

"with the azure CI run" sometime I understand, sometimes it is pure chineese. ;)

Anyway, final result of " cmake --build build -j1 -- -k 0" is clear

[327/368] Building CXX object CMakeFiles/OCP.dir/XCAFDoc.cpp.o
[367/368] Building CXX object CMakeFiles/OCP.dir/math_pre.cpp.o
ninja: build stopped: cannot make progress due to previous errors.

If you have that(OCP) produced under the conda env, then, some kind of include file is bad between gcc clang python3 on openSuse.

[adam-urbanczyk]

I meant, here you have the intermediate results: https://dev.azure.com/cadquery/OCP/_build/results?buildId=820&view=artifacts&type=publishedArtifacts

[Franck78]

python -m bindgen -n 7 -v generate ocp.toml out_f.pkl

ok, i have ~ the same errors/warnings at the beginning. That doesn't bother you ?

Outputs are not the same between my thing and azure. There is no 'bindgen' module in the OCT git repository Maybe I'm wrong since the beginning and Pywrap repo is good and OCT repo is 'well, something' ?

[adam-urbanczyk]

ok, i have ~ the same errors/warnings at the beginning. That doesn't bother you ?

Feel free to improve things here.

Outputs are not the same between my thing and azure. There is no 'bindgen' module in the OCT git repository

Pywrap is a submodule of this repo, not sure what you mean here.

[Franck78]

Feel free to improve things here.

Oh, I'm sure you tried before me to fix them. Not a fan a c c++ and the includes headers and the syntax .

It make no sense to me. Why would the thing process hundred of headers and suddenly decree "I know nothing about TColStd_HArray1OfReal"

[W 201022 22:14:42 translation_unit:47] dummy.cxx:210:66: error: use of undeclared identifier 'TColStd_HArray1OfReal'

grep (on the OCCT headers) returns tons of use of this name. Why one error.

[greyltc]

I absolutely hate conda package manager.

I hate it more! ;-) Too bad it seems we can't convince @adam-urbanczyk to hate it with us! @Grawp Like you, I loath conda. I've spent a lot of time and effort to understand how this is built and how I can strip the conda garbage out of it so that I can package it properly for the Arch Linux family. I doubt I've done this all the best way possible, but at least I have it all working right now. I'm sad to say I feel that that this a pretty fragile package today because of all the manipulation I've had to do to get it packaged correctly.

Feel free to use my PKGBUILD as a reference for your noble unconda packaging efforts! https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-ocp

[WizardUli]

Feel free to use my PKGBUILD as a reference for your noble unconda packaging efforts! https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-ocp

@greyltc I've already used your PKGBUILD as a reference! It helped me understand how the OCP should be build. Sorry it is taking too long. I haven't had much time and I need to learn CMake first.

Btw. @adam-urbanczyk Are you aware that FindOpenCascade.cmake in OCP and pywrap repositories are conceptually redundant or even wrong since OpenCASCADE installations on their own bring their own cmake files. Moreover CMake which is already being used as a last step in the build process could have (probably) been used to get rid of all the hardcoded paths in bindgen/pywrap/ocp.. on all OSes.

[greyltc]

Great. I'm glad you're aware of it. The fewer people reinventing wheels, the better!

[adam-urbanczyk]

@greyltc @Grawp I'm very much against using language like "hate","garbage","perverted" with respect to other people’s hard work. If you don't like conda, don't use it. If there are ways to improve OCP or pywrap please open a PR (see my comment above for details).

@Grawp if the the cmake file is wrong, then don't use it. It does produce a working result for me. I'd rather expose the paths as an command line parameter to pywrap than make it call cmake.

[greyltc]

@adam-urbanczyk We're all working hard to give the community free and cool stuff and maybe none of us are even getting compensated in any way for our efforts! I have no doubt that we all want to do this work in the most efficient way. I apologize for offending you with my expression of dislike for anaconda. I didn't mean to direct that at you or this project. I used those strong words because I get frustrated with how difficult it can be for me every time I want to package a project that only considers the anaconda distribution channel. I don't necessarily blame the project owners for that, I blame Continuum Analytics/Anaconda Inc/whatever is behind that thing. I spent two of my days on getting this to build only so that the Arch community (myself included of course) can use it without being subjected to the conda package manager.

Probably a ton of stuff can be copied/learned from the pythonocc-core build system. As I understand it, from a 50,000ft view, that project does the exact same thing this project does: it makes python bindings for OCCT. I know there are some major differences going on under the hood with pybind and maybe it's naive to think that this project could have a similar build interface. Building and packaging that project in linux is very familiar and thus efficient:

mkdir -p build
cd build
cmake \
  -DCMAKE_INSTALL_PREFIX=/usr/local \
  ..
make
python run_tests.py
make DESTDIR="${pkgdir}" install

cmake there does magic to figure everything out. For a properly set up system, that works fine. However, locations for some important hard to find things can be overridden if needed by feeding parameters to cmake like this: -DOCE_INCLUDE_PATH=/opt/oce/include/oce It would be really neat if we could work all together to get OCP to that state! I can imagine packaging like that can work in the conda cloud and on all supported platforms natively without the need for distributing any precompiled binary blobs and then everyone is happy and we could even get neat stuff like ARM architecture support for free.

I can imagine that there is a bunch of duplicated effort between here and pythonocc-core. I think historically they've been generating bindings to their intermediate fork of OCCT, so they've been really slow on keeping up with releases/features since they had so many moving parts, but it seems that they've now cut that all out and are binding to the same code base you're binding to here so they might now be a bit more agile than they were in the past. I wonder if it's worth revisiting a collaboration there. Of course I understand that each binding is meant to be in support of the owners' own projects, so maybe this sort of collaboration is not workable.

[WizardUli]

Everything what @greyltc said. And I'd like to apologize for my wording. I really went over what would be considered constructive criticism.

@Grawp if the the cmake file is wrong, then don't use it. It does produce a working result for me. I'd rather expose the paths as an command line parameter to pywrap than make it call cmake.

Maybe I've misunderstood so please forgive me for possibly unnecessarily stating that: I do not mean for pywrap/bindgen to call cmake. That would make things even worse. Exactly the opposite. CMake should either:

• Call the pywrap with correct paths as CLI arguments exactly like you suggested.
• Or prepare those paths into a configuration file (e.g. extend the ocp.toml with additional items).

Azure pipelines would be of course able to call pywrap directly without the cmake as well as call the cmake with parameters specified using -Dxyz and possibly even call just cmake without -Dxyzs. And this is not an either-or solution. Should you just by a chance decide to call the cmake in the azure pipelines the possibility to use it without cmake wouldn't go away.

[adam-urbanczyk]

@adam-urbanczyk We're all working hard to give the community free and cool stuff and maybe none of us are even getting compensated in any way for our efforts! I have no doubt that we all want to do this work in the most efficient way. I apologize for offending you with my expression of dislike for anaconda. I didn't mean to direct that at you or this project. I used those strong words because I get frustrated with how difficult it can be for me every time I want to package a project that only considers the anaconda distribution channel. I don't necessarily blame the project owners for that, I blame Continuum Analytics/Anaconda Inc/whatever is behind that thing. I spent two of my days on getting this to build only so that the Arch community (myself included of course) can use it without being subjected to the conda package manager.

@greyltc just to be clear, I don't think conda is to be blamed for anything. I'd never be able to set up all this without it (and conda-forge) given the time I have.

Probably a ton of stuff can be copied/learned from the pythonocc-core build system. As I understand it, from a 50,000ft view, that project does the exact same thing this project does: it makes python bindings for OCCT. I know there are some major differences going on under the hood with pybind and maybe it's naive to think that this project could have a similar build interface. Building and packaging that project in linux is very familiar and thus efficient:

I'm very well aware of the existence of PythonOCC. The problem is not inspiration or knowledge (i.e. how), it's rather the resources (i.e. when). If you want to contribute said universal CMakeLists.txt for this project I'll merge it, but don't expect me to work on it anytime soon.

@Grawp we are on the same page. I'll try to get rid of the hardcoded paths in pywrap via a command line option.

[greyltc]

I'd never be able to set up all this without it (and conda-forge) given the time I have.

You might be suprised by how much improvement the CI tools on GitHub and GitLab have seen recently! They're pretty powerful and easy to use these days.

If you want to contribute said universal CMakeLists.txt for this project I'll merge it

Knowing you'd be willing to merge something like that is great.

[adam-urbanczyk]

You can now specify libclang location and headers as command line parameters. Additionally pywrap is pip-installable and provides a pywrap command:

pywrap --help
Usage: pywrap [OPTIONS] COMMAND [ARGS]...

Options:
  -n, --njobs INTEGER
  -v, --verbose
  -c, --clean
  -i, --include PATH       additional inlcude paths
  -l, --libclang FILENAME  libclang location
  --help                   Show this message and exit.

Commands:
  all
  generate
  parse
  transform
  validate

[greyltc]

Nice! I'll try this out on my build shortly

[adam-urbanczyk]

FYI: I found some issues with the code, but all should be fixed in #37 .

[greyltc]

Thanks @adam-urbanczyk, is there a CI build script or something you can point me to that uses the new system that I can use as a reference for updating my packaging?

[adam-urbanczyk]

This should do on the latest master (of pywrap):

pywrap all ocp.toml
cmake -S OCP -B build
cmake --build build

Of course depending on the env you might need to use the additional -i and -l options for pywrap (as mentioned above).

[greyltc]

I've given this a shot on the latest master branch here (using the pywrap submodule as configured). I think I'm pretty close to getting the build to work, but I don't quite have it working. Part of the problem I'm having is that I'm not 100% confident that I'm choosing the correct headers and ocp.toml to use. Today I seem to have the following choices for headers:

(1) /usr/include/opencascade (2) https://github.com/CadQuery/pywrap/tree/367eb8de984344d3a7cde85b3bef6c5d2c1092c9/opencascade (3) https://github.com/CadQuery/OCP/tree/6b105a338094771cb5d11106a46a1404f42b3767/opencascade

Those in (1) are the official headers on my system as shipped with the occt 7.5.0 release tarball. None of these three choices for headers agree with ether of the two others.

Furthermore, I have the ocp.toml provided in the pywrap repo and that provided in this repo to choose between.

Please correct me if I'm wrong: If I do nothing, it seems the headers that the various build steps use get chosen for me (I can't say I've really looked into exactly how they're chosen). I can override the headers chosen in the pywrap step using -i and I can override the headers in the cmake prepare step using -DOPENCASCADE_INCLUDE_DIR

I get differeing build errors based on what choices I make for the inputs to the various build steps. @adam-urbanczyk If you can advise me on which headers and ocp.toml to use, then I'll post a more meaningful build error that I hope you can help me overcome!

[adam-urbanczyk]

Please go for (3). For the generate step headers are specified in the ocp.toml file. For the rest you are spot on.

[greyltc]

Ok, here's what I get when I do it that way (abbreviated output):

$ git clone https://github.com/CadQuery/OCP.git
$ cd OCP
$ git submodule update --init --recursive
$ export CONDA_PREFIX=/usr
$ export PYTHONPATH=pywrap
$ python -m bindgen --clean --libclang /usr/lib/libclang.so --include opencascade all ocp.toml
# ...
# [W 210111 20:37:08 translation_unit:47] ./opencascade/AppDef_BSpParFunctionOfMyBSplGradientOfBSplineCompute.hxx
# [W 210111 20:37:08 translation_unit:48] /../lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cstddef:50:10: fatal error: 'stddef.h' file not found
# ...success (? after hundreds/thousands of "warnings" that say "fatal error" as above) and an OCP folder is created
$ cmake -D OPENCASCADE_INCLUDE_DIR=opencascade -D CMAKE_BUILD_TYPE=Release -B build_dir -G Ninja -S OCP
-- The CXX compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python (found version "3.9.1") 
-- Found PythonLibs: /usr/lib/libpython3.9.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /usr/include (found version "2.6.1" )
CMake Warning (dev) at /usr/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
  The package name passed to `find_package_handle_standard_args`
  (OPENCASCADE) does not match the name of the calling package (OpenCascade).
  This can lead to problems in calling code that expects `find_package`
  result variables (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  FindOpenCascade.cmake:78 (find_package_handle_standard_args)
  CMakeLists.txt:19 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found OPENCASCADE: /usr/lib/libTKMath.so;/usr/lib/libTKernel.so;/usr/lib/libTKG2d.so;/usr/lib/libTKG3d.so;/usr/lib/libTKGeomBase.so;/usr/lib/libTKBRep.so;/usr/lib/libTKGeomAlgo.so;/usr/lib/libTKTopAlgo.so;/usr/lib/libTKPrim.so;/usr/lib/libTKShHealing.so;/usr/lib/libTKHLR.so;/usr/lib/libTKBO.so;/usr/lib/libTKBool.so;/usr/lib/libTKFeat.so;/usr/lib/libTKOffset.so;/usr/lib/libTKFillet.so;/usr/lib/libTKMesh.so;/usr/lib/libTKXMesh.so;/usr/lib/libTKXSBase.so;/usr/lib/libTKService.so;/usr/lib/libTKV3d.so;/usr/lib/libTKOpenGl.so;/usr/lib/libTKMeshVS.so;/usr/lib/libTKBin.so;/usr/lib/libTKBinL.so;/usr/lib/libTKBinTObj.so;/usr/lib/libTKCAF.so;/usr/lib/libTKCDF.so;/usr/lib/libTKLCAF.so;/usr/lib/libTKStd.so;/usr/lib/libTKStdL.so;/usr/lib/libTKTObj.so;/usr/lib/libTKVCAF.so;/usr/lib/libTKXml.so;/usr/lib/libTKXmlL.so;/usr/lib/libTKXmlTObj.so;/usr/lib/libTKIGES.so;/usr/lib/libTKSTEP.so;/usr/lib/libTKSTEP209.so;/usr/lib/libTKSTEPAttr.so;/usr/lib/libTKSTEPBase.so;/usr/lib/libTKSTL.so;/usr/lib/libTKXDESTEP.so;/usr/lib/libTKXCAF.so;/usr/lib/libTKBinXCAF.so;/usr/lib/libTKXmlXCAF.so;/usr/lib/libTKVRML.so;/usr/lib/libTKRWMesh.so  
-- Configuring done
-- Generating done
-- Build files have been written to: OCP/build_dir
$ cmake --build build_dir -- -j1 > build.log.txt 2>&1

build.log.txt attached. build.log.txt

[adam-urbanczyk]

Ok, here's what I get when I do it that way (abbreviated output):

$ git clone https://github.com/CadQuery/OCP.git
$ cd OCP
$ git submodule update --init --recursive
$ export CONDA_PREFIX=/usr
$ export PYTHONPATH=pywrap
$ python -m bindgen --clean --libclang /usr/lib/libclang.so --include opencascade all ocp.toml
# ...
# [W 210111 20:37:08 translation_unit:47] ./opencascade/AppDef_BSpParFunctionOfMyBSplGradientOfBSplineCompute.hxx
# [W 210111 20:37:08 translation_unit:48] /../lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cstddef:50:10: fatal error: 'stddef.h' file not found
# ...success (? after hundreds/thousands of "warnings" that say "fatal error" as above) and an OCP folder is created

Looks like you need to specify the correct include location for clang, something like (NB: made up path):

 python -m bindgen --clean --libclang /usr/lib/libclang.so --include opencascade --include /usr/lib/clang/10.0.1/include/ all ocp.toml

If you use one -i/--include different path in the code is followed so you need to specify all includes. Specifying explicitly --include opencascade is not needed AFAIR.

BTW: @greyltc here https://github.com/CadQuery/OCP/issues/33 there is more discussion on building on BSD.

[greyltc]

Thanks! That works! Here's what I have for the meat of my build commands now:

  CONDA_PREFIX=/usr PYTHONPATH=pywrap python -m bindgen \
    --clean \
    --libclang "$(ldconfig -p | grep 'libclang.so$' | awk '{print $NF}')" \
    --include "$(clang -print-resource-dir)"/include \
    all ocp.toml

  cmake \
    -W no-dev \
    -D OPENCASCADE_INCLUDE_DIR=opencascade \
    -D CMAKE_BUILD_TYPE=None \
    -B build_dir \
    -G Ninja \
    -S OCP

  ninja -C build_dir

Should hopefully be pretty portable.

[adam-urbanczyk]

I think you can even strip few more things now:

PYTHONPATH=pywrap python -m bindgen \
  --clean \
  --libclang "$(ldconfig -p | grep 'libclang.so$' | awk '{print $NF}')" \
  --include "$(clang -print-resource-dir)"/include \
  all ocp.toml

cmake \
  -W no-dev \
  -D OPENCASCADE_INCLUDE_DIR=opencascade \
  -D CMAKE_BUILD_TYPE=None \
  -B build_dir \
  -S OCP

cmake --build build_dir

[greyltc]

Ah, neat! I didn't realize I could get rid of CONDA_PREFIX. Thanks

[adam-urbanczyk]

So is the current solution standard enough, i.e. is the issue resolved?

BTW: I'm still thinking about adding the possibility to override the CMakeLists template

[greyltc]

I think the new cmake stuff in this repo is great and this issue can be closed here in my opinion. Thank you! It would be a good improvement if cmake --build build_dir -- install could put the OCP .so into the python site packages directory.

Do you think pywrap deserves the same or similar cmake treatment? I discovered https://github.com/CadQuery/pywrap/issues/31 in my recent builds and thought that I noticed a few hardcoded paths and library and platform discovery logic in pywrap/bindgen/utils.py that might be better done by a tool like cmake. What do you think? Would it be appropriate for the pywrap call to be rolled into OCP's cmake invocation? Should pywrap have its own?

[adam-urbanczyk]

It could be done using https://cmake.org/cmake/help/latest/command/execute_process.html but for now I don't want to spend time on writing something that likely will not be generic.

I was wrong regarding the custom CMakeLists template - it is already supported. Regarding https://github.com/CadQuery/pywrap/issues/31 I think you are wrong - as long as you provide libcalng path explicitly via --libclang, CONDA_PREFIX is not used.

[greyltc]

Regarding CadQuery/pywrap#31 I think you are wrong...

I'm seeing the following on 6b105a3 (which is configured to pull in https://github.com/CadQuery/pywrap/commit/367eb8de984344d3a7cde85b3bef6c5d2c1092c9 as a submodule)

$ PYTHONPATH=pywrap python -m bindgen --clean --libclang /usr/lib/libclang.so --include /usr/lib/clang/11.0.1/include all ocp.toml
100%|██████████| 309/309 [00:00<00:00, 311.52it/s]                                                                     
joblib.externals.loky.process_executor._RemoteTraceback: 
"""                                                                                                                    
Traceback (most recent call last):                                                                                     
  File "/usr/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line 431, in _process_worker
    r = call_item()                                      
  File "/usr/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line 285, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)                 
  File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 262, in __call__                    
    return [func(*args, **kwargs)        
  File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 262, in <listcomp>
    return [func(*args, **kwargs)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__init__.py", line 241, in _process_module
    return ModuleInfo(name,path,files,module_names,settings)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/module.py", line 45, in __init__
    self.headers.append(process_header(p,prefix,settings,name))
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/header.py", line 715, in process_header
    hi.parse(path,input_folder,settings,module_name)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/header.py", line 662, in parse
    tr_unit = parse_tu(path,
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/translation_unit.py", line 34, in parse_tu
    ix = get_index()
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/utils.py", line 99, in get_index
    if not initialized: init_clang()
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/utils.py", line 56, in init_clang
    conda_prefix = Path(getenv('CONDA_PREFIX'))
  File "/usr/lib/python3.9/site-packages/path/__init__.py", line 164, in __init__
    raise TypeError("Invalid initial value for path: None") 
TypeError: Invalid initial value for path: None
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code 
    exec(code, run_globals)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__main__.py", line 107, in <module>
    main()
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx)) 
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__main__.py", line 101, in all
    ctx.invoke(parse, configuration=configuration, output=tmp_parsed, platform=platform)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__main__.py", line 40, in parse
    result = parse_modules(obj.verbose,obj.njobs,settings,module_mapping,module_settings,platform)
  File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__init__.py", line 243, in parse_modules
    modules = Parallel(prefer='processes',n_jobs=n_jobs)\
  File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 1054, in __call__
    self.retrieve()
  File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 933, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
TypeError: Invalid initial value for path: None

v.s.

$ CONDA_PREFIX=/usr PYTHONPATH=pywrap python -m bindgen --clean --libclang /usr/lib/libclang.so --include /usr/lib/clang/11.0.1/include all ocp.toml
100%|██████████| 309/309 [00:47<00:00,  6.57it/s]
[W 210116 16:19:26 translation_unit:47] ./opencascade/Standard_WarningsRestore.hxx
[W 210116 16:19:26 translation_unit:48] dummy.cxx:38:28: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
[W 210116 16:19:27 translation_unit:47] ./opencascade/Graphic3d_WorldViewProjState.hxx
[W 210116 16:19:27 translation_unit:48] dummy.cxx:154:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_NUMERICAL'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:154:60: error: expected ';' after expression
[W 210116 16:19:27 translation_unit:48] dummy.cxx:155:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_POINTER'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:155:57: error: expected ';' after expression
[W 210116 16:19:27 translation_unit:48] dummy.cxx:156:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_NUMERICAL'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:156:68: error: expected ';' after expression
[W 210116 16:19:27 translation_unit:48] dummy.cxx:157:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_NUMERICAL'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:157:67: error: expected ';' after expression
100%|██████████| 309/309 [00:17<00:00, 17.99it/s] 
[I 210116 16:22:41 __init__:275] sorting
[I 210116 16:22:41 __init__:293] removing duplicate typedefs
Processing module Standard                        
Processing module gp           
Processing module math          
Processing module BVH       
Processing module NCollection        
Processing module TColgp       
Processing module OSD           
Processing module Quantity        

...

Processing module XSControl 
Processing module IFSelect           
Processing module IFGraph      
Processing module STEPCAFControl
Processing module XCAFApp         
100%|██████████| 309/309 [00:02<00:00, 135.29it/s]

Processing module XCAFDoc   
Processing module XCAFNoteObjects
Processing module XCAFPrs    
Processing module XCAFView   
Processing module BinXCAFDrivers
Processing module XmlXCAFDrivers
Processing module Vrml     
Processing module VrmlAPI      
Processing module RWGltf      
Processing module Xw    

I assume you can recreate the issue by calling unset CONDA_PREFIX before you wrap.

[adam-urbanczyk]

Thanks, you are right! I solved it in https://github.com/CadQuery/pywrap/commit/420c92f56d2d8bb416744a11fd078cddf6cec8be , will pull this into OCP later.

[adam-urbanczyk]

Solved