Closed WizardUli closed 3 years ago
I, on the contrary, like conda a lot and it saved me months of work by now. My advice is to use pure local install and do not do any magic conda init
stuff. That way your environment will be untouched.
As stated in the README, if you want to build locally you'll need to understand what is in the azure-piplines.yml
. The code of OCP is generated by pywrap (which BTW generates a cmake file too). Assuming you have all the dependencies it boils down to running something along these lines:
python -m bindgen -n $n_cores parse ocp.toml out.pkl
python -m bindgen -n $n_cores transform ocp.toml out.pkl out_f.pkl
python -m bindgen -n $n_cores generate ocp.toml out_f.pkl
cmake -B build -S $OCP_src -G Ninja
cmake --build build -j $n_cores -- -k 0
Development in this repo boils down to modifying ocp.toml
and rerunning parts of the script above.
Hello,
I'm running without conda. Not sure i will get something out one day....
Anyway, it looks like some OCCT headers are not 'compiled' well and it's disturbing. You flagged them Warnings so I think it should produce something in the end.
[D 201022 01:49:55 module:44] ./opencascade/BVH_DistanceField.hxx
[D 201022 01:49:55 module:44] ./opencascade/NCollection_CellFilter.hxx
[D 201022 01:49:55 module:44] ./opencascade/Standard_LicenseError.hxx
[D 201022 01:49:57 module:44] ./opencascade/gp_Parab.hxx
[W 201022 01:49:57 translation_unit:46] ./opencascade/NCollection_CellFilter.hxx
[W 201022 01:49:57 translation_unit:47] dummy.cxx:510:74: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:514:9: error: incomplete result type 'NCollection_CellFilter_InspectorXYZ::Point' (aka 'gp_XYZ') in function definition
[W 201022 01:49:57 translation_unit:47] dummy.cxx:515:25: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:515:46: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:515:67: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXYZ::Point' (aka 'const gp_XYZ')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:533:74: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXY::Point' (aka 'const gp_XY')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:537:9: error: incomplete result type 'NCollection_CellFilter_InspectorXY::Point' (aka 'gp_XY') in function definition
[W 201022 01:49:57 translation_unit:47] dummy.cxx:538:25: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXY::Point' (aka 'const gp_XY')
[W 201022 01:49:57 translation_unit:47] dummy.cxx:538:46: error: member access into incomplete type 'const NCollection_CellFilter_InspectorXY::Point' (aka 'const gp_XY')
[D 201022 01:49:58 module:44] ./opencascade/Standard_PrimitiveTypes.hxx
[D 201022 01:49:58 module:44] ./opencascade/math_MultipleVarFunction.hxx
D 201022 02:07:31 module:44] ./opencascade/Quantity_Temperature.hxx
[D 201022 02:07:31 module:44] ./opencascade/OSD_Exception_PRIV_INSTRUCTION.hxx
[W 201022 02:07:33 translation_unit:46] ./opencascade/Standard_WarningsRestore.hxx
[W 201022 02:07:33 translation_unit:47] dummy.cxx:33:28: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
[D 201022 02:07:34 module:44] ./opencascade/TColStd_Array1OfExtendedString.hxx
[D 201022 02:07:34 module:44] ./opencascade/TCollection.hxx
[D 201022 02:19:42 module:44] ./opencascade/gce_MakeDir.hxx
[D 201022 02:19:42 module:44] ./opencascade/IntAna_Quadric.hxx
[D 201022 02:19:43 module:44] ./opencascade/Extrema_FuncExtCS.hxx
[D 201022 02:19:44 module:44] ./opencascade/TopoDS_Shell.hxx
[W 201022 02:19:45 translation_unit:46] ./opencascade/IntAna_Quadric.hxx
[W 201022 02:19:45 translation_unit:47] dummy.cxx:41:40: error: unknown type name 'gp_Pln'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:44:40: error: unknown type name 'gp_Sphere'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:47:40: error: unknown type name 'gp_Cylinder'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:50:40: error: unknown type name 'gp_Cone'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:53:42: error: unknown type name 'gp_Pln'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:56:42: error: unknown type name 'gp_Sphere'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:59:42: error: unknown type name 'gp_Cone'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:62:42: error: unknown type name 'gp_Cylinder'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:75:255: error: unknown type name 'gp_Ax3'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:78:26: error: use of undeclared identifier 'gp_Pnt'
[W 201022 02:19:45 translation_unit:47] dummy.cxx:104:20: error: use of undeclared identifier 'gp_Pnt'
[D 201022 02:19:45 module:44] ./opencascade/ProjLib_Projector.hxx
[D 201022 02:29:13 module:44] ./opencascade/IntSurf_Allocator.hxx
[D 201022 02:29:14 module:44] ./opencascade/GeomFill_Tensor.hxx
[W 201022 02:29:16 translation_unit:46] ./opencascade/BRepApprox_SurfaceTool.hxx
[W 201022 02:29:16 translation_unit:47] ./opencascade/BRepApprox_SurfaceTool.lxx:31:108: error: member access into incomplete type 'const BRepAdaptor_Surface'
[W 201022 02:29:16 translation_unit:47] ./opencascade/BRepApprox_SurfaceTool.lxx:32:108: error: member access into incomplete type 'const BRepAdaptor_Surface'
[D 201022 02:31:16 module:44] ./opencascade/IntPatch_PolyhedronTool.hxx
[D 201022 02:31:17 module:44] ./opencascade/BRepBuilderAPI_PipeError.hxx
[W 201022 02:31:19 translation_unit:46] ./opencascade/IntPatch_PolyhedronTool.hxx
[W 201022 02:31:19 translation_unit:47] ./opencascade/IntPatch_PolyhedronTool.lxx:19:18: error: member access into incomplete type 'const IntPatch_Polyhedron'
[W 201022 02:31:19 translation_unit:47] ./opencascade/IntPatch_PolyhedronTool.lxx:24:18: error: member access into incomplete type 'const IntPatch_Polyhedron'
[D 201022 02:34:32 module:41] Processing headers
[D 201022 02:34:32 module:44] ./opencascade/ShapeFix_SequenceOfWireSegment.hxx
[D 201022 02:34:32 module:44] ./opencascade/ShapeCustom_Curve.hxx
[D 201022 02:34:33 module:44] ./opencascade/ShapeConstruct_ProjectCurveOnSurface.hxx
[D 201022 02:34:33 module:44] ./opencascade/MAT2d_Array2OfConnexion.hxx
[D 201022 02:34:33 module:44] ./opencascade/ShapeExtend_Explorer.hxx
[W 201022 02:34:36 translation_unit:46] ./opencascade/ShapeConstruct_ProjectCurveOnSurface.hxx
[W 201022 02:34:36 translation_unit:47] dummy.cxx:210:66: error: use of undeclared identifier 'TColStd_HArray1OfReal'
[W 201022 02:34:36 translation_unit:47] dummy.cxx:213:138: error: use of undeclared identifier 'TColStd_HArray1OfReal'
and more but ttgtb!
@Grawp , are you working on something about that (avoiding conda) ?
ok, .OCP full of new c++ and the CMakelists.txt
Next step: get something out of Cmake/make
@Franck78 Yes, I've started working on that although just very recently so don't expect results tomorrow :D My goals are to:
Config.set_library_file(find_library('clang'))
seems to be sufficient on Linux systems).good.
for now I run basic, run.sh is from pywrap, notice I force 3 time 'python3':
rm -rf ./OCP
rm out.pkl out_f.pkl
mkdir ./OCP
CONDA_PREFIX=/usr python3 -m bindgen -n 7 -v parse ocp.toml out.pkl && python3 -m bindgen -n 7 -v transform ocp.toml out.pkl out_f.pkl && python3 -m bindgen -n 7 -v generate ocp.toml out_f.pkl
cd ./OCP
CONDA_PREFIX=/usr cmake -B build -S . -G Ninja
CONDA_PREFIX=/usr cmake --build build -j2 -- -k 0
with
@@ -42,7 +42,7 @@
else:
rv.append(Path(prefix) / 'lib/clang/8.0.0/include/')
rv.append(Path(prefix) / 'lib/clang/6.0.1/include/')
- rv.append(Path(prefix) / 'lib/clang/9.0.1/include/')
+ rv.append(Path(prefix) / 'lib64/clang/9.0.1/include/')
rv.append(Path(prefix) / 'lib/clang/10.0.1/include/')
rv.append(Path(prefix) / 'include/c++/v1/')
@@ -58,7 +58,7 @@
if platform.startswith('win'):
Config.set_library_file(conda_prefix / 'Library' / 'bin' / 'libclang.dll')
elif platform.startswith('linux') or platform.startswith('freebsd'):
- Config.set_library_file(conda_prefix / 'lib' / 'libclang.so')
+ Config.set_library_file(conda_prefix / 'lib64' / 'libclang.so')
elif platform.startswith('darwin'):
Config.set_library_file(conda_prefix / 'lib' / 'libclang.dylib')
CMakeLists.j2: force python3. Why is my opensuse still installing python2 and making it 'default'......
if(WIN32)
set( PYTHON_EXECUTABLE "$ENV{CONDA_PREFIX}/python.exe" )
add_compile_definitions( _hypot=hypot _AMD64_ _WINDOWS)
execute_process(COMMAND ${PYTHON_EXECUTABLE} -c "from sys import version_info as vi; print(f'{vi.major}{vi.minor}')"
OUTPUT_VARIABLE PYTHON_VER )
string(STRIP ${PYTHON_VER} PYTHON_VER)
set( PYTHON_LIBRARY $ENV{CONDA_PREFIX}/libs/python${PYTHON_VER}.lib )
else()
set( PYTHON_EXECUTABLE "$ENV{CONDA_PREFIX}/bin/python3" ) <===== python3
endif()
FindOpenCascade.cmake: headers are in /usr/local/include/opencascade after a local build of OCCT 7.4
include( FindPackageHandleStandardArgs )
find_path( OPENCASCADE_INCLUDE_DIR Standard.hxx PATHS
$ENV{CONDA_PREFIX}/local/include/opencascade <======== +local
$ENV{CONDA_PREFIX}/Library/include/opencascade )
as you saw in previous report, some warnings during the python3 -m bindgen -n 7 -v transform (I think) Now waiting 4 hours on my slow laptop to see what I get.
probably consequence of warnings in phase 2 (bindgen transform)
[10/618] Building CXX object CMakeFiles/OCP.dir/AIS.cpp.o
../AIS.cpp: In function ‘void register_AIS(pybind11::module&)’:
../AIS.cpp:1150:501: warning: ‘void AIS_InteractiveContext::Display(const opencascade::handle<AIS_InteractiveObject>&, Standard_Integer, Standard_Integer, Standard_Boolean, Standard_Boolean, AIS_DisplayStatus)’ is deprecated: Deprecated method Display() with obsolete argument theToAllowDecomposition [-Wdeprecated-declarations]
(void (AIS_InteractiveContext::*)( const opencascade::handle<AIS_InteractiveObject> & , const Standard_Integer , const Standard_Integer , const Standard_Boolean , const Standard_Boolean , const AIS_DisplayStatus ) ) static_cast<void (AIS_InteractiveContext::*)( const opencascade::handle<AIS_InteractiveObject> & , const Standard_Integer , const Standard_Integer , const Standard_Boolean , const Standard_Boolean , const AIS_DisplayStatus ) >(&AIS_InteractiveContext::Display),
+ a lot more
[27/618] Building CXX object CMakeFiles/OCP.dir/BOPAlgo.cpp.o
FAILED: CMakeFiles/OCP.dir/BOPAlgo.cpp.o
/usr/bin/c++ -DOCP_EXPORTS -I../ -I/usr/local/include/opencascade -isystem /usr/include/python3.6m -fPIC -fpermissive -fvisibility=hidden -fvisibility-inlines-hidden -std=c++14 -std=gnu++1z -MD -MT CMakeFiles/OCP.dir/BOPAlgo.cpp.o -MF CMakeFiles/OCP.dir/BOPAlgo.cpp.o.d -o CMakeFiles/OCP.dir/BOPAlgo.cpp.o -c ../BOPAlgo.cpp
../BOPAlgo.cpp: In function ‘void register_BOPAlgo(pybind11::module&)’:
../BOPAlgo.cpp:1568:314: error: ‘TreatCompound’ is not a member of ‘BOPAlgo_Tools’
(void (*)( const TopoDS_Shape & , NCollection_Map<TopoDS_Shape, TopTools_ShapeMapHasher> & , NCollection_List<TopoDS_Shape> & ) ) static_cast<void (*)( const TopoDS_Shape & , NCollection_Map<TopoDS_Shape, TopTools_ShapeMapHasher> & , NCollection_List<TopoDS_Shape> & ) >(&BOPAlgo_Tools::TreatCompound),
@Grawp improvements are welcome. If you want to contribute I'd have some suggestions:
1) Please use cmake, we are targeting 3 platforms here. 2) Indeed pywrap has some relative paths hardcoded based on the conda env layout, I think the best way would be to expose those to the CLI and use the current values as defaults. I don't know how well find_package works in different situations (Windows) 3) I'm quite happy with the current azure setup so I'd be only interested in adding additional jobs.
@Franck78 you can compare the results with the azure CI run (there are warnings being generated there as well). Intermediate results are available as pipeline artifacts.
"with the azure CI run" sometime I understand, sometimes it is pure chineese. ;)
Anyway, final result of " cmake --build build -j1 -- -k 0" is clear
[327/368] Building CXX object CMakeFiles/OCP.dir/XCAFDoc.cpp.o
[367/368] Building CXX object CMakeFiles/OCP.dir/math_pre.cpp.o
ninja: build stopped: cannot make progress due to previous errors.
If you have that(OCP) produced under the conda env, then, some kind of include file is bad between gcc clang python3 on openSuse.
I meant, here you have the intermediate results: https://dev.azure.com/cadquery/OCP/_build/results?buildId=820&view=artifacts&type=publishedArtifacts
python -m bindgen -n 7 -v generate ocp.toml out_f.pkl
ok, i have ~ the same errors/warnings at the beginning. That doesn't bother you ?
Outputs are not the same between my thing and azure. There is no 'bindgen' module in the OCT git repository Maybe I'm wrong since the beginning and Pywrap repo is good and OCT repo is 'well, something' ?
ok, i have ~ the same errors/warnings at the beginning. That doesn't bother you ?
Feel free to improve things here.
Outputs are not the same between my thing and azure. There is no 'bindgen' module in the OCT git repository
Pywrap is a submodule of this repo, not sure what you mean here.
Feel free to improve things here.
Oh, I'm sure you tried before me to fix them. Not a fan a c c++ and the includes headers and the syntax .
It make no sense to me. Why would the thing process hundred of headers and suddenly decree "I know nothing about TColStd_HArray1OfReal"
[W 201022 22:14:42 translation_unit:47] dummy.cxx:210:66: error: use of undeclared identifier 'TColStd_HArray1OfReal'
grep (on the OCCT headers) returns tons of use of this name. Why one error.
I absolutely hate conda package manager.
I hate it more! ;-) Too bad it seems we can't convince @adam-urbanczyk to hate it with us! @Grawp Like you, I loath conda. I've spent a lot of time and effort to understand how this is built and how I can strip the conda garbage out of it so that I can package it properly for the Arch Linux family. I doubt I've done this all the best way possible, but at least I have it all working right now. I'm sad to say I feel that that this a pretty fragile package today because of all the manipulation I've had to do to get it packaged correctly.
Feel free to use my PKGBUILD as a reference for your noble unconda packaging efforts! https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-ocp
Feel free to use my PKGBUILD as a reference for your noble unconda packaging efforts! https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-ocp
@greyltc I've already used your PKGBUILD as a reference! It helped me understand how the OCP should be build. Sorry it is taking too long. I haven't had much time and I need to learn CMake first.
Btw. @adam-urbanczyk Are you aware that FindOpenCascade.cmake
in OCP and pywrap repositories are conceptually redundant or even wrong since OpenCASCADE installations on their own bring their own cmake files.
Moreover CMake which is already being used as a last step in the build process could have (probably) been used to get rid of all the hardcoded paths in bindgen/pywrap/ocp.. on all OSes.
Great. I'm glad you're aware of it. The fewer people reinventing wheels, the better!
@greyltc @Grawp I'm very much against using language like "hate","garbage","perverted" with respect to other people’s hard work. If you don't like conda, don't use it. If there are ways to improve OCP or pywrap please open a PR (see my comment above for details).
@Grawp if the the cmake file is wrong, then don't use it. It does produce a working result for me. I'd rather expose the paths as an command line parameter to pywrap than make it call cmake.
@adam-urbanczyk We're all working hard to give the community free and cool stuff and maybe none of us are even getting compensated in any way for our efforts! I have no doubt that we all want to do this work in the most efficient way. I apologize for offending you with my expression of dislike for anaconda. I didn't mean to direct that at you or this project. I used those strong words because I get frustrated with how difficult it can be for me every time I want to package a project that only considers the anaconda distribution channel. I don't necessarily blame the project owners for that, I blame Continuum Analytics/Anaconda Inc/whatever is behind that thing. I spent two of my days on getting this to build only so that the Arch community (myself included of course) can use it without being subjected to the conda package manager.
Probably a ton of stuff can be copied/learned from the pythonocc-core build system. As I understand it, from a 50,000ft view, that project does the exact same thing this project does: it makes python bindings for OCCT. I know there are some major differences going on under the hood with pybind and maybe it's naive to think that this project could have a similar build interface. Building and packaging that project in linux is very familiar and thus efficient:
mkdir -p build
cd build
cmake \
-DCMAKE_INSTALL_PREFIX=/usr/local \
..
make
python run_tests.py
make DESTDIR="${pkgdir}" install
cmake there does magic to figure everything out. For a properly set up system, that works fine. However, locations for some important hard to find things can be overridden if needed by feeding parameters to cmake like this:
-DOCE_INCLUDE_PATH=/opt/oce/include/oce
It would be really neat if we could work all together to get OCP to that state! I can imagine packaging like that can work in the conda cloud and on all supported platforms natively without the need for distributing any precompiled binary blobs and then everyone is happy and we could even get neat stuff like ARM architecture support for free.
I can imagine that there is a bunch of duplicated effort between here and pythonocc-core. I think historically they've been generating bindings to their intermediate fork of OCCT, so they've been really slow on keeping up with releases/features since they had so many moving parts, but it seems that they've now cut that all out and are binding to the same code base you're binding to here so they might now be a bit more agile than they were in the past. I wonder if it's worth revisiting a collaboration there. Of course I understand that each binding is meant to be in support of the owners' own projects, so maybe this sort of collaboration is not workable.
Everything what @greyltc said. And I'd like to apologize for my wording. I really went over what would be considered constructive criticism.
@Grawp if the the cmake file is wrong, then don't use it. It does produce a working result for me. I'd rather expose the paths as an command line parameter to pywrap than make it call cmake.
Maybe I've misunderstood so please forgive me for possibly unnecessarily stating that: I do not mean for pywrap/bindgen to call cmake. That would make things even worse. Exactly the opposite. CMake should either:
ocp.toml
with additional items).Azure pipelines would be of course able to call pywrap directly without the cmake as well as call the cmake with parameters specified using -Dxyz
and possibly even call just cmake without -Dxyz
s.
And this is not an either-or solution. Should you just by a chance decide to call the cmake in the azure pipelines the possibility to use it without cmake wouldn't go away.
@adam-urbanczyk We're all working hard to give the community free and cool stuff and maybe none of us are even getting compensated in any way for our efforts! I have no doubt that we all want to do this work in the most efficient way. I apologize for offending you with my expression of dislike for anaconda. I didn't mean to direct that at you or this project. I used those strong words because I get frustrated with how difficult it can be for me every time I want to package a project that only considers the anaconda distribution channel. I don't necessarily blame the project owners for that, I blame Continuum Analytics/Anaconda Inc/whatever is behind that thing. I spent two of my days on getting this to build only so that the Arch community (myself included of course) can use it without being subjected to the conda package manager.
@greyltc just to be clear, I don't think conda is to be blamed for anything. I'd never be able to set up all this without it (and conda-forge) given the time I have.
Probably a ton of stuff can be copied/learned from the pythonocc-core build system. As I understand it, from a 50,000ft view, that project does the exact same thing this project does: it makes python bindings for OCCT. I know there are some major differences going on under the hood with pybind and maybe it's naive to think that this project could have a similar build interface. Building and packaging that project in linux is very familiar and thus efficient:
I'm very well aware of the existence of PythonOCC. The problem is not inspiration or knowledge (i.e. how), it's rather the resources (i.e. when). If you want to contribute said universal CMakeLists.txt for this project I'll merge it, but don't expect me to work on it anytime soon.
@Grawp we are on the same page. I'll try to get rid of the hardcoded paths in pywrap via a command line option.
I'd never be able to set up all this without it (and conda-forge) given the time I have.
You might be suprised by how much improvement the CI tools on GitHub and GitLab have seen recently! They're pretty powerful and easy to use these days.
If you want to contribute said universal CMakeLists.txt for this project I'll merge it
Knowing you'd be willing to merge something like that is great.
You can now specify libclang location and headers as command line parameters. Additionally pywrap is pip-installable and provides a pywrap
command:
pywrap --help
Usage: pywrap [OPTIONS] COMMAND [ARGS]...
Options:
-n, --njobs INTEGER
-v, --verbose
-c, --clean
-i, --include PATH additional inlcude paths
-l, --libclang FILENAME libclang location
--help Show this message and exit.
Commands:
all
generate
parse
transform
validate
Nice! I'll try this out on my build shortly
FYI: I found some issues with the code, but all should be fixed in #37 .
Thanks @adam-urbanczyk, is there a CI build script or something you can point me to that uses the new system that I can use as a reference for updating my packaging?
This should do on the latest master (of pywrap):
pywrap all ocp.toml
cmake -S OCP -B build
cmake --build build
Of course depending on the env you might need to use the additional -i
and -l
options for pywrap (as mentioned above).
I've given this a shot on the latest master branch here (using the pywrap submodule as configured). I think I'm pretty close to getting the build to work, but I don't quite have it working. Part of the problem I'm having is that I'm not 100% confident that I'm choosing the correct headers and ocp.toml to use. Today I seem to have the following choices for headers:
(1) /usr/include/opencascade (2) https://github.com/CadQuery/pywrap/tree/367eb8de984344d3a7cde85b3bef6c5d2c1092c9/opencascade (3) https://github.com/CadQuery/OCP/tree/6b105a338094771cb5d11106a46a1404f42b3767/opencascade
Those in (1) are the official headers on my system as shipped with the occt 7.5.0 release tarball. None of these three choices for headers agree with ether of the two others.
Furthermore, I have the ocp.toml provided in the pywrap repo and that provided in this repo to choose between.
Please correct me if I'm wrong: If I do nothing, it seems the headers that the various build steps use get chosen for me (I can't say I've really looked into exactly how they're chosen). I can override the headers chosen in the pywrap step using -i
and I can override the headers in the cmake prepare step using -DOPENCASCADE_INCLUDE_DIR
I get differeing build errors based on what choices I make for the inputs to the various build steps. @adam-urbanczyk If you can advise me on which headers and ocp.toml to use, then I'll post a more meaningful build error that I hope you can help me overcome!
Please go for (3). For the generate step headers are specified in the ocp.toml file. For the rest you are spot on.
Ok, here's what I get when I do it that way (abbreviated output):
$ git clone https://github.com/CadQuery/OCP.git
$ cd OCP
$ git submodule update --init --recursive
$ export CONDA_PREFIX=/usr
$ export PYTHONPATH=pywrap
$ python -m bindgen --clean --libclang /usr/lib/libclang.so --include opencascade all ocp.toml
# ...
# [W 210111 20:37:08 translation_unit:47] ./opencascade/AppDef_BSpParFunctionOfMyBSplGradientOfBSplineCompute.hxx
# [W 210111 20:37:08 translation_unit:48] /../lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cstddef:50:10: fatal error: 'stddef.h' file not found
# ...success (? after hundreds/thousands of "warnings" that say "fatal error" as above) and an OCP folder is created
$ cmake -D OPENCASCADE_INCLUDE_DIR=opencascade -D CMAKE_BUILD_TYPE=Release -B build_dir -G Ninja -S OCP
-- The CXX compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python (found version "3.9.1")
-- Found PythonLibs: /usr/lib/libpython3.9.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /usr/include (found version "2.6.1" )
CMake Warning (dev) at /usr/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
The package name passed to `find_package_handle_standard_args`
(OPENCASCADE) does not match the name of the calling package (OpenCascade).
This can lead to problems in calling code that expects `find_package`
result variables (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
FindOpenCascade.cmake:78 (find_package_handle_standard_args)
CMakeLists.txt:19 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found OPENCASCADE: /usr/lib/libTKMath.so;/usr/lib/libTKernel.so;/usr/lib/libTKG2d.so;/usr/lib/libTKG3d.so;/usr/lib/libTKGeomBase.so;/usr/lib/libTKBRep.so;/usr/lib/libTKGeomAlgo.so;/usr/lib/libTKTopAlgo.so;/usr/lib/libTKPrim.so;/usr/lib/libTKShHealing.so;/usr/lib/libTKHLR.so;/usr/lib/libTKBO.so;/usr/lib/libTKBool.so;/usr/lib/libTKFeat.so;/usr/lib/libTKOffset.so;/usr/lib/libTKFillet.so;/usr/lib/libTKMesh.so;/usr/lib/libTKXMesh.so;/usr/lib/libTKXSBase.so;/usr/lib/libTKService.so;/usr/lib/libTKV3d.so;/usr/lib/libTKOpenGl.so;/usr/lib/libTKMeshVS.so;/usr/lib/libTKBin.so;/usr/lib/libTKBinL.so;/usr/lib/libTKBinTObj.so;/usr/lib/libTKCAF.so;/usr/lib/libTKCDF.so;/usr/lib/libTKLCAF.so;/usr/lib/libTKStd.so;/usr/lib/libTKStdL.so;/usr/lib/libTKTObj.so;/usr/lib/libTKVCAF.so;/usr/lib/libTKXml.so;/usr/lib/libTKXmlL.so;/usr/lib/libTKXmlTObj.so;/usr/lib/libTKIGES.so;/usr/lib/libTKSTEP.so;/usr/lib/libTKSTEP209.so;/usr/lib/libTKSTEPAttr.so;/usr/lib/libTKSTEPBase.so;/usr/lib/libTKSTL.so;/usr/lib/libTKXDESTEP.so;/usr/lib/libTKXCAF.so;/usr/lib/libTKBinXCAF.so;/usr/lib/libTKXmlXCAF.so;/usr/lib/libTKVRML.so;/usr/lib/libTKRWMesh.so
-- Configuring done
-- Generating done
-- Build files have been written to: OCP/build_dir
$ cmake --build build_dir -- -j1 > build.log.txt 2>&1
build.log.txt attached. build.log.txt
Ok, here's what I get when I do it that way (abbreviated output):
$ git clone https://github.com/CadQuery/OCP.git $ cd OCP $ git submodule update --init --recursive $ export CONDA_PREFIX=/usr $ export PYTHONPATH=pywrap $ python -m bindgen --clean --libclang /usr/lib/libclang.so --include opencascade all ocp.toml # ... # [W 210111 20:37:08 translation_unit:47] ./opencascade/AppDef_BSpParFunctionOfMyBSplGradientOfBSplineCompute.hxx # [W 210111 20:37:08 translation_unit:48] /../lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cstddef:50:10: fatal error: 'stddef.h' file not found # ...success (? after hundreds/thousands of "warnings" that say "fatal error" as above) and an OCP folder is created
Looks like you need to specify the correct include location for clang, something like (NB: made up path):
python -m bindgen --clean --libclang /usr/lib/libclang.so --include opencascade --include /usr/lib/clang/10.0.1/include/ all ocp.toml
If you use one -i/--include
different path in the code is followed so you need to specify all includes. Specifying explicitly --include opencascade
is not needed AFAIR.
BTW: @greyltc here https://github.com/CadQuery/OCP/issues/33 there is more discussion on building on BSD.
Thanks! That works! Here's what I have for the meat of my build commands now:
CONDA_PREFIX=/usr PYTHONPATH=pywrap python -m bindgen \
--clean \
--libclang "$(ldconfig -p | grep 'libclang.so$' | awk '{print $NF}')" \
--include "$(clang -print-resource-dir)"/include \
all ocp.toml
cmake \
-W no-dev \
-D OPENCASCADE_INCLUDE_DIR=opencascade \
-D CMAKE_BUILD_TYPE=None \
-B build_dir \
-G Ninja \
-S OCP
ninja -C build_dir
Should hopefully be pretty portable.
I think you can even strip few more things now:
PYTHONPATH=pywrap python -m bindgen \
--clean \
--libclang "$(ldconfig -p | grep 'libclang.so$' | awk '{print $NF}')" \
--include "$(clang -print-resource-dir)"/include \
all ocp.toml
cmake \
-W no-dev \
-D OPENCASCADE_INCLUDE_DIR=opencascade \
-D CMAKE_BUILD_TYPE=None \
-B build_dir \
-S OCP
cmake --build build_dir
Ah, neat! I didn't realize I could get rid of CONDA_PREFIX. Thanks
So is the current solution standard enough, i.e. is the issue resolved?
BTW: I'm still thinking about adding the possibility to override the CMakeLists template
I think the new cmake stuff in this repo is great and this issue can be closed here in my opinion. Thank you! It would be a good improvement if cmake --build build_dir -- install
could put the OCP .so into the python site packages directory.
Do you think pywrap deserves the same or similar cmake treatment? I discovered https://github.com/CadQuery/pywrap/issues/31 in my recent builds and thought that I noticed a few hardcoded paths and library and platform discovery logic in pywrap/bindgen/utils.py that might be better done by a tool like cmake. What do you think? Would it be appropriate for the pywrap call to be rolled into OCP's cmake invocation? Should pywrap have its own?
It could be done using https://cmake.org/cmake/help/latest/command/execute_process.html but for now I don't want to spend time on writing something that likely will not be generic.
I was wrong regarding the custom CMakeLists template - it is already supported. Regarding https://github.com/CadQuery/pywrap/issues/31 I think you are wrong - as long as you provide libcalng path explicitly via --libclang
, CONDA_PREFIX
is not used.
Regarding CadQuery/pywrap#31 I think you are wrong...
I'm seeing the following on 6b105a3 (which is configured to pull in https://github.com/CadQuery/pywrap/commit/367eb8de984344d3a7cde85b3bef6c5d2c1092c9 as a submodule)
$ PYTHONPATH=pywrap python -m bindgen --clean --libclang /usr/lib/libclang.so --include /usr/lib/clang/11.0.1/include all ocp.toml
100%|██████████| 309/309 [00:00<00:00, 311.52it/s]
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line 431, in _process_worker
r = call_item()
File "/usr/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line 285, in __call__
return self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 595, in __call__
return self.func(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 262, in __call__
return [func(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 262, in <listcomp>
return [func(*args, **kwargs)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__init__.py", line 241, in _process_module
return ModuleInfo(name,path,files,module_names,settings)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/module.py", line 45, in __init__
self.headers.append(process_header(p,prefix,settings,name))
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/header.py", line 715, in process_header
hi.parse(path,input_folder,settings,module_name)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/header.py", line 662, in parse
tr_unit = parse_tu(path,
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/translation_unit.py", line 34, in parse_tu
ix = get_index()
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/utils.py", line 99, in get_index
if not initialized: init_clang()
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/utils.py", line 56, in init_clang
conda_prefix = Path(getenv('CONDA_PREFIX'))
File "/usr/lib/python3.9/site-packages/path/__init__.py", line 164, in __init__
raise TypeError("Invalid initial value for path: None")
TypeError: Invalid initial value for path: None
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__main__.py", line 107, in <module>
main()
File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__main__.py", line 101, in all
ctx.invoke(parse, configuration=configuration, output=tmp_parsed, platform=platform)
File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__main__.py", line 40, in parse
result = parse_modules(obj.verbose,obj.njobs,settings,module_mapping,module_settings,platform)
File "/var/tmp/python-ocp-git/src/OCP/pywrap/bindgen/__init__.py", line 243, in parse_modules
modules = Parallel(prefer='processes',n_jobs=n_jobs)\
File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 1054, in __call__
self.retrieve()
File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 933, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/usr/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
return future.result(timeout=timeout)
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
TypeError: Invalid initial value for path: None
v.s.
$ CONDA_PREFIX=/usr PYTHONPATH=pywrap python -m bindgen --clean --libclang /usr/lib/libclang.so --include /usr/lib/clang/11.0.1/include all ocp.toml
100%|██████████| 309/309 [00:47<00:00, 6.57it/s]
[W 210116 16:19:26 translation_unit:47] ./opencascade/Standard_WarningsRestore.hxx
[W 210116 16:19:26 translation_unit:48] dummy.cxx:38:28: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
[W 210116 16:19:27 translation_unit:47] ./opencascade/Graphic3d_WorldViewProjState.hxx
[W 210116 16:19:27 translation_unit:48] dummy.cxx:154:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_NUMERICAL'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:154:60: error: expected ';' after expression
[W 210116 16:19:27 translation_unit:48] dummy.cxx:155:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_POINTER'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:155:57: error: expected ';' after expression
[W 210116 16:19:27 translation_unit:48] dummy.cxx:156:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_NUMERICAL'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:156:68: error: expected ';' after expression
[W 210116 16:19:27 translation_unit:48] dummy.cxx:157:5: error: use of undeclared identifier 'OCCT_DUMP_FIELD_VALUE_NUMERICAL'
[W 210116 16:19:27 translation_unit:48] dummy.cxx:157:67: error: expected ';' after expression
100%|██████████| 309/309 [00:17<00:00, 17.99it/s]
[I 210116 16:22:41 __init__:275] sorting
[I 210116 16:22:41 __init__:293] removing duplicate typedefs
Processing module Standard
Processing module gp
Processing module math
Processing module BVH
Processing module NCollection
Processing module TColgp
Processing module OSD
Processing module Quantity
...
Processing module XSControl
Processing module IFSelect
Processing module IFGraph
Processing module STEPCAFControl
Processing module XCAFApp
100%|██████████| 309/309 [00:02<00:00, 135.29it/s]
Processing module XCAFDoc
Processing module XCAFNoteObjects
Processing module XCAFPrs
Processing module XCAFView
Processing module BinXCAFDrivers
Processing module XmlXCAFDrivers
Processing module Vrml
Processing module VrmlAPI
Processing module RWGltf
Processing module Xw
I assume you can recreate the issue by calling unset CONDA_PREFIX
before you wrap.
Thanks, you are right! I solved it in https://github.com/CadQuery/pywrap/commit/420c92f56d2d8bb416744a11fd078cddf6cec8be , will pull this into OCP later.
Solved
I absolutely hate conda package manager. Not only it does not play well with existing OS packages whether binary or python, it also by default pollutes or even break one's environment in a very perverted way which is exactly what at least one other user stated in https://github.com/CadQuery/cadquery/issues/153 .
Having said that I decided to try to improve the experience for myself and perhaps for other people as well by creating ebuilds for Gentoo and probably also rpms for Fedora etc..
But after examining this repository I'm confused at how to even build it. If it was a standard conda package as far as I understand there would be a
build.sh
with build steps or calls to another build system.But all commands which seem to be doing something useful are in
azure-pipelines.yml
andbuild-bindings-job.yml
? How does this work? In order to build it you send it to some Microsoft machine?! How do you even develop in such repo then?Would it be possible to use some normal build system like Make, Cmake or some other so one can then easily integrate it into OS specific packaging solutions?