search

CadQuery / cadquery

A python parametric CAD scripting framework based on OCCT
https://cadquery.readthedocs.io
Other
2.98k stars 281 forks source link

Self-contained OCP wheel to enable installation via pip #1048

Closed fpq473 closed 2 years ago

fpq473 commented 2 years ago

I have found a way to create self-contained OCP wheels on Linux, allowing cadquery and all its dependencies to be installed via pip. I am sharing it here in the hopes that others may find it helpful, either for personal pip workflows or as a step towards official PyPI wheels.

I note that https://github.com/CadQuery/cadquery/issues/153#issuecomment-1033053777 requests a working pipeline, presumably Azure pipelines for all platforms. This is NOT that yet; this is only a way to build a wheel, and has only been tested on Linux.

The wheel is created from a conda environment with OCP installed. The OCP and vtk files in the python import path are blindly copied into a wheel. This wheel is of course unusable, but then we simply use auditwheel to bundle external libraries and produce a working manylinux_2_31_x86_64 wheel. This wheel is 160 MB, which is large but probably can be accommodated by PyPI.

A working cadquery venv can be created by pip installing this wheel along with git+https://github.com/CadQuery/cadquery, ezdxf, ipython, multimethod, nlopt, and nptyping<2. Thanks to @roipoussiere who has also tested this.

The code is available in this gist; see the docstring of setup.py for details. Testers on macOS and Windows will have to replace the use of auditwheel with delocate and delvewheel respectively. The created wheels may not be distributed until the requisite license files have been added into the wheel.

roipoussiere commented 2 years ago

Thanks a lot for this work. I created a pyproject.toml that uses this ocpvtk wheel. I tested it on Linux but it should work on other platforms.

This way, CadQuery can be easily installed and published with Poetry:

poetry install
poetry publish

For now it uses url of a local server for wheels, but once the weels will be generated by the CI, a CadQuery package could be published (by the CI) on Pypi (ie. on https://pypi.org/project/cadquery/), and I could create a PR to add this pyproject.toml.

roipoussiere commented 2 years ago

I published the docker image on the Docker hub:

So the Linux wheel can be retrieved with:

docker run --name ocpvtk cadquery/ocpvtk
docker cp ocpvtk:/ocpvtk-linux.whl .

I updated the Dockerfile in order to download Miniforge during the build.

At first, my motivation for this was to enable automated builds on the Docker Hub in order to automatically build the image on each release, but sadly this option is now only available to paid plans...

But at least, if anyone want to try the wheel, it's easier to retrieve it now. :)

roipoussiere commented 2 years ago

I'm trying to configure the CI. For now I was able to build the Linux wheel, it's a work in progress.

jmwright commented 2 years ago

I've done some work on creating a CQ wheel with conda-press. I've got other things to do first, but I'll see if I can combine this work with mine to get something working in CI.

The created wheels may not be distributed until the requisite license files have been added into the wheel.

As noted, we'll definitely need to address this in CI before we can push to PyPi.

roipoussiere commented 2 years ago

Tadaa:

image

ocp wheels for Windows, macOS and Linux, and python 3.8, 3.9 and 3.10, generated by the CI! This is the result of a work by @fpq473 and me these last days. 🙂

All of them has been successfully tested in the CI with import cadquery; cadquery.Workplane('XY').box(1,2,3).toSvg().

I just created a request to Pypi in order to ask for an increase file size limit (default is 100MB), I'm waiting for their response. 🙂

roipoussiere commented 2 years ago

The ocp-vtk wheels generated by the CI are now available in a github release, which means that they can be downloaded programmatically. As a result, it is now possible to install cadquery with pip!

There is some work to do to improve the installation process, but you can still give a try:

pip install <ocp-vtk wheel link> 'git+https://github.com/CadQuery/cadquery' ezdxf multimethod nlopt nptyping typish

Where <ocp-vtk wheel link> is one of the links published here: https://github.com/roipoussiere/OCP/releases/tag/7.5.3, according to your platform and Python version.

Each wheel is tested on the CI, but we will be happy to know if it works on your platform, in particular if you use Windows or macOS.

roipoussiere commented 2 years ago

pip install cadquery2* is now a thing. Does someone want to try it out?

Just create a venv first, to avoid messing up you system:

python -m venv .venv
source .venv/bin/activate
pip install cadquery2
python -c "import cadquery2 as cq; print(cq.Workplane('XY').box(1,2,3).toSvg())"

*Temporary name, I would suggest to move on cadquery later.

jmwright commented 2 years ago

I'm used the ocp-vtk wheel to set up a CadQuery development environment on Linux and it works well. I've also tested the cadquery2 wheel in a couple of different ways, but only on Linux. I'll post this issue to the Discord server to see if we can get some testers on Windows and MacOS.

dcowden commented 2 years ago

I have windows and will test this tonight and report back.

dcowden commented 2 years ago

Ok, I have tested and can't get it to work.

Summary: OS: Windows 11 pro x64, build 10.0.22000 Build 22000 try 1: miniforge/conda/python try 2: python 3.9.0 /venv

All failed with this error: >>> import cadquery2 as cq Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\tmp\cq2\lib\site-packages\cadquery2\__init__.py", line 2, in <module> from .occ_impl.geom import Plane, BoundBox, Vector, Matrix, Location File "c:\tmp\cq2\lib\site-packages\cadquery2\occ_impl\geom.py", line 5, in <module> from OCP.gp import ( ImportError: DLL load failed while importing OCP: The specified module could not be found.

More Detail is below

. Using conda, installation succeeded, but with pip warnings. Ultimately the installation failed: it could not find OCP dlls:

` To activate this environment, use

 $ conda activate cq2

To deactivate an active environment, use

 $ conda deactivate

(base) C:\Users\davec\gitwork\CQ-editor>conda activate cq2

(cq2) C:\Users\davec\gitwork\CQ-editor>pip install cadquery2 Defaulting to user installation because normal site-packages is not writeable Collecting cadquery2 Downloading cadquery2-2.1.1-py3-none-any.whl (136 kB) |████████████████████████████████| 136 kB 6.4 MB/s Collecting typish<2.0.0,>=1.9.3 Downloading typish-1.9.3-py3-none-any.whl (45 kB) |████████████████████████████████| 45 kB ... Collecting multimethod<2.0,>=1.8 Downloading multimethod-1.8-py3-none-any.whl (9.8 kB) Collecting ezdxf<0.18.0,>=0.17.2 Downloading ezdxf-0.17.2-cp39-cp39-win_amd64.whl (1.3 MB) |████████████████████████████████| 1.3 MB ... Collecting nlopt<3.0.0,>=2.7.1 Downloading nlopt-2.7.1-cp39-cp39-win_amd64.whl (325 kB) |████████████████████████████████| 325 kB ... Collecting nptyping<3.0.0,>=2.0.0 Downloading nptyping-2.0.0-py3-none-any.whl (19 kB) Collecting ocp-vtk<8.0.0,>=7.5.3 Downloading ocp_vtk-7.5.3.0-cp39-cp39-win_amd64.whl (63.1 MB) |████████████████████████████████| 63.1 MB 6.4 MB/s Requirement already satisfied: pyparsing>=2.0.1 in c:\users\davec\appdata\roaming\python\python39\site-packages (from ezdxf<0.18.0,>=0.17.2->cadquery2) (2.4.7) Collecting typing-extensions Downloading typing_extensions-4.2.0-py3-none-any.whl (24 kB) Requirement already satisfied: numpy>=1.14 in c:\users\davec\appdata\roaming\python\python39\site-packages (from nlopt<3.0.0,>=2.7.1->cadquery2) (1.20.1) Installing collected packages: typish, multimethod, typing-extensions, ezdxf, nlopt, nptyping, ocp-vtk, cadquery2 WARNING: The script ezdxf.exe is installed in 'C:\Users\davec\AppData\Roaming\Python\Python39\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed cadquery2-2.1.1 ezdxf-0.17.2 multimethod-1.8 nlopt-2.7.1 nptyping-2.0.0 ocp-vtk-7.5.3.0 typing-extensions-4.2.0 typish-1.9.3 WARNING: You are using pip version 20.2.3; however, version 22.0.4 is available. You should consider upgrading via the 'c:\program files\python39\python.exe -m pip install --upgrade pip' command.

(cq2) C:\Users\davec\gitwork\CQ-editor>python -c "import cadquery2 as cq; print(cq.Workplane('XY').box(1,2,3).toSvg())" Traceback (most recent call last): File "", line 1, in File "C:\Users\davec\AppData\Roaming\Python\Python39\site-packages\cadquery2__init__.py", line 2, in from .occ_impl.geom import Plane, BoundBox, Vector, Matrix, Location File "C:\Users\davec\AppData\Roaming\Python\Python39\site-packages\cadquery2\occ_impl\geom.py", line 5, in from OCP.gp import ( ImportError: DLL load failed while importing OCP: The specified module could not be found.

(cq2) C:\Users\davec\gitwork\CQ-editor>python Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import cadquery2 Traceback (most recent call last): File "", line 1, in File "C:\Users\davec\AppData\Roaming\Python\Python39\site-packages\cadquery2__init__.py", line 2, in from .occ_impl.geom import Plane, BoundBox, Vector, Matrix, Location File "C:\Users\davec\AppData\Roaming\Python\Python39\site-packages\cadquery2\occ_impl\geom.py", line 5, in from OCP.gp import ( ImportError: DLL load failed while importing OCP: The specified module could not be found.`

I upgraded pip, and forced re-installation, and got the same error:

`(cq2) C:\Users\davec\gitwork\CQ-editor>pip install --force-reinstall cadquery2 Defaulting to user installation because normal site-packages is not writeable Collecting cadquery2 Using cached cadquery2-2.1.1-py3-none-any.whl (136 kB) Collecting nlopt<3.0.0,>=2.7.1 Using cached nlopt-2.7.1-cp39-cp39-win_amd64.whl (325 kB) Collecting ezdxf<0.18.0,>=0.17.2 Using cached ezdxf-0.17.2-cp39-cp39-win_amd64.whl (1.3 MB) Collecting multimethod<2.0,>=1.8 Using cached multimethod-1.8-py3-none-any.whl (9.8 kB) Collecting typish<2.0.0,>=1.9.3 Using cached typish-1.9.3-py3-none-any.whl (45 kB) Collecting nptyping<3.0.0,>=2.0.0 Using cached nptyping-2.0.0-py3-none-any.whl (19 kB) Collecting ocp-vtk<8.0.0,>=7.5.3 Using cached ocp_vtk-7.5.3.0-cp39-cp39-win_amd64.whl (63.1 MB) Collecting pyparsing>=2.0.1 Downloading pyparsing-3.0.8-py3-none-any.whl (98 kB) ---------------------------------------- 98.5/98.5 KB 5.5 MB/s eta 0:00:00 Collecting typing-extensions Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB) Collecting numpy>=1.14 Downloading numpy-1.22.3-cp39-cp39-win_amd64.whl (14.7 MB) ---------------------------------------- 14.7/14.7 MB 9.1 MB/s eta 0:00:00 Installing collected packages: typish, ocp-vtk, typing-extensions, pyparsing, numpy, multimethod, nptyping, nlopt, ezdxf, cadquery2 Attempting uninstall: typish Found existing installation: typish 1.9.3 Uninstalling typish-1.9.3: Successfully uninstalled typish-1.9.3 Attempting uninstall: ocp-vtk Found existing installation: ocp-vtk 7.5.3.0 Uninstalling ocp-vtk-7.5.3.0: Successfully uninstalled ocp-vtk-7.5.3.0 Attempting uninstall: typing-extensions Found existing installation: typing_extensions 4.2.0 Uninstalling typing_extensions-4.2.0: Successfully uninstalled typing_extensions-4.2.0 Attempting uninstall: pyparsing Found existing installation: pyparsing 2.4.7 Uninstalling pyparsing-2.4.7: Successfully uninstalled pyparsing-2.4.7 Attempting uninstall: numpy Found existing installation: numpy 1.20.1 Uninstalling numpy-1.20.1: Successfully uninstalled numpy-1.20.1 WARNING: The script f2py.exe is installed in 'C:\Users\davec\AppData\Roaming\Python\Python39\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Attempting uninstall: multimethod Found existing installation: multimethod 1.8 Uninstalling multimethod-1.8: Successfully uninstalled multimethod-1.8 Attempting uninstall: nptyping Found existing installation: nptyping 2.0.0 Uninstalling nptyping-2.0.0: Successfully uninstalled nptyping-2.0.0 Attempting uninstall: nlopt Found existing installation: nlopt 2.7.1 Uninstalling nlopt-2.7.1: Successfully uninstalled nlopt-2.7.1 Attempting uninstall: ezdxf Found existing installation: ezdxf 0.17.2 Uninstalling ezdxf-0.17.2: Successfully uninstalled ezdxf-0.17.2 WARNING: The script ezdxf.exe is installed in 'C:\Users\davec\AppData\Roaming\Python\Python39\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Attempting uninstall: cadquery2 Found existing installation: cadquery2 2.1.1 Uninstalling cadquery2-2.1.1: Successfully uninstalled cadquery2-2.1.1 Successfully installed cadquery2-2.1.1 ezdxf-0.17.2 multimethod-1.8 nlopt-2.7.1 nptyping-2.0.0 numpy-1.22.3 ocp-vtk-7.5.3.0 pyparsing-3.0.8 typing-extensions-4.2.0 typish-1.9.3

(cq2) C:\Users\davec\gitwork\CQ-editor>import cadquery2 as cq 'import' is not recognized as an internal or external command, operable program or batch file.

(cq2) C:\Users\davec\gitwork\CQ-editor>python Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import cadquery2 as cq Traceback (most recent call last): File "", line 1, in File "C:\Users\davec\AppData\Roaming\Python\Python39\site-packages\cadquery2__init__.py", line 2, in from .occ_impl.geom import Plane, BoundBox, Vector, Matrix, Location File "C:\Users\davec\AppData\Roaming\Python\Python39\site-packages\cadquery2\occ_impl\geom.py", line 5, in from OCP.gp import ( ImportError: DLL load failed while importing OCP: The specified module could not be found.`

I retried using a virtual env, and got the same result: ` C:\Users\davec\gitwork\CQ-editor>python3 -m venv c:/tmp/cq2 Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

C:\Users\davec\gitwork\CQ-editor>python -m venv c:/tmp/cq2

C:\Users\davec\gitwork\CQ-editor>cd c:/tmp

c:\tmp>ls 'ls' is not recognized as an internal or external command, operable program or batch file.

c:\tmp>dir Volume in drive C is Windows Volume Serial Number is 5E31-75AF

Directory of c:\tmp

04/27/2022 05:58 PM

. 04/03/2022 02:16 PM 23,333 basicsplitflap-brd.svg 04/03/2022 01:32 PM 22,331 basicsplitflap-B_Cu.svg 04/03/2022 01:32 PM 2,019 basicsplitflap-Edge_Cuts.svg 12/06/2020 07:48 PM battlepoint 04/27/2022 05:58 PM cq2 04/03/2022 01:36 PM 27,965 drawing.hpgl 04/03/2022 01:36 PM 6,649 drawing.pdf 04/03/2022 01:35 PM 27,949 drawing.svg 10/16/2021 03:48 PM 155,679 Face.SLDPRT 12/23/2021 10:12 AM flash_sound 04/03/2022 01:52 PM 1,542,455 g1188.png 04/03/2022 02:10 PM 58,539 g535.png 04/03/2022 02:13 PM 59,320 g536.png 03/14/2021 07:42 PM 63 intro.py 03/14/2021 07:50 PM pgz 06/12/2021 04:41 PM 1,502 SeeduinoBreakout-Edge_Cuts.svg 03/14/2021 07:28 PM 762 test.py 04/03/2022 02:22 PM 615,400 test_board.lbrn 12/23/2021 10:35 AM 340 tf_sound.log 15 File(s) 2,544,306 bytes 5 Dir(s) 39,225,929,728 bytes free

c:\tmp>cd cq2

c:\tmp\cq2>dir Volume in drive C is Windows Volume Serial Number is 5E31-75AF

Directory of c:\tmp\cq2

04/27/2022 05:58 PM

. 04/27/2022 05:58 PM .. 04/27/2022 05:58 PM Include 04/27/2022 05:58 PM Lib 04/27/2022 05:58 PM 89 pyvenv.cfg 04/27/2022 05:58 PM Scripts 1 File(s) 89 bytes 5 Dir(s) 39,225,929,728 bytes free

c:\tmp\cq2>Scripts/activate 'Scripts' is not recognized as an internal or external command, operable program or batch file.

c:\tmp\cq2>cd Scripts

c:\tmp\cq2\Scripts>activate

(cq2) c:\tmp\cq2\Scripts>pip install --upgrade pip Collecting pip Using cached pip-22.0.4-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.2.3 Uninstalling pip-20.2.3: Successfully uninstalled pip-20.2.3 ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\Users\davec\AppData\Local\Temp\pip-uninstall-ed7h8o__\pip.exe' Consider using the --user option or check the permissions.

(cq2) c:\tmp\cq2\Scripts>pip install cadquery2 Collecting cadquery2 Using cached cadquery2-2.1.1-py3-none-any.whl (136 kB) Collecting multimethod<2.0,>=1.8 Using cached multimethod-1.8-py3-none-any.whl (9.8 kB) Collecting typish<2.0.0,>=1.9.3 Using cached typish-1.9.3-py3-none-any.whl (45 kB) Collecting nlopt<3.0.0,>=2.7.1 Using cached nlopt-2.7.1-cp39-cp39-win_amd64.whl (325 kB) Collecting nptyping<3.0.0,>=2.0.0 Using cached nptyping-2.0.0-py3-none-any.whl (19 kB) Collecting ocp-vtk<8.0.0,>=7.5.3 Using cached ocp_vtk-7.5.3.0-cp39-cp39-win_amd64.whl (63.1 MB) Collecting ezdxf<0.18.0,>=0.17.2 Using cached ezdxf-0.17.2-cp39-cp39-win_amd64.whl (1.3 MB) Collecting pyparsing>=2.0.1 Using cached pyparsing-3.0.8-py3-none-any.whl (98 kB) Collecting typing-extensions Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB) Collecting numpy>=1.14 Using cached numpy-1.22.3-cp39-cp39-win_amd64.whl (14.7 MB) Installing collected packages: typish, ocp-vtk, typing-extensions, pyparsing, numpy, multimethod, nptyping, nlopt, ezdxf, cadquery2 Successfully installed cadquery2-2.1.1 ezdxf-0.17.2 multimethod-1.8 nlopt-2.7.1 nptyping-2.0.0 numpy-1.22.3 ocp-vtk-7.5.3.0 pyparsing-3.0.8 typing-extensions-4.2.0 typish-1.9.3

(cq2) c:\tmp\cq2\Scripts>python Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import cadquery2 as cq Traceback (most recent call last): File "", line 1, in File "c:\tmp\cq2\lib\site-packages\cadquery2__init__.py", line 2, in from .occ_impl.geom import Plane, BoundBox, Vector, Matrix, Location File "c:\tmp\cq2\lib\site-packages\cadquery2\occ_impl\geom.py", line 5, in from OCP.gp import ( ImportError: DLL load failed while importing OCP: The specified module could not be found. `

the dlls are definitely installed, I can see them in what I think is the right spot: `(cq2) c:\tmp\cq2\Lib\site-packages>dir Volume in drive C is Windows Volume Serial Number is 5E31-75AF

Directory of c:\tmp\cq2\Lib\site-packages

04/27/2022 06:00 PM

. 04/27/2022 05:58 PM .. 04/27/2022 05:59 PM 4,603 .load-order-ocp_vtk-7.5.3.0 04/27/2022 06:00 PM cadquery2 04/27/2022 06:00 PM cadquery2-2.1.1.dist-info 04/27/2022 05:59 PM 10,752 charset-f1ff420e.dll 04/27/2022 05:58 PM 126 easy_install.py 04/27/2022 05:59 PM ezdxf 04/27/2022 06:00 PM ezdxf-0.17.2.dist-info 04/27/2022 05:59 PM 634,880 freetype-4118af61.dll 04/27/2022 05:59 PM 95,744 gl2ps-d5be8415.dll 04/27/2022 05:59 PM 451,584 glew32-45501540.dll 04/27/2022 05:59 PM 739,840 hdf-839eff43.dll 04/27/2022 05:59 PM 3,164,672 hdf5-4fe83a80.dll 04/27/2022 05:59 PM 117,248 hdf5_hl-1e917a47.dll 04/27/2022 05:59 PM 939,008 iconv-cfade252.dll 04/27/2022 05:59 PM 57,856 jbig-283e2ee0.dll 04/27/2022 05:59 PM 227,840 jsoncpp-74e0510f.dll 04/27/2022 05:59 PM 385,024 Lerc-d98a2f21.dll 04/27/2022 05:59 PM 187,904 libbz2-bf72f086.dll 04/27/2022 05:59 PM 5,070,848 libcrypto-3-x64-79e66d5d.dll 04/27/2022 05:59 PM 513,024 libcurl-4715d16f.dll 04/27/2022 05:59 PM 66,048 libdeflate-b8e589f8.dll 04/27/2022 05:59 PM 407,040 libexpat-f770ecdf.dll 04/27/2022 05:59 PM 113,664 liblz4-b9e0f074.dll 04/27/2022 05:59 PM 154,624 liblzma-f8552861.dll 04/27/2022 05:59 PM 193,536 libpng16-f2a3ac70.dll 04/27/2022 05:59 PM 227,840 libssh2-54993272.dll 04/27/2022 05:59 PM 1,281,024 libxml2-64eb6d5f.dll 04/27/2022 05:59 PM 253,659 LICENSES_bundled.txt 04/27/2022 05:59 PM 137,728 mfhdf-b1f9baf5.dll 04/27/2022 05:59 PM 565,640 msvcp140.dll 04/27/2022 05:59 PM multimethod 04/27/2022 05:59 PM multimethod-1.8.dist-info 04/27/2022 05:59 PM 1,092,608 netcdf-b8bee6dc.dll 04/27/2022 05:59 PM nlopt 04/27/2022 05:59 PM nlopt-2.7.1.dist-info 04/27/2022 05:59 PM nptyping 04/27/2022 05:59 PM nptyping-2.0.0.dist-info 04/27/2022 05:59 PM numpy 04/27/2022 05:59 PM numpy-1.22.3.dist-info 04/27/2022 05:59 PM 113,632,256 OCP.cp39-win_amd64.pyd 04/27/2022 05:59 PM ocp_vtk-7.5.3.0.dist-info 04/27/2022 05:59 PM 29,184 ogg-e38dc37e.dll 04/27/2022 05:59 PM pip 04/27/2022 05:59 PM pip-22.0.4.dist-info 04/27/2022 05:58 PM pkg_resources 04/27/2022 05:59 PM 2,999,808 proj_7_2-671605b1.dll 04/27/2022 05:59 PM pyparsing 04/27/2022 05:59 PM pyparsing-3.0.8.dist-info 04/27/2022 05:58 PM setuptools 04/27/2022 05:58 PM setuptools-49.2.1.dist-info 04/27/2022 05:59 PM 1,460,736 sqlite3-dd39700b.dll 04/27/2022 05:59 PM 287,744 tbb-17b02d2c.dll 04/27/2022 05:59 PM 131,584 tbbmalloc-24341326.dll 04/27/2022 05:59 PM tests 04/27/2022 05:59 PM 210,432 theora-ad1efaaf.dll 04/27/2022 05:59 PM 698,368 tiff-9eb9c421.dll 04/27/2022 05:59 PM 1,738,240 TKernel-d4900e62.dll 04/27/2022 05:59 PM 315,392 TKG2d-af7ff42f.dll 04/27/2022 05:59 PM 974,336 TKG3d-2f65fb9e.dll 04/27/2022 05:59 PM 1,787,392 TKMath-e14c04cc.dll 04/27/2022 05:59 PM typing_extensions-4.2.0.dist-info 04/27/2022 05:59 PM 70,613 typing_extensions.py 04/27/2022 05:59 PM typish 04/27/2022 05:59 PM typish-1.9.3.dist-info 04/27/2022 05:59 PM 932,352 vtkChartsCore-9.0-a07ade5b.dll 04/27/2022 05:59 PM 106,496 vtkCommonColor-9.0-87ee4902.dll 04/27/2022 05:59 PM 166,400 vtkCommonComputationalGeometry-9.0-4f06c132.dll 04/27/2022 05:59 PM 4,744,192 vtkCommonCore-9.0-4ec86a33.dll 04/27/2022 05:59 PM 3,540,480 vtkCommonDataModel-9.0-8d940f29.dll 04/27/2022 05:59 PM 774,656 vtkCommonExecutionModel-9.0-953aa4ba.dll 04/27/2022 05:59 PM 124,416 vtkCommonMath-9.0-899fa293.dll 04/27/2022 05:59 PM 117,760 vtkCommonMisc-9.0-b4b3de35.dll 04/27/2022 05:59 PM 25,600 vtkCommonPython-9.0-da6b908b.dll 04/27/2022 05:59 PM 95,744 vtkCommonSystem-9.0-1a27d27d.dll`

dcowden commented 2 years ago

@roipoussiere see above, installation said successful using the wheel, but then dlls are not found.
Let me know if there are things i should try. This result seems very par for the course in windows.

jmwright commented 2 years ago

There is also a user in issue #1068 who seems to be having the same issue while trying to install in Blender.

fpq473 commented 2 years ago

Thanks for testing.

I've stumbled on at least one issue on Windows which may explain the reported errors above. Looking through the build action logs, I see that only four TK*.dll files are included in the wheel:

The following DLLs will be copied into the wheel.
    Lerc.dll (C:\Miniconda3\envs\ocp-vtk\Library\bin\Lerc.dll)
    TKG2d.dll (C:\Miniconda3\envs\ocp-vtk\Library\bin\TKG2d.dll)
    TKG3d.dll (C:\Miniconda3\envs\ocp-vtk\Library\bin\TKG3d.dll)
    TKMath.dll (C:\Miniconda3\envs\ocp-vtk\Library\bin\TKMath.dll)
    TKernel.dll (C:\Miniconda3\envs\ocp-vtk\Library\bin\TKernel.dll)
    charset.dll (C:\Miniconda3\envs\ocp-vtk\Library\bin\charset.dll)
    [...]

The listed TK*.dll files match the directory listing in the error report above. My understanding is that many more (or maybe all) TK*.dll files should have been included.

The list of dll's to include is generated by delvewheel, which in turn uses pefile to parse imports. pefile imposes an upper limit on the number of imports to parse, causing the import list to be truncated.

A simple fix to try would be to replace the two calls to python -m delvewheel in roipoussiere/OCP/setup.py with python -c "import pefile;pefile.MAX_IMPORT_SYMBOLS*=20;from delvewheel.__main__ import main;main()". With this 20x increase, I observe pefile finding many more TK*.dll.

@roipoussiere

fpq473 commented 2 years ago

A new version of delvewheel (0.0.22) has been released with some fixes, so the workaround above is no longer needed. We can just trigger the build action to get new wheels and test again on Windows.

@roipoussiere

dcowden commented 2 years ago

i'm happy to retest whenever we're ready

jmwright commented 2 years ago

@fpq473 @roipoussiere I'm trying to get a parallel CI setup going as a precursor to integrating this work into the main pipeline. I'm getting this encoding error, but only on Windows. Did either of your run into this?

File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\pep517\in_process\_in_process.py", line 363, in <module>
not_found=['ocp_vtk-7.5.3.0.data/platlib/.load-order-ocp_vtk-7.5.3.0', 'ocp_vtk-7.5.3.0.dist-info/DELVEWHEEL']
    main()
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\pep517\in_process\_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\pep517\in_process\_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\build_meta.py", line 244, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\build_meta.py", line 229, in _build_with_temp_dir
    self.run_setup()
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\build_meta.py", line 281, in run_setup
    super(_BuildMetaLegacyBackend,
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\build_meta.py", line 174, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 294, in <module>
    setup(
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\_distutils\core.py", line 148, in setup
    return run_commands(dist)
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\_distutils\core.py", line 163, in run_commands
    dist.run_commands()
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\dist.py", line 1214, in run_command
    super().run_command(command)
  File "C:\Miniconda3\envs\ocp-vtk\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
    cmd_obj.run()
  File "setup.py", line 111, in run
    add_licenses_bundled(conda_prefix, repaired_whl, added_files)
  File "setup.py", line 202, in add_licenses_bundled
    write_licenses(conda_prefix, whl, ["ocp", "vtk"], added_files, f)
  File "setup.py", line 272, in write_licenses
    print(f"> {l.rstrip()}", file=out)
  File "C:\Miniconda3\envs\ocp-vtk\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]

Also, because this method uses a conda environment to build the wheels, we can't trigger this build off of a push to the master branch as the changes will not be available in the Anaconda package repository yet. I'm considering putting this and one or two other builds in a separate repo designed for that purpose.

fpq473 commented 2 years ago

I am unable to verify this in a Windows environment on Github Actions, but my theory is that the default encoding there is cp1252. So what may be happening is that we are trying to write a UTF-8 string to a cp1252-encoded file, causing the error. Demo:

>>> import sys
>>> sys.stdout.reconfigure(encoding='cp1252')
>>> print('\ufffd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/encodings/cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in position 0: character maps to <undefined>
>>>

One fix to try would be to specify the encoding when opening the licenses file:

    with open("LICENSES_bundled.txt", "w", encoding="utf-8") as f:
#                                           ^^^^^^^^^^^^^^

in setup.py: https://github.com/jmwright/ocp-build-system/blob/741e6c579d43e3281420549923ea3e3c1a2f822e/setup.py#L199

You would have to remove print(l) on L272 because that would fail when writing to stdout, which appears to be cp1252-encoded.

There are other workarounds like defaulting to UTF-8, but I'm not so familiar with best practices on Windows: https://dev.to/methane/python-use-utf-8-mode-on-windows-212i

jmwright commented 2 years ago

@fpq473 Thanks for the tip. Setting the PYTHONUTF8 variable to 1 seems to have fixed the problem. Tomorrow I'll check to see if the build is using the new version of delvewheel and if the Windows wheels work correctly.

jmwright commented 2 years ago

Here's a link to the new builds if anyone else wants to take a look: https://github.com/jmwright/ocp-build-system/actions/runs/2272471797

fpq473 commented 2 years ago

Thanks for building. I just tested on Windows with Python 3.9 and it looks fine (and the build is much faster, which is expected with the new version of delvewheel).

Note that there is a test step in the build action, but I'm thinking that maybe it's running inside the conda env (which has OCP installed!) instead of starting from scratch in an empty environment. That's why the tests passed before here but we got the error reported above.

I think that conda is being activated on L59 (see the discussion about shell: in the setup-miniconda step) -- I think we can just delete it. Maybe we can add something like ! python -c "import OCP" to test that OCP isn't importable, before building the env and testing.

dcowden commented 2 years ago

Ok I will test this again tonight also

jmwright commented 2 years ago

@dcowden You'll want to use one of the install lines that I posted here for the Blender user to try.

I think I have a method worked out to trigger a wheel build/release in Azure after the OCP uploads to the Anaconda package repo complete sucessfully, but I want to get all the pieces of the full integration with PyPi working before I create a PR on OCP's repo.

jmwright commented 2 years ago

@fpq473 I think I have the testing step fixed. Before, the test was checking what was already on PyPi, not the wheel that had just been generated locally. https://github.com/jmwright/ocp-build-system/actions/runs/2276238784

I have also confirmed that the Python 3.8 wheel works in MacOS 10.14.6.

dcowden commented 2 years ago

ok @jmwright @roipoussiere @fpq473 I tested. I am successful using a virtual environment, + python 3.9, on windows 11 x64 !

(testcq3) C:\Users\davec\testcq3\Scripts>python -c "import cadquery as cq; print(cq.Workplane('XY').box(1,2,3).toSvg())"
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   width="800.0"
   height="240.0"

>
    <g transform="scale(69.20188809083672, -69.20188809083672)   translate(3.857549370751361,-1.5895520055119063)" stroke-width="0.014450472777380966"  fill="none">
       <!-- hidden lines -->
       <g  stroke="rgb(160,160,160)" fill="none" stroke-dasharray="0.014450472777380966,0.014450472777380966" >
                        <path d="M-0.02359645890915052,-0.6576855014574596 L0.967454815275168,-1.233378380717305 " />
                        <path d="M-0.02359645890915052,-0.6576855014574596 L-0.02359645890915052,1.300542549964287 " />
                        <path d="M-0.967454815275168,-0.7248496707044416 L-0.02359645890915052,-0.6576855014574596 " />

       </g>

       <!-- solid lines -->
       <g  stroke="rgb(0,0,0)" fill="none">
                        <path d="M-0.967454815275168,-0.7248496707044416 L0.02359645890915052,-1.300542549964287 " />
                        <path d="M0.02359645890915052,-1.300542549964287 L0.02359645890915052,0.6576855014574596 " />
                        <path d="M-0.967454815275168,1.233378380717305 L0.02359645890915052,0.6576855014574596 " />
                        <path d="M-0.967454815275168,-0.7248496707044416 L-0.967454815275168,1.233378380717305 " />
                        <path d="M0.967454815275168,-1.233378380717305 L0.967454815275168,0.7248496707044416 " />
                        <path d="M-0.02359645890915052,1.300542549964287 L0.967454815275168,0.7248496707044416 " />
                        <path d="M0.02359645890915052,-1.300542549964287 L0.967454815275168,-1.233378380717305 " />
                        <path d="M-0.967454815275168,1.233378380717305 L-0.02359645890915052,1.300542549964287 " />
                        <path d="M0.02359645890915052,0.6576855014574596 L0.967454815275168,0.7248496707044416 " />

       </g>
    </g>
    <g transform="translate(20,210.0)" stroke="rgb(0,0,255)">
        <line x1="30" y1="-30" x2="75" y2="-33" stroke-width="3" stroke="#000000" />
         <text x="80" y="-30" style="stroke:#000000">X </text>

        <line x1="30" y1="-30" x2="30" y2="-75" stroke-width="3" stroke="#000000" />
         <text x="25" y="-85" style="stroke:#000000">Y </text>

        <line x1="30" y1="-30" x2="58" y2="-15" stroke-width="3" stroke="#000000" />
         <text x="65" y="-5" style="stroke:#000000">Z </text>
        <!--
            <line x1="0" y1="0" x2="69.20188809083672" y2="0" stroke-width="3" />
            <text x="0" y="20" style="stroke:#000000">1  in </text>
        -->
    </g>
</svg>
jmwright commented 2 years ago

That's great news. I'm considering manually building a release on CadQuery's PyPi account so we can get people using these wheels, and then starting on the OCP integration after that. I think that doing the release manually this time will make it easier to request the size increase from PyPi for our packages anyway.

fpq473 commented 2 years ago

In case you missed it, and in case there's a clash of names, @roipoussiere obtained a PyPI file limit increase to 200 MB for the ocp-vtk wheels he published. See https://github.com/CadQuery/cadquery/issues/1048#issuecomment-1096874385 above.

jmwright commented 2 years ago

That's one of the questions I have. The ocp package is already being used by the "Open Collaboration Platform" on PyPi, and so we will end up with a naming clash there too. I think my current approach would be to use the package name ocp-cq, but I'm open to suggestions.

jmwright commented 2 years ago

Maybe the package name should be cadquery-ocp? That would be similar to what some other projects do where the overall project name is first, with the sub-project's is second.

dcowden commented 2 years ago

@jmwright yes i agree, isnt it useful sometimes to be able to refer to packages with wildcards? cadquery-*

fpq473 commented 2 years ago

Either works. It's probably a good idea to drop the "vtk" part of the name in case we figure out how to avoid bundling it, and include it as a dependency instead.

jmwright commented 2 years ago

I've updated the release to use the cadquery-ocp naming scheme. So the command line template above becomes:

pip install <cadquery-ocp wheel link> 'git+https://github.com/CadQuery/cadquery' ezdxf multimethod nlopt nptyping typish

For example (Linux with Python 3.8):

pip install https://github.com/jmwright/ocp-build-system/releases/download/7.5.3/cadquery_ocp-7.5.3.0-cp38-cp38-manylinux_2_31_x86_64.whl 'git+https://github.com/CadQuery/cadquery' ezdxf multimethod nlopt nptyping typish

I'll work on getting these wheels pushed to PyPi next.

hastelloy commented 2 years ago

For example (Linux with Python 3.8):

pip install https://github.com/jmwright/ocp-build-system/releases/download/7.5.3/cadquery_ocp-7.5.3.0-cp38-cp38-manylinux_2_31_x86_64.whl 'git+https://github.com/CadQuery/cadquery' ezdxf multimethod nlopt nptyping typish

For Debian11 with Python 3.9.2, all these work except nptyping package I am not sure why using pip3 install nptyping , version 1.4.4 is installed while the latest pypi version is 2.0.1, manual download the latest wheel from pypi and reinstall solve the problem. Hope this helps if someone else meets the same problem.

    import cadquery as cq
  File "/home/AAA/.local/lib/python3.9/site-packages/cadquery/__init__.py", line 32, in <module>
    from .assembly import Assembly, Color, Constraint
  File "/home/AAA/.local/lib/python3.9/site-packages/cadquery/assembly.py", line 11, in <module>
    from .occ_impl.solver import (
  File "/home/AAA/.local/lib/python3.9/site-packages/cadquery/occ_impl/solver.py", line 395, in <module>
    class ConstraintSolver(object):
  File "/home/AAA/.local/lib/python3.9/site-packages/cadquery/occ_impl/solver.py", line 452, in ConstraintSolver
    Callable[[Array[Any, Float]], float],
  File "/home/AAA/.local/lib/python3.9/site-packages/nptyping/types/_ndarray_meta.py", line 117, in __getitem__
    raise TypeError('Invalid parameter for NDArray: "{}"'.format(item))
TypeError: Invalid parameter for NDArray: "(typing.Any, Float)"
fpq473 commented 2 years ago

Thanks for pointing this out @hastelloy. I can replicate, assuming you had nptyping 1.4.4 already installed when you ran your pip command -- pip will not upgrade in this scenario. I am not able to replicate from a new venv.

That said, this is a good reminder that we need to remember to specify the nptyping version when building cadquery wheels, as was done for conda.

jmwright commented 2 years ago

I'm going to close this issue since we now have OCP wheels. If anyone is still having issues with the wheels, please feel free to open a new issue to get help. The documentation here shows how to install CadQuery via pip and what the limitations are, and the OCP wheel can be installed by itself using pip install cadquery-ocp.

jmwright commented 2 years ago

Big thanks to @fpq473 for getting this going.