search

CadQuery / ocp-build-system

A system to build Python wheel PyPI packages for OCP.
Apache License 2.0
6 stars 8 forks source link

Build wheels for 7.7.0 #7

Closed fpq473 closed 1 year ago

fpq473 commented 1 year ago

Match latest versions of ocp and vtk conda packages.

VTK manylinux wheels are now available on PyPI for 3.10, so remove use of third-party wheel.

jmwright commented 1 year ago

@fpq473 Is this going to run into the same glibc versioning problem that we have with 7.6.*?

fpq473 commented 1 year ago

@jmwright The fix in a54b3c759b51ab5089cc18077e7ba6c1d53bca7d allowed us to produce manylinux_2_31 wheels for 7.6, matching what we had for 7.5. So the requirement is debian 11 or ubuntu 20.04, which isn't the greatest, but still useful.

This change will (hopefully) build the same for 7.7.

fpq473 commented 1 year ago

And just FTR, here are the 7.6.3a0 wheels released last October:

https://pypi.org/project/cadquery-ocp/7.6.3a0/#files

It's an alpha release (matching the conda package), so the PyPI page still says 7.5.3.0.

jmwright commented 1 year ago

I've tried building a final 7.6.3.0 release before merging this, but things aren't working again.

https://github.com/CadQuery/ocp-build-system/actions/runs/3884118392/jobs/6626257305#step:8:98879

fpq473 commented 1 year ago

I committed some fixes to try to get 7.6.3.0 to build.

But there is still one problem which you might know how to fix: the conda install command is installing 7.6.3.alpha from the cadquery channel instead of 7.6.3.0 from conda-forge.

I already tried 2102e5d225b5e946af19989e44597391b7914c5d but it seemingly had no effect.

fpq473 commented 1 year ago

Okay, I managed to get 7.6.3.0 to install. But now there are glibc versioning problems like you noted.

I'll think about it.

jmwright commented 1 year ago

@fpq473 Thanks.

I think the proper way to fix this kind of problem and to support other platforms like ARM properly, is to do from-scratch compilation of the toolchain (VTK, OCCT, OCP, etc) so we have more control over the glibc version. It's very involved to build everything from scratch though.

fpq473 commented 1 year ago

I did some debugging. The workflow logs say:

2023-01-10T17:55:46.2566145Z DEBUG:auditwheel.policy.versioned_symbols:Package requires GLIBCXX_3.4.29, incompatible with policy manylinux_2_31_x86_64 ... 2023-01-10T17:55:46.2566962Z DEBUG:auditwheel.policy.versioned_symbols:Package requires CXXABI_1.3.13, incompatible with policy manylinux_2_31_x86_64 ...

This requirement is coming from OCP.so file:

./OCP.cpython-310-x86_64-linux-gnu.so:     file format elf64-x86-64
  9264f7:       ff 15 a3 05 d0 07       callq  *0x7d005a3(%rip)        # 8626aa0 <std::__exception_ptr::exception_ptr::_M_addref()@CXXABI_1.3.13>
  926512:       ff 15 90 ff cf 07       callq  *0x7cfff90(%rip)        # 86264a8 <std::__exception_ptr::exception_ptr::_M_release()@CXXABI_1.3.13>
  926527:       ff 15 73 05 d0 07       callq  *0x7d00573(%rip)        # 8626aa0 <std::__exception_ptr::exception_ptr::_M_addref()@CXXABI_1.3.13>
  c7f09e:       ff 15 cc 2e 98 07       callq  *0x7982ecc(%rip)        # 8601f70 <std::__throw_bad_array_new_length()@GLIBCXX_3.4.29>
  c7f42e:       ff 15 3c 2b 98 07       callq  *0x7982b3c(%rip)        # 8601f70 <std::__throw_bad_array_new_length()@GLIBCXX_3.4.29>
  c7f796:       ff 15 d4 27 98 07       callq  *0x79827d4(%rip)        # 8601f70 <std::__throw_bad_array_new_length()@GLIBCXX_3.4.29>

This is specific to the 7.6.3.0 package on conda-forge. The 7.6.3.alpha and 7.7.0.alpha packages on the cadquery channel do not have this problem. So there is some difference in how these packages are compiled (but not sure how/where these builds are done).

Should we (1) try to identify the differences, or (2) just build and release 7.7.0.alpha, or (3) throw in the towel?

fpq473 commented 1 year ago

Fourth option: release 7.6.3.0 as a manylinux_2_34 wheel. This requires newer distros, e.g. fedora 35, ubuntu 21.10, debian testing.

jmwright commented 1 year ago

It looks like all the OCP builds, even the OCCT 7.7 ones are using Ubuntu 18.04, so the system's glibc version shouldn't be the problem. It must be something with how conda is solving and setting up the environments.

https://dev.azure.com/cadquery/OCP/_build/results?buildId=2552&view=results

Maybe the best thing to do is your option #4 first, followed by a build and release of 7.7 alpha.

What do you think @fpq473 ?

fpq473 commented 1 year ago

We can do (4) but after additional reflection, it seems like a potential pain point to support -- when users are unable to install cadquery-ocp, it may be because they didn't say pip install --pre or maybe it's their linux distro version.

I can try to test your theory by comparing build logs, but I can't find the 7.6.3.0 build log. Can you?

The 7.6.3.0 files were uploaded to conda-forge around Dec 30. The Azure pipeline runs you linked to go back to September, but none of them appear to be for 7.6.3.0 -- all the December runs are titled "Start with 7.7.0".

https://dev.azure.com/cadquery/OCP/_build?definitionId=6

jmwright commented 1 year ago

The latest commit I see on master for 7.6.3 is Sept. https://github.com/CadQuery/OCP/commits/master

The latest release is the 7.6.3.0 tag on Nov 23rd. I don't see any Azure logs for that tag, but that tag points back to the September commit for 7.6.3, which does have logs. https://github.com/CadQuery/OCP/releases

fpq473 commented 1 year ago

@jmwright Are you still referring to the Azure builds? None of them are for 7.6.3.0, as far as I can tell. The log output indicates that 7.7.0.alpha or 7.6.3.alpha are being built.

image

Also I see that this line {% set OCP_TWEAK = "alpha" %} has existed in OCP for over a year, so if it means what I think it means, it seems unlikely that a non-alpha build was done via github/azure.

https://github.com/CadQuery/OCP/blame/2cb2dba8674c58d6ede38fd27619a09e850856e8/conda/meta.yaml#L2

Is it possible the upload was done as a one-off elsewhere?

jmwright commented 1 year ago

@fpq473 Unless you object, I'm going to do the following.

  1. Do a "fake" 7.6.3.0 release.
  2. Start work on reviewing/merging this branch
fpq473 commented 1 year ago

@jmwright Sounds like a plan!

jmwright commented 1 year ago

@fpq473 Sorry that I made such a mess of this. 7.6.3 has been published and we can move ahead with this PR. I ended up using 7.6.3 as the version number rather than 7.6.3.0 because we didn't have the trailing 0 in the last release version number.

fpq473 commented 1 year ago

I see 7.6.3 on pypi, awesome, nicely done.

Do you want to release 7.7.0.alpha now, or just a straight up 7.7.0?

jmwright commented 1 year ago

Do you want to release 7.7.0.alpha now, or just a straight up 7.7.0?

Let's do an alpha first just to be safe.

fpq473 commented 1 year ago

@jmwright I updated this PR for 7.7.0.alpha. Also tried to centralize the version wrangling. Tested on Linux.

jmwright commented 1 year ago

@fpq473 Looks like there's a version conflict.

https://github.com/CadQuery/ocp-build-system/actions/runs/3954442641/jobs/6771785613#step:5:56