Open nijel opened 5 months ago
@nijel The workaround solution would be to force lxml not to use the pre-compiled wheel: pip install lxml xmlsec --no-binary=lxml
We already have pre-compile wheels for linux and macos (see release note: https://github.com/xmlsec/python-xmlsec/releases/tag/1.3.14) but I didn't upload them in pypi because of this version checking that was added in the latest release.
I'm gonna note this issue for the next release, meanwhile can you provide the following information:
I know the workaround, the problem is that there is no way to tell this inside dependencies. Instead, you need to tell users to not use pip install <package>
, but pip install --no-binary=lxml <package>
, what half of them won't do and complain that things do not work then :-).
I'm on Debian 12 (Python 3.11, libxml2 2.9.14, xmlsec1 1.2.37), it happens on GitHub CI environment with Ubuntu 22.04 (any Python version installed by actions/setup-python, libxml 2.9.13, xmlsec1 1.2.33).
PS: I'm happy user of https://github.com/pypa/cibuildwheel and https://github.com/pypa/gh-action-pypi-publish to automate all the wheel builds and publishing.
@nijel You're right, I put it in the todo list for the next release. Hope we can solve it better :)
@mxamin i like the idea of providing Linux wheels for the next release as well. We are already making the wheels with cibuildwheel, so it's just a matter of distributing them.
I can work on a PR to automate attaching the wheels to the release in GitHub and uploading them to PyPI if you would like.
@mxamin i like the idea of providing Linux wheels for the next release as well. We are already making the wheels with cibuildwheel, so it's just a matter of distributing them.
I can work on a PR to automate attaching the wheels to the release in GitHub and uploading them to PyPI if you would like.
I could have do it manually (simply downloading the artificats and uploading them to pypi) but I decided not to do it because of libxml2 version check. Assume that lxml wheel is build with version 2.9.x and python-xmlsec with 2.11.x, we would end up with the mismatch error which is not good. So the real problem here is to fix the lxml/python-xmlsec compatibility issue for real :)
Making the integration with lxml safer, like you are suggesting, would be a win for sure.
Even if we do that though, I think there is still a strong argument for providing wheels for Linux. It makes things much easier for end users of the package, especially when installing inside a container where you don't necessarily have all the build tooling available.
Definitely it would help. I wonder if the created wheels would help in reported cases. I probably check that later.
I was thinking if it is possible to build a private xmlsec1 against libxml2 from lxml and ship it inside the wheel.
@nijel would you be able to manually install one of the Linux wheels attached the the release in GitHub https://github.com/xmlsec/python-xmlsec/releases/tag/1.3.14 and see if it solves your issue?
The wheels are statically linked against libxml2 and libxmlsec, so I believe it will sort your issue. Those are the wheels we are discussing uploading to pypi.
Thanks, that seems to work locally just fine. Pushed that to CI now to see the outcome there.
@nijel yesterday afternoon, after some testing, @mxamin uploaded these same wheels to pypi. You should just be able to install xmlsec
normally in CI now.
See: https://github.com/xmlsec/python-xmlsec/issues/315#issuecomment-2064368226
I think that satisfies this ticket.
Is it possible that after this change, I get the lxml & xmlsec libxml2 library version mismatch
error with lxml
built locally? The CI started to fail for us meanwhile, so this might be the reason.
Maybe the version check probably can be skipped in case xmlsec is linked statically with the libxml?
Is it possible that after this change, I get the
lxml & xmlsec libxml2 library version mismatch
error withlxml
built locally? The CI started to fail for us meanwhile, so this might be the reason.
It looks like maybe you were installing lxml
with no-binary, but installing the binary wheel for xmlsec
. Now that we have wheels uploaded both lxml
and xmlsec
need to be built the same way. Either installed from wheels or built with no-binary.
Maybe the version check probably can be skipped in case xmlsec is linked statically with the libxml?
Because xmlsec
passes data into lxml
both need to be built with the same version of libxml2
. This situation is unfortunate, and we'd like to fix it, so this requirement doesn't exist. But for now it does.
This means that either they both need to be installed from wheels, which includes same version of libxml2
or they both need to be compiled using no-binary so they both get the system version of libxml2
. If you mix and match these approaches you will see version mismatch
error.
Thanks for explanation, it now works for me. Publishing binary wheels while I was testing this in CI made results for me quite unpredictable, but now with wheels-only setup, everything works just fine. Thanks for your effort to make this work!
I know this error has been added intentionally to catch other breakages, but is there a way to provide binary wheels that will work with current lxml? So that
pip install lxml xmlsec
will bring working combo.Presently, it doesn't:
The problem is that lxml wheels are built with internal libxml while xmlsec is using system one.
Rebuilding lxml against system libxml does address the issue, but working combo out of the box would be better.