Segmentation fault on `xmlsec.tree.add_ids()` in Python Development Mode

[Anthchirp]

boom.py:

from lxml import etree

import xmlsec

template = etree.parse('sign1-res.xml').getroot()
xmlsec.tree.add_ids(template, ["ID"])

with sign-res.xml coming from the xmlsec examples.

Expected output

$ python boom.py
$

Observed output

$ PYTHONDEVMODE=1 python boom.py
Debug memory block at address p=0x5556aed07c40: API '!'
    15987178197214944733 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x00 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0x00 *** OUCH
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0xddde33348cae5a1d are Fatal Python error: Segmentation fault

Current thread 0x00007fb7df1d8000 (most recent call first):
  File "boom.py", line 6 in <module>

Extension modules: lxml._elementpath, lxml.etree, xmlsec (total: 3)
Segmentation fault
$

Environment

I'm running on Ubuntu 22.04 with system python 3.10.6 and xmlsec 1.3.13.

pip list

Package            Version
------------------ -------------
blinker            1.4
chardet            4.0.0
cryptography       3.4.8
devscripts         2.22.1ubuntu1
dh-virtualenv      1.2.2
distro             1.7.0
httplib2           0.20.2
importlib-metadata 4.6.4
isodate            0.6.1
jeepney            0.7.1
keyring            23.5.0
launchpadlib       1.10.16
lazr.restfulclient 0.14.4
lazr.uri           1.0.6
lxml               4.9.2
more-itertools     8.10.0
netifaces          0.11.0
oauthlib           3.2.0
pip                22.0.2
PyGObject          3.42.1
PyJWT              2.3.0
pyparsing          2.4.7
python-apt         2.4.0+ubuntu1
python-debian      0.1.43ubuntu1
python3-saml       1.15.0
SecretStorage      3.3.1
setuptools         59.6.0
six                1.16.0
supervisor         4.2.1
VapourSynth        54
wadllib            1.3.6
wheel              0.37.1
xmlsec             1.3.13
zipp               1.0.0

apt list | grep xmlsec | grep installed

libxmlsec1-dev/jammy,now 1.2.33-1build2 amd64 [installed]
libxmlsec1-gcrypt/jammy,now 1.2.33-1build2 amd64 [installed,automatic]
libxmlsec1-gnutls/jammy,now 1.2.33-1build2 amd64 [installed,automatic]
libxmlsec1-nss/jammy,now 1.2.33-1build2 amd64 [installed,automatic]
libxmlsec1-openssl/jammy,now 1.2.33-1build2 amd64 [installed,automatic]
libxmlsec1/jammy,now 1.2.33-1build2 amd64 [installed,automatic]

[Anthchirp]

I checked whether this is an lxml <-> xmlsec issue, but it is not:

$ pip install --no-binary :all: --force-reinstall lxml
$ PYTHONDEVMODE=1 python boom.py
(...)
Segmentation fault

[jlthorel]

Hello just had a similar problem with xmlsec.tree.add_ids program crash on this line without any error message I solve the issue by falling back to lxml==4.9.3 instead of lxml==5.1.0

Brgds JL

[seda3091]

@jlthorel You saved my life...!! :thumbsup:

[downpat]

This fixed a problem for me too. It was causing a SAML SSO login failure for me. Has anyone submitted a bug to lxml? Their bug tracker is here: https://launchpad.net/lxml

[MaylinJeong]

@jlthorel You saved my life as well..! 💯

[rhenanbartels]

@jlthorel You're a life saver!! Thank you very much!🎉🎉 I had the same problem as @downpa and my SAML login was failing because of lxml library

[jake-lester]

@jlthorel thank you for saving us as well! You are the best!!! We had issues recently with SSO on heroku where our ACS was failing. We isolated it to python3-saml validate_sign --> xmlsec xmlsec.tree.add_ids(elem, ["ID"]) ... thank you ❤️

[Daviazuos]

@jlthorel You saved my life!!

[vpatov]

We had the same issue with our SAML SSO failing due to a seg fault, downgrading lxml resolved the issue

[quique]

Hello just had a similar problem with xmlsec.tree.add_ids program crash on this line without any error message I solve the issue by falling back to lxml==4.9.3 instead of lxml==5.1.0

OMG, this was driving me crazy: my SAML SSO was failing most of the times, but sometimes it did work 😠 . Downgrading xml fixed the issue :-) Thanks a lot for the tip!!!

[quique]

This fixed a problem for me too. It was causing a SAML SSO login failure for me. Has anyone submitted a bug to lxml? Their bug tracker is here: https://launchpad.net/lxml

I couldn't find any bug report about this, so I just submitted one: https://bugs.launchpad.net/lxml/+bug/2054606

[andpena]

Hello just had a similar problem with xmlsec.tree.add_ids program crash on this line without any error message I solve the issue by falling back to lxml==4.9.3 instead of lxml==5.1.0

Brgds JL

Thank you very much! This solution worked for me too.

[quique]

This fixed a problem for me too. It was causing a SAML SSO login failure for me. Has anyone submitted a bug to lxml? Their bug tracker is here: https://launchpad.net/lxml

I couldn't find any bug report about this, so I just submitted one: https://bugs.launchpad.net/lxml/+bug/2054606

The folks at lxml closed the report. They consider it is a third party issue (ie, that the problem lies in xmlsec).

[quique]

I believe commit ee9fbd03ac4c466447f0f67e3032866657dcb19d by @jonathangreen avoids the segfault without any error message, and emits instead an exception ("lxml & xmlsec libxml2 library version mismatch").

However, it's still not clear to me what we should do to avoid it.

[jonathangreen]

Hi @quique. My comment here gives some context https://github.com/xmlsec/python-xmlsec/issues/316#issuecomment-2066350808 that might be helpful.

Basically to avoid the issue, either both lxml and xmlsec should be installed from wheels, or should both be installed with --no-binary set.

The most recent release for xmlsec supplies wheels for most platforms, so updating to the most recent and making sure that it it installed from via a wheel is probably the easiest fix.

[quique]

@jonathangreen Thank you for your helpful comment!

Unfortunately it seems I'm quite thick today, because I am unable to make it work.

This is what I do:

docker run -it python:3.12-slim-bookworm /bin/bash
# Place sign1-res.xml and boom.py inside the container (see the issue report by @Anthchirp )
pip install lxml --only-binary=lxml
pip install xmlsec --only-binary=xmlsec
PYTHONDEVMODE=1 python boom.py  # See it crash

If I understand it correctly, this way both lxml and xmlsec are installed from wheels, but I stil get a segmentation fault. This is what I see:

root@d199d2820803:/# pip install lxml --only-binary=lxml
Collecting lxml
  Downloading lxml-5.2.1-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.4 kB)
Downloading lxml-5.2.1-cp312-cp312-manylinux_2_28_x86_64.whl (4.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 7.5 MB/s eta 0:00:00
Installing collected packages: lxml
Successfully installed lxml-5.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@d199d2820803:/# pip install xmlsec --only-binary=xmlsec
Collecting xmlsec
  Downloading xmlsec-1.3.14-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Requirement already satisfied: lxml>=3.8 in /usr/local/lib/python3.12/site-packages (from xmlsec) (5.2.1)
Downloading xmlsec-1.3.14-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.9/3.9 MB 7.2 MB/s eta 0:00:00
Installing collected packages: xmlsec
Successfully installed xmlsec-1.3.14
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@d199d2820803:/# python boom.py 
root@d199d2820803:/# PYTHONDEVMODE=1 python boom.py
Debug memory block at address p=0x56499b02ca70: API '!'
    6873730420813529088 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x00 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0x00 *** OUCH
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0x5f64bbafc965ca70 are Fatal Python error: Segmentation fault

Current thread 0x00007fb0f0a63740 (most recent call first):
  File "//boom.py", line 6 in <module>

Extension modules: lxml._elementpath, lxml.etree, xmlsec (total: 3)
Segmentation fault (core dumped)

Any hint? Thank you!