Getting Segfault issue when login using SAML

[ahujarajesh]

We are using onelogin for saml login via okta, recently we figured out after updating lxml == 5.0.0 we are seeing segmentation fault and python application crashes.

After downgrading version of lxml to 4.9.3 this issue resolves.

Here are the traces for your reference.

System Info:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

Python 3.10.12
python3-saml==1.16.0

dmesg logs:

Jan  2 15:07:25 ip-172-28-5-29 kernel: python3.10[14382]: segfault at c ip 00007f1d9751cce0 sp 00007f1d7a7f7c70 error 4 in libxml2.so.2.9.13[7f1d974d7000+153000] likely on CPU 3 (core 1, socket 0)
Jan  2 15:07:25 ip-172-28-5-29 kernel: Code: e0 04 49 03 45 00 8b 50 28 85 d2 0f 84 ec 00 00 00 49 83 7d 10 00 74 6c 48 89 c2 eb 0d 0f 1f 44 00 00 48 8b 12 48 85 d2 74 5a <48> 39 5a 08 75 f2 48 39 6a 10 75 ec 4c 39 62 18 75 e6 48 8b 42 20
Jan  2 15:07:25 ip-172-28-5-29 systemd[1]: webui.service: Main process exited, code=dumped, status=11/SEGV
Jan  2 15:07:25 ip-172-28-5-29 systemd[1]: webui.service: Failed with result 'core-dump'.
Jan  2 15:07:25 ip-172-28-5-29 systemd[1]: webui.service: Consumed 2min 48.996s CPU time.

PYTHONFAULTHANDLER=1:

Thread 0x00007f416effd640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/process/plugins.py", line 514 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f416f7fe640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f416ffff640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f418cff9640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f418d7fa640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f418dffb640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f418e7fc640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f418effd640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait 
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00007f418f7fe640 (most recent call first): 
  File "/usr/local/lib/python3.10/dist-packages/onelogin/saml2/utils.py", line 826 in validate_sign
  File "/usr/local/lib/python3.10/dist-packages/onelogin/saml2/utils.py", line 48 in exceptfalse
  File "/usr/local/lib/python3.10/dist-packages/onelogin/saml2/response.py", line 305 in is_valid
  File "/usr/local/lib/python3.10/dist-packages/onelogin/saml2/auth.py", line 131 in process_response
  File "/efs/ubuntu/App/trunk/App/src/modsaml.py", line 29 in consume
  File "/home/ubuntu/App/trunk/App/src/webui.py", line 291 in wrsaml_consume
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpdispatch.py", line 54 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/lib/encoding.py", line 223 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 697 in _do_respond
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 638 in respond
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 604 in run
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 335 in run
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 236 in __init__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 423 in tail
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 104 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 184 in trap
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 166 in __init__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 148 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cpwsgi.py", line 435 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cptree.py", line 168 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cptree.py", line 302 in __call__
  File "/usr/local/lib/python3.10/dist-packages/cheroot/wsgi.py", line 136 in respond
  File "/usr/local/lib/python3.10/dist-packages/cheroot/server.py", line 1077 in respond
  File "/usr/local/lib/python3.10/dist-packages/cheroot/server.py", line 1287 in communicate
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 120 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f418ffff640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f419cbfd640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/usr/local/lib/python3.10/dist-packages/cheroot/workers/threadpool.py", line 110 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f419d3fe640 (most recent call first):
  File "/usr/lib/python3.10/selectors.py", line 469 in select
  File "/usr/local/lib/python3.10/dist-packages/cheroot/connections.py", line 102 in select
  File "/usr/local/lib/python3.10/dist-packages/cheroot/connections.py", line 233 in _run
  File "/usr/local/lib/python3.10/dist-packages/cheroot/connections.py", line 198 in run
  File "/usr/local/lib/python3.10/dist-packages/cheroot/server.py", line 1814 in serve
  File "/usr/local/lib/python3.10/dist-packages/cheroot/server.py", line 1841 in start
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/process/servers.py", line 225 in _start_http_thread
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f419dbff640 (most recent call first):
  File "/home/ubuntu/App/trunk/App/src/webui.py", line 728 in _log_compact_thread
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f41a413f1c0 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/process/wspbus.py", line 376 in wait
  File "/usr/local/lib/python3.10/dist-packages/cherrypy/process/wspbus.py", line 336 in block
  File "/home/ubuntu/App/trunk/App/src/webui.py", line 254 in main
  File "/home/ubuntu/App/trunk/App/src/webui.py", line 1046 in <module>

Extension modules: lxml._elementpath, lxml.etree, xmlsec, charset_normalizer.md, _cffi_backend, MySQLdb._mysql (total: 6)
Segmentation fault (core dumped)

GDB:

Thread 5 "python3.10" received signal SIGBUS, Bus error.
[Switching to Thread 0x7ffff09fd640 (LWP 18581)]
0x00007ffff61e6bf0 in xmlDictLookup () from /lib/x86_64-linux-gnu/libxml2.so.2
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) 
Continuing.
Couldn't get registers: No such process.
(gdb) py-bt
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb)
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) 
Unable to locate python frame
Couldn't get registers: No such process.
(gdb) py-bt[Thread 0x7fffc2ffd640 (LWP 18606) exited]
[Thread 0x7fffc37fe640 (LWP 18590) exited]
[Thread 0x7fffc3fff640 (LWP 18589) exited]
[Thread 0x7fffe0ff9640 (LWP 18588) exited]
[Thread 0x7fffe17fa640 (LWP 18587) exited]
[Thread 0x7fffe1ffb640 (LWP 18586) exited]
[Thread 0x7fffe27fc640 (LWP 18585) exited]
[Thread 0x7fffe2ffd640 (LWP 18584) exited]
[Thread 0x7fffe37fe640 (LWP 18583) exited]
[Thread 0x7fffe3fff640 (LWP 18582) exited]
[Thread 0x7ffff09fd640 (LWP 18581) exited]
[Thread 0x7ffff11fe640 (LWP 18580) exited]
[Thread 0x7ffff7e7d1c0 (LWP 18483) exited]
[Thread 0x7ffff19ff640 (LWP 18487) exited]
[New process 18483]

[mayormaier]

We are having the same issue, +1

[CTV-2023]

Same issue with gunicorn on Netbox https://github.com/netbox-community/netbox/discussions/14649

[paulzakin]

Can confirm, and also that downgrading solved our problem

[eljeffeg]

Me too - wish I would have seen this solution sooner. I created an issue on https://github.com/xmlsec/python-xmlsec/issues/277 to address the dependency chain.

[johnnymetz]

+1

[taesungh]

I provided a minimal example in #388 and concluded my issue was fixed by pinning lxml<5 or installing lxml not from binary as noted in the project README from #323, although that increased the installation time by a few minutes.

pip install python3-saml --no-binary lxml

[nosnilmot]

The correct solution is not to downgrade lxml! Use pip install --no-binary lxml lxml to install lxml from source and link to system libxml2, same as xmlsec, to avoid conflicting libxml2 libraries. See also https://bugs.launchpad.net/lxml/+bug/1960668#yui_3_10_3_1_1705879387759_1639 and https://github.com/lxml/lxml/blob/master/doc/FAQ.txt#L623 and https://github.com/SAML-Toolkits/python3-saml?tab=readme-ov-file#note

[paulzakin]

Yup, @nosnilmot is correct - we switched from downgrading lxml to doing a no binary install, which was a better way to solve this problem!

[vinodpandey]

Installing lxml from source didn't fix the error for us. The error occurs 30-40% for the time. Rest of the time, the SAML login works as expected.

Below is our configuration

Macbook Pro Apple M2 Max 

Python 3.10.12

python3-saml==1.16.0
lxml==4.9.3

I have created a demo project to replicate this issue: https://github.com/vinodpandey/saml-allauth-demo. I have uploaded video at https://youtu.be/NV2FokoH1zI which shows the above repository setup steps and replicates the segfault issue.

In above same project, I ran below commands and tested it again. But, even after that, the issue occurs approx. 40-50% times.

pip uninstall lxml
pip install --no-binary lxml lxml==4.9.3

Error log

Fatal Python error: Segmentation fault

Current thread 0x000000030bc96000 (most recent call first):
  File "/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54 in wrapped_view
  File "/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 46 in _wrapper
  File "/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 84 in view
  File "/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197 in _get_response
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/cachalot/panels.py", line 41 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/logging.py", line 77 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/staticfiles.py", line 117 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/headers.py", line 50 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/timer.py", line 65 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/panels/__init__.py", line 193 in process_request
  File "/venv/lib/python3.10/site-packages/debug_toolbar/middleware.py", line 57 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/allauth/account/middleware.py", line 14 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/apps/util/middleware.py", line 48 in __call__
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 140 in get_response
  File "/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 132 in __call__
  File "/venv/lib/python3.10/site-packages/django/contrib/staticfiles/handlers.py", line 80 in __call__
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/wsgiref/handlers.py", line 137 in run
  File "/venv/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 227 in handle_one_request
  File "/venv/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 202 in handle
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 747 in __init__
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 360 in finish_request
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 683 in process_request_thread
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953 in run
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000030ac93000 (most recent call first):
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/selectors.py", line 416 in select
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/socketserver.py", line 232 in serve_forever
  File "/venv/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 246 in run
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 158 in inner_run
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 64 in wrapper
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953 in run
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/vinodpandey/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00000002044a82c0 (most recent call first):
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 405 in tick
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 350 in run_loop
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 344 in run
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 661 in start_django
  File "/venv/lib/python3.10/site-packages/django/utils/autoreload.py", line 680 in run_with_reloader
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 118 in run
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 111 in handle
  File "/venv/lib/python3.10/site-packages/django/core/management/base.py", line 460 in execute
  File "/venv/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 74 in execute
  File "/venv/lib/python3.10/site-packages/django/core/management/base.py", line 414 in run_from_argv
  File "/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440 in execute
  File "/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446 in execute_from_command_line
  File "/manage.py", line 39 in <module>

Extension modules: yaml._yaml, charset_normalizer.md, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, MySQLdb._mysql, markupsafe._speedups, lxml._elementpath, lxml.etree, PIL._imaging, _cffi_backend, xmlsec (total: 22)

[nosnilmot]

As discussed extensively here, @vinodpandey's problem is also caused by mismatch between libxml2 versions, albeit with the added twist of macos system & brew-installed libraries adding another variable.

[blazefury-fire]

confim, i will down version lxml from 5. -> 4.9.2, and it worked

[nosnilmot]

At the risk of sounding like a broken record:

The correct solution is not to downgrade lxml! Use pip install --no-binary lxml lxml to install lxml from source and link to the same libxml2 as xmlsec, to avoid conflicting libxml2 libraries. See also https://bugs.launchpad.net/lxml/+bug/1960668 and https://lxml.de/FAQ.html#my-application-crashes and https://github.com/SAML-Toolkits/python3-saml?tab=readme-ov-file#note

On macos you will need to make sure libxml2 from brew is linked, not the system libxml2, by setting PKG_CONFIG_PATH appropriately.

[ryanhiebert]

Because xmlsec also depends on lxml, this feels like a problem that xmlsec should resolve. It seems that there is some activity on the xmlsec repo toward this: https://github.com/xmlsec/python-xmlsec/issues/283, though it's not clear whether xmlsec is maintained well enough to address this or whether it will need to be forked.

[pitbulk]

@ryanhiebert I have good news, the maintainers of the project started to work again on python-xmlsec, and the release of a new version is pretty close.

See https://github.com/xmlsec/python-xmlsec/issues/284

[pencil]

While we wait for the bug to be fixed upstream, this is what worked for me in requirements.txt:

python3-saml==1.16.0

lxml==5.1.0
--no-binary=lxml
xmlsec==1.3.13
--no-binary=xmlsec

[CymeriaHub]

Dear All,

Even if i set : python3-saml==1.16.0

lxml==5.1.0 --no-binary=lxml xmlsec==1.3.13 --no-binary=xmlsec

as mentionned in the previous post, my server partially crashed after calling the IDP, the IDP login windows is displayed but all the session are lost (for all user), so the users are no more identified. (but the server is still running). As i am behind "Apache passenger" it is difficult to get the exact trace.

Somebody has the same behavior?

[vdboor]

Looks like xmlsec 1.3.14 addressed this issue!

[fsargent]

I'm still having this issue even with xmlsec 1.3.14, and lxml 5.2.1 installed. Am I missing something?

Debug memory block at address p=0xfffd280104d0: API '%'
 2305843009213693952 bytes originally requested
 The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
     at p-7: 0x00 *** OUCH
     at p-6: 0x00 *** OUCH
     at p-5: 0x00 *** OUCH
     at p-4: 0x00 *** OUCH
     at p-3: 0x00 *** OUCH
     at p-2: 0x00 *** OUCH
     at p-1: 0x00 *** OUCH
 Because memory is corrupted at the start, the count of bytes requested
    may be bogus, and checking the trailing pad bytes may segfault.
 The 8 pad bytes at tail=0x2000fffd280104d0 are not all FORBIDDENBYTE (0xfd):
     at tail+0: 0x90 *** OUCH
     at tail+1: 0x12 *** OUCH
     at tail+2: 0x06 *** OUCH
     at tail+3: 0x86 *** OUCH
     at tail+4: 0xff *** OUCH
     at tail+5: 0xff *** OUCH
     at tail+6: 0x00 *** OUCH
     at tail+7: 0x00 *** OUCH
 Data at p: 90 12 06 86 ff ff 00 00 ... 25 00 00 00 00 00 00 00

Enable tracemalloc to get the memory block allocation traceback

Fatal Python error: _PyMem_DebugRawFree: bad ID: Allocated using API '%', verified using API 'm'
Python runtime state: initialized

Current thread 0x0000ffff5e41e1a0 (most recent call first):
File "/usr/local/lib/python3.11/site-packages/onelogin/saml2/utils.py", line 826 in validate_sign
File "/usr/local/lib/python3.11/site-packages/onelogin/saml2/utils.py", line 48 in exceptfalse
File "/usr/local/lib/python3.11/site-packages/onelogin/saml2/response.py", line 312 in is_valid
File "/usr/local/lib/python3.11/site-packages/onelogin/saml2/auth.py", line 131 in process_response
File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/saml/views.py", line 58 in dispatch
File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56 in wrapper_view
File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46 in _wrapper
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104 in view
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197 in _get_response
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/allauth/account/middleware.py", line 34 in middleware
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django_structlog/middlewares/request.py", line 73 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/app/app/middleware.py", line 188 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/app/app/middleware.py", line 147 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/app/app/middleware.py", line 75 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/oauth2_provider/middleware.py", line 36 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/app/app/middleware.py", line 67 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/whitenoise/middleware.py", line 124 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 140 in get_response
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/wsgi.py", line 124 in __call__
File "/usr/local/lib/python3.11/wsgiref/handlers.py", line 137 in run
File "/usr/local/lib/python3.11/site-packages/django/core/servers/basehttp.py", line 252 in handle_one_request
File "/usr/local/lib/python3.11/site-packages/django/core/servers/basehttp.py", line 227 in handle
File "/usr/local/lib/python3.11/socketserver.py", line 755 in __init__
File "/usr/local/lib/python3.11/socketserver.py", line 361 in finish_request
File "/usr/local/lib/python3.11/socketserver.py", line 691 in process_request_thread
File "/usr/local/lib/python3.11/threading.py", line 982 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 1118 in __call__

Thread 0x0000ffff65b9d1a0 (most recent call first):
File "/usr/local/lib/python3.11/socket.py", line 706 in readinto
File "/usr/local/lib/python3.11/site-packages/django/core/servers/basehttp.py", line 237 in handle_one_request
File "/usr/local/lib/python3.11/site-packages/django/core/servers/basehttp.py", line 229 in handle
File "/usr/local/lib/python3.11/socketserver.py", line 755 in __init__
File "/usr/local/lib/python3.11/socketserver.py", line 361 in finish_request
File "/usr/local/lib/python3.11/socketserver.py", line 691 in process_request_thread
File "/usr/local/lib/python3.11/threading.py", line 982 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 1118 in __call__

Thread 0x0000ffff665ee1a0 (most recent call first):
File "/usr/local/lib/python3.11/selectors.py", line 415 in select
File "/usr/local/lib/python3.11/socketserver.py", line 233 in serve_forever
File "/usr/local/lib/python3.11/site-packages/django/core/servers/basehttp.py", line 281 in run
File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 140 in inner_run
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 64 in wrapper
File "/usr/local/lib/python3.11/threading.py", line 982 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 1118 in __call__

Thread 0x0000ffff66dfe1a0 (most recent call first):
File "/usr/local/lib/python3.11/threading.py", line 331 in wait
File "/usr/local/lib/python3.11/threading.py", line 629 in wait
File "/usr/local/lib/python3.11/site-packages/devcycle_python_sdk/managers/event_queue_manager.py", line 75 in _sleep
File "/usr/local/lib/python3.11/site-packages/devcycle_python_sdk/managers/event_queue_manager.py", line 149 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 1118 in __call__

Thread 0x0000ffff6760e1a0 (most recent call first):
File "/usr/local/lib/python3.11/site-packages/devcycle_python_sdk/managers/config_manager.py", line 96 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 1118 in __call__

Thread 0x0000ffff811211a0 (most recent call first):
File "/usr/local/lib/python3.11/threading.py", line 331 in wait
File "/usr/local/lib/python3.11/threading.py", line 629 in wait
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 261 in _on_run
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x0000ffff819311a0 (most recent call first):
File "/usr/local/lib/python3.11/threading.py", line 331 in wait
File "/usr/local/lib/python3.11/threading.py", line 629 in wait
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd.py", line 215 in _on_run
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x0000ffff821411a0 (most recent call first):
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 204 in _read_line
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 222 in _on_run
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x0000ffff829511a0 (most recent call first):
File "/usr/local/lib/python3.11/threading.py", line 331 in wait
File "/usr/local/lib/python3.11/queue.py", line 180 in get
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 368 in _on_run
File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x0000ffff88259020 (most recent call first):
File "<frozen posixpath>", line 451 in _joinrealpath
File "<frozen posixpath>", line 416 in realpath
File "/usr/local/lib/python3.11/pathlib.py", line 993 in resolve
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 168 in iter_modules_and_files
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 120 in iter_all_python_module_files
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 304 in watched_files
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 411 in snapshot_files
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 390 in tick
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 350 in run_loop
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 344 in run
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 660 in start_django
File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 671 in run_with_reloader
File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 118 in run
File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 111 in handle
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458 in execute
File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 74 in execute
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412 in run_from_argv
File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436 in execute
File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442 in execute_from_command_line
File "/app/manage.py", line 10 in <module>

Extension modules: yaml._yaml, lxml._elementpath, lxml.etree, charset_normalizer.md, psycopg2._psycopg, _cffi_backend, google._upb._message, xmlsec (total: 8)
Frontend with code 250

[pencil]

Not sure if it helps, but here is the combination of dependencies I'm now using with Python 3.12.3:

python3-saml==1.16.0
lxml==5.2.1
xmlsec==1.3.14

I no longer get any crashes and everything seems to work as expected (tested with Mock SAML).

I ran pip uninstall -y python3-saml lxml xmlsec before running pip install -r requirements.txt but not sure if that actually made a difference.

[ryanhiebert]

but not sure if that actually made a difference

fwiw, yes, it makes a difference. If it's already installed lxml with --no-binary, unless you remove it first it won't reinstall it from a binary. At least that made the difference for me when I did this a few days ago.

[fsargent]

Here's my docker configuration. I believe I'm using all the versions that everyone is recommending. https://gist.github.com/fsargent/4375eb4a99f4b902184a6a7412f72de7 Do I need to pin my apt versions of libxml2-dev also? This is running on the python 3.8 docker image based on ubuntu-bookworm.

I've managed to kind of get it working with Python 3.12, running locally on my M1 mac (not in docker), but even then if I refresh the .../acs/ page then it'll segfault in OS X.

This is the command where it's segfaulting: xmlsec.tree.add_ids(elem, ["ID"])

[jeremyschiff]

If none of the above fixes work, make sure that you don't have the parameter export PYTHONDEVMODE=1 enabled. It will cause the same issue.