Error importing xmlsec: null argument to internal routine

[jnorton001]

I running into some trouble when building and using a manylinux1 wheel (https://github.com/pypa/manylinux). The docker image uses an old distro, CentOS 5, so I can't use yum to install all the dependencies as the packages don't meet the minimum requirements for python-xmlsec.

So, I'm manually assembling most of the dependencies before building the wheel. I'm using openssl for libxmlsec's crypto library, e.g. ./configure --with-openssl=/usr/local/ssl && make && make install.

Everything builds just fine, but when I then pip install my wheel on my server and try to import xmlsec in python, I get this error:

jnorton$ pip install xmlsec-1.3.3-cp35-cp35m-manylinux1_x86_64.whl
jnorton$ python -c "import xmlsec"
func=xmlSecPtrListAdd:file=list.c:line=305:obj=unknown:subj=xmlSecPtrListIsValid(list):error=100:assertion:
func=xmlSecKeyDataIdsRegister:file=keysdata.c:line=110:obj=unknown:subj=xmlSecPtrListAdd:error=1:xmlsec library function failed:dataId=aes
func=xmlSecCryptoDLFunctionsRegisterKeyDataAndTransforms:file=dl.c:line=615:obj=aes:subj=xmlSecKeyDataIdsRegister:error=1:xmlsec library function failed:
func=xmlSecOpenSSLInit:file=crypto.c:line=340:obj=unknown:subj=xmlSecCryptoDLFunctionsRegisterKeyDataAndTransforms:error=1:xmlsec library function failed:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
SystemError: null argument to internal routine

Python version: 3.5.1 python-xmlsec version: 1.3.3 libxmlsec1 version: 1.2.25 libxml2 version: 2.9.8 libxslt version: 1.1.29 openssl version: 1.0.2o

jnorton$ pkg-config --cflags xmlsec1
-D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_NO_CRYPTO_DYNAMIC_LOADING=1 -DXMLSEC_OPENSSL_110=1 -DXMLSEC_CRYPTO_OPENSSL=1 -I/usr/include/libxml2 -I/usr/include/xmlsec1

Any tips for solving this? I realize this is a non-standard way of installing python-xmlsec, but it'd be great to find a way to make this work.

Side-note: I have gotten this wheel building to work with python-xmlsec 0.6.0 by using an older version of libxmlsec1, but I'd like to use the newest version of python-xmlsec if possible.

Also, if it's helpful, here's the libraries that get packaged in the xmlsec wheel:

jnorton$ ls venv/lib/python3.5/site-packages/.libsxmlsec/
libltdl-c1a91d42.so.7.3.1  libxml2-ef3ec899.so.2.9.8  libxmlsec1-177bee2c.so.1.2.25  libxslt-bcf758f0.so.1.1.29  libz-a147dcb0.so.1.2.3

[sivictorallen]

I'm having a similar issue on Alpine, where an earlier version of the Docker image with libxmlsec 1.2.22 works fine, but 1.2.25 results in this same error (both in python-xmlsec 1.3.3). Not sure if this helps you or not.

[ghost]

@sivictorallen I'm having a similar issue on Alpine, where an earlier version of the Docker image with libxmlsec 1.2.22 works fine, but 1.2.25 results in this same error (both in python-xmlsec 1.3.3). Not sure if this helps you or not.

I investigated further, on alpine it is the Alpine 3.8 version (libxmlsec-1.2.25r2) versus r1 on Alpine 3.7 that causes the problem. So if you need it: alpine 3.7 + python-xmlsec-1.3.3 works.

[xaralis]

Is there any ETA for fix? This affects Alpine 3.8 which is a serious problem due to its widespread use in docker environments.

[bgaifullin]

there is problem with xmlsec1 on linux alpine. report an issue for xmlsec1 https://github.com/lsh123/xmlsec/issues/244

as workaround you can build xmlsec1 with static crypto library, this should help

[bgaifullin]

there was error with exception handling during load module, I fix it. https://github.com/mehcode/python-xmlsec/pull/94

[bgaifullin]

seem like there is no package xmlsec1-openssl for alpine

[alexa-infra]

~There is a fix for Alpine 3.9: apk add libcrypto1.1~ Just use Alpine 3.9. It looks like libxmlsec1-openssl.so is built with different openssl/libressl version (alpine is shipped with libressl2.7-libcrypto openssl version 1.0.0). To see the issue run ldd /usr/lib/libxmlsec1-openssl.so and see missing symbols. ~libcrypto1.1 (openssl version 1.1.1) from alpine repositories fixes this issue.~ Another way would be to build libressl/openssl from sources for alpine:3.8..

[caleb15]

I got this too when installing the necessary apt packages via brew :(

brew install libxml2 libxmlsec1 pkg-config

I already installed them normally via apt but that didn't work (xmlsec not found) hence why I tried brew. But that doesn't work either. Yikes.

OS: ubuntu 18

[caleb15]

Gave another shot at fixing it and failed. Brew installs libxmlsec1@1.2.29 but apt installs libxmlsec1-dev=1.2.25. Maybe that's the issue? Unfortunately brew doesn't have a earlier version of xmlsec to install.

It works in docker so I'll just use that I guess.

[caleb15]

I tried uninstall the apt xmlsec, installing xmlsec via brew, then reinstalling xmlsec python package with --no-cache-dir but I got:

└─[$] python -c "import xmlsec"                                                                                                                                                          [16:15:00]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libxmlsec1-openssl.so.1: cannot open shared object file: No such file or directory

I uninstalled brew xmlsec, brew remove libxmlsec1, installed xmlsec via apt, sudo apt install libxmlsec1-dev libxmlsec1-openssl xmlsec1, and finally reinstalled the xmlsec python package but when reinstalling I got

└─[$] pip install xmlsec==1.3.3 --no-cache-dir                                                                                                                                           [16:20:29]
Collecting xmlsec==1.3.3
  Downloading xmlsec-1.3.3.tar.gz (29 kB)
Requirement already satisfied: pkgconfig in /home/caleb/.local/lib/python3.6/site-packages (from xmlsec==1.3.3) (1.5.1)
Requirement already satisfied: lxml>=3.0 in /home/caleb/.pyenv/versions/3.6.8/lib/python3.6/site-packages (from xmlsec==1.3.3) (4.3.3)
Installing collected packages: xmlsec
    Running setup.py install for xmlsec ... error
    ERROR: Command errored out with exit status 1:
     command: /home/caleb/.pyenv/versions/3.6.8/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bx8hajvs/xmlsec/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bx8hajvs/xmlsec/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qb5x45l1/install-record.txt --single-version-externally-managed --compile --install-headers /home/caleb/.pyenv/versions/3.6.8/include/python3.6m/xmlsec
         cwd: /tmp/pip-install-bx8hajvs/xmlsec/
    Complete output (44 lines):
    running install
    running build
    running build_ext
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-bx8hajvs/xmlsec/setup.py", line 60, in <module>
        'Topic :: Text Processing :: Markup :: XML'
      File "/home/caleb/.local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/caleb/.local/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/caleb/.pyenv/versions/3.6.8/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-bx8hajvs/xmlsec/setup.py", line 11, in run
        self.patch_options()
      File "/tmp/pip-install-bx8hajvs/xmlsec/setup.py", line 16, in patch_options
        ext.define_macros.extend(xmlsec_setupinfo.define_macros())
      File "/tmp/pip-install-bx8hajvs/xmlsec/xmlsec_setupinfo.py", line 48, in define_macros
        macros.extend(xmlsec_config()['define_macros'])
      File "/tmp/pip-install-bx8hajvs/xmlsec/xmlsec_setupinfo.py", line 111, in xmlsec_config
        __XMLSEC_CONFIG = load_xmlsec1_config()
      File "/tmp/pip-install-bx8hajvs/xmlsec/xmlsec_setupinfo.py", line 164, in load_xmlsec1_config
        config = pkgconfig.parse('xmlsec1')
      File "/home/caleb/.local/lib/python3.6/site-packages/pkgconfig/pkgconfig.py", line 248, in parse
        _raise_if_not_exists(package)
      File "/home/caleb/.local/lib/python3.6/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
        raise PackageNotFoundError(package)
    pkgconfig.pkgconfig.PackageNotFoundError: xmlsec1 not found
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/caleb/.pyenv/versions/3.6.8/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bx8hajvs/xmlsec/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bx8hajvs/xmlsec/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qb5x45l1/install-record.txt --single-version-externally-managed --compile --install-headers /home/caleb/.pyenv/versions/3.6.8/include/python3.6m/xmlsec Check the logs for full command output.

I debugged pkgconfig and it calls pkg-config --exists xmlsec1. I checked pkg-config -h and found a option --print-errors so I ran pkg-config --exists xmlsec1 --print-errors and got the following:

Package xmlsec1 was not found in the pkg-config search path.
Perhaps you should add the directory containing `xmlsec1.pc'
to the PKG_CONFIG_PATH environment variable
No package 'xmlsec1' found

I checked the version of xmlsec with apt policy pkg-config and found that the installed version was a different version than what pkg-config --version reported on the command line. That made me suspicous so I ran which pkg-config and it was the linuxbrew pkg-config! After some googling I found that you can check the pkg config search paths by running pkg-config --variable pc_path pkg-config.

I got the right search paths by running /usr/bin/pkg-config --variable pc_path pkg-config, exported that as PKG_CONFIG_PATH var, and then it worked! :D

# https://askubuntu.com/questions/210210/pkg-config-path-environment-variable
# This is necessary so pkg-config can find xmlsec
export PKG_CONFIG_PATH=/usr/local/lib/x86_64-linux-gnu/pkgconfig:/usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig

TL;DR: Don't use brew for installing xmlsec. If you get a xmlsec1 not found error and have pkg-config installed via brew you can alter the pkg-config search paths with PKG_CONFIG_PATH

[hoefling]

Since 1.3.8, you can build PEP 571-compliant static wheels (manylinux2010) that don't require libxml2 or libxmlsec1 installed. Use docker to build them:

$ mkdir io
$ docker run --rm -v$(pwd)/io:/io -it quay.io/pypa/manylinux2010_x86_64
# export STATIC_DEPS=1
# /opt/python/cp38-cp38/bin/pip wheel xmlsec
# auditwheel repair xmlsec-1.3.8-cp38-cp38-linux_x86_64.whl
# exit

You should now have a wheel ./io/wheelhouse/xmlsec-1.3.8-cp38-cp38-manylinux2010_x86_64.whl on your host. Installing it for Python 3.8:

$ pip3.8 install xmlsec --find-links=/io/wheelhouse

For different interpreter target, adjust the path to pip when in the manylinux2010 container.

By default, the following library versions are used for building:

Library	Version	Environment variable
OpenSSL	1.1.1g	OPENSSL_VERSION
zlib	1.2.11	ZLIB_VERSION
libiconv	1.16	LIBICONV_VERSION
libxml2	latest (currently the 2.9.10)	LIBXML2_VERSION
libxslt	latest (currently the 1.1.34)	LIBXLST_VERSION
xmlsec1	1.2.30	XMLSEC1_VERSION

If you need to change the library version, use the environment variables listed in the above table.

When building from the current master, prepend PYMXLSEC_ to env var names, e.g. PYXMLSEC_STATIC_DEPS=1, PYXMLSEC_OPENSSL_VERSION="1.1.1g" etc.

This should resolve the issue, thus closing. Feel free to ping me if you think this should be reopened, though!