search

LudovicRousseau / PyKCS11

PKCS#11 Wrapper for Python
GNU General Public License v2.0
96 stars 35 forks source link

PyKCS11 in error when installing on Ubuntu 24.04 (Noble) #107

Closed SixK closed 8 months ago

SixK commented 8 months ago

PyKCS11 seem's to not install on Ubuntu 24.04 (Noble) anymore (seem's it was working about a week ago)

The problem can be reproduced using Ubuntu 24.04 docker image like this:

docker pull ubuntu:24.04
docker run -it ubuntu:24.04 bash
apt update
apt install python3 python3-pip swig
pip install PyKCS11 --break-system-packages

Note that v1.5.0 also finish with the same error: pip install PyKCS11==1.5.0 --break-system-packages --verbose But this command work: pip install PyKCS11==1.4.4 --break-system-packages --verbose

SWIG library version is 4.2.0 on Ubuntu 24.04. PyKCS11 installation work with Ubuntu 22.04 (Swig 4.0.2) and Debian Bookworm (Swig 4.1.0)

Problem seem's to be related to this variable: SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t

Corresponding error seem's to be in the following file near line 281: https://github.com/LudovicRousseau/PyKCS11/blob/master/src/pykcs11.i

Find below full log:

root@be58da524838:/# pip install PyKCS11 --break-system-packages --verbose
Using pip 23.3 from /usr/lib/python3/dist-packages/pip (python 3.11)
Collecting PyKCS11
  Using cached PyKCS11-1.5.13.tar.gz (78 kB)
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB)
  Using cached setuptools-69.0.3-py3-none-any.whl (819 kB)
  Installing collected packages: setuptools
  Successfully installed setuptools-69.0.3
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  running egg_info
  writing PyKCS11.egg-info/PKG-INFO
  writing dependency_links to PyKCS11.egg-info/dependency_links.txt
  writing top-level names to PyKCS11.egg-info/top_level.txt
  file PyKCS11/LowLevel.py (for module PyKCS11.LowLevel) not found
  reading manifest file 'PyKCS11.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'COPYING'
  writing manifest file 'PyKCS11.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
  Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.42.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info
  writing /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/SOURCES.txt'
  file PyKCS11/LowLevel.py (for module PyKCS11.LowLevel) not found
  reading manifest file '/tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'COPYING'
  writing manifest file '/tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-asi8l97q/PyKCS11-1.5.13.dist-info'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: PyKCS11
  Running command Building wheel for PyKCS11 (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  running build_ext
  building 'PyKCS11._LowLevel' extension
  swigging src/pykcs11.i to src/pykcs11_wrap.cpp
  swig -python -c++ -o src/pykcs11_wrap.cpp src/pykcs11.i
  creating build
  creating build/temp.linux-x86_64-cpython-311
  creating build/temp.linux-x86_64-cpython-311/src
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usr/local/incude -I/usr/include/python3.11 -c src/ck_attribute_smart.cpp -o build/temp.linux-x86_64-cpython-311/src/ck_attribute_smart.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usrlocal/include -I/usr/include/python3.11 -c src/dyn_unix.c -o build/temp.linux-x86_64-cpython-311/src/dyn_unix.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -wrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usr/ocal/include -I/usr/include/python3.11 -c src/pkcs11lib.cpp -o build/temp.linux-x86_64-cpython-311/src/pkcs11lib.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g-fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/us/local/include -I/usr/include/python3.11 -c src/pykcs11.cpp -o build/temp.linux-x86_64-cpython-311/src/pykcs11.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrpv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usr/locl/include -I/usr/include/python3.11 -c src/pykcs11_wrap.cpp -o build/temp.linux-x86_64-cpython-311/src/pykcs11_wrap.o
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_MECHANISM_pParameter_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:15849:64: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  15849 |     if (SWIG_IsOK(SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0)))
        |                                                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:343:37: note: in definition of macro ‘SWIG_IsOK’
    343 | #define SWIG_IsOK(r)               (r >= 0)
        |                                     ^
  src/pykcs11_wrap.cpp:1130:57: note: in expansion of macro ‘SWIG_Python_ConvertPtr’
   1130 | #define SWIG_ConvertPtr(obj, pptr, type, flags)         SWIG_Python_ConvertPtr(obj, pptr, type, flags)
        |                                                         ^~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:15849:19: note: in expansion of macro ‘SWIG_ConvertPtr’
  15849 |     if (SWIG_IsOK(SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0)))
        |                   ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_GCM_PARAMS_pIv_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:16027:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  16027 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:16027:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  16027 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_GCM_PARAMS_pAAD_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:16145:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  16145 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:16145:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  16145 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_RSA_PKCS_OAEP_PARAMS_pSourceData_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:16527:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  16527 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:16527:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  16527 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_ECDH1_DERIVE_PARAMS_pSharedData_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:17011:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  17011 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:17011:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  17011 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_ECDH1_DERIVE_PARAMS_pPublicData_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:17129:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  17129 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:17129:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  17129 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for PyKCS11 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 /usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpciub54do
  cwd: /tmp/pip-install-q1hwk9le/pykcs11_35eada89245c4008885373382292fd38
  Building wheel for PyKCS11 (pyproject.toml) ... error
  ERROR: Failed building wheel for PyKCS11
Failed to build PyKCS11
ERROR: Could not build wheels for PyKCS11, which is required to install pyproject.toml-based projects
LudovicRousseau commented 8 months ago

What happens if you follow the documentation to build PyKCS11? https://github.com/LudovicRousseau/PyKCS11/blob/master/README.md#unix-build-howto

SixK commented 8 months ago

Got the same error if building with this commands in a virtual python env (following the unix-build-howto).

If compairing generated pykcs11_wrap.cpp from ubuntu 22.04 and 24.04, I see the following difference:
Ubuntu 22.04 --> src/pykcs11_wrap.cpp:#define SWIGTYPE_p_std__vectorT_unsigned_char_stdallocatorT_unsigned_char_t_t swig_types[29]
Ubuntu 24.04 --> src/pykcs11_wrap.cpp:#define SWIGTYPE_p_stdvectorT_unsigned_char_t swig_types[29]

That can explain that SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t definition is not found.

At this state, I can't figure out if this is a swig 4.2.0 (or swig 4.1.1) change or something else.

In swig #defines seem's to be generated here: https://github.com/swig/swig/blob/f762ab35b29b993e868ea4e0750bdb608e0084cd/Source/Swig/typesys.c#L2263

But this doesn't help much to understand

I replaced SWIGTYPE_p_std__vectorT_unsigned_char_stdallocatorT_unsigned_char_t_t by SWIGTYPE_p_stdvectorT_unsigned_char_t in pykcs11.i file. Build is now working without any error, but I didn't have tested the generated package. I have no idea if it's the right modification to do for Ubuntu 24.04 or Swig 4.2.0 Maybe it will be more obvious for you ?

lmiguelmh commented 8 months ago

We had a similar error using swig==4.2.0 (latest) and pykcs==1.5.12

Workaround: use swig==4.1.1.post1

pip install swig==4.1.1.post1
SixK commented 8 months ago

If I follow the sequence below on an official ubuntu 24.04 image, I can confirm that your work-around is working to build PyKCS11 for Ubuntu 24.04:

    2  apt update
    3  apt install python3 python3-pip
    4  apt install git
    6  apt install python3-venv
    8  apt install build-essential
    9  python3 -m venv temp
   10  source temp/bin/activate
   11  pip3 install -r dev-requirements.txt
   12  git clone https://github.com/LudovicRousseau/PyKCS11
   13  cd PyKCS11/
   14  pip3 install -r dev-requirements.txt
   16  pip3 install swig==4.1.1.post1
   17  make

But If I simply want to install PyKCS11 to the system, I can't find a simple way : pip3 install swig==4.1.1.post1 --break-system-packages pip3 install PyKCS11 --break-system-packages


Collecting PyKCS11
  Using cached PyKCS11-1.5.13.tar.gz (78 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: PyKCS11
  Building wheel for PyKCS11 (pyproject.toml) ... error
  error: subprocess-exited-with-error
  × Building wheel for PyKCS11 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'PyKCS11._LowLevel' extension
      swigging src/pykcs11.i to src/pykcs11_wrap.cpp
      swig -python -c++ -o src/pykcs11_wrap.cpp src/pykcs11.i
      Traceback (most recent call last):
        File "/usr/local/bin/swig", line 5, in <module>
          from swig import swig
      ModuleNotFoundError: No module named 'swig'
      error: command '/usr/local/bin/swig' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for PyKCS11
Failed to build PyKCS11
ERROR: Could not build wheels for PyKCS11, which is required to install pyproject.toml-based projects

It's weird that swig module is not found, in this case. If I install swig using apt, I will get v4.2.0 and will have the undefined variable error.

LudovicRousseau commented 8 months ago

Thanks for the bug report. I will make a new release of PyKCS11 "soon".

LudovicRousseau commented 8 months ago

New release available https://blog.apdu.fr/posts/2024/02/new-version-of-pykcs11-1514/

SixK commented 8 months ago

Thank's for new release, according to wsfulton on swig project, the right way to solve the error would be to use descriptor: $descriptor(std::vector

corresponding swig issue: https://github.com/swig/swig/issues/2793#issuecomment-1926509914

LudovicRousseau commented 8 months ago

I tried with $descriptor(std::vector<unsigned char>) but I then have regressions in my tests. It may be a problem in my code. I will have a look. Thanks for the info @SixK.

LudovicRousseau commented 8 months ago

Fixed in https://github.com/LudovicRousseau/PyKCS11/commit/808199fb969bbb26f6e98a1829cc407f640c6e2c

The correct fix is to use $descriptor(vector<unsigned char> *)