In OpenBSD oscrypto is not depending on cffi package

[oherrala]

OpenBSD (-current, snapshot 2016-06-14) using Python 3.6 from Ports is missing Python cffi package. I didn't test other Python versions, but let me know if it's needed.

Should oscrypto depend on cffi and install it if missing? This helps users who install Python packages with pip install.

Workaround (or solution) for this issue in OpenBSD is to install py3-cffi package from Ports. However, hunting missing Python packages from ports is not solution for users without root access.

When cffi is missing, the user gets following traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_cffi.py", line 13, in <module>
    from cffi import FFI
ModuleNotFoundError: No module named 'cffi'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto.py", line 8, in <module>
    from ._libcrypto_cffi import (
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_cffi.py", line 16, in <module>
    raise FFIEngineError('Error importing cffi')
oscrypto._ffi.FFIEngineError: Error importing cffi

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_ctypes.py", line 258, in <module>
    libcrypto.EVP_MD_CTX_new.argtypes = []
  File "/usr/local/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: Unable to resolve symbol

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/trytls", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/trytls/runner.py", line 177, in main
    bundle = bundles.load_bundle(bundle_name)
  File "/usr/local/lib/python3.6/site-packages/trytls/bundles/__init__.py", line 11, in load_bundle
    return entry.load()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2268, in load
    return self.resolve()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2274, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/site-packages/trytls/bundles/https.py", line 15, in <module>
    from ..gencert import gencert
  File "/usr/local/lib/python3.6/site-packages/trytls/gencert.py", line 3, in <module>
    from oscrypto import asymmetric
  File "/usr/local/lib/python3.6/site-packages/oscrypto/asymmetric.py", line 15, in <module>
    from .kdf import pbkdf2, pbkdf2_iteration_calculator
  File "/usr/local/lib/python3.6/site-packages/oscrypto/kdf.py", line 9, in <module>
    from .util import rand_bytes
  File "/usr/local/lib/python3.6/site-packages/oscrypto/util.py", line 14, in <module>
    from ._openssl.util import rand_bytes
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/util.py", line 6, in <module>
    from ._libcrypto import libcrypto, libcrypto_version_info, handle_openssl_error
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto.py", line 14, in <module>
    from ._libcrypto_ctypes import (
  File "/usr/local/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_ctypes.py", line 668, in <module>
    raise FFIEngineError('Error initializing ctypes')
oscrypto._ffi.FFIEngineError: Error initializing ctypes

For cross referencing this issue is also in TryTLS repo: https://github.com/ouspg/trytls/issues/307

[jviide]

When cffi is missing oscrypto falls back to ctypes. In this case that happens to fail, as the line libcrypto.EVP_MD_CTX_new.argtypes = [] raises AttributeError: Unable to resolve symbol.

Seeing that you use OpenBSD I assume you're using LibreSSL, so could this be some kind of LibreSSL vs. OpenSSL thing? It seems that LibreSSL doesn't have EVP_MD_CTX_new, instead there's EVP_MD_CTX_create (see evp.h). Based on the related lines in _libcrypto_ctypes.py oscrypto does support both EVP_MD_CTX_new and EVP_MD_CTX_create, but picks one based on a version check. Maybe the check is the problem?

Just an idea: Instead of an explicit version check oscrypto could try to access libcrypto.EVP_MD_CTX_new and try the *_create variant if there's an AttributeError. Or vice versa.

[wbond]

@oherrala No, cffi is not required. As @jviide has pointed out, this seems to be an issue with function availability in LibreSSL. I don’t regularly test on OpenBSD with LibreSSL, mostly just because I don’t know of a CI service that offers that.

@jviide Trying to see if the functions are available is certainly a possibility. In general I’d rather fail loudly than guess what the user meant, sort of like how the various functions do strict type checking to make sure that incorrect parameters aren’t being passed.

@oherrala If you wouldn’t mind doing a little debugging, I’d love to know what the values of version_string and version_info are at line 46 in oscrypto/_openssl/_libcrypto_ctypes.py. This way I can make sure that the version checks used in a few different places are properly accounting for recent versions of LibreSSL.

[walshbm15]

Hi, not sure if this is still being looked at but think I might have a similar issue (LibreSSL vs. OpenSSL)? I noticed my computer and Python was using LibreSSL 2.2.7 and have since forced Python to use OpenSSL by updating OpenSSL using Brew and reinstalling Python 2.7 with brew install --with-brewed-openssl. I was then able to verify my Python is now using 'OpenSSL 1.0.2n 7 Dec 2017'

To continue with your debugging questions: at line 46 in oscrypto/_openssl/_libcrypto_ctypes.py I see version_string=u'LibreSSL 2.2.7' version_info=<type 'tuple'>: (2, 2, 7)

[wbond]

I haven't had an OpenBSD box, nor time to set one up in a while, so I haven't done anything here.

The info you provided is definitely useful. Next I'll need to check and see what API LibreSSL 2.2.7 supports compared to OpenSSL, so we can select the appropriate bindings to use.

[wbond]

This should be fixed by 83e2a06085b5b09ee5cd6c28a2ae1c52352c9e53

[krash84]

Hi!

I get similiar error on ALT Linux 64-bit p8

$ python
Python 3.6.6 (default, Sep  8 2018, 19:06:55) 
[GCC 5.3.1 20151207 (ALT Linux 5.3.1-alt3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Module readline not available.
>>> from ocspbuilder import OCSPBuilder
Traceback (most recent call last):
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_cffi.py", line 13, in <module>
    from cffi import FFI
ModuleNotFoundError: No module named 'cffi'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto.py", line 8, in <module>
    from ._libcrypto_cffi import (
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_cffi.py", line 16, in <module>
    raise FFIEngineError('Error importing cffi')
oscrypto._ffi.FFIEngineError: Error importing cffi

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_ctypes.py", line 258, in <module>
    libcrypto.EVP_MD_CTX_new.argtypes = []
  File "/home/jetbird/sdk/python3.6/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/home/jetbird/sdk/python3.6/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib64/libcrypto.so.41: undefined symbol: EVP_MD_CTX_new

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/ocspbuilder/__init__.py", line 12, in <module>
    from oscrypto import asymmetric, util
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/asymmetric.py", line 15, in <module>
    from .kdf import pbkdf2, pbkdf2_iteration_calculator
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/kdf.py", line 9, in <module>
    from .util import rand_bytes
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/util.py", line 14, in <module>
    from ._openssl.util import rand_bytes
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/util.py", line 6, in <module>
    from ._libcrypto import libcrypto, libcrypto_version_info, handle_openssl_error
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto.py", line 14, in <module>
    from ._libcrypto_ctypes import (
  File "/home/jetbird/.envs/cprocsp_alt/lib/python3.6/site-packages/oscrypto/_openssl/_libcrypto_ctypes.py", line 668, in <module>
    raise FFIEngineError('Error initializing ctypes')
oscrypto._ffi.FFIEngineError: Error initializing ctypes

version_string = 'LibreSSL 2.5.5' version_info = (2, 5, 5)

[wbond]

@krash84 If you are running a release from PyPi, this is to be expected since I haven't had time to do a release in a while.