[3.4.0] `oscrypto` does not respect `LD_LIBRARY_PATH`

[maximsmol]

In Python 3.3, find_library (used in oscrypto/_ffi.py) ignores the LD_LIBRARY_PATH environment variable.

Right now this means that 3.4.0 crashes on startup for users on systems that do not package OpenSSL in a default location e.g. NixOS (where dependencies are under /nix/store and get exposed to applications using wrapper shell scripts that set LD_LIBRARY_PATH).

I tried disabling oscrypto in the settings file but it seems like the crash happens before the settings are read.

A quick and dirty fix is to backport the following from CPython 3.6 into _ffi.py:

def _findLib_ld(name):
    # See issue #9998 for why this is needed
    expr = r'[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name)
    cmd = ['ld', '-t']
    libpath = os.environ.get('LD_LIBRARY_PATH')
    if libpath:
        for d in libpath.split(':'):
            cmd.extend(['-L', d])
    cmd.extend(['-o', os.devnull, '-l%s' % name])
    result = None
    try:
        p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE,
                             universal_newlines=True)
        out, _ = p.communicate()
        res = re.search(expr, os.fsdecode(out))
        if res:
            result = res.group(0)
    except Exception:
        pass  # result will be None
    return result

library = find_library(name) or _findLib_ld(name)

[wbond]

Do you have a backtrace?

I thought I put code in to not even try using OscryptoDownloader on Linux with Python 3.3.

Either way, ultimately any patch would need to go in oscrypto directly. I’m sort of surprised that ctypes.find_library() doesn’t take into account LD_LIBRARY_PATH.

I’m definitely not keen on launching a subprocess to find OpenSSL. There is a way in oscrypto to explicitly tell it where OpenSSL is, which would be the best solution.

[maximsmol]

Here is a traceback on my NixOS install (with debug: true even though it did not seem to change anything):

Traceback (most recent call last):
  File "/nix/store/y93ll4jrwanhiadn2s32m1zzs9vk5jrd-sublimetext3-bin-3210/sublime_plugin.py", line 125, in reload_plugin
    m = importlib.import_module(modulename)
  File "./python3.3/importlib/__init__.py", line 90, in import_module
  File "<frozen importlib._bootstrap>", line 1584, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/2_bootstrap.py", line 26, in <module>
    from .package_control.bootstrap import bootstrap_dependency, mark_bootstrapped
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/bootstrap.py", line 20, in <module>
    from .download_manager import downloader
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/download_manager.py", line 24, in <module>
    from .downloaders import DOWNLOADERS
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/downloaders/__init__.py", line 20, in <module>
    from .oscrypto_downloader import OscryptoDownloader
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/downloaders/oscrypto_downloader.py", line 44, in <module>
    from ..deps.oscrypto import tls  # noqa
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/deps/oscrypto/tls.py", line 23, in <module>
    from ._openssl.tls import (
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/deps/oscrypto/_openssl/tls.py", line 10, in <module>
    from ._libssl import libssl, LibsslConst
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/deps/oscrypto/_openssl/_libssl.py", line 7, in <module>
    from ._libcrypto import libcrypto_version_info
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/deps/oscrypto/_openssl/_libcrypto.py", line 15, in <module>
    from ._libcrypto_ctypes import (
  File "/home/maximsmol/.config/sublime-text-3/Packages/Package Control/package_control/deps/oscrypto/_openssl/_libcrypto_ctypes.py", line 28, in <module>
    raise LibraryNotFoundError('The library libcrypto could not be found')
Package Control.package_control.deps.oscrypto.errors.LibraryNotFoundError: The library libcrypto could not be found

Here are the relevant environment variables, from the Sublime console (binutils is there since find_library runs ld under the hood):

>>> import os
>>> os.environ["LD_LIBRARY_PATH"]
'/nix/store/5cgzrk1cyq1sr218xvjgw05z0jcyki92-libX11-1.6.8/lib:/nix/store/vlsc0gm44gdvw7l69yk4z3xbwfvk3v13-glib-2.64.5/lib:/nix/store/02xr4yq8svlg0vc2nzkbrw1s4sg4bk38-gtk+3-3.24.21/lib:/nix/store/60jw8v2k3qps31n7071ik60yyis1l5dx-cairo-1.16.0/lib:/nix/store/bn0j5dsd6fkzhw9wlq76ggd0x5gnzr6f-pango-1.45.3/lib:/nix/store/aqafh2kgahm2hv3nkihmgnvsg7y4ihcj-openssl-1.1.1g/lib'
>>> os.environ["PATH"]
'/nix/store/5sj06x18pd8an12ndl65hlwmp8afnrwa-binutils-wrapper-2.31.1/bin:/home/maximsmol/.nix-profile/share/git/contrib/diff-highlight/:/home/maximsmol/.cargo/bin:/home/maximsmol/opt/cs164-software/fsasim:/home/maximsmol/bin:/run/wrappers/bin:/home/maximsmol/.nix-profile/bin:/etc/profiles/per-user/maximsmol/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin'

I thought I put code in to not even try using OscryptoDownloader on Linux with Python 3.3.

The failure is on module import, so it might be that it really never uses it but still crashes anyway.

I’m sort of surprised that ctypes.find_library() doesn’t take into account LD_LIBRARY_PATH.

It does starting with Python 3.6 (there is a note on the docs), so I guess that this is more of an issue with the really old Python runtime that Sublime uses.

I’m definitely not keen on launching a subprocess to find OpenSSL.

FYI the standard library (ctypes.find_library()) launches 3 different ones every time (ldconfig, gcc, ld, in this order).

There is a way in oscrypto to explicitly tell it where OpenSSL is, which would be the best solution.

I saw that there is a backend_config thing, but could not figure out how to set it since it is not in the default settings file. Also, could we have an option to set this using an environment variable? In the NixOS use case in particular, config files are usually untouched (since they are outside the package manager store), but providing a wrapper script that sets env is pretty common. The path to OpenSSL also includes a hash of its configuration so it could change (and thus break Package Control) a lot if the user has to specify it manually.

[wbond]

Can you try running the following in your Sublime Text console:

import sys; sys.platform; sys.executable

[maximsmol]

There you go: (added sys.version just in case)

>>> import sys; sys.platform; sys.executable
'linux'
'python3'
>>> sys.version
'3.3.6 (default, Mar 30 2018, 11:25:50) \n[GCC 4.6.3]'

[wbond]

Ok, perfect. For some reason I thought sys.executable was returning plugin_host. Because of that, it is trying to load OscryptoDownloader when it really should not be.

I'm not sure if I'll roll this into a 3.4.1 release, or if it will be part of the 4.0.0 release that will be coming soon.

[wbond]

I just cut 3.4.1 to fix this issue. It will be installed if you install PC fresh, and will hit the default channel soon, triggering automatic updates.