Python 2.7 ctypes shared library import fails because of a hardcoded path, and crle default settings.

[goekesmi]

I'm not sure exactly how to report this, so I'm going to try here.

Context: I'm working on getting https://letsencrypt.org/ up and running on SmartOS.

Reproduction steps: Boot a SmartOS zone using the bcfdd92c-1f2f-11e5-8386-73310a2d6492 (ghost 15.1.2) image.

----------------------------------Console log ------------------------------------------ root@70-54-d2-1a-a2-3d /usbkey/scratch]# vmadm create -f ghost2.json Successfully created VM 1f991fb1-fa0e-ebef-ef25-862ca94e54be [root@70-54-d2-1a-a2-3d /usbkey/scratch]# zlogin 1f991fb1-fa0e-ebef-ef25-862ca94e54be [Connected to zone '1f991fb1-fa0e-ebef-ef25-862ca94e54be' pts/5] . . | | | .-. . . .-. :--. |- | | ;| || |(.-' | | | || --'-' ;-|-' ' ' -' / ; Instance (ghost 15.1.2) -' http://docs.joyent.com/images/smartos/ghost

[root@ghost2 ~]# pkgin update reading local summary... processing local summary... updating database: 100% pkg_summary.bz2 100% 1990KB 1.9MB/s 1.9MB/s 00:00
processing remote summary (http://pkgsrc.joyent.com/packages/SmartOS/2015Q1/x86_64/All)... updating database: 100% [root@ghost2 ~]# pkgin install python-augeas calculating dependencies... done.

nothing to upgrade. 4 packages to be installed (3204K to download, 12M to install):

xmlcatmgr-2.2nb1 libxml2-2.9.2nb2 augeas-1.3.0 python-augeas-0.5.0

proceed ? [Y/n] y downloading packages... xmlcatmgr-2.2nb1.tgz 100% 32KB 32.4KB/s 32.4KB/s 00:00
libxml2-2.9.2nb2.tgz 100% 2373KB 2.3MB/s 2.3MB/s 00:01
augeas-1.3.0.tgz 100% 785KB 784.7KB/s 784.7KB/s 00:00
python-augeas-0.5.0.tgz 100% 13KB 13.4KB/s 13.4KB/s 00:00
installing packages... installing xmlcatmgr-2.2nb1... xmlcatmgr-2.2nb1: copying /opt/local/share/examples/xmlcatmgr/catalog.etc.sgml to /opt/local/etc/sgml/catalog xmlcatmgr-2.2nb1: copying /opt/local/share/examples/xmlcatmgr/catalog.etc.xml to /opt/local/etc/xml/catalog xmlcatmgr-2.2nb1: copying /opt/local/share/examples/xmlcatmgr/catalog.share.sgml to /opt/local/share/sgml/catalog

xmlcatmgr-2.2nb1: copying /opt/local/share/examples/xmlcatmgr/catalog.share.xml to /opt/local/share/xml/catalog

$NetBSD: MESSAGE,v 1.5 2004/01/23 17:12:16 jmmv Exp $

The following catalogs have been installed:

    * /opt/local/etc/sgml/catalog
      System wide SGML catalog.  Can be edited by the administrator and
      is not changed by packages.  This is the *default* catalog when
      working in SGML mode.

    * /opt/local/etc/xml/catalog
      System wide XML catalog.  Can be edited by the administrator and
      is not changed by packages.  This is the *default* catalog when
      working in XML mode.

    * /opt/local/share/sgml/catalog
      Handles SGML stuff installed under /opt/local/share/sgml.
      Automatically handled by packages.

    * /opt/local/share/xml/catalog
      Handles XML stuff installed under /opt/local/share/xml.
      Automatically handled by packages.

installing libxml2-2.9.2nb2... installing augeas-1.3.0... installing python-augeas-0.5.0... pkg_install warnings: 0, errors: 0 reading local summary... processing local summary... updating database: 100% marking python-augeas-0.5.0 as non auto-removable [root@ghost2 ~]# python Python 2.7.9 (default, Apr 21 2015, 19:01:11) [GCC 4.7.4] on sunos5 Type "help", "copyright", "credits" or "license" for more information.

import augeas Traceback (most recent call last): File "", line 1, in File "/opt/local/lib/python2.7/site-packages/augeas.py", line 78, in class Augeas(object): File "/opt/local/lib/python2.7/site-packages/augeas.py", line 82, in Augeas _libaugeas = _dlopen("augeas") File "/opt/local/lib/python2.7/site-packages/augeas.py", line 75, in _dlopen raise ImportError("Unable to import lib%s!" % args[0]) ImportError: Unable to import libaugeas!

---------------------------------- End Console log ------------------------------------------

I've located two problems. First, in /opt/local/lib/python2.7/ctypes/util.py, line 117 to 131.

117 if sys.platform == "sunos5": 118 # use /usr/ccs/bin/dump on solaris 119 def _get_soname(f): 120 if not f: 121 return None 122 cmd = "/usr/ccs/bin/dump -Lpv 2>/dev/null " + f 123 f = os.popen(cmd) 124 try: 125 data = f.read() 126 finally: 127 f.close() 128 res = re.search(r'[.*]\sSONAME\s+([^\s]+)', data) 129 if not res: 130 return None 131 return res.group(1)

On line 122, the command referenced '/usr/ccs/bin/dump doesn't exist in that location on SmartOS.

Work around: cd /usr/ccs/bin/; ln -s which dump I'm reasonably sure that's horrible.

Second. crle doesn't have /opt/local/lib in it's search path.

workaround: crle -l /lib:/usr/lib:/opt/local/lib I'm reasonably sure that's not a good way to do this either.

Post both changes: ----------------------------------Console log ------------------------------------------ root@ghost2 /usr/ccs/bin]# python Python 2.7.9 (default, Apr 21 2015, 19:01:11) [GCC 4.7.4] on sunos5 Type "help", "copyright", "credits" or "license" for more information.

import augeas

---------------------------------- End Console log ------------------------------------------

Post both of these work arounds, Lets encrypt seems to be functional, but I haven't completed a full sequence yet.

While I don't know how it's relevant, I'm on SmartOS platform image: joyent_20160204T173339Z.

I apologize if I'm mixing two bugs, but they seem to be strongly related.

I'm happy to help sort this out. Questions welcome.

[mamash]

Hi Jeff,

yes, I've hit that problem before, when I was bringing letsencrypt into pkgsrc. I patched py-augeas to work around the problem in 62d1bf83ae6950d971fb0ebaffdcde612b35b3c4, but the right approach of course would be to fix the ctypes find_library function to not rely on /usr/ccs/bin/dump.

Looks like OI did that in https://github.com/OpenIndiana/oi-userland/commit/0d693fe3f9cd9e7d00d11238484c615b1b0706b2, but I'd need to make sure to handle all the SunOS worlds at the same time.

BTW letsencrypt is in pkgsrc and working (looks like I should update it to the latest though), but work is needed for make it handle pkgsrc style Nginx and Apache for sure.

https://github.com/joyent/pkgsrc-wip/tree/master/py-letsencrypt

[goekesmi]

Hi Filip,

It's nice to know I'm not off in this wilderness alone.

I've solved my immediate problem. After doing the above two workarounds, I'm up and running with Let's encrypt.

I agree with your observations. I wasn't aware of letsencrypt being in pkgsrc, and that's because I didn't go looking for wip.

How can I best be of assistance in fixing, both/some/all of this?

My goal was to have letsencrypt work correctly by default on the ghost image. This was to support helping my friends run blogs. However, I'm willing to re-target my work if there's a better objective.

Thanks.

[mamash]

I assume you use 'letsencrypt certonly' for your needs? The base letsencrypt package seems to work fine for such setup. I was talking about py-letsencrypt-apache and py-letsencrypt-nginx (pre-created in wip) that do not currently support the layout of the respective config files, how pkgsrc sets them up, so more work is needed there. I guess there there's not enough overlap there for you to work on those.

[rmustacc]

github autoclose, re-opening