Update Python to 2.7.13

[Anthchirp]

We tried updating Python to 2.7.12 before, and had to revert to 2.7.8 when we came across SSL issues on macOS and Debian/Ubuntu, cf. 22b772785819394c1636d23243e83431f8a84479

This needs more investigating

[bkpoon]

Just built everything with Python 2.7.13 on macOS 10.12.4 without any problems. Maybe updating OpenSSL helped? I'll be building on macOS 10.9 - 10.11 next.

[bkpoon]

Running with Python 2.7.13 on macOS 10.12.4 has issues when trying to use SSL for validating connections. An example is iotbx.blast_pdb sending a request to the NCBI https server. The error that pops up is "<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>."

The problem is that macOS keeps its root certificates in the Keychain, which is not accessible via OpenSSL (macOS use SecureTransport, or an old/patched version of OpenSSL). Windows might have this issue. Linux should not have this issue because the certificates are available via packages ("ca-certificates" for both CentOS and Ubuntu). It looks like Python might have a general solution in 3.7 (https://www.python.org/dev/peps/pep-0543/).

A solution for now would be to use pip to install certifi (https://pypi.python.org/pypi/certifi), which is just a repackaged certificate bundle from Mozilla. The following commands provide the location of the certificates,

import certifi
ca_path = certifi.where()

In the dispatcher, the SSL_CERT_FILE environment variable can be set, which lets the ssl module find the certificate bundle. I tested this on macOS 10.12.4 and iotbx.blast_pdb works correctly.

Previously, the SSL error occurred during the base installation step when pip was called to install additional dependencies. It's unclear why installation worked without issue on 10.12.4. I still have to test 10.11. The workaround would be to install certifi after Python and set the SSL_CERT_FILE environment variable for the rest of install_base_packages.py.

[Anthchirp]

Good stuff. Thank you for looking into this!

[bkpoon]

Let's shoot for late April/early May for the upgrade. I'm double-checking the installation on our CentOS and Ubuntu virtual machines.

[bkpoon]

Hi everyone,

The update to Python 2.7.13 is live. To update an existing installation, you will want to delete your existing "base" and "base_tmp" directories and rebuild base with bootstrap.py. You will want to do this in a new shell without the cctbx environment so that python does not point to the one in base.

cd <installation directory>
rm -fr base*
python bootstrap.py base build --builder=cctbx --nproc=<cpus>

The builder flag is for the basic cctbx dependencies. You may need another builder (e.g. "phenix," "dials," or "xfel") if you need more dependencies. Also, "build" is run to reconstruct the dispatchers.

The certifi package has been added for macOS builds on 10.11 and greater. This package provides the root certificates needed for OpenSSL to work. Normally, the root certificates are stored in your Keychain, but the standard, unpatched OpenSSL cannot access them.

Some additional quirks about using OpenSSL:

1. macOS binary installers built on 10.9 and 10.10 have patched versions of OpenSSL that can access the root certificates in the Keychain. These installations can work on 10.11 and greater without the certifi package. This may change in future macOS releases, but we can just include certifi for all macOS builds.
2. CentOS and Ubuntu may have different library names for OpenSSL (e.g. CentOS 6 and Ubuntu 16.04), so Linux binary installers built on CentOS may not be functionally the same when installed on an Ubuntu machine. To check, you can try importing the ssl module in python.

   libtbx.python -c "import ssl"

   This is only a problem if you use anything that requires OpenSSL to access a https website (e.g. iotbx.blast_pdb)

[alyubimov]

Hi Billy,

My install on Mac OS (10.12.4) failed when installing lz4_plugin (error traceback below). I was doing a clean install in an empty folder.

Art

Installing lz4_plugin... log file is /Users/art/xfel/base_tmp/lz4_plugin_install_log ===== Running in /Users/art/xfel/base_tmp: git clone --recursive https://github.com/dectris/HDF5Plugin.git hdf5_lz4 --progress --verbose Cloning into 'hdf5_lz4'... POST git-upload-pack (206 bytes) remote: Counting objects: 98, done. remote: Total 98 (delta 0), reused 0 (delta 0), pack-reused 98 Unpacking objects: 100% (98/98), done. ===== Running in /Users/art/xfel/base_tmp: git clone --recursive https://github.com/kiyo-masui/bitshuffle.git bitshuffle --progress --verbose Cloning into 'bitshuffle'... POST git-upload-pack (1006 bytes) remote: Counting objects: 1042, done. remote: Compressing objects: 100% (6/6), done. remote: Total 1042 (delta 1), reused 0 (delta 0), pack-reused 1036 Receiving objects: 100% (1042/1042), 307.25 KiB | 0 bytes/s, done. Resolving deltas: 100% (675/675), done. installing lz4 plugin... Traceback (most recent call last): File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1466, in installer(args=sys.argv, log=sys.stdout) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 217, in init self.build_dependencies(packages=packages) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 648, in builddependencies getattr(self, 'build%s'%i)() File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 915, in build_lz4_plugin self.call("%s setup.py build"%self.python_exe,log=log) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 312, in call return call(args, log=log, verbose=self.verbose, **kwargs) File "/Users/art/xfel/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to '/Users/art/xfel/base/bin/python setup.py build' failed with exit code 1 Process failed with return code 1

On Mon, May 1, 2017 at 4:48 PM, Billy K. Poon notifications@github.com wrote:

Hi everyone,

The update to Python 2.7.13 is live. To update an existing installation, you will want to delete your existing "base" and "base_tmp" directories and rebuild base with bootstrap.py. You will want to do this in a new shell without the cctbx environment so that python does not point to the one in base.

cd rm -fr base* python bootstrap.py base --builder=cctbx --nproc=

The certifi package has been added for macOS builds on 10.11 and greater. This package provides the root certificates needed for OpenSSL to work. Normally, the root certificates are stored in your Keychain, but the standard, unpatched OpenSSL cannot access them.

Some additional quirks about using OpenSSL:

1. macOS binary installers built on 10.9 and 10.10 have patched versions of OpenSSL that can access the root certificates in the Keychain. These installations can work on 10.11 and greater without the certifi package. This may change in future macOS releases, but we can just include certifi for all macOS builds.
2. CentOS and Ubuntu may have different library names for OpenSSL (e.g. CentOS 6 and Ubuntu 16.04), so Linux binary installers built on CentOS may not be functionally the same when installed on an Ubuntu machine. To check, you can try importing the ssl module in python.

libtbx.python -c "import ssl"

This is only a problem if you use anything that requires OpenSSL to access a https website (e.g. iotbx.blast_pdb)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cctbx/cctbx_project/issues/25#issuecomment-298462327, or mute the thread https://github.com/notifications/unsubscribe-auth/AOnoOb0TgAkxLw5iiKKl9piQqyEPatnSks5r1m9cgaJpZM4M1YuF .

-- Artem Y. Lyubimov, PhD Research Specialist Howard Hughes Medical Institute 318 Campus Dr. Room E300 Stanford, CA 94304 lyubimov@stanford.edu

[bkpoon]

Let me double-check. I just compiled on 10.9 - 10.12 last week.

[bkpoon]

It looks like bitshuffle just added a pkg-config dependency on 4/27 (bitshuffle/setup.py) Can we pick a version of bitshuffle before the change?

[Anthchirp]

https://github.com/dials/bitshuffle Not a long-term fix though. Having looked at the bitshuffle sources I think this is a bug.

[Anthchirp]

should now be fixed via patching of setup.py in the installer. I'll make a pull request to bitshuffle: https://github.com/kiyo-masui/bitshuffle/pull/62

[bkpoon]

Thanks Markus! The bitshuffle dependency is working for me now.

[bkpoon]

Since no new problems have popped up, I'm closing this issue.

[Anthchirp]

Thank you!