Bundling a Python interpreter and some essential Python libraries

[joerick]

I'm creating this issue as a python-specific #12.

I think it's pretty crucial to the user experience (and a common stumbling block for programming) for sample code to just work. I also think it's good hygiene not to be installing anything system-wide, where we could get into version conflicts and other complications.

Each install of Tide-Electron should have a bundled Python interpreter, and the following libraries preinstalled-

• python-tingbot
  • pygame (including C extensions)
  • SDL 1.2.14
  • libpng16.16
  • libfreetype.6
  • zmq (including C extensions, and a bundled version of libzmq)
  • Pillow (including C extensions)
  • paramiko (pure Python)
  • cryptography (including C extensions)
  • A few more pure-python extensions (pyasn1, pycparser, cffi, six, idna, ipaddress, enum34)
• requests

[joerick]

When I wrote the previous version of Tide for Mac, I bundled libraries by finding wheels, unzipping them and including manually. Libraries not available in wheel form (pygame) were-

• extracted from a working installation (grabbed from site-packages)
• their depending dylibs found (using otool -L, recursively for each dependency found)
• the dylibs install names rewritten (using install_name_tool)

This was a pain in the arse. I'm thinking we can find wheels for almost everything this time round, and if we've got a bundled Python install, just pip install them as part of the build process.

Pygame looks to be making some progress with their wheel creation, I've found recent wheels for Windows, Mac, and Linux. Hooray!

[RogueM]

Isn't what you are trying to achieve exactly what PyInstaller performs? http://pythonhosted.org/PyInstaller/

[joerick]

Mayyybe! Our use-case is a little different as the Tide app is built using Electron, and we need a python environment with those deps installed, which will be bundled inside an Electron app. PyInstaller might be able to build that for us too, but the dependency auto-discovery implies that it's not really built for that purpose.

[joerick]

I'm working on a solution that looks a bit like this-

scripts/bundlePython.bat

pushd %~dp0
cd ..
mkdir build\bundlePython
cd build\bundlePython

set ABS_PATH=%CD%

REM Download the Python MSI
DEL python-2.7.12.msi
powershell -Command "(new-object System.Net.WebClient).DownloadFile('https://www.python.org/ftp/python/2.7.12/python-2.7.12.msi', 'python-2.7.12.msi')"

REM Expand it to a local location
RMDIR /S /Q python27
msiexec -a %ABS_PATH%\python-2.7.12.msi -qb TARGETDIR=%ABS_PATH%\python27

REM Install Pip
python27\python.exe -m ensurepip

REM Move the Python enviroment into place
RMDIR /S /Q ..\..\vendor\python27
xcopy python27 ..\..\vendor\python27

My batch scripting is ropey, but I think the approach is good. I'm going to try to replicate the same approach (download Python binary dist, extract a build root, install pip and deps into there) on Mac and see if it works okay there – then look at Windows again.

[joerick]

Okay, so after looking at Mac... I'm using a different approach there to on Windows.

On Mac, it's not as easy as Windows to pull a built version from an installer from Python.org. And my homemade binaries lacked OpenSSL support, probably due to missing headers. However, there's a version of Python 2.7 preinstalled with the OS. So on Mac, we'll just bundle a folder full of packages. Those with C extensions are installed with wheels. That's implemented in 425f38373ac8b4ae6384b5417346bde175b620f5.

So, for Windows, we'll need the procedure in the above batch script replicated, followed by a similar approach to installing dependencies as Mac.

There's a python script waiting for this code - /scripts/build-python-env.py. I haven't got around to it yet, contributions welcome.

@Rob4001 is taking care of Linux I believe :)

[joerick]

I'm going to tackle Windows today!