Build and package Nikola with pyinstaller

[akielbowicz]

Discussion started in #PythonArgentina Telegram and the purpose is to generate an executable for Windows using pyinstaller

I've tried bundling executing

pyinstaller setup.py pyinstaller nikola/nikola.py

And both failed when executing

dist/setup/setup.exe
dist/nikola/nikola.exe

Traceback (most recent call last):
  File "C:\Users\pc\Miniconda3\envs\pyinstaller-nikola\Lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_pkgres.py", line 13, in <module>
    import pkg_resources as res
  File "c:\users\pc\miniconda3\envs\pyinstaller-nikola\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\pkg_resources\__init__.py", line 86, in <module>
ModuleNotFoundError: No module named 'pkg_resources.py2_warn'
[2428] Failed to execute script pyi_rth_pkgres

[tritium21]

running pyinstaller on a typical setup.py is never the right thing to do, and running it on a file in a package is not right either. You need to write a python module that is stand alone, and imports nikola - this module is the entry point that pyinstaller would need.

The error message you are seeing is also because of a bug in pyinstaller, and unrelated to nikola.

[Kwpolska]

Also, I would personally prefer any sort of Nikola Windows distribution to include the data files (mostly themes) in a way that can be easily browsed. While users should not modify those assets on their own, it is beneficial to have easy access to them while making one’s own theme.

[akielbowicz]

@tritium21 Thanks, yes looking around found this https://github.com/pyinstaller/pyinstaller/issues/4872 that explains the issue

[tritium21]

the first thing you need is a file to load the main entry point. Usually this is in the root of the project, above the package. This file is just called entrypoint.py on my system.

import sys
from nikola.__main__ import main

if __name__ == '__main__':
    sys.exit(main(sys.argv[1:]))

Then there's the fun part - the spec file. This is called entrypoint.spec in my system, sitting in parallel to entrypoint.py

# -*- mode: python ; coding: utf-8 -*-
import sys
import os.path

from PyInstaller.utils.hooks import collect_data_files

def expands_symlinks_for_windows():
    """replaces the symlinked files with a copy of the original content.

    In windows (msysgit), a symlink is converted to a text file with a
    path to the file it points to. If not corrected, installing from a git
    clone will end with some files with bad content

    After install the working copy will be dirty (symlink markers overwritten
    with real content)
    """
    if sys.platform != 'win32':
        return

    # apply the fix
    localdir = os.path.abspath(SPECPATH)
    oldpath = sys.path[:]
    sys.path.insert(0, os.path.join(localdir, 'nikola'))
    winutils = __import__('winutils')
    failures = winutils.fix_all_git_symlinked(localdir)
    sys.path = oldpath
    del sys.modules['winutils']
    if failures != -1:
        print('WARNING: your working copy is now dirty by changes in '
              'samplesite, sphinx and themes')
    if failures > 0:
        raise Exception("Error: \n\tnot all symlinked files could be fixed." +
                        "\n\tYour best bet is to start again from clean.")

expands_symlinks_for_windows()

block_cipher = None

datas = [
    ('npm_assets', 'npm_assets'),
    ('docs', 'docs'),
    ('logo', 'logo'),
    ('nikola', 'nikola'),
    ('scripts', 'scripts'),
    ('tests', 'tests'),
    ('translations', 'translations'),
    ('*.txt', '.'),
    ('*.rst', '.'),
    ('*.py', '.'),
]
datas.extend(collect_data_files('pyphen'))
datas.extend(collect_data_files('dateutil.zoneinfo'))

a = Analysis(['entrypoint.py'],
    pathex=[os.path.abspath(SPECPATH)],
    binaries=[],
    datas=datas,
    hiddenimports=[
        'pkg_resources.py2_warn',
        'urllib.robotparser',
        'piexif',
        'mako',
        'mako.template',
        'mako.exceptions',
        'mako.util',
        'mako.lexer',
        'mako.lookup',
        'mako.parsetree',
        'docutils',
        'docutils.core',
        'PIL.ExifTags',
        'aiohttp',
        'watchdog.observers',
    ],
    hookspath=[],
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False
)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='nikola',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='Nikola')

Yes, this is just a python file. Yes, those names did just come out of nowhere. No, there really isn't much you can do about it, pyinstaller is kinda ugly that way.

then, you just do env\Scripts\pyinstaller.exe entrypoint.spec, make a coffee, and you get a dist directory, with Nikola/nikola.exe and a LOT of data files. The single file option would not work without making substantial changes to nikola, along the lines of entirely reimplementing it, so thats not going to happen.

This... is still a work in progress, but it runs init, build, clean, check, serve, and auto. It could probably use a lot more testing and fixing, and the logical next step of building an installer (innosetup is the tool i would use for that, but i have experience with it, so I'm biased). I don't know if I have the time for that though.

[tritium21]

Also, I would personally prefer any sort of Nikola Windows distribution to include the data files (mostly themes) in a way that can be easily browsed. While users should not modify those assets on their own, it is beneficial to have easy access to them while making one’s own theme.

In a one-directory (rather than one-file) pyinstaller build, the themes files are MORE accessible than they are in site-packages. Of course, that changes the second you write an installer, and it becomes exactly as accessible as they are in site-packages. Themes would be in Nikola\nikola\data\themes with my previously provided spec file.

[devilgate]

I used PyInstaller to build a multiplatform installer for a Java app I worked on a year or two back: https://github.com/smallAreaHealthStatisticsUnit/rapidInquiryFacility/tree/master/installer. It pulls in and packages all the files it needs, so that on Windows you just get a file called rifInstaller.exe.

When you run it a series of preexisting scripts is called, and it builds a database schema in Postgres or SQL Server.

There's a list of tuples called added_files in the .spec file, and the datas parameter gets set to that. That was key to packaging up all the files.

That said, it was fairly clunky to get it all set up, so I could well understand if it's not quite practical for Nikola.

[tritium21]

Is there a part of the test suite that just runs all the nikola commands with all known options? Whats holding me back is not knowing if I have caught all the dependencies nikola needs. There is something to be said about the beeware approach to this - "Use pip and --target, ok, we'll just ship that" - the dumbest thing that could possibly work. Sadly pyinstaller doesn't support that.

[Kwpolska]

@tritium21 There isn’t anything. requirements-extras.txt should have everything in it, and you can search for req_missing to identify parts that use optional dependencies.

[tritium21]

@Kwpolska Unfortunately, doesn't help with transient dependencies, and any use of __import__ is just utterly hopeless for pyinstaller to ever find modules. Its not intractable, just a very manual process.

[ralsina]

If you want all the transient dependencies, you can just

pip install -r requirements-extras.txt pip freeze

Here is what I got right now:

aiohttp==3.6.2 async-timeout==3.0.1 attrs==19.3.0 Babel==2.8.0 backcall==0.1.0 bleach==3.1.5 blinker==1.4 certifi==2020.4.5.1 chardet==3.0.4 cloudpickle==1.4.1 decorator==4.4.2 defusedxml==0.6.0 docutils==0.16 doit==0.32.0 entrypoints==0.3 ghp-import2==1.0.1 husl==4.0.3 idna==2.9 ipykernel==5.3.0 ipython==7.14.0 ipython-genutils==0.2.0 jedi==0.17.0 Jinja2==2.11.2 jsonschema==3.2.0 jupyter-client==6.1.3 jupyter-core==4.6.3 lxml==4.5.1 Mako==1.1.2 Markdown==3.2.2 MarkupSafe==1.1.1 micawber==0.5.1 mistune==0.8.4 multidict==4.7.6 natsort==7.0.1 nbconvert==5.6.1 nbformat==5.0.6 notebook==6.0.3 packaging==20.4 pandocfilters==1.4.2 parso==0.7.0 pathtools==0.1.2 pexpect==4.8.0 phpserialize==1.3 pickleshare==0.7.5 piexif==1.1.3 Pillow==7.1.2 prometheus-client==0.8.0 prompt-toolkit==3.0.5 ptyprocess==0.6.0 pygal==2.4.0 Pygments==2.6.1 pyinotify==0.9.6 pyparsing==2.4.7 Pyphen==0.9.5 pyrsistent==0.16.0 PyRSS2Gen==1.1 python-dateutil==2.8.1 pytz==2020.1 pyzmq==19.0.1 requests==2.23.0 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 Send2Trash==1.5.0 six==1.15.0 smartypants==2.0.1 terminado==0.8.3 testpath==0.4.4 toml==0.10.1 tornado==6.0.4 traitlets==4.3.3 typogrify==2.0.7 Unidecode==1.1.1 urllib3==1.25.9 watchdog==0.10.2 wcwidth==0.1.9 webencodings==0.5.1 Yapsy==1.12.2 yarl==1.4.2

On Thu, May 28, 2020 at 4:23 PM Alexander Walters notifications@github.com wrote:

@Kwpolska https://github.com/Kwpolska Unfortunately, doesn't help with transient dependencies, and any use of import is just utterly hopeless for pyinstaller to ever find modules. Its not intractable, just a very manual process.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/getnikola/nikola/issues/3410#issuecomment-635547127, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMKYJYQKI2OXCCF72CILRT2223ANCNFSM4NADNC5Q .

[tritium21]

As I keep saying ... the requirements file is useless for this, the actual module graph that get imported, even with __import__ is what you ultimately need to trick pyinstaller into bundling everything. Which is why i was asking if there was something that stressed all code paths, or at least all code paths that generate imports, to test it.

I wish I could just point pyinstaller at a list of requirements and have it work.

Edit: TIL dunder names don't render in markdown without fencing....

[ralsina]

Then this is probably not doable, since that is going to break every release.

On Fri, May 29, 2020 at 1:39 AM Alexander Walters notifications@github.com wrote:

As I keep saying ... the requirements file is useless for this, the actual module graph that get imported, even with import is what you ultimately need to trick pyinstaller into bundling everything. Which is why i was asking if there was something that stressed all code paths, or at least all code paths that generate imports, to test it.

I wish I could just point pyinstaller at a list of requirements and have it work.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/getnikola/nikola/issues/3410#issuecomment-635754370, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMK4DS6AKTXL2BT2YU4DRT4377ANCNFSM4NADNC5Q .

[ralsina]

What about pyoxidizer? It looks 10x more usable.

On Fri, May 29, 2020 at 4:55 AM Roberto Alsina ralsina@kde.org wrote:

Then this is probably not doable, since that is going to break every release.

On Fri, May 29, 2020 at 1:39 AM Alexander Walters < notifications@github.com> wrote:

As I keep saying ... the requirements file is useless for this, the actual module graph that get imported, even with import is what you ultimately need to trick pyinstaller into bundling everything. Which is why i was asking if there was something that stressed all code paths, or at least all code paths that generate imports, to test it.

I wish I could just point pyinstaller at a list of requirements and have it work.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/getnikola/nikola/issues/3410#issuecomment-635754370, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMK4DS6AKTXL2BT2YU4DRT4377ANCNFSM4NADNC5Q .

[tritium21]

Why don't we try the Dumbest Thing That Could Possibly Work? https://gist.github.com/tritium21/71acb9091845ce0f63ee049aae7b9172 Requires inno setup. Does nothing magical.

Pyoxidizer is even more magical than pyinstaller. Nikola is fairly magical. Magic + Magic = Broken.

[ralsina]

I don't see why not.

On Fri, May 29, 2020 at 5:12 AM Alexander Walters notifications@github.com wrote:

Why don't we try the Dumbest Thing That Could Possibly Work? https://gist.github.com/tritium21/71acb9091845ce0f63ee049aae7b9172 Requires inno setup. Does nothing magical.

Pyoxidizer is even more magical than pyinstaller. Nikola is fairly magical. Magic + Magic = Broken.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/getnikola/nikola/issues/3410#issuecomment-635835928, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMK6KU6JCYRTXJB3UJELRT5U4ZANCNFSM4NADNC5Q .