itamarst / eliot

Eliot: the logging system that tells you *why* it happened
https://eliot.readthedocs.io
Apache License 2.0
1.11k stars 66 forks source link

Eliot is incompatible with PyInstaller "one-file programs" #386

Closed exarkun closed 5 years ago

exarkun commented 5 years ago

PyInstaller (https://www.pyinstaller.org/) gets sad when eliot._util.load_module is used inside a "one-file program" (https://pyinstaller.readthedocs.io/en/stable/operating-mode.html#how-the-one-file-program-works) to create a no-I/O version of the traceback module by trying to read the source file of the traceback module. In the PyInstaller execution context there is no readable source file in the filesystem.

exarkun commented 5 years ago

Looking at the users of load_module in Eliot, it seems like all mutations are shallow and so a shallow copy of the module might be suitable here.

This can easily be accomplished without reading the module source:

def load_module(name, original_module):
    new_module = ModuleType(name)
    new_module.__dict__.update(original_module.__dict__)
    return new_module
exarkun commented 5 years ago

Apparently that simple, straightforward, understandable fix appears not to work. I should have heeded the warning in the comment, I guess. I'm looking at other options.

crwood commented 5 years ago

A minor correction: this issue is not exclusive to PyInstaller-generated "one-file" binaries; it also occurs in "one-folder" mode.

itamarst commented 5 years ago

I suspect https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly would work.

itamarst commented 5 years ago
>>> import os, importlib.util
>>> spec = importlib.util.find_spec("os")
>>> module = importlib.util.module_from_spec(spec)
>>> spec.loader.exec_module(module)
>>> module is os
False
>>> module.curdir
'.'
>>> module.environ == os.environ
True
exarkun commented 5 years ago

So this suggests a load_module like:

import importlib.util
def load_module(name, original_module):
    spec = importlib.util.find_spec(original_module.__name__)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    module.__name__ = name
    return module

Unlike my previously suggested work-around this one actually gives you a module where, when you override globals on it, its contents respect your override. But also unlikely the previous suggestion, this is Python 3-only solution. If it works under pyinstaller and we find a Python 2 analog then together they might make a solution to this issue.

itamarst commented 5 years ago

I don't really care about Python 2 very much, insofar as I'm going to drop support real soon now. As such, I'd be happy to accept a workaround for Python 2 that's less ideal, e.g. not bothering copying the module at all and just risking I/O in logging codepath for people on PyInstaller on Python 2.

itamarst commented 5 years ago

zipimporter doesn't implement exec_module :angry: