zipimport is not PEP 3147 or PEP 488 compliant

[586616f8-21e9-4aac-9781-20cea90584c8]

BPO	25710
Nosy	@Yhg1s, @brettcannon, @ncoghlan, @ambv, @ericsnowcurrently, @serhiy-storchaka, @matrixise
Dependencies	bpo-25711: Rewrite zipimport from scratch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['extension-modules', 'type-bug'] title = 'zipimport is not PEP 3147 or PEP 488 compliant' updated_at = user = 'https://bugs.python.org/byrnes' ``` bugs.python.org fields: ```python activity = actor = 'lkollar' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules'] creation = creator = 'byrnes' dependencies = ['25711'] files = [] hgrepos = [] issue_num = 25710 keywords = [] message_count = 4.0 messages = ['255178', '255184', '259446', '325732'] nosy_count = 10.0 nosy_names = ['twouters', 'brett.cannon', 'ncoghlan', 'lukasz.langa', 'eric.snow', 'serhiy.storchaka', 'matrixise', 'superluser', 'byrnes', 'lkollar'] pr_nums = [] priority = 'normal' resolution = None stage = 'resolved' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue25710' versions = ['Python 3.6'] ```

[586616f8-21e9-4aac-9781-20cea90584c8]

zipimport is not PEP-3147 compliant: i.e., it looks for foo.pyc (in the same directory as foo.py) instead of __pycache__/foo.cpython-35.pyc. This is counterintuitive, and unfortunate because it means that installation directories (that obey PEP-3147 conventions) can't be zip archived from filesystems and then used by zipimport.

When support for PEP-488 was added to Python 3.5, zipimport was modified to just delete references to .pyo files, but it needs to know about the new .opt-[12] filename components (as well as the PEP-3147 version tags). As far as I can tell, the current version of zipimport can't import optimized bytecode files, using either the old or new filename conventions.

Finally, none of this behavior is documented. The zipimport docs still mention .pyo files (which were eliminated by PEP-488), and say nothing about the filename conventions expected within zip archives.

[brettcannon]

Not finding bytecode files is not that big of a deal. While it's a very minor performance loss, it isn't critical to how Python works (and since it looks for bytecode-only files that use-case isn't broken either).

But the real problem is that zipimport is such a nasty chunk of code that no one wants to fix it. Yet more evidence we just need to rewrite the whole module. I created issue bpo-25711 to track a rewrite.

[ambv]

If you're affected, there's a workaround. Use the following script to rewrite paths in your .zip file to be zipimport-compatible:

https://gist.github.com/ambv/909d38bdac4f3e719b7c

It preserves the preamble and all file metadata.

A little context why this is useful: for archives that are effectively "static" environments with entire dependency trees in them, this bug causes both a big slowdown in startup time (counted in seconds), as well as tens of megabytes of .pyc created on the fly. Examples include zipapp-style .pyz, Twitter's .pex or Facebook's .par.

So while we're waiting for a new zipimport implementation, the workaround listed above should unblock people uzing this form of Python package distribution.

[serhiy-storchaka]

zipimport has been rewritten in pure Python (bpo-25711).