bdist_egg --exclude-source-files does not work with Python3

[dmiyakawa]

(Coming from here: https://bugs.python.org/issue28630)

I'm trying to prepare an egg file without source code. I had used python setup.py bdist_egg --exclude-source-files, which does not work with Python3.

After digging into the issue more, I found PEP-3147 is related to the problem. https://www.python.org/dev/peps/pep-3147/

$ git clone https://github.com/dmiyakawa/python_greeting.git
..
$ cd python_greeting
$ python --version
Python 3.5.2
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py3.5.egg
Archive:  dist/greeting-1.0.0-py3.5.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:31   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:31   EGG-INFO/entry_points.txt
      319  11-09-2016 16:31   EGG-INFO/PKG-INFO
      213  11-09-2016 16:31   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:31   EGG-INFO/top_level.txt
        1  11-09-2016 16:31   EGG-INFO/zip-safe
      139  11-09-2016 16:31   greeting/__pycache__/__init__.cpython-35.pyc
      392  11-09-2016 16:31   greeting/__pycache__/greeting.cpython-35.pyc
---------                     -------
     1131                     8 files

Here, greeting/__pycache__/__init__.cpython-35.pyc and greeting/__pycache__/greeting.cpython-35.pyc are not where they should be. See the following results (with Python 2.7.12)

$ python --version
Python 2.7.12
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py2.7.egg
Archive:  dist/greeting-1.0.0-py2.7.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:32   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:32   EGG-INFO/entry_points.txt
      319  11-09-2016 16:32   EGG-INFO/PKG-INFO
      213  11-09-2016 16:32   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:32   EGG-INFO/top_level.txt
        1  11-09-2016 16:32   EGG-INFO/zip-safe
      143  11-09-2016 16:32   greeting/__init__.pyc
      482  11-09-2016 16:32   greeting/greeting.pyc
---------                     -------
     1225                     8 files

According to the chart in PEP-3147, original py file must also exist to make pyc cache in __pycache__ effective.

To let bdist_egg's --exclude-sourcce-files option work as it worked in Python2 era, pyc files in __pycache__ must be at where legacy Python2 placed. I tried tweaking the Py3's egg content with the following code, and the tweaked egg worked nicely.

RE_PYCACHE_FILE = re.compile('(.+/)__pycache__/(.+)\.cpython-3\d(.pyc)$')

with tempfile.TemporaryDirectory() as tmpdir:
    names_lst = []
    tmpdir_abs_path = os.path.abspath(tmpdir)
    with ZipFile(egg_path, 'r') as zf:
        for name in zf.namelist():
            m = RE_PYCACHE_FILE.match(name)
            if m:
                w_name = m.group(1) + m.group(2) + m.group(3)
            else:
                w_name = name
            w_path = os.path.join(tmpdir_abs_path, w_name)
            dir_path = os.path.dirname(w_path)
            if not os.path.exists(dir_path):
                os.makedirs(dir_path)
            with open(w_path, 'wb') as file_to_write:
                with zf.open(name) as file_to_read:
                    file_to_write.write(file_to_read.read())
            names_lst.append(w_name)
    with ZipFile(egg_path, 'w', compression=ZIP_DEFLATED) as zf:
        for name in names_lst:
            r_path = os.path.join(tmpdir_abs_path, name)
            zf.write(r_path, arcname=name)

[gaw19]

may i ask how would you use this code ? in a separate script ? or maybe contribute back into setuptools as a separate flag to enable it ?

[dmiyakawa]

Use it as a separate script toward the egg file that is discussed as problematic here.

[ghost]

Could you tell me where exactly, we need to keep the above script? Below is the structure of my folder. root_folder/ --------app_folder --------setup.py --------README.rst --------MANIFEST.in

[haobibo]

@dmiyakawa I created a PR (https://github.com/pypa/setuptools/pull/1145) to mend this issue. The PR directly rename files in the bdist_egg command. Your comments or suggestions are welcome.

[dmiyakawa]

Sorry I had been satisfied with my workaround and haven't seen this issue for a while.

Seems there are people who want the functionality, while I am still not sure if mainstream developers think the feature itself is still important enough, or, should be deprecated and removed in the future (I hope the former of course). #1145 looks quite nice.

Maybe wheel should have same functionality?

@bRsatUoh Try it after building the egg. dist/ dir and compiled egg will be needed.

@haobibo Looks very nice but CI complains something.

[laggardkernel]

Today, I encountered the same problem when build .egg package for Scrapy deployment. (.egg packages are required by scrapyd for deployment).

Even there's no __pycache__ in my source files, __pycache__ folders are added within the .egg pacakge, which doesn't happen with .wheel packages.

Seems a bug.

Environment:

• Python 3.7.3
• pip 19.0.3
• setuptools 40.8.0
• wheel 0.34.2