Packaging dependencies with data files

[dcnieho]

Describe the bug I am not sure this is a bug, but i cannot find how to solve this in the manual/FAQ or with googling. This is different from the usual data_files question. In this case i do not want to include data_files of my app, but data files that come with a package used by my app (so data files contained in site_packages). How do i do that?

To Reproduce In a clean venv, run pip install glassesValidator

Test script to freeze (test.py):

import tempfile
import os
from glassesValidator import config

with tempfile.TemporaryDirectory() as tmpdirname:
    config.deploy_validation_config(tmpdirname)
    print(os.listdir(tmpdirname))

script to build the exe:

import cx_Freeze
import pathlib
import site
import sys

path = pathlib.Path(__file__).absolute().parent

def get_include_files():
    files = []

    # ffpyplayer bin deps
    for d in site.getsitepackages():
        d=pathlib.Path(d)/'share'/'ffpyplayer'
        for lib in ('ffmpeg', 'sdl'):
            d2 = d/lib/'bin'
            if d2.is_dir():
                for f in d2.iterdir():
                    if f.is_file() and f.suffix=='' or f.suffix in ['.dll', '.exe']:
                        files.append((f,pathlib.Path('lib')/f.name))
    return files

build_options = {
    "build_exe": {
        "optimize": 1,
        "packages": ['OpenGL','glassesValidator',           # packages needed for glassesValidator to import when frozen, not relevant for this problem
            'ffpyplayer.player','ffpyplayer.threading',     # some specific subpackages that need to be mentioned to be picked up correctly
            'imgui_bundle._imgui_bundle'
        ],
        "excludes":["tkinter"],
        "zip_include_packages": "*",
        "zip_exclude_packages": [   # needed for glassesValidator to import when frozen, not relevant for this problem
            "OpenGL_accelerate",
            "glfw",
            "imgui_bundle",
        ],
        "silent_level": 1,
        "include_msvcr": True
    }
}
if sys.platform.startswith("win"):
    build_options["build_exe"]["include_files"] = get_include_files()

cx_Freeze.setup(
    name="test",
    version="0.0.1",
    description="test",
    executables=[
        cx_Freeze.Executable(
            script=path / "test.py",
            target_name="test"
        )
    ],
    options=build_options,
    py_modules=[]
)

(that is a little more complicated than you'd hope, not all these packages work well from the zip, or are picked up correctly, but thats not the problem here).

When i run test.py from the venv, it works as expected, printing ['markerPositions.csv', 'poster', 'targetPositions.csv', 'validationSetup.txt']. When i run the frozen exe, i get:

Traceback (most recent call last):
  File "C:\dat\projects\gazeMapper\cxFreeze\.venv\Lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 140, in run
    module_init.run(f"__main__{name}")
  File "C:\dat\projects\gazeMapper\cxFreeze\.venv\Lib\site-packages\cx_Freeze\initscripts\console.py", line 25, in run
    exec(code, main_globals)
  File "C:\dat\projects\gazeMapper\cxFreeze\test.py", line 6, in <module>
    config.deploy_validation_config(tmpdirname)
  File "C:\dat\projects\gazeMapper\cxFreeze\.venv\lib\site-packages\glassesValidator\config\__init__.py", line 62, in deploy_validation_config
    with importlib.resources.path(__package__, r) as p:
  File "C:\Program Files\Python310\lib\importlib\resources.py", line 121, in path
    _path_from_reader(reader, _common.normalize_path(resource))
  File "C:\Program Files\Python310\lib\importlib\resources.py", line 130, in _path_from_reader
    return _path_from_resource_path(reader, resource) or _path_from_open_resource(
  File "C:\Program Files\Python310\lib\importlib\resources.py", line 141, in _path_from_open_resource
    saved = io.BytesIO(reader.open_resource(resource).read())
  File "C:\Program Files\Python310\lib\importlib\readers.py", line 35, in open_resource
    return super().open_resource(resource)
  File "C:\Program Files\Python310\lib\importlib\abc.py", line 433, in open_resource
    return self.files().joinpath(resource).open('rb')
  File "C:\Program Files\Python310\lib\zipfile.py", line 2342, in open
    raise FileNotFoundError(self)
FileNotFoundError: C:\dat\projects\gazeMapper\cxFreeze\build\exe.win-amd64-3.10\lib\library.zip/glassesValidator/config/markerPositions.csv

When i inspect library.zip, indeed the markerPositions.csv (and other non-python files) are missing from glassesValidator/config, while they are present in the site_packages of the environment used for building. If relevant, note that these data files are included in the package's manifest.

Expected behavior Data files of a package used by the script that is being frozen are included in the library.zip.

Desktop (please complete the following information):

• Windows 11 Enterprise
• amd64
• cx_Freeze version 7.2.3
• Python version 3.10

[marcelotduarte]

i do not want to include data_files of my app, but data files that come with a package used by my app

You say 'i do not want', but apparently you need zip_includes; It is similar to include_files but copies the data files to the zip file.

[dcnieho]

ah ok! So, since these are not python files, I should manually make sure that the data files are found and included in the zip. Thanks for the pointer to the right functionality. Looking at for instance the hooks for OpenCV, i see that is what you do as well: https://github.com/marcelotduarte/cx_Freeze/blob/0dd2a31b42937c30739cab94bdb7451556fe4763/cx_Freeze/hooks/cv2.py

Is this always what you have to do if you're dealing with packaged with data files that are not handled by a hook? I am asking that also because i see another package, imgui_bundle, for which cx_Freeze has no specific support, but which does get all its data files included. I list that package in "zip_exclude_packages" however, are those handled differently?

[marcelotduarte]

In fact, I force OpenCV to be kept outside of zip, because it doesn't work well in zip. And when I use this option (see line 59), data files are not copied, in the same way that they are not copied when using zip. Directories that do not have __init__ are generally not copied. If they have __init__, they are considered a package and are treated differently by Python, which is then inherited by cx_Freeze. In other words, we need to create hooks when we need to optimize a very large package (for example, PyTorch) or when it does not follow some Python rules (PyQt*). If it fails at some point, I create the hook. In general, packages that have a C interface need a hook.

An example where I needed to use zip-includes in a hook is tzdata when zip_include_packages is used, the data is not copied, so I copy it in this case.

[dcnieho]

Ok, thanks!

I've added the following to solve my problem:

def get_zip_include_files():
    files = []
    for d in site.getsitepackages():
        base = pathlib.Path(d)
        p = base / 'glassesValidator' / 'config'

        for f in p.rglob('*'):
            if f.is_file() and f.suffix not in ['.py','.pyc']:
                files.append((f, pathlib.Path(os.path.relpath(f,base))))
    return files

So in conclusion, it doesn't matter whether we're talking about data files of my project, or of packages, custom handling may be needed for them to be picked up.

[marcelotduarte]

We can make a hook for glassesValidator, which is simpler and can be reused by all users. I'll do it later.

[dcnieho]

I am not sure that is a good idea, I am almost certainly the only one that will be freezing an application using this package, its rather niche :)