Tensorflow hook AttributeError: module 'site' has no attribute 'getsitepackages'

[jjwall]

First I'd like to say thank you to the contributors of pyinstaller! I'm relatively new to Python and have found working with the ecosystem can be pretty challenges at times, but pyinstaller has helped immensely in several of my software deliveries.

Now to the bug at hand:

I am getting the following error when trying to package Tensorflow version 2.16.1 (also tested with 2.17.0):

File "Lib\site-packages\_pyinstaller_hooks_contrib\rthooks\pyi_rth_tensorflow.py", line 52, in <module>

File "Lib\site-packages\_pyinstaller_hooks_contrib\rthooks\pyi_rth_tensorflow.py",
line 37, in _pyi_rthook AttributeError: module 'site' has no attribute 'getsitepackages'
[30184] Failed to execute script 'pyi_rth_tensorflow' due to unhandled exception!

Note that I do have a venv set up and am invoking pyinstaller from it activated.

Error gets raised when running my pyinstaller outputted .exe program.

The hook appears to compensate for older versions of Tensorflow (< 2.12) that may cause issues for newer versions of Tensorflow (>= 2.12).

To Reproduce

Write a simple module that consumes a tf method. Below is my app.py file:

import tensorflow as tf

test = "test"
tf.is_tensor(test)

My pyinstaller spec file (pyinstaller.spec) is set up like so:

#-*- mode: python ; coding: utf-8 -*-
import sys ; sys.setrecursionlimit(sys.getrecursionlimit() * 5)

a = Analysis (
    ['app.py'],
    pathex=[], 
    binaries=[], 
    datas=[],
    hiddenimports=[], 
    hookspath=[], 
    hooksconfig=[],
    runtime_hooks=[], 
    excludes=[], 
    noarchive=False,
)
pyz = PYZ(a, pure)

exe = EXE(
    pyz,
    a.scripts,
    [],
    exclude_binaries=True, 
    name='app', 
    debug=False, 
    bootloader_ignore_signals=False,
    upx=True, 
    console=True, 
    disable_windowed_traceback=Flase
    argv_emulation=False,
    target_arch=None, 
    codesign_identity=None, 
    entitlements_file=None,
)

coll = COLLECT(
    exe,
    a. binaries,
    a. datas,
    strip=False,
    upx=True, 
    upx_exclude=[],
    name='app'
)

And invoked by running:

pyinstaller pyinstaller.spec

Desktop Environment:

• OS: Windows
• Python Version: python 3.10.11
• Version of pyinstaller-hooks-contrib: 2024.8
• Version of PyInstaller: 6.10

Workaround Workaround has been to comment out these lines in the Tensorflow hook and add the following snippet to my Python program so the dynamic library gets properly picked up:

import os

tensorflow_dynamic_lib = "_pywrap_tensorflow_internal.pyd"

# For pyinstaller builds, check if tensorflow pyd file is at root.
# Then set Tensorflow environment variable: TF_PLUGGABLE_DEVICE_LIBRARY_PATH
if os.path.isfile(tensorflow_dynamic_lib):
    os.environ["TF_PLUGGABLE_DEVICE_LIBRARY_PATH"] = tensorflow_dynamic_lib

Expected Behavior Expected behavior would be that the Tensorflow dynamic library gets picked up without having to do this manual edit to my venv's site-packages.

Additional Notes Again, I'm new to pyinstaller so I imagine there's a better work around than having to manually edit a file in my venv's site-packages. My intuition tells me adding an excludes entry wouldn't be a bad idea.

Also, I read that pyinstaller dropped support for fake-modules, so I'm curious why the Tensorflow hook is even monkey patching the getsitepackages attribute given these changes.

[rokm]

Also, I read that pyinstaller dropped support for fake-modules, so I'm curious why the Tensorflow hook is even monkey patching the getsitepackages attribute given these changes.

To work around the issues with older tensorflow versions.

---

That said, I cannot reproduce this. There seem to be some mentions of site not having getsitepackages when using virtualenv instead of stdlib's venv for virtual environments, but the latest version of virtualenv seems to work for me as well. So are you using virtual environment? If so, which one (and if virtualenv, what version)?

Can you check build/<name>/Analysis-00.toc˙ (orPYZ-00.toc) and check wheresite.py` was collected from?

Also, I suspect your modification to the program is actually a no-op; with PyInstaller 6.x on Windows, _pywrap_tensorflow_internal.pyd should not exist in the top-level application directory. Can you verify that?

Does changing https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/35ac0e065370cfa09577252f6ad0347b7ed6d8be/_pyinstaller_hooks_contrib/rthooks/pyi_rth_tensorflow.py#L37 into

_orig_getsitepackages = getattr(site, 'getsitepackages', None) 

fix the problem (without trying to set TF_PLUGGABLE_DEVICE_LIBRARY_PATH in your program)?

[jjwall]

Thanks for the quick response!

---

So are you using virtual environment? If so, which one (and if virtualenv, what version)?

I'm using the stdlib's venv with Python 3.10.11 and created my virtual environment at my project's root dir by running:

python -m venv .venv

I believe this uses virtualenv under the hood and if that's the case it would be virtualenv version 20.26.3.

Can you check build//Analysis-00.toc˙ (or PYZ-00.toc) and check where site.py` was collected from?

site.py is being collected from /.venv/Lib/site-packages/PyInstaller/fake-modules/site.py

...and with this check I'm realizing I'm running pyinstaller version 5.1 not 6.10.

Does making this modification: _orig_getsitepackages = getattr(site, 'getsitepackages', None) to pyi_rth_rensorflow.py (without trying to set TF_PLUGGABLE_DEVICE_LIBRARY_PATH) fix the problem?

Tested this on my current build setup without upgrading pyinstaller, and yes it does appear to fix the problem.

Retrospective After upgrading to pyinstaller version 6.10 from version 5.1 this resolved my problem without having to set TF_PLUGGABLE_DEVICE_LIBRARY_PATH or modifying my venv's site-packages.

Thanks for the help!

---

To work around the issues with older tensorflow versions.

To clear up my confusion: If pyinstaller dropped support for fake-modules, wouldn't this workaround still throw errors? Perhaps I'm confused when a venv's or system installed python interpreter's use of site-packages begins and pyinstaller's use of fake-modules ends.

[rokm]

To work around the issues with older tensorflow versions.

To clear up my confusion: If pyinstaller dropped support for fake-modules, wouldn't this workaround still throw errors? Perhaps I'm confused when a venv's or system installed python interpreter's use of site-packages begins and pyinstaller's use of fake-modules ends.

"Dropping support for fake site module" means that we are now using stdlib one. And because PyInstaller-frozen application runs with site disabled, stdlib site module does not set some of the variables that our old fake site module did (which was, essentially, a non-compliant behavior). That's why we need https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/35ac0e065370cfa09577252f6ad0347b7ed6d8be/_pyinstaller_hooks_contrib/rthooks/pyi_rth_tensorflow.py#L21-L22, because some older versions of tensorflow require USER_SITE to be different from None.

The other work-around, https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/35ac0e065370cfa09577252f6ad0347b7ed6d8be/_pyinstaller_hooks_contrib/rthooks/pyi_rth_tensorflow.py#L37-L46, is required for the reasons outlined in the comment above it. And this work-around breaks with early PyInstaller 5.x versions (<= 5.4, to be exact) that were still using our fake site module, which does not provide getsitepackages.