Session open error using pyinstaller

[Villanut0]

Hi !

I'm trying to create an executable for my python application with Pyinstaller but I'm running into some issues. My application is flashing STM32L475 MCUs by using 2 STLinksV3 and is working fine before using Pyinstaller.

First, I had the same problems as #1422 and #1358 but I was able to resolve them thanks to the discussion #1274. I added the following lines :

from pyocd.probe.aggregator import PROBE_CLASSES
from pyocd.probe.stlink_probe import StlinkProbe
PROBE_CLASSES["stlink"] = StlinkProbe

And I'm now able to get the sessions for my STLinks using ConnectHelper.get_sessions_for_all_connected_probes()

The problem now, is that when I try to open these sessions my application crashes with the following output:

Traceback (most recent call last):
  File "src/ProbesHandler.py", line 44, in getSessionDevice
    session.open()
  File "pyocd/core/session.py", line 525, in open
  File "pyocd/probe/stlink_probe.py", line 140, in open
  File "pyocd/probe/stlink/stlink.py", line 253, in set_prescaler
AssertionError

My .spec file is the following:

# -*- mode: python ; coding: utf-8 -*-
import platform
from PyInstaller.utils.hooks import (get_package_paths, collect_dynamic_libs)

block_cipher = None

is_windows = (platform.system() == "Windows")

cpm_path = get_package_paths('cmsis_pack_manager')[1]
if is_windows:
    cpm_lib_name = "native.so"
    cpm_lib_path = os.path.join(cpm_path, 'cmsis_pack_manager', cpm_lib_name)
    cpm_lib_path_deploy = 'cmsis_pack_manager/cmsis_pack_manager'
else:
    cpm_lib_name = "_native__lib.so"
    cpm_lib_path = os.path.join(cpm_path, cpm_lib_name)
    cpm_lib_path_deploy = 'cmsis_pack_manager'

pyocd_path = get_package_paths('pyocd')[1]
svd_path = os.path.join(pyocd_path, 'debug', 'svd', 'svd_data.zip')

a = Analysis(
    ['../programmer.py'],
    pathex=[],
    binaries=[],
    datas=[('../img', 'img'), ('../ui', 'ui'), ('../src', 'src'), ('../binaries', 'binaries'), (cpm_lib_path, cpm_lib_path_deploy), (svd_path, 'pyocd/debug/svd/.')],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
    pyz,
    a.scripts,
    [],
    exclude_binaries=True,
    name='programmer',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)
coll = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='programmer',
)

I don't know if I'm doing something wrong. The same application is working fine until I try to pack it with Pyinstaller.

Do you have an idea of what's happening?

Thanks

[flit]

The problem is related to package entry points used by pyocd for plugins. Another user (Giovanni I.) has the solution below. Please note that I haven't tested this out myself. And Linux/Mac may be slightly different than Windows. At some point I'd like to build and release a standard binary for pyocd, but it will take some time and isn't particularly high priority (this would be a great contribution from someone!).

From Giovanni: I found a way to overcome this problem. I added in the .spec file of pyinstaller these lines and it works for me :

from PyInstaller.utils.hooks import collect_entry_point,copy_metadata
datas,hiddenimports= collect_entry_point('pyocd.probe')
datas2,hiddenimports2=collect_entry_point('pyocd.rtos')

[Villanut0]

Hi,

Thanks for your answer.

I did what you told me:

• Add collect_entry_point for pyocd.probe and pyocd.rtos in the spec file
• Add entire cmsis_pack_manager folder in the datas
• Override __init__ of Cache class

I now have a different issue. When I run the executable generated by PyInstaller on Linux, I get the following error:

Traceback (most recent call last):
  File "programmer.py", line 16, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "src/ProbesHandler.py", line 2, in <module>
    from pyocd.probe.aggregator import PROBE_CLASSES
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "pyocd/__init__.py", line 21, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "pyocd/gdbserver/__init__.py", line 17, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "pyocd/gdbserver/gdbserver.py", line 41, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "pyocd/rtos/__init__.py", line 29, in <module>
  File "pyocd/core/plugin.py", line 97, in load_plugin_classes_of_type
  File "pkg_resources/__init__.py", line 2444, in load
  File "pkg_resources/__init__.py", line 2467, in require
  File "pkg_resources/__init__.py", line 787, in resolve
pkg_resources.DistributionNotFound: The 'intelhex<3.0,>=2.0' distribution was not found and is required by the application
[488348] Failed to execute script 'programmer' due to unhandled exception!

The distribution not found is different everytime I run the executable. It misses intelhex, cmsis-pack-manager, natsort, intervaltree and a lot of other modules. Even if I had them in the datas of the spec file.

Here is my new spec file:

# -*- mode: python ; coding: utf-8 -*-
import platform
from PyInstaller.utils.hooks import get_package_paths, collect_entry_point, copy_metadata, collect_all

datas_probe, hiddenimports_probe = collect_entry_point('pyocd.probe')
datas_rtos, hiddenimports_rtos = collect_entry_point('pyocd.rtos')

block_cipher = None

is_windows = (platform.system() == "Windows")

cpm_path = get_package_paths('cmsis_pack_manager')[1]
if is_windows:
    # cpm_lib_name = "native.so"
    # cpm_lib_path = os.path.join(cpm_path, 'cmsis_pack_manager', cpm_lib_name)
    cpm_lib_path = cpm_path
    cpm_lib_path_deploy = 'cmsis_pack_manager/cmsis_pack_manager'
else:
    # cpm_lib_name = "_native__lib.so"
    # cpm_lib_path = os.path.join(cpm_path, cpm_lib_name)
    cpm_lib_path = cpm_path
    cpm_lib_path_deploy = 'cmsis_pack_manager'

pyocd_path = get_package_paths('pyocd')[1]
svd_path = os.path.join(pyocd_path, 'debug', 'svd', 'svd_data.zip')

datas = [
    ('../img', 'img'),
    ('../ui', 'ui'),
    ('../src', 'src'),
    ('../binaries', 'binaries'),
    ('../patchs', 'patchs'),
    (cpm_lib_path, cpm_lib_path_deploy),
    (svd_path, 'pyocd/debug/svd/.')
]
datas = datas + datas_probe + datas_rtos
hiddenimports = hiddenimports_probe + hiddenimports_rtos

a = Analysis(
    ['../programmer.py'],
    pathex=[],
    binaries=[],
    datas=datas,
    hiddenimports=hiddenimports,
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
    pyz,
    a.scripts,
    [],
    exclude_binaries=True,
    name='programmer',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)
coll = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='programmer',
)

Maybe I did not understand something in your instructions but the problem appears when I add the collect_entry_point in the datas.

[flit]

Well, I was just passing along instructions from another user, so I can't really tell you what's wrong, unfortunately. However, it sounds more like a general pyInstaller usage issue rather than something specific to pyocd—just a guess, though.

However, it turns out that the changes to cmsis-pack-manager are unnecessary unless you are intending to package CMSIS-Packs directly in the pyocd executable. (I'll update the instructions from earlier to remove this.)

[edcloudcycle]

Hi @Villanut0, Did you make any progress with this? I just got to the point of my application being finished and have got stuck on pyocd not packaging with everything else. I will have a look at the suggestions above to see if they work for me. Ed

[edcloudcycle]

I think I got this working for my application (tm_fct). I am just including pyocd as a module so maybe it is different but using this specfile it does seem to be working. The main problem was that things were just not getting copied so I had to add them explicitly.

# -*- mode: python ; coding: utf-8 -*-
#
# Custom spec file for pyinstaller DO NOT CALL pyinstaller DIRECTLY
# Instead use the package_to_exe.py script provided which will also copy all other files needed
#

from PyInstaller.utils.hooks import get_package_paths, collect_entry_point, copy_metadata, collect_all

datas_probe, hiddenimports_probe = collect_entry_point('pyocd.probe')
datas_rtos, hiddenimports_rtos = collect_entry_point('pyocd.rtos')

pyocd_path = get_package_paths('pyocd')[1]
cmsis_path = ".\\.venv\\Lib\\site-packages\\cmsis_pack_manager"
pylink_path = get_package_paths('pylink')[1]

datas = [(pyocd_path, 'pyocd/.'),
         (cmsis_path, 'cmsis_pack_manager/.'),
         (pylink_path, 'pylink/.')]
datas += copy_metadata('nidaqmx')
datas = datas + datas_probe + datas_rtos
hiddenimports = hiddenimports_probe + hiddenimports_rtos

a = Analysis(
    ['tm_fct.py'],
    pathex=[],
    binaries=[],
    datas=datas,
    hiddenimports=hiddenimports,
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
)
pyz = PYZ(a.pure)

exe = EXE(
    pyz,
    a.scripts,
    [],
    exclude_binaries=True,
    name='tm_fct',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)
coll = COLLECT(
    exe,
    a.binaries,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='tm_fct',
)

[Villanut0]

Hi @edcloudcycle! Sorry for the late reply. I'm no longer working on this project and it has been somewhat abandoned. I'm glad to hear you've managed to make it work!

[edcloudcycle]

Hi,

I think you will need the _internal directory as well as the exe. If you want everything inside the exe I think there is an option for that.

Cheers

Ed

From: Julián Ordóñez @.> Sent: Monday, January 8, 2024 2:48 AM To: pyocd/pyOCD @.> Cc: Ed Waugh @.>; Mention @.> Subject: Re: [pyocd/pyOCD] Session open error using pyinstaller (Issue #1529)

Hi @edcloudcycle https://github.com/edcloudcycle I wanted to thank you for providing the .spec file, it worked fine, I just had to remove the line datas += copy_metadata('nidaqmx') since I didn't need it.

The command I used was pyinstaller file.spec, this generated the .exe and the _internal directory, but if I move the .exe to another location it does not work.

Is it related to this comment ?

Custom spec file for pyinstaller DO NOT CALL pyinstaller DIRECTLY

Instead use the package_to_exe.py script provided which will also copy all other files needed

I couldn't find the package_to_exe.py file.

Any suggestion?

— Reply to this email directly, view it on GitHub https://github.com/pyocd/pyOCD/issues/1529#issuecomment-1880316849 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AQOYBKG44WYLRFJT2ZWPR2TYNNM7TAVCNFSM6AAAAAAWUJRCX2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBQGMYTMOBUHE . You are receiving this because you were mentioned.Message ID: @.***>

[julian-0]

Hi all!

Somehow I deleted my previous comment :sweat_smile:

I was able to generate an executable that runs but when i try to open the session with: ConnectHelper.session_with_chosen_probe(blocking=False, options={"chip_erase": "sector", "target_override": "stm32l431vctx"})

I get this error:

Target type stm32l431vctx not recognized. Use 'pyocd list --targets' to see currently available target types. See <https://pyocd.io/docs/target_support.html> for how to install additional target support.
Traceback (most recent call last):
  File "pyocd\board\board.py", line 111, in __init__
    self.target = TARGET[self._target_type](session)
KeyError: 'stm32l431vctx'

This is the .spec I use to generate the exe:

from PyInstaller.utils.hooks import get_package_paths, collect_entry_point, copy_metadata, collect_all

datas_probe, hiddenimports_probe = collect_entry_point('pyocd.probe')
datas_rtos, hiddenimports_rtos = collect_entry_point('pyocd.rtos')

pyocd_path = get_package_paths('pyocd')[1]
cmsis_path = "..\\.\\venv\\Lib\\site-packages\\cmsis_pack_manager"
pylink_path = get_package_paths('pylink')[1]

datas = [(pyocd_path, 'pyocd/.'),
         (cmsis_path, 'cmsis_pack_manager/.'),
         (pylink_path, 'pylink/.')]
datas = datas + datas_probe + datas_rtos + datas_targets
hiddenimports = hiddenimports_probe + hiddenimports_rtos

a = Analysis(
    ['controller.py'],
    pathex=[],
    binaries=[],
    datas=datas,
    hiddenimports=hiddenimports,
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
)
pyz = PYZ(a.pure)

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    [],
    name='controller',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
)

Do you know how I can include the target types installed in my environment?

[edcloudcycle]

Hi Julian, I am afraid I am not sure. I guess it works ok when it is not packaged? All I can notice is I have a collect step in my script that references the datas. Maybe that is doing the copy? Thanks Ed

[julian-0]

Yes, when it is not packaged it works perfectly. I am going to try what you tell me, thanks!

[julian-0]

Unfortunately it didn't work.

I found that the managed packages I installed are stored in my %APPDATA%/Local/cmsis-pack-manager and not in the virtual enviroment as i thought

I managed to copy these files to the package but pyocd was not using it as I was getting the same error.

Then I tried adding the pack session option with the path to the managed_packs directory of the image but I get the same error again.

Target type stm32l431vctx not recognized. Use 'pyocd list --targets' to see currently available target types. See <https://pyocd.io/docs/target_support.html> for how to install additional target support.
Traceback (most recent call last):
  File "pyocd\board\board.py", line 111, in __init__
    self.target = TARGET[self._target_type](session)
KeyError: 'stm32l431vctx'

Can you think of anything else I could try?

Thanks

[edcloudcycle]

Maybe if you try activating your venv before installing pyocd and the packs they will all be in the right place. That might help. You could also try copying any missing files manually to the folders after they are created.

[clopez-a2e]

This worked for me to generate a single .exe for my application, which is a flash loader (thanks to the help from the posts in this thread). I run this with pyinstaller my_flashdaq.spec:

# -*- mode: python ; coding: utf-8 -*-
#

from PyInstaller.utils.hooks import get_package_paths, collect_entry_point, copy_metadata, collect_all
from PyInstaller.__main__ import run

datas_probe, hiddenimports_probe = collect_entry_point('pyocd.probe')
datas_rtos, hiddenimports_rtos = collect_entry_point('pyocd.rtos')

pyocd_path = get_package_paths('pyocd')[1]
cmsis_path = "C:\\Python312\\Lib\\site-packages\\cmsis_pack_manager"
pylink_path = get_package_paths('pylink')[1]

datas = [(pyocd_path, 'pyocd/.'),
         (cmsis_path, 'cmsis_pack_manager/.'),
         (pylink_path, 'pylink/.')]
datas = datas + datas_probe + datas_rtos
hiddenimports = hiddenimports_probe + hiddenimports_rtos

a = Analysis(
    ['flashdaq.py'],
    pathex=[],
    binaries=[('c:\\libusb\\libusb-1.0.dll', '.')],
    datas=datas,
    hiddenimports=hiddenimports,
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
)

pyz = PYZ(a.pure, a.zipped_data)

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    name='flashdaq',
    debug=False,
    strip=False,
    upx=True,
    console=True,  # Change to False if you want no console to appear
    icon=None,  # Add path to .ico file here if you want a custom icon
    upx_exclude=[],
    runtime_tmpdir=None,
    bootloader_ignore_signals=False
)

if __name__ == '__main__':
    # Use PyInstaller directly to handle the build as a one-dir if desired:
    run([
        '--name=%s' % exe.name,
        '--onefile',
        '--noconfirm',
        '--log-level=INFO',
        'flashdaq.py'
    ])

[julian-0]

I have some updates too, I finally got it working.

I was having this error triying to open the session in the .exe: Target type stm32l431vctx not recognized.

So I copied the missing pack files (from the image of my previous comment) into my project and then when I opened the session I added the option "pack" with the path to the files:

keil_path = [os.path.join(managed_packs_path, 'Keil.STM32L4xx_DFP.2.6.2.pack'), os.path.join(managed_packs_path, 'Keil.STM32F4xx_DFP.2.17.1.pack')]
session = ConnectHelper.session_with_chosen_probe(blocking=False, options={"chip_erase": "sector", "target_override": target_str, "pack": keil_path})
session.open()

I hope this helps someone :)