When building a Windows .exe file via packaging/PackageWithPyInstaller.py, we include the Operations modules as Hidden Imports as most of them are imported programmatically with Mantid Imaging. Due to recent changes in conda/PyInstaller/pkgutil (it is not clear exactly where the change is coming from), superfluous modules are being picked up which do not exist in MantidImaging and are causing AssertionErrors during runtime.
This is not caused by recent commits/merges to MI as even if you checkout an old branch (e.g. release-2.7.0), the errors still persist.
Steps To Reproduce
Update Conda (issues found with 23.3.1)
Rebuild Developer Environment
Follow instructions in the Release Instructions to create a Windows build with PyInstaller.
Expected Behaviour
The Windows executable should build with no errors and also have no errors during runtime.
Current Behaviour
Packaging errors and runtime errors show up.
Context
Conda 23.3.1
PyInstaller 6.7.0
Python 3.10.14
Failure Logs
Errors when running PackageWithPyInstaller.py:
100727 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support'
100728 ERROR: Hidden import 'mantidimaging.core.operations.test.support' not found
100728 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.bytecode_helper'
100728 ERROR: Hidden import 'mantidimaging.core.operations.test.support.bytecode_helper' not found
100728 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.hashlib_helper'
100729 ERROR: Hidden import 'mantidimaging.core.operations.test.support.hashlib_helper' not found
100729 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.import_helper'
100729 ERROR: Hidden import 'mantidimaging.core.operations.test.support.import_helper' not found
100729 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.interpreters'
100729 ERROR: Hidden import 'mantidimaging.core.operations.test.support.interpreters' not found
100729 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.logging_helper'
100730 ERROR: Hidden import 'mantidimaging.core.operations.test.support.logging_helper' not found
100730 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.os_helper'
100730 ERROR: Hidden import 'mantidimaging.core.operations.test.support.os_helper' not found
100730 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.script_helper'
100730 ERROR: Hidden import 'mantidimaging.core.operations.test.support.script_helper' not found
100730 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.socket_helper'
100730 ERROR: Hidden import 'mantidimaging.core.operations.test.support.socket_helper' not found
100730 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.testresult'
100730 ERROR: Hidden import 'mantidimaging.core.operations.test.support.testresult' not found
100731 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.threading_helper'
100733 ERROR: Hidden import 'mantidimaging.core.operations.test.support.threading_helper' not found
100733 INFO: Analyzing hidden import 'mantidimaging.core.operations.test.support.warnings_helper'
100734 ERROR: Hidden import 'mantidimaging.core.operations.test.support.warnings_helper' not found
Errors during runtime (with Windows build make with PyInstaller)
C:\Users\ddb29996\mantidimaging\packaging\dist\MantidImaging>MantidImaging.exe
2024-07-31 15:20:39,811 [mantidimaging.core.parallel.manager:L40] INFO: Creating process pool with 3 processes
2024-07-31 15:20:39,853 [perf.mantidimaging.core.parallel.manager:L44] INFO: Process pool started in 0.046999999991385266
2024-07-31 15:20:43,205 [perf.mantidimaging.gui.windows.main.view:L183] INFO: Mantid Imaging ready in 3.4210000000020955
2024-07-31 15:20:43,206 [perf.mantidimaging.gui.mvp_base.view:L65] INFO: MainWindowView shown in 3.3430000000080327
Process SpawnPoolWorker-1:
Process SpawnPoolWorker-2:
Traceback (most recent call last):
File "multiprocessing\process.py", line 314, in _bootstrap
File "multiprocessing\process.py", line 108, in run
File "multiprocessing\pool.py", line 109, in worker
File "mantidimaging\core\parallel\manager.py", line 49, in worker_setup
File "mantidimaging\core\operations\loader.py", line 52, in load_filter_packages
Traceback (most recent call last):
File "multiprocessing\process.py", line 314, in _bootstrap
File "mantidimaging\core\operations\loader.py", line 30, in _find_operation_modules
File "multiprocessing\process.py", line 108, in run
AssertionError
File "multiprocessing\pool.py", line 109, in worker
File "mantidimaging\core\parallel\manager.py", line 49, in worker_setup
File "mantidimaging\core\operations\loader.py", line 52, in load_filter_packages
File "mantidimaging\core\operations\loader.py", line 30, in _find_operation_modules
AssertionError
Summary
When building a Windows .exe file via
packaging/PackageWithPyInstaller.py, we include the Operations modules as Hidden Imports as most of them are imported programmatically with Mantid Imaging. Due to recent changes in conda/PyInstaller/pkgutil (it is not clear exactly where the change is coming from), superfluous modules are being picked up which do not exist in MantidImaging and are causing AssertionErrors during runtime.This is not caused by recent commits/merges to MI as even if you checkout an old branch (e.g. release-2.7.0), the errors still persist.
Steps To Reproduce
Update Conda (issues found with
23.3.1) Rebuild Developer Environment Follow instructions in the Release Instructions to create a Windows build with PyInstaller.Expected Behaviour
The Windows executable should build with no errors and also have no errors during runtime.
Current Behaviour
Packaging errors and runtime errors show up.
Context
Conda 23.3.1 PyInstaller 6.7.0 Python 3.10.14
Failure Logs
Errors when running
PackageWithPyInstaller.py:Errors during runtime (with Windows build make with PyInstaller)