search

agronholm / apscheduler

Task scheduling library for Python
MIT License
6.3k stars 713 forks source link

PyInstaller fails to package APScheduler various `pkg_resources.DistributionNotFound` #556

Closed iamliamc closed 3 years ago

iamliamc commented 3 years ago

Describe the bug When using PyInstaller to compile an executable python application containing apscheduler the application fails to add all of apschedulers dependencies....

It seems that the apscheduler code is making some "on the run imports" and therefore PyInstaller does not find the packages it needs to integrate. Similar to 131.

Pyinstaller executable package can be configured using a "hook files" but I can't determine which imports are required...

Running the executable raises various pkg_resources.DistributionNotFound errors. What is strange is that running the same executable multiple times raises different exceptions. 

2021-10-12 11:02:17,694 - apscheduler.scheduler - INFO - Scheduler started
Exception in thread Thread-3:
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'setuptools>=0.7' distribution was not found and is required by the application
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'pytz' distribution was not found and is required by the application
2021-10-12 10:55:04,156 - apscheduler.scheduler - INFO - Scheduler started
Exception in thread Thread-3:
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'six>=1.4.0' distribution was not found and is required by the application

To Reproduce Working on this now... https://github.com/iamliamc/pyinstaller_apscheduler_test

Expected behavior Python standalone application works fine. Expect compiled version to run similarly.

Additional context Possibly related tickets: 131 554

As 131 describes I have tried switching over to explicit trigger declarations like: 

        self.scheduler.add_job(
            self.restore_defaults.__call__,
            id="RestoreDefaults",
            trigger=IntervalTrigger(
                start_date=self.scheduler.calc_start_for_interval_minutes(self.start_offset) + dt.timedelta(minutes=60),
                seconds=600,
            )
        )
agronholm commented 3 years ago

I think this is happening because pyinstaller omits the package metadata for these dependencies. Is there a way to tell it to package all that metadata, I wonder?

iamliamc commented 3 years ago

I think this is happening because pyinstaller omits the package metadata for these dependencies. Is there a way to tell it to package all that metadata, I wonder?

I will try this for apscheduler: https://pyinstaller.readthedocs.io/en/stable/hooks.html#PyInstaller.utils.hooks.copy_metadata

iamliamc commented 3 years ago

Since copy_metadata returns a subset of the collect_all:

>>> data1 = copy_metadata('apscheduler')
>>> data1
[('/home/liam/spectral/test/test_pyinstaller/venv/lib/python3.9/site-packages/APScheduler-3.8.0.dist-info', 'APScheduler-3.8.0.dist-info')]
>>> datas, binaries, hiddenimports = collect_all('apscheduler')
427475 WARNING: Unable to find package for requirement six from package apscheduler.
427475 INFO: Packages required by apscheduler:
['tzlocal', 'pytz', 'setuptools']
>>> datas[0] == data1[0]
True

I am trying this ap-scheduler-hook.py:

from PyInstaller.utils.hooks import collect_all

datas, binaries, hiddenimports = collect_all('apscheduler')
iamliamc commented 3 years ago

That didn't change anything.

The weirdest part is my standalone "minimal example" works... what does that point me towards... dependencies in my larger application? The stack trace still points me at apscheduler 

Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'pytz' distribution was not found and is required by the application
Sentry is attempting to send 0 pending error messages
agronholm commented 3 years ago

Did you add the recursive=True option?

agronholm commented 3 years ago

For reference, this is the currently distributed APScheduler pyinstaller hook: https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-apscheduler.py Note the lack of recursive=True there.

iamliamc commented 3 years ago

For reference, this is the currently distributed APScheduler pyinstaller hook: https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-apscheduler.py Note the lack of recursive=True there.

Thank you for the dialogue!

I was using collect_metadata('apscheduler') we'll see what happens copying the above but adding the recursive=True!

I guess the next step if this fails is figuring out what from my full application to add to the minimal example to get it to fail similarly (threading, other packages that use tzlocal 554 orrrr...)

iamliamc commented 3 years ago

No luck! Darn. I'll brain storm with some colleagues work on a minimal failing example... and be back. 

2021-10-13 08:30:03,309 - apscheduler.scheduler - INFO - Scheduler started
Exception in thread Thread-3:
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'setuptools>=0.7' distribution was not found and is required by the application
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

For future reference pyinstaller output: 

$ python -m PyInstaller "$SPEC_FILE"
786 INFO: PyInstaller: 4.5.1
789 INFO: Python: 3.9.7
813 INFO: Platform: Linux-4.14.108-ti-r113-armv7l-with-glibc2.24
836 INFO: UPX is not available.
852 INFO: Extending PYTHONPATH with paths
['/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/bacnet_read_publisher']
4295 INFO: checking Analysis
4298 INFO: Building Analysis because Analysis-00.toc is non existent
4304 INFO: Initializing module dependency graph...
4332 INFO: Caching module graph hooks...
4398 WARNING: Several hooks defined for module 'apscheduler'. Please take care they do not conflict.
4524 INFO: Analyzing base_library.zip ...
51312 INFO: Processing pre-find module path hook distutils from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
51321 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.9'
111912 INFO: Caching module dependency graph...
115679 INFO: running Analysis Analysis-00.toc
115868 INFO: Analyzing bacnet_read_publisher/__main__.py
129452 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-urllib3.packages.six.moves.py'.
189133 INFO: Processing pre-safe import module hook six.moves from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/tzlocal/unix.py:141: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  while start is not 0:
234843 INFO: Processing module hooks...
234846 INFO: Loading module hook 'hook-apscheduler.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/pyinstaller_hookspath'...
242137 INFO: Loading module hook 'hook-sentry_sdk.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
253061 INFO: Loading module hook 'hook-certifi.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
253118 INFO: Loading module hook 'hook-google.api.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
253143 INFO: Loading module hook 'hook-pytz.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
254505 INFO: Loading module hook 'hook-distutils.util.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
254572 INFO: Loading module hook 'hook-sqlite3.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
255497 INFO: Loading module hook 'hook-xml.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
256673 INFO: Loading module hook 'hook-_tkinter.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/tkinter/__init__.py", line 37, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: libtk8.6.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libtk8.6.so: cannot open shared object file: No such file or directory
257936 ERROR: Tcl/Tk improperly installed on this system.
257940 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
258015 INFO: Loading module hook 'hook-heapq.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
258080 INFO: Loading module hook 'hook-sysconfig.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
258235 INFO: Loading module hook 'hook-lib2to3.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
258830 INFO: Loading module hook 'hook-encodings.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
259870 INFO: Loading module hook 'hook-pkg_resources.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
266060 INFO: Processing pre-safe import module hook win32com from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
269206 WARNING: Hidden import "pkg_resources.py2_warn" not found!
274340 WARNING: Hidden import "pkg_resources.markers" not found!
274417 INFO: Loading module hook 'hook-pickle.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274483 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274491 INFO: Loading module hook 'hook-difflib.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274557 INFO: Loading module hook 'hook-distutils.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274565 INFO: Loading module hook 'hook-packaging.py' from '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks'...
275626 INFO: Looking for ctypes DLLs
276461 INFO: Analyzing run-time hooks ...
276645 INFO: Including run-time hook '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
276693 INFO: Including run-time hook '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
276753 INFO: Including run-time hook '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
276781 INFO: Including run-time hook '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py'
276882 INFO: Including run-time hook '/home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/rthooks/pyi_rth_certifi.py'
277248 INFO: Looking for dynamic libraries
282806 INFO: Looking for eggs
282809 INFO: Using Python library /usr/local/lib/libpython3.9.so.1.0
283049 INFO: Warnings written to /home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/build/pyinstaller/warn-pyinstaller.txt
284996 INFO: Graph cross-reference written to /home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/build/pyinstaller/xref-pyinstaller.html
286160 INFO: checking PYZ
286162 INFO: Building PYZ because PYZ-00.toc is non existent
286164 INFO: Building PYZ (ZlibArchive) /home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/build/pyinstaller/PYZ-00.pyz
300092 INFO: Building PYZ (ZlibArchive) /home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/build/pyinstaller/PYZ-00.pyz completed successfully.
300618 INFO: checking PKG
300621 INFO: Building PKG because PKG-00.toc is non existent
300623 INFO: Building PKG (CArchive) PKG-00.pkg
315317 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
315837 INFO: Bootloader /home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/venv/lib/python3.9/site-packages/PyInstaller/bootloader/Linux-32bit-arm/run
315839 INFO: checking EXE
315841 INFO: Building EXE because EXE-00.toc is non existent
315843 INFO: Building EXE from EXE-00.toc
315848 INFO: Appending archive to ELF section in EXE /home/gitlab-runner/builds/Bb9rLvYx/0/spectral/sbp/control/bacnet_read_publisher/dist/bacnet_read_publisher
316059 INFO: Building EXE from EXE-00.toc completed successfully.
agronholm commented 3 years ago

Does it complain about setuptools on every run though?

iamliamc commented 3 years ago

Does it complain about setuptools on every run though?

I just reran the same executable and it produces: 

2021-10-13 08:38:45,478 - apscheduler.scheduler - INFO - Scheduler started
Exception in thread Thread-3:
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'tzlocal~=2.0' distribution was not found and is required by the application
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

And a third time: 

Exception in thread Thread-3:
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'six>=1.4.0' distribution was not found and is required by the application
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Fourth time: 

Exception in thread Thread-3:
Traceback (most recent call last):
  File "apscheduler/schedulers/base.py", line 901, in _create_plugin_instance
KeyError: 'interval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 973, in _bootstrap_inner
  File "sentry_sdk/integrations/threading.py", line 69, in run
  File "sentry_sdk/_compat.py", line 54, in reraise
  File "sentry_sdk/integrations/threading.py", line 67, in run
  File "adapters/primary/scheduler/schedule_runner.py", line 128, in run
  File "adapters/primary/scheduler/scheduler.py", line 51, in add_job
  File "apscheduler/schedulers/base.py", line 425, in add_job
  File "apscheduler/schedulers/base.py", line 926, in _create_trigger
  File "apscheduler/schedulers/base.py", line 904, in _create_plugin_instance
  File "pkg_resources/__init__.py", line 2449, in load
  File "pkg_resources/__init__.py", line 2472, in require
  File "pkg_resources/__init__.py", line 772, in resolve
pkg_resources.DistributionNotFound: The 'setuptools>=0.7' distribution was not found and is required by the application
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
agronholm commented 3 years ago

Also, is this a problem? 4398 WARNING: Several hooks defined for module 'apscheduler'. Please take care they do not conflict.

iamliamc commented 3 years ago

Also, is this a problem? 4398 WARNING: Several hooks defined for module 'apscheduler'. Please take care they do not conflict.

I think this is because I am adding my own apscheduler-hook.py to include the recursive=True did I misunderstand what you meant?

agronholm commented 3 years ago

Could you try ditching that and modifying the hook from pyinstaller-hooks-contrib?

iamliamc commented 3 years ago

Could you try ditching that and modifying the hook from pyinstaller-hooks-contrib?

I see what you mean I went into my venv site packages found the _pyinstaller_hooks_contrib found the existing hook and edited it there... let's see!

iamliamc commented 3 years ago

So now: 234067 INFO: Loading module hook 'hook-apscheduler.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...

And: 

(bacnet_read_publisher_venv) debian@beaglebone-sbp-3:~/compile_test/bacnet_read_publisher$ cat /home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-apscheduler.py 
# ------------------------------------------------------------------
# Copyright (c) 2020 PyInstaller Development Team.
#
# This file is distributed under the terms of the GNU General Public
# License (version 2.0 or later).
#
# The full license is available in LICENSE.GPL.txt, distributed with
# this software.
#
# SPDX-License-Identifier: GPL-2.0-or-later
# ------------------------------------------------------------------

"""
APScheduler uses entry points to dynamically load executors, job
stores and triggers.
This hook was tested against APScheduler 3.6.3.
"""

from PyInstaller.utils.hooks import collect_submodules, copy_metadata

datas = copy_metadata('APScheduler', recursive=True)
hiddenimports = collect_submodules('apscheduler')

(bacnet_read_publisher_venv) debian@beaglebone-sbp-3:~/compile_test/bacnet_read_publisher$ python -m PyInstaller pyinstaller.spec 
875 INFO: PyInstaller: 4.5.1
877 INFO: Python: 3.9.7
901 INFO: Platform: Linux-4.14.108-ti-r113-armv7l-with-glibc2.24
925 INFO: UPX is not available.
940 INFO: Extending PYTHONPATH with paths
['/home/debian/compile_test/bacnet_read_publisher/bacnet_read_publisher']
4297 INFO: checking Analysis
4550 INFO: Building because hiddenimports changed
4553 INFO: Initializing module dependency graph...
4578 INFO: Caching module graph hooks...
4756 INFO: Analyzing base_library.zip ...
51621 INFO: Processing pre-find module path hook distutils from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
51635 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.9'
112183 INFO: Caching module dependency graph...
115965 INFO: running Analysis Analysis-00.toc
116136 INFO: Analyzing bacnet_read_publisher/__main__.py
129646 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-urllib3.packages.six.moves.py'.
188891 INFO: Processing pre-safe import module hook six.moves from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/tzlocal/unix.py:141: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  while start is not 0:
234057 INFO: Processing module hooks...
234067 INFO: Loading module hook 'hook-apscheduler.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
241716 INFO: Loading module hook 'hook-sentry_sdk.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
252608 INFO: Loading module hook 'hook-certifi.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
252666 INFO: Loading module hook 'hook-google.api.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
252694 INFO: Loading module hook 'hook-pytz.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
254077 INFO: Loading module hook 'hook-distutils.util.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
254151 INFO: Loading module hook 'hook-sqlite3.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
255089 INFO: Loading module hook 'hook-xml.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
256439 INFO: Loading module hook 'hook-_tkinter.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/tkinter/__init__.py", line 37, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: libtk8.6.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libtk8.6.so: cannot open shared object file: No such file or directory
257657 ERROR: Tcl/Tk improperly installed on this system.
257660 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
257740 INFO: Loading module hook 'hook-heapq.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
257848 INFO: Loading module hook 'hook-sysconfig.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
258002 INFO: Loading module hook 'hook-lib2to3.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
258600 INFO: Loading module hook 'hook-encodings.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
259621 INFO: Loading module hook 'hook-pkg_resources.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
265711 INFO: Processing pre-safe import module hook win32com from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
268599 WARNING: Hidden import "pkg_resources.py2_warn" not found!
274035 WARNING: Hidden import "pkg_resources.markers" not found!
274114 INFO: Loading module hook 'hook-pickle.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274182 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274192 INFO: Loading module hook 'hook-difflib.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274260 INFO: Loading module hook 'hook-distutils.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
274270 INFO: Loading module hook 'hook-packaging.py' from '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks'...
275116 INFO: Looking for ctypes DLLs
275976 INFO: Analyzing run-time hooks ...
276162 INFO: Including run-time hook '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
276211 INFO: Including run-time hook '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
276271 INFO: Including run-time hook '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
276299 INFO: Including run-time hook '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py'
276401 INFO: Including run-time hook '/home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/rthooks/pyi_rth_certifi.py'
276771 INFO: Looking for dynamic libraries
281815 INFO: Looking for eggs
281820 INFO: Using Python library /usr/local/lib/libpython3.9.so.1.0
282074 INFO: Warnings written to /home/debian/compile_test/bacnet_read_publisher/build/pyinstaller/warn-pyinstaller.txt
284001 INFO: Graph cross-reference written to /home/debian/compile_test/bacnet_read_publisher/build/pyinstaller/xref-pyinstaller.html
285122 INFO: checking PYZ
285268 INFO: Building because toc changed
285270 INFO: Building PYZ (ZlibArchive) /home/debian/compile_test/bacnet_read_publisher/build/pyinstaller/PYZ-00.pyz
298939 INFO: Building PYZ (ZlibArchive) /home/debian/compile_test/bacnet_read_publisher/build/pyinstaller/PYZ-00.pyz completed successfully.
299460 INFO: checking PKG
299814 INFO: Building because toc changed
299816 INFO: Building PKG (CArchive) PKG-00.pkg
314530 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
315098 INFO: Bootloader /home/debian/compile_test/bacnet_read_publisher_venv/lib/python3.9/site-packages/PyInstaller/bootloader/Linux-32bit-arm/run
315101 INFO: checking EXE
315296 INFO: Rebuilding EXE-00.toc because bacnet_read_publisher missing
315298 INFO: Building EXE from EXE-00.toc
316081 INFO: Appending archive to ELF section in EXE /home/debian/compile_test/bacnet_read_publisher/dist/bacnet_read_publisher
316293 INFO: Building EXE from EXE-00.toc completed successfully.

And it looks like it's working!!!

2021-10-13 09:09:31,130 - start_up_sequence - INFO - Revision already exists, skipping start-up sequence
2021-10-13 09:09:31,489 - apscheduler.scheduler - INFO - Scheduler started
2021-10-13 09:09:33,184 - apscheduler.scheduler - INFO - Added job "PublishRegisterReadings.__call__" to job store "default"
2021-10-13 09:09:33,192 - apscheduler.scheduler - INFO - Added job "CARPullerJob.__call__" to job store "default"
2021-10-13 09:09:33,200 - apscheduler.scheduler - INFO - Added job "ScheduleJobsJob.__call__" to job store "default"
2021-10-13 09:09:33,208 - apscheduler.scheduler - INFO - Added job "RestoreDefaults.__call__" to job store "default"
2021-10-13 09:09:33,212 - ScheduleRunner - INFO - CARPullerJob.__call__: next run time 2021-10-13 11:10:00+02:00
2021-10-13 09:09:33,216 - ScheduleRunner - INFO - PublishRegisterReadings.__call__: next run time 2021-10-13 11:10:00+02:00
2021-10-13 09:09:33,218 - ScheduleRunner - INFO - ScheduleJobsJob.__call__: next run time 2021-10-13 11:10:00+02:00
2021-10-13 09:09:33,220 - ScheduleRunner - INFO - RestoreDefaults.__call__: next run time 2021-10-13 12:10:00+02:00
2021-10-13 09:09:33,222 - ScheduleRunner - INFO - Starting AsyncSchedule interface
2021-10-13 09:10:00,035 - CARPuller - INFO - Requesting jobs from CAR...
agronholm commented 3 years ago

Yup, so just as I thought: the "official" hook took precedence.

iamliamc commented 3 years ago

If you think this is appropriate for the standard hook here is a PR... if I need to do it custom for my own workflow so be it. . .

https://github.com/pyinstaller/pyinstaller-hooks-contrib/pull/333

agronholm commented 3 years ago

Closing as the problem was resolved via the above PR.

jhselvik commented 2 years ago

For an additional datapoint - I had to do something similar to include the JSON report pytest plugin module. However, the discrepancy between the package name pytest-json-report and the importable package name pytest_jsonreport was causing issues until I specified the two.

Admittedly, I don't totally understand how this is working, but I thought I'd leave this here to help a future developer.

datas = copy_metadata('pytest-json-report', recursive=True)   # package name
hiddenimports = collect_submodules('pytest_jsonreport')       # importable package name