search

Miksus / rocketry

Modern scheduling library for Python
https://rocketry.readthedocs.io
MIT License
3.23k stars 105 forks source link

BUG crashed in pickling #170

Open tekumara opened 1 year ago

tekumara commented 1 year ago

Describe the bug

On the second run of my task, the following log line triggers continuously in a loop forever:

CRITICAL:rocketry.task:Task 'daily_notification' crashed in pickling. Cannot pickle: {'__dict__': {'session': <rocketry.session.Session object at 0x2f6d4f9a0>, 'permanent': False, 'fmt_log_message': "Task '{task}' status: '{action}'", 'daemon': None, 'batches': [], 'name': 'daily_notification', 'description': None, 'logger_name': 'rocketry.task', 'execution': None, 'priority': 0, 'disabled': False, 'force_run': False, 'force_termination': False, 'status': 'run', 'timeout': None, 'parameters': Parameters(), 'start_cond': None, 'end_cond': None, 'multilaunch': None, 'on_startup': False, 'on_shutdown': False, 'func_run_id': None, 'func': <function daily_notification at 0x1038cd4c0>, 'path': None, 'func_name': 'main', 'cache': False, 'sys_paths': []}}

The log line is the same each time, except for the Session object reference.

On CTRL+C the trace back contains:

INFO:rocketry.scheduler:Interupted. Shutting down...
Traceback (most recent call last):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/schedule.py", line 128, in serve
    await self.run_cycle()
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/schedule.py", line 182, in run_cycle
    await self.run_task(task)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/schedule.py", line 211, in run_task
    await task.start_async(log_queue=self._log_queue)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 514, in start_async
    self.run_as_process(params=params, direct_params=direct_params, task_run=task_run, **kwargs)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 735, in run_as_process
    process.start()
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in __getstate__
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in <dictcomp>
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in __getstate__
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in <dictcomp>
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in __getstate__
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in <dictcomp>
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1326, in __getstate__
    self.logger.critical(f"Task '{self.name}' crashed in pickling. Cannot pickle: {unpicklable}", extra={"action": "fail", "task_name": self.name})
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1835, in critical
    self.log(CRITICAL, msg, *args, **kwargs)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1844, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1512, in log
    self._log(level, msg, args, **kwargs)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
KeyboardInterrupt
INFO:rocketry.scheduler:Shutdown completed. Good bye.

My task:

@app.task(hourly)
async def daily_notification() -> None:
    file = "summary.png"
    # we are using the sync clients so run them on a thread
    await to_thread.run_sync(
        glance.browser.screenshot,
        file
    )
    await to_thread.run_sync(glance.slack.upload_screenshot, file)

Desktop (please complete the following information):

Additional context

rocketry 2.5.1

tekumara commented 1 year ago

As a workaround, switching the task execution to thread or async (rather than default of process) avoids this problem, presumably because pickling doesn't need to happen.

Miksus commented 1 year ago

Thanks for reporting this! These pickling issues are often hard to find as they are often OS-dependent or come up in special situations.

I think an unpicklable attribute from Rocketry's session leaks to the process for some reason.

If interested, the task sort of prunes the session:

If I remember correctly, the process task needs some config options from the session thus most of the session configs are pickled. I tried to make it so that it would throw away only known unpicklable attributes, but I am not sure if it should work the other way around (including only the ones we know we can pickle).

Another could be the function but that function looks pretty standard.

How are your app configurations, and do you perhaps set some attributes to the session?

tekumara commented 1 year ago

Thanks for the explanation!

This is the only config I have:

app = Rocketry(
    # see https://rocketry.readthedocs.io/en/stable/handbooks/config.html
    config={
        "task_execution": "process",
    }
)

Interestingly, if I remove that config and just use app = Rocketry() it works even though I thought that process was the default.