search

northpowered / temporal-boost

Extemely fast development for Temporal-based microservices
https://northpowered.github.io/temporal-boost/
MIT License
6 stars 0 forks source link

TypeError: cannot pickle 'TextIOWrapper' instances #24

Open derevyan opened 2 months ago

derevyan commented 2 months ago

Hey! great idea, i play around with temporal for few days and try to solve Pydentic problem, looks like you have out of the box solution here and loggin looks just great! But unfortunately i had this error on start, probably because o Python 3.12.5 but not sure. 

python playground/temporal/boost.py run all
{"text": "Worker langchain_worker was registered in CLI", "record": {"elapsed": {"repr": "0:00:00.843833", "seconds": 0.843833}, "exception": null, "extra": {}, "file": {"name": "core.py", "path": "/Users/lib/python3.12/site-packages/temporal_boost/core.py"}, "function": "add_worker", "level": {"icon": "ℹ️", "name": "INFO", "no": 20}, "line": 133, "message": "Worker langchain_worker was registered in CLI", "module": "core", "name": "temporal_boost.core", "process": {"id": 53320, "name": "MainProcess"}, "thread": {"id": 140704491470272, "name": "MainThread"}, "time": {"repr": "2024-08-26 14:52:24.006149+08:00", "timestamp": 1724655144.006149}}}
{"text": "Internal worker was registered in CLI", "record": {"elapsed": {"repr": "0:00:00.845894", "seconds": 0.845894}, "exception": null, "extra": {}, "file": {"name": "core.py", "path": "/Users/lib/python3.12/site-packages/temporal_boost/core.py"}, "function": "add_internal_worker", "level": {"icon": "ℹ️", "name": "INFO", "no": 20}, "line": 160, "message": "Internal worker was registered in CLI", "module": "core", "name": "temporal_boost.core", "process": {"id": 53320, "name": "MainProcess"}, "thread": {"id": 140704491470272, "name": "MainThread"}, "time": {"repr": "2024-08-26 14:52:24.008210+08:00", "timestamp": 1724655144.00821}}}
{"text": "Use all-in-one mode only in development!", "record": {"elapsed": {"repr": "0:00:00.849280", "seconds": 0.84928}, "exception": null, "extra": {}, "file": {"name": "core.py", "path": "/Userslib/python3.12/site-packages/temporal_boost/core.py"}, "function": "register_all", "level": {"icon": "⚠️", "name": "WARNING", "no": 30}, "line": 185, "message": "Use all-in-one mode only in development!", "module": "core", "name": "temporal_boost.core", "process": {"id": 53320, "name": "MainProcess"}, "thread": {"id": 140704491470272, "name": "MainThread"}, "time": {"repr": "2024-08-26 14:52:24.011596+08:00", "timestamp": 1724655144.011596}}}
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/lib/python3.12/site-packages/temporal_boost/core.py:193 in  │
│ register_all                                                                                     │
│                                                                                                  │
│   190 │   │   │   │   target=worker.run,                                                         │
│   191 │   │   │   )                                                                              │
│   192 │   │   │   procs.append(proc)                                                             │
│ ❱ 193 │   │   │   proc.start()                                                                   │
│   194 │   │                                                                                      │
│   195 │   │   for proc in procs:                                                                 │
│   196 │   │   │   proc.join()                                                                    │
│                                                                                                  │
│ ╭────────────────────────────── locals ──────────────────────────────╮                           │
│ │   proc = <Process name='Process-1' parent=53320 initial>           │                           │
│ │  procs = [<Process name='Process-1' parent=53320 initial>]         │                           │
│ │   self = <temporal_boost.core.BoostApp object at 0x10c50c350>      │                           │
│ │ worker = <temporal_boost.worker.BoostWorker object at 0x111203fe0> │                           │
│ ╰────────────────────────────────────────────────────────────────────╯                           │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/process.py:121 in start      │
│                                                                                                  │
│   118 │   │   assert not _current_process._config.get('daemon'), \                               │
│   119 │   │   │      'daemonic processes are not allowed to have children'                       │
│   120 │   │   _cleanup()                                                                         │
│ ❱ 121 │   │   self._popen = self._Popen(self)                                                    │
│   122 │   │   self._sentinel = self._popen.sentinel                                              │
│   123 │   │   # Avoid a refcycle if the target function holds an indirect                        │
│   124 │   │   # reference to the process object (see bpo-30775)                                  │
│                                                                                                  │
│ ╭──────────────────────── locals ────────────────────────╮                                       │
│ │ self = <Process name='Process-1' parent=53320 initial> │                                       │
│ ╰────────────────────────────────────────────────────────╯                                       │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/context.py:224 in _Popen     │
│                                                                                                  │
│   221 │   _start_method = None                                                                   │
│   222 │   @staticmethod                                                                          │
│   223 │   def _Popen(process_obj):                                                               │
│ ❱ 224 │   │   return _default_context.get_context().Process._Popen(process_obj)                  │
│   225 │                                                                                          │
│   226 │   @staticmethod                                                                          │
│   227 │   def _after_fork():                                                                     │
│                                                                                                  │
│ ╭─────────────────────────── locals ────────────────────────────╮                                │
│ │ process_obj = <Process name='Process-1' parent=53320 initial> │                                │
│ ╰───────────────────────────────────────────────────────────────╯                                │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/context.py:289 in _Popen     │
│                                                                                                  │
│   286 │   │   @staticmethod                                                                      │
│   287 │   │   def _Popen(process_obj):                                                           │
│   288 │   │   │   from .popen_spawn_posix import Popen                                           │
│ ❱ 289 │   │   │   return Popen(process_obj)                                                      │
│   290 │   │                                                                                      │
│   291 │   │   @staticmethod                                                                      │
│   292 │   │   def _after_fork():                                                                 │
│                                                                                                  │
│ ╭──────────────────────────── locals ─────────────────────────────╮                              │
│ │       Popen = <class 'multiprocessing.popen_spawn_posix.Popen'> │                              │
│ │ process_obj = <Process name='Process-1' parent=53320 initial>   │                              │
│ ╰─────────────────────────────────────────────────────────────────╯                              │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/popen_spawn_posix.py:32 in   │
│ __init__                                                                                         │
│                                                                                                  │
│   29 │                                                                                           │
│   30 │   def __init__(self, process_obj):                                                        │
│   31 │   │   self._fds = []                                                                      │
│ ❱ 32 │   │   super().__init__(process_obj)                                                       │
│   33 │                                                                                           │
│   34 │   def duplicate_for_child(self, fd):                                                      │
│   35 │   │   self._fds.append(fd)                                                                │
│                                                                                                  │
│ ╭─────────────────────────────────── locals ────────────────────────────────────╮                │
│ │ process_obj = <Process name='Process-1' parent=53320 initial>                 │                │
│ │        self = <multiprocessing.popen_spawn_posix.Popen object at 0x113c52240> │                │
│ ╰───────────────────────────────────────────────────────────────────────────────╯                │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/popen_fork.py:19 in __init__ │
│                                                                                                  │
│   16 │   │   util._flush_std_streams()                                                           │
│   17 │   │   self.returncode = None                                                              │
│   18 │   │   self.finalizer = None                                                               │
│ ❱ 19 │   │   self._launch(process_obj)                                                           │
│   20 │                                                                                           │
│   21 │   def duplicate_for_child(self, fd):                                                      │
│   22 │   │   return fd                                                                           │
│                                                                                                  │
│ ╭─────────────────────────────────── locals ────────────────────────────────────╮                │
│ │ process_obj = <Process name='Process-1' parent=53320 initial>                 │                │
│ │        self = <multiprocessing.popen_spawn_posix.Popen object at 0x113c52240> │                │
│ ╰───────────────────────────────────────────────────────────────────────────────╯                │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/popen_spawn_posix.py:47 in   │
│ _launch                                                                                          │
│                                                                                                  │
│   44 │   │   set_spawning_popen(self)                                                            │
│   45 │   │   try:                                                                                │
│   46 │   │   │   reduction.dump(prep_data, fp)                                                   │
│ ❱ 47 │   │   │   reduction.dump(process_obj, fp)                                                 │
│   48 │   │   finally:                                                                            │
│   49 │   │   │   set_spawning_popen(None)                                                        │
│   50                                                                                             │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │               fp = <_io.BytesIO object at 0x113c5d260>                                       │ │
│ │        prep_data = {                                                                         │ │
│ │                    │   'log_to_stderr': False,                                               │ │
│ │                    │   'authkey':                                                            │ │
│ │                    b'U\xf2Od\xb7\x95\xda\xaf\x16\xc7\xfc\x95\x10\x071\x8f\x05S\xed\xae\x83\… │ │
│ │                    │   'name': 'Process-1',                                                  │ │
│ │                    │   'sys_path': [                                                         │ │
│ │                    │   │                                                                     │ │
│ │                    '/Users/derevo/MEV.nosync/SANTA/GIT/ECU/HOMA/playground/temporal/langcha… │ │
│ │                    │   │   '/Users/derevo/.pyenv/versions/3.12.5/lib/python312.zip',         │ │
│ │                    │   │   '/Users/derevo/.pyenv/versions/3.12.5/lib/python3.12',            │ │
│ │                    │   │                                                                     │ │
│ │                    '/Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/lib-dynload',        │ │
│ │                    │   │                                                                     │ │
│ │                    '/Users/lib/python3.12/site-packages',      │ │
│ │                    │   │   '/Users/derevo/MEV.nosync/SANTA/GIT/ECU/HOMA'                     │ │
│ │                    │   ],                                                                    │ │
│ │                    │   'sys_argv': [                                                         │ │
│ │                    │   │   'playground/temporal/langchain/boost.py',                         │ │
│ │                    │   │   'run',                                                            │ │
│ │                    │   │   'all'                                                             │ │
│ │                    │   ],                                                                    │ │
│ │                    │   'orig_dir': '/Users/derevo/MEV.nosync/SANTA/GIT/ECU/HOMA',            │ │
│ │                    │   'dir': '/Users/derevo/MEV.nosync/SANTA/GIT/ECU/HOMA',                 │ │
│ │                    │   'start_method': 'spawn',                                              │ │
│ │                    │   'init_main_from_path':                                                │ │
│ │                    '/Users/derevo/MEV.nosync/SANTA/GIT/ECU/HOMA/playground/temporal/langcha… │ │
│ │                    }                                                                         │ │
│ │      process_obj = <Process name='Process-1' parent=53320 initial>                           │ │
│ │ resource_tracker = <module 'multiprocessing.resource_tracker' from                           │ │
│ │                    '/Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/res… │ │
│ │             self = <multiprocessing.popen_spawn_posix.Popen object at 0x113c52240>           │ │
│ │       tracker_fd = 7                                                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/derevo/.pyenv/versions/3.12.5/lib/python3.12/multiprocessing/reduction.py:60 in dump      │
│                                                                                                  │
│    57                                                                                            │
│    58 def dump(obj, file, protocol=None):                                                        │
│    59 │   '''Replacement for pickle.dump() using ForkingPickler.'''                              │
│ ❱  60 │   ForkingPickler(file, protocol).dump(obj)                                               │
│    61                                                                                            │
│    62 #                                                                                          │
│    63 # Platform specific definitions                                                            │
│                                                                                                  │
│ ╭────────────────────────── locals ──────────────────────────╮                                   │
│ │     file = <_io.BytesIO object at 0x113c5d260>             │                                   │
│ │      obj = <Process name='Process-1' parent=53320 initial> │                                   │
│ │ protocol = None                                            │                                   │
│ ╰────────────────────────────────────────────────────────────╯                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: cannot pickle 'TextIOWrapper' instances

Have a great day and thanks for trying to make temporal easy to use with python🙌

northpowered commented 2 months ago

@derevyan Tnx for thew reporting? Are you using Windows? I`d already cought bug like this on Windows platforms - troubles with multiprocessing lib. I`ll fix it asap

derevyan commented 2 months ago

@northpowered Hi, im using - MacOS Sonoma (14.6.1) with Intel Core i9 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:13:00 PDT 2024; root:xnu-10063.141.2~1/RELEASE_X86_64 x86_64

P.S: accidentally misscliks and close issue😅

northpowered commented 2 months ago

@derevyan Mb this can help? https://github.com/northpowered/temporal-boost/blob/main/examples/example_app.py#L95

derevyan commented 2 months ago

@derevyan Mb this can help? https://github.com/northpowered/temporal-boost/blob/main/examples/example_app.py#L95

Hm, interesting, but not really helps, got same error. Seems like some fork-spawn problem to me, but i not that good on python to find out exactly.