search

reflex-dev / reflex

🕸️ Web apps in pure Python 🐍
https://reflex.dev
Apache License 2.0
20.44k stars 1.18k forks source link

Picking/Unpickling Issues when using Redis (Memorystore) #4074

Open gniewus opened 1 month ago

gniewus commented 1 month ago

Describe the bug Hey, im having a very hard time self-hosting the reflex app on GCP.

reflex==0.5.10 Untitled Diagram drawio (1)

I've tried connecting Redis (tried both 7.0 and 7.2) and I'm having issues with the app as it's throwing serialization errors:

2024-10-06 00:02:22,940] [runners.py:118 -        run() ] Task exception was never retrieved
future: <Task finished name='Task-146' coro=<AsyncServer._handle_event_internal() done, defined at /opt/anaconda3/envs/py311/lib/python3.11/site-packages/socketio/async_server.py:608> exception=TypeError("cannot pickle 'ComputedVar' object")>
Traceback (most recent call last):
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/socketio/async_server.py", line 610, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/socketio/async_server.py", line 646, in _trigger_event
    return await handler.trigger_event(event, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/socketio/async_namespace.py", line 37, in trigger_event
    ret = await handler(*args)
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/reflex/app.py", line 1524, in on_event
    async for update in process(self.app, event, sid, headers, client_ip):
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/reflex/app.py", line 1275, in process
    async with app.state_manager.modify_state(event.substate_token) as state:
  File "/opt/anaconda3/envs/py311/lib/python3.11/contextlib.py", line 211, in __aexit__
    await anext(self.gen)
  File "/opt/anaconda3/envs/py311/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/reflex/state.py", line 2833, in _lock
    yield lock_id
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/reflex/state.py", line 2733, in modify_state
    await self.set_state(token, state, lock_id)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/reflex/state.py", line 2717, in set_state
    await t
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/reflex/state.py", line 2707, in set_state
    pickle_state = dill.dumps(state, byref=True)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 280, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 252, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 420, in dump
    StockPickler.dump(self, obj)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 717, in save_reduce
    save(state)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/anaconda3/envs/py311/lib/python3.11/pickle.py", line 578, in save
    rv = reduce(self.proto)
         ^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle 'ComputedVar' object

Upgrading reflex to 0.6.x does not solve the issue but it causes some other issues when picking state objects:

_pickle.PicklingError: args[0] from __newobj__ args has the wrong class
[Reflex Backend Exception]
 Traceback (most recent call last):
  File "/opt/anaconda3/envs/reflex/lib/python3.12/site-packages/reflex/app.py", line 1264, in process
    async with app.state_manager.modify_state(event.substate_token) as state:
  File "/opt/anaconda3/envs/reflex/lib/python3.12/contextlib.py", line 217, in __aexit__
    await anext(self.gen)
  File "/opt/anaconda3/envs/reflex/lib/python3.12/site-packages/reflex/state.py", line 2834, in modify_state
    await self.set_state(token, state)
  File "/opt/anaconda3/envs/reflex/lib/python3.12/site-packages/reflex/state.py", line 2811, in set_state
    await self.set_state_for_substate(client_token, state)
  File "/opt/anaconda3/envs/reflex/lib/python3.12/site-packages/reflex/state.py", line 2800, in set_state_for_substate
    await self.set_state_for_substate(client_token, substate_substate)
  File "/opt/anaconda3/envs/reflex/lib/python3.12/site-packages/reflex/state.py", line 2794, in set_state_for_substate
    state_dilled = substate._serialize()
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/reflex/lib/python3.12/site-packages/reflex/state.py", line 1941, in _serialize
    return pickle.dumps((state_to_schema(self), self))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.PicklingError: args[0] from __newobj__ args has the wrong class

[2024-10-05 21:25:32,010] [base_events.py:1821 - default_exception_handler() ] Task exception was never retrieved
future: <Task finished name='Task-210' coro=<AsyncServer._handle_event_internal() done, defined at /opt/anaconda3/envs/reflex/lib/python3.12/site-packages/socketio/async_server.py:608> exception=PicklingError('args[0] from __newobj__ args has the wrong class')>
Traceback (most recent call last):

This makes me think that there must be an issue with my implementation and a huge State class that im using. However i can't tell which attribute/compute var is causing the problem + I also made sure wrap all attributes different from bool/dict/list into rx.Base objects. On 0.5.10 the error goes away when i don't use Redis :/

Specifics (please complete the following information):

masenf commented 1 month ago

Minimal repro

import reflex as rx

class State(rx.State):
    @rx.var
    def computed(self) -> int:
        return 42

class Other(rx.State):
    ref: int

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.ref = State.computed

def index() -> rx.Component:
    return rx.container(
        rx.vstack(
            rx.text(f"Bug bug {State.computed} {Other.ref}")
        ),
        rx.logo(),
    )

app = rx.App()
app.add_page(index)

This seems to be caused when assigning a ComputedVar to a regular attribute, instead of using get_state to get the actual value.

adhami3310 commented 1 month ago

We can possibly detect this if it's a common error and print a more reasonable error message.

@masenf let me know if i should get on that.

gniewus commented 1 month ago

Can you explain how would the correct way of doing that using get_state look like ?

masenf commented 1 month ago

Can you explain how would the correct way of doing that using get_state look like ?

  1. Either make TenKDialogState a substate of BaseState and access the value directly from self
  2. Get the value as part of the open toggle
class TenKDialogState(rx.State):
    is_open: bool = False
    ten_k: TenK = None

    async def toggle_dialog(self):
        self.is_open = not self.is_open
        if self.is_open:
            # Fetch the latest value
            base_state = await self.get_state(BaseState)
            self.ten_k = base_state.current_company.ten_k

    def close_dialog(self):
        self.is_open = False