Serialization issue when using REDIS

[PasqualePuzio]

When we run the application storing the state in memory, everything works fine. However, when we use Redis, we get the following exception.

We're not storing any custom or complex object in our states, we only use primitive types such dict, list, int and str. I'm not sure if it's related to this issue, but the some of these variables can be set to None sometimes.

Do you have any clue on what might be the root cause here?

Task exception was never retrieved future: <Task finished name='Task-27' coro=<AsyncServer._handle_event_internal() done, defined at /root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/socketio/async_server.py:605> exception=NotImplementedError('object proxy must define reduce_ex()')> Traceback (most recent call last): File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/socketio/async_server.py", line 607, in _handle_event_internal r = await server._trigger_event(data[0], namespace, sid, data[1:]) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/socketio/async_server.py", line 643, in _trigger_event return await handler.trigger_event(event, args) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/socketio/async_namespace.py", line 37, in trigger_event ret = await handler(*args) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/reflex/app.py", line 1079, in on_event async for update in process(self.app, event, sid, headers, client_ip): File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/reflex/app.py", line 871, in process async with app.state_manager.modify_state(event.token) as state: File "/usr/lib/python3.10/contextlib.py", line 206, in aexit__ await anext(self.gen) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/reflex/state.py", line 1697, in modify_state await self.set_state(token, state, lock_id) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/reflex/state.py", line 1682, in set_state await self.redis.set(token, cloudpickle.dumps(state), ex=self.token_expiration) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps cp.dump(obj) File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/cloudpickle/cloudpickle_fast.py", line 632, in dump return Pickler.dump(self, obj) NotImplementedError: object proxy must define reduce_ex__()

• Python Version: 3.10
• Reflex Version: 0.3.7
• OS: Ubuntu 22.04
• Browser (Optional): Chrome

pip freeze output:

alembic==1.13.1 anyio==4.2.0 async-timeout==4.0.3 bidict==0.22.1 certifi==2023.11.17 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==2.2.1 coverage==7.4.0 distro==1.9.0 dnspython==2.4.2 docopt==0.6.2 exceptiongroup==1.2.0 fastapi==0.96.1 filelock==3.13.1 greenlet==3.0.3 gunicorn==20.1.0 h11==0.14.0 httpcore==0.17.3 httpx==0.24.1 idna==3.6 Jinja2==3.1.3 Mako==1.3.0 markdown-it-py==3.0.0 MarkupSafe==2.1.3 mdurl==0.1.2 packaging==23.2 pipdeptree==2.13.2 pipreqs==0.4.13 platformdirs==3.11.0 psutil==5.9.7 py3-validate-email @ git+https://gitea.ksol.io/karolyi/py3-validate-email@3f3fcc241c03e85f0f8466992cded1f715f04d67 pycryptodome==3.19.0 pydantic==1.10.13 Pygments==2.17.2 python-dateutil==2.8.2 python-engineio==4.8.2 python-multipart==0.0.5 python-socketio==5.11.0 pytz==2023.3.post1 redis==4.6.0 reflex==0.3.7 reflex-hosting-cli==0.1.6 requests==2.31.0 rich==13.7.0 simple-websocket==1.0.0 six==1.16.0 sniffio==1.3.0 SQLAlchemy==2.0.25 sqlmodel==0.0.14 starlette==0.27.0 starlette-admin==0.9.0 tabulate==0.9.0 typer==0.9.0 typing_extensions==4.9.0 urllib3==2.1.0 uvicorn==0.20.0 watchdog==2.3.1 watchfiles==0.19.0 websockets==12.0 wrapt==1.16.0 wsproto==1.2.0 yarg==0.1.9

[PasqualePuzio]

Also, looks like some attributes aren't stored in the state when using Redis.

Traceback (most recent call last): File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/reflex/state.py", line 1095, in _process_event events = fn(**payload) File "/root/fantacalcio-frontend-web/fantacalcio_frontend_web/pages/free_players.py", line 139, in on_role_changed self.update_free_players() File "/root/fantacalcio-frontend-web/fantacalcio_frontend_web/pages/free_players.py", line 164, in update_free_players if self._all_free_players is None or len(self._all_free_players) == 0: File "/root/fantacalcio-frontend-web/.venv/lib/python3.10/site-packages/reflex/state.py", line 850, in getattribute value = super().getattribute(name) AttributeError: 'LeagueFreePlayersState' object has no attribute '_all_free_players'. Did you mean: 'load_free_players'?

[benedikt-bartscher]

Seems like you are storing a non-pickleable object in a reflex backend var (prefixed with _)

[PasqualePuzio]

I wish it was so simple :)

First of all, the object in question is just a list of dictionaries parsed from json, there’s no reason why it shouldn’t be pickleable. Second, _ is a perfectly valid prefix for vars, it’s stated in the docs that it’s intended for backend only variables. Third, the issue occurs on every single page, it can’t be due to a single object. Last but not least, after some research I found out that the exception is thrown by objectproxy (which is used by Reflex) whenever pickle tries to perform dump or dumps (version 1.11.0 changelog https://wrapt.readthedocs.io/en/latest/changes.html), therefore I suspect there’s something wrong under the hood.

Seems like you are storing a non-pickleable object in a reflex backend var (prefixed with _)

[benedikt-bartscher]

I would be pretty helpful if you could add a minimal reproduction example

[PasqualePuzio]

I would be pretty helpful if you could add a minimal reproduction example

Sure, I'll try to isolate the issue and reproduce it in a simpler example.

For now, what I have figured out is that the issue is happening because of this:

If pickle.dump() or pickle.dumps() is used on an instance of the ObjectProxy class, a NotImplementedError exception is raised, with a message indicating that the object proxy must implement the __reduce_ex__() method.

[PasqualePuzio]

We have probably figured it out.

The issue occurs when we have for instance two vars, where one is a list of dictionaries and the other one is a single dictionary (taken from that list), or a similar case where the first var is a list of dictionaries and the second var is a dictionary where values are the items of that list.

In those cases, looks like something goes wrong with pickling/unpickling, therefore some objects don't get pickled.

We have apparently fixed the issue by calling .copy() on all dictionaries that are copied from a data structure to another. However it looks very ugly :)

Is that the intended behavior of serialization?

[PasqualePuzio]

One more detail: we confirm that by performing deepcopy on all duplicated objects (objects that appear in more than one state var) the problem is gone.

Is that expected behavior? Or is it a bug?

[PasqualePuzio]

No feedback on this? Could you please at least confirm that because of pickle an object cannot appear in more than on state var?

[benedikt-bartscher]

Hi @PasqualePuzio I am not part of the reflex team. If I had more time, I would look deeper into this. You would make reproducing the issue much easier if you could provide a minimal reproduction example/repo.

[masenf]

@PasqualePuzio do you have a repro case that demonstrates what is going on? I've read through your comments up to yesterday and it sounds like cloudpickle (only used when redis is in the picture) isn't liking some of the state objects when they are copied (shallowly?) and thus they are not available on the unpickled object.

The team wants to fix this issue, but we are currently slogging through the radix-ui component refactor and have not yet had a chance to design a repro case which would allow us to investigate this issue.

If you're able to provide a repro case, then someone on the team can dig in and try to investigate.