Closed chamini2 closed 6 months ago
I get the impression from your description/branch name you saw this as a v2 regression @chamini2 but this reproduces on main
actually.
The hello_dataclass
function does not actually serialise successfully in Pydantic 1.x from what I can see: but the serve=True
parameter hides this by successfully starting a Uvicorn server.
There might indeed be problems with Pydantic 2, but I'd think it'd be best to start from a working v1.x baseline here.
serve=False
to reproduce a little more minimally.We can run hello_query
$ fal fn run demo.py hello_query
2024-01-18 22:11:38.933 [stdout ] Cached function
2024-01-18 22:11:38.933 [stdout ] 2024-01-18 22:11:38.932910 Messaged by: Louis
We cannot run hello_dataclass
$ fal fn run demo.py hello_dataclass
Error while serializing the given object
I would try a debugging approach that stays in Python rather than CLI (shell level) to get more visibility.
The stack trace shows it's emerging from dill
What's more if I use a Pydantic model it fails more verbosely [error shown with no --debug
flag] and seems to indicate that the virtualenv cached directory does not have the necessary Pydantic module...
$ fal fn run demo.py hello_data_model
2024-01-18 22:38:55.311 [error ] Traceback (most recent call last):
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/isolate/connections/common.py", line 40, in _step
yield
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/isolate/connections/common.py", line 77, in load_serialized_object
result = serialization_backend.loads(raw_object)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/dill/_dill.py", line 301, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/dill/_dill.py", line 287, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/dill/_dill.py", line 442, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/dill/_dill.py", line 432, in find_class
return StockUnpickler.find_class(self, module, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pydantic'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/isolate/connections/grpc/agent.py", line 113, in execute_function
function = from_grpc(function)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/functools.py", line 909, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/isolate/connections/grpc/interface.py", line 29, in _
return load_serialized_object(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/isolate/connections/common.py", line 76, in load_serialized_object
with _step("deserializing the given object"):
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/root/.cache/isolate/virtualenv/dd85ba8a65db645667152bcadfce1749d4fbd40765c63ba1d368c48eafe9c34e/lib/python3.11/site-packages/isolate/connections/common.py", line 42, in _step
raise SerializationError("Error while " + message) from exception
isolate.connections.common.SerializationError: Error while deserializing the given object
Function execution quit unexpectedly. Error contents: The function function could not be deserialized.
I get the impression this is the same error as the one raised from the Pydantic dataclass, but a bit clearer (no need to --debug
to see it)
I checked and in fact this seems to have been cut off from the traceback you shared, it's the upstream cause of the 2nd exception in both cases though.
Discussed this with @isidentical who suggested an initial repro which eventually turned into the following Pytest repro
It is possible to, in effect, reproduce in the form of a pytest test (which is desirable for maintaining future coverage of this bug when contributing a fix) by resorting to subprocess, since Python is ultimately just another program we can run within Python.
The following code is a standalone pytest test that reproduces the bug
# demo_4_pytest_subprocess.py
import subprocess
import sys
import dill
from pydantic import BaseModel, Field
class Input(BaseModel):
"""A simple Pydantic model used to demonstrate deserialisation via dill.
Attributes:
prompt: An input prompt for a generative AI model.
num_steps: The number of steps to run a generative AI model for.
"""
prompt: str
num_steps: int = Field(default=2, ge=1, le=10)
def deserialise_pydantic_model():
"""Serialise (`dill.dumps`) then deserialise (`dill.loads`) a Pydantic model.
The `recurse` setting must be set, counterintuitively, to prevent excessive
recursion (refer to e.g. dill issue
[#482](https://github.com/uqfoundation/dill/issues/482#issuecomment-1139017499)):
to limit the amount of recursion that dill is doing to pickle the function, we
need to turn on a setting called recurse, but that is because the setting
actually recurses over the global dictionary and finds the smallest subset that
the function needs to run, which will limit the number of objects that dill
needs to include in the pickle.
"""
dill.settings["recurse"] = True
# Run the function locally
cls = dill.loads(dill.dumps(Input))
model = cls(prompt="a")
return model
def test_main():
"""Test that deserialisation of a Pydantic model succeeds.
The deserialisation failure mode reproduction is incompatible with pytest (see
[#29](https://github.com/fal-ai/fal/issues/29#issuecomment-1902241217) for
discussion) so we directly invoke the current Python executable on this file.
"""
proc = subprocess.run([sys.executable, __file__], capture_output=True, text=True)
assert not proc.returncode, f"Pydantic model deserialisation failed:\n{proc.stderr}"
if __name__ == "__main__":
deserialise_pydantic_model()
Tested with the diff https://github.com/fal-ai/fal/compare/main...matteo/pydantic2-test
I was trying out what the migration to Pydantic 2 could look like and I get the following:
Runing the query version it works as expected and can be used
But when I try to use the dataclass version I get an error
Notice this error while deserializing seems to be local (gRPC deserializing?)
Specifically, running the above command with
--debug
I get