search

tardunge / prefect-spark-on-k8s-operator

Prefect integrations for orchestrating and monitoring apache spark jobs on kubernetes using spark-on-k8s-operator.
Apache License 2.0
4 stars 1 forks source link

”TypeError: Object of type 'FieldInfo' is not JSON serializable“ with prefect-2.14.13 prefect-kubernetes-0.2.3 in python3.8 #8

Open NewAlice opened 5 months ago

NewAlice commented 5 months ago

Traceback (most recent call last): File "test_spark_application.py", line 8, in from prefect_spark_on_k8s_operator import SparkApplication File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect_spark_on_k8s_operator/init.py", line 2, in from prefect_spark_on_k8s_operator.app import ( # noqa F401 File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect_spark_on_k8s_operator/app.py", line 39, in class SparkApplication(JobBlock): File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/main.py", line 282, in new cls = super().new(mcs, name, bases, new_namespace, kwargs) File "/usr/lib/python3.8/abc.py", line 85, in new cls = super().new(mcls, name, bases, namespace, kwargs) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/utilities/dispatch.py", line 99, in _register_subclass_of_base_type register_type(cls) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/utilities/dispatch.py", line 153, in register_type key = get_dispatch_key(cls) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/utilities/dispatch.py", line 76, in get_dispatch_key dispatch_key = dispatch_key() File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/blocks/core.py", line 315, in dispatch_key__ return block_schema_to_key(cls._to_block_schema()) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/blocks/core.py", line 487, in _to_block_schema fields = cls.schema() File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/main.py", line 664, in schema s = model_schema(cls, by_alias=by_alias, ref_template=ref_template) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 188, in model_schema m_schema, m_definitions, nested_models = model_process_schema( File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 582, in model_process_schema m_schema, m_definitions, nested_models = model_type_schema( File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 623, in model_type_schema f_schema, f_definitions, f_nested_models = field_schema( File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 249, in field_schema s, schema_overrides = get_field_info_schema(field) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 217, in get_field_infoschema schema['default'] = encode_default(field.default) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 996, in encode_default return pydantic_encoder(dft) File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/json.py", line 90, in pydantic_encoder raise TypeError(f"Object of type '{obj.class.name__}' is not JSON serializable") TypeError: Object of type 'FieldInfo' is not JSON serializable

NewAlice commented 5 months ago

could you help identify what's the problem?

kkondamadugula commented 2 months ago

@NewAlice @tardunge I am facing the exact issue. Is this resolved? Or how did you overcome this?

tardunge commented 2 months ago

can you post me the steps to reproduce the issue?

kkondamadugula commented 2 months ago

I am trying to run a simple spark-operator job on prefect server by following the instructions provided here: https://github.com/tardunge/prefect-spark-on-k8s-operator/blob/main/README.md. Attaching the flow and spark-operator yaml to this message. When I try to run the job locally (python3.11 job.py), I run into "prefect.exceptions.ScriptError: Script at 'simple-so-k8s-on-prefect.py' encountered an exception: TypeError("Object of type 'FieldInfo' is not JSON serializable")". Attaching entire stack trace for reference. Note: I am running the code inside a container. simple-so-k8s-on-prefect.zip Steps to reproduce: 1) Create a container on top of an image that installs prefect 2.14.16, python 3.11, pyspark 3.2. 2) Bash into the container and pip install prefect-spark-on-k8s-operator 3) I am using kube config extracted from ~/.kube/config and hardcoding it into the simple-so-k8s-on-prefect.py for testing purposes. 4) Trigger the flow using python3.11 simple-so-k8s-on-prefect.py

kkondamadugula commented 1 month ago

@tardunge downgrading to pydantic 1.10 from 2.x makes the code work. Were you able to reproduce the issue? Is there a way to fix the Pydantic issues in 2.x?

tardunge commented 3 weeks ago

@NewAlice @kkondamadugula Yes. I can see the tests failing with the same error when using pydantic 2.x, however they are produced from the prefect core.