Apparently valid model cannot be serialized: `TypeError: Object of type 'dict' is not XML serializable`

[Jacob-Flasheye]

Hi.

I'm having an issue that is very hard to figure out, where I cannot serialize certain nested models. The error looks like this:

  File "<redacted>/python3.11/site-packages/pydantic_xml/serializers/encoder.py", line 49, in default
    raise TypeError(f"Object of type '{obj.__class__.__name__}' is not XML serializable")
TypeError: Object of type 'dict' is not XML serializable

But the model can still be serialized into json with .json(). I would share the models here but they are work related :/ I have tried to create a reproducer outside the work-related code but I have not managed to do it, but I did some debugging and it seems like the serializer at some point cannot tell what type a model is and then defaults to dict, which causes this error. I will try to debug some more next week to get to the bottom of this problem.

[Jacob-Flasheye]

This is the full traceback after I run mymodel.to_xml_tree():

  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/model.py", line 308, in to_xml_tree
    self.__xml_serializer__.serialize(root, self, encoder=encoder, skip_empty=skip_empty)
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/factories/model.py", line 68, in serialize
    field_serializer.serialize(element, getattr(value, field_name), encoder=encoder, skip_empty=skip_empty)
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/factories/model.py", line 138, in serialize
    super().serialize(sub_element, value, encoder=encoder, skip_empty=skip_empty)
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/factories/model.py", line 103, in serialize
    return self._model.__xml_serializer__.serialize(element, value, encoder=encoder, skip_empty=skip_empty)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/factories/model.py", line 68, in serialize
    field_serializer.serialize(element, getattr(value, field_name), encoder=encoder, skip_empty=skip_empty)
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/factories/primitive.py", line 79, in serialize
    super().serialize(sub_element, value, encoder=encoder, skip_empty=skip_empty)
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/factories/primitive.py", line 24, in serialize
    encoded = encoder.encode(value)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/encoder.py", line 20, in encode
    return self._encode(obj, set())
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/encoder.py", line 40, in _encode
    return self._encode(value, seen_values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/encoder.py", line 33, in _encode
    value = self.default(obj)
            ^^^^^^^^^^^^^^^^^
  File "/home/jacob/.local/lib/python3.11/site-packages/pydantic_xml/serializers/encoder.py", line 49, in default
    raise TypeError(f"Object of type '{obj.__class__.__name__}' is not XML serializable")
TypeError: Object of type 'dict' is not XML serializable

where the model essentially looks like this:

import pydantic_xml as pxml

import other_models as tom

NSMAP={"tns": "this_namespace", "tom": "other_namespace"}

# tom.InnerElement is another BaseXmlModel
class MyModel(pxml.BaseXmlModel, ns="tns", nsmap=NSMAP):
    inner_element: "tom.InnerElement" = element(tag="InnerElement")

# More BaseXmlModels defined here
...

for obj in list(globals().values()):
    if isinstance(obj, type) and issubclass(obj, BaseXmlModel):
        obj.update_forward_refs()

I didn't think about it before, but could this error be caused by using strings for the types instead of plain types?

[dapper91]

@Jacob-Flasheye Hi. I can't figure out what the problem is from the stack trace. It would be helpful if you share the model description you are trying to serialize.

[Jacob-Flasheye]

Thanks for the quick reply.

I will create an anonymized full version of the models either this weekend or on Monday.

[dapper91]

I didn't think about it before, but could this error be caused by using strings for the types instead of plain types?

I used a string instead of a plain type, but managed to serialize the model successfully. So it seems not to be a problem.

[Jacob-Flasheye]

This script reproduces the problem:

from pydantic_xml import BaseXmlModel, element

NSMAP = {"tns": "this/namespace"}

class DateTime(BaseXmlModel, ns="tns", nsmap=NSMAP):
    time: "Time" = element(tag="Time")
    date: "Date" = element(tag="Date")

class Date(BaseXmlModel, ns="tns", nsmap=NSMAP):
    year: "int" = element(tag="Year")
    month: "int" = element(tag="Month")
    day: "int" = element(tag="Day")

class Time(BaseXmlModel, ns="tns", nsmap=NSMAP):
    hour: "int" = element(tag="Hour")
    minute: "int" = element(tag="Minute")
    second: "int" = element(tag="Second")

DateTime.update_forward_refs()

date_time = DateTime(
    date=Date(
        year=2023,
        month=6,
        day=19,
    ),
    time=Time(
        hour=9,
        minute=0,
        second=0,
    ),
)

date_time.to_xml()

I tried debugging and it seems that the time and date fields of the DateTime model are given/initialized with the primitives.PrimitiveTypeSerializerFactory.ElementSerializer serializer, which is causing the failure because they are not primitive fields. I think the likely cause order of declaration of the elements, if I move the Time and Date model declarations before the DateTime model declaration the issue goes away. My takeaway is that running update_forward_refs does not update the serializers of a model, is that correct?

Now the trivial solution here would be to just move the dependent model declaration to a later stage, but I am generating these models from XMLSchema, and I would prefer not to have to do a bunch of analysis to make sure everything goes in the correct order. But if that is the proper solution, I will do it.

[dapper91]

I debugged the model you provided and the figured out the problem. The reason is that pydantic marks ForwardRef as SHAPE_SINGLETON regardless whether the type behind the reference is singleton or not. I will try to fix it today, but for now the simplest solution is to reorder the models declaration to get rid of forward refs.

[Jacob-Flasheye]

A funny little edge case. I was confused for the longest time (I initially found this like a month-ish ago but thought it was restricted to a specific model) because I couldn't reproduce it in the REPL, but in the REPL I'm a good citizen and declare them in the order of least depedence.

[Jacob-Flasheye]

The 0.6.3 update has resolved my issues. Thanks a bunch!