Mapping between non odin resources

[jokiefer]

I want to map two different datastructures by using this package.

For example one resource is a django model and one is a simple python object. The datastructure of the two classes differs. So i wrote a mapper:

from odin.mapping import Mapping, map_field
from ows_lib.xml_mapper.capabilities.wms.wms130 import \
    WebMapService as XmlWebMapService
from registry.models.service import WebMapService

class WebMapServiceToXml(Mapping):
    from_obj = WebMapService
    to_obj = XmlWebMapService

    @map_field(from_field='title', to_field='service_metadata.title')
    def title(self, value):
        return value

To update the existing xml from the django object i tried:

    @property
    def updated_capabilitites(self) -> XmlObject:
        """Returns the current version of the capabilities document.

            The values from the database overwrites the values inside the xml document.
        """
        xml_object: OGCServiceMixin = self.xml_backup
        from registry.mapping.service import WebMapServiceToXml

        mapper = WebMapServiceToXml(source_obj=self, )
        mapper.update(destination_obj=xml_object)

        return xml_object

This results in an exception:

Traceback (most recent call last):
django-tests_1         |   File "/opt/mrmap/tests/django/registry/models/tests_document.py", line 40, in test_current_capabilities
django-tests_1         |     capabilities: XmlWebMapService = wms.updated_capabilitites
django-tests_1         |   File "/opt/mrmap/registry/models/document.py", line 108, in updated_capabilitites
django-tests_1         |     from registry.mapping.service import WebMapServiceToXml
django-tests_1         |   File "/opt/mrmap/registry/mapping/service.py", line 7, in <module>
django-tests_1         |     class WebMapServiceToXml(Mapping):
django-tests_1         |   File "/opt/venv/lib/python3.10/site-packages/odin/mapping/__init__.py", line 184, in __new__
django-tests_1         |     raise MappingSetupError(
django-tests_1         | odin.exceptions.MappingSetupError: `from_obj` <class 'registry.models.service.WebMapService'> does not have an attribute resolver defined.

What does dies exception means to me? Do i need to make clear that any field of the both resources has type hints?

[timsavage]

Mappers need to resolve what fields are available to be mapped, this is done by registering a FieldResolver against the type (or super type) to be mapped. The development release has updated documentation covering this.

There is also an example of how to do this with Django models here: https://odin.readthedocs.io/en/development/integration/django/index.html

[jokiefer]

seems like that the resolvers now works for the django models and my plain python object with the following example:

from django.db.models import Model
from odin import registration
from odin.mapping import FieldResolverBase
from odin.utils import getmeta
from ows_lib.xml_mapper.mixins import CustomXmlObject

class ModelFieldResolver(FieldResolverBase):
    """
    Field resolver for Django Models
    """

    def get_field_dict(self):
        meta = getmeta(self.obj)
        return {f.attname: f for f in meta.fields}

class XmlMapperFieldResolver(FieldResolverBase):
    """
    Field resolver for XML Objects
    """

    def get_field_dict(self):
        xml_obj = self.obj
        return {key: getattr(xml_obj, key) for key in filter(lambda key: not key.startswith('_'), xml_obj._fields.keys())}

registration.register_field_resolver(ModelFieldResolver, Model)
registration.register_field_resolver(XmlMapperFieldResolver, CustomXmlObject)

But now the custom field mapping crashes:

Traceback (most recent call last):
django-tests_1         |   File "/opt/mrmap/tests/django/registry/models/tests_document.py", line 40, in test_current_capabilities
django-tests_1         |     capabilities: XmlWebMapService = wms.updated_capabilitites
django-tests_1         |   File "/opt/mrmap/registry/models/document.py", line 110, in updated_capabilitites
django-tests_1         |     from registry.mapping.service import WebMapServiceToXml
django-tests_1         |   File "/opt/mrmap/registry/mapping/service.py", line 7, in <module>
django-tests_1         |     class WebMapServiceToXml(Mapping):
django-tests_1         |   File "/opt/venv/lib/python3.10/site-packages/odin/mapping/__init__.py", line 303, in __new__
django-tests_1         |     mapping_rule = attr_mapping_to_mapping_rule(
django-tests_1         |   File "/opt/venv/lib/python3.10/site-packages/odin/mapping/__init__.py", line 253, in attr_mapping_to_mapping_rule
django-tests_1         |     raise MappingSetupError(
django-tests_1         | odin.exceptions.MappingSetupError: Field `service_metadata.title` of custom mapping `<function WebMapServiceToXml.title at 0x7fdee0c6d360>` not found on to object.

How can is map sub hierarchical properties like service_metadata.title in the WebMapServiceToXml class above?

[timsavage]

It might help if you could provide the data structures (or a basic outline) so I can understand the structures you are attempting to map.

The mappers are largely to map to an object rather than into an attribute on a nested object. Rather than mapping to a model containing the object to be updated, could you just map directly to that object? eg provide model.service_metadata as the update destination.

[jokiefer]

I want to map and update from my django db model WebMapService to the xml representing WebMapService object for example. For some reasons the xml object structure differs from the orm structure.

The ORM model inheritance the service metadata, for example title so that this attribute is directly owned by the orm model. The xml object does not directly ownes the title attribute. It is an attribute of an child object called service_metadata

[jokiefer]

In my pov i determined that what iam thinking about to do with your lib is not possible for now.

For example the update routine is a simple straightforward looping over the applied mapping rules on a single destination object. So those rules are always on the same hierarchy level.

There should be something like a deep_update

    def update(
        self,
        destination_obj,
        ignore_fields=None,
        fields=None,
        ignore_not_provided=False,
    ):
        """
        Update an existing object with fields from the provided source object.

        :param destination_obj: The existing destination object.
        :param ignore_fields: A list of fields that should be ignored eg ID fields
        :param fields: Collection of fields that should be mapped.
        :param ignore_not_provided: Ignore field values that are `NotDefined`

        """
        ignore_fields = ignore_fields or []

        for mapping_rule in self._mapping_rules:
            for name, value in self._apply_rule(mapping_rule).items():
                if not (
                    (name in ignore_fields)
                    or (fields and name not in fields)
                    or (ignore_not_provided and value is NotProvided)
                ):
                    if '.' in name:
                        # deep updating attribute of sub object
                        sub_object_name, sub_attribute_name = name.split('.')
                        if not isinstance(_name, str):
                            raise NotImplementedError("can't update sub-sub attributes")
                        sub_object = getattr(destination_obj, sub_object_name)
                        # FIXME: handle None objects
                        setattr(sub_object, sub_attribute_name, value)
                    else:
                        setattr(destination_obj, name, value)

[timsavage]

While this is a solution to your particular problem it is not something the Mapping is designed to do.

Mappers are designed to prepare and verify mapping rules when the mapping is defined (this is done using Metaclasses) to detect invalid mappings as early as possible, and so each concrete instance shares a pre-prepared set of rules that just need to be applied.

Typically to map into a sub-object a second mapper is used specifically for that type, a mapping rule can pass the value to another mapper to generate the new object.

Reading of values from a nested datastructure is supported using TraversalPaths. Extending TraversalPath with a set_value method might do what you are after. TraversalPath also handles multiple levels of nesting as well as indexing into lists.

The original use-case for Mapping was to handle migrating from Version 1 of a datastructure to Version 2 of a data structure and later as an API translation layer between Django Models and native Odin Resource objects.

[jokiefer]

Typically to map into a sub-object a second mapper is used specifically for that type, a mapping rule can pass the value to another mapper to generate the new object.

Could you provide an example how i can do this?

[timsavage]

I'm simulating your existing object with Odin Resources. For a single field this is overkill, but if you need to update a lot of values the mappings become much easier.

import odin

# Data model

class DBModel(odin.AnnotatedResource):
    title: str

class WebServiceMetadata(odin.AnnotatedResource):
    title: str

class WebServiceModel(odin.AnnotatedResource):
    service_metadata: WebServiceMetadata

# Mappings

class DBModelToWebServiceMetadata(odin.Mapping):
    from_obj = DBModel
    to_obj = WebServiceMetadata

class DBModelToWebServiceModel(odin.Mapping):
    from_obj = DBModel
    to_obj = WebServiceModel

    @odin.assign_field
    def service_metadata(self):
        return DBModelToWebServiceMetadata.apply(self.source)

# Using apply and generating a new instance
my_db_model = DBModel(title="This is the new title")
web_service_model = DBModelToWebServiceModel.apply(my_db_model)
print(f"Created with title: {web_service_model.service_metadata.title}")

# Using update to update existing parts of and existing data structure
my_web_service_model = WebServiceModel(
    service_metadata=WebServiceMetadata(title="Old Title")
)
DBModelToWebServiceMetadata(my_db_model).update(my_web_service_model.service_metadata)
print(f"Updated title to: {my_web_service_model.service_metadata.title}")

[jokiefer]

thanks for the solution. I adapt it for my special use case like this:

from copy import deepcopy

from odin.mapping import Mapping, assign_field
from ows_lib.xml_mapper.capabilities.wms.wms130 import \
    ServiceMetadata as XmlServiceMetadata
from ows_lib.xml_mapper.capabilities.wms.wms130 import \
    WebMapService as XmlWebMapService
from registry.models.service import WebMapService

class ServiceMetadataToXml(Mapping):
    from_obj = WebMapService
    to_obj = XmlServiceMetadata

class WebMapServiceToXml(Mapping):
    from_obj = WebMapService
    to_obj = XmlWebMapService

    def __init__(self, xml_root: XmlWebMapService, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.xml_root: WebMapService = xml_root

    @assign_field
    def service_metadata(self):
        """ Updating the service metadata field by using the concrete mapper and a deep copy of the old xml object

        .. note::
           Can't be handled by the apply function as predicted in https://github.com/python-odin/odin/issues/137#issuecomment-1408750868
           Cause the xml mapper objects only handles a subset of all xpaths which are present in a capabilities document (only database relevant fields), 
           the usage of apply will create a new fresh xml mapper object with just the xml structure of the concrete attributes which are handled by it. 
           So this will not represent the full valid xml structure of a valid capabilities document. So we need to update the existing xml objects. 

        .. note:: 
           Update routine from the odin package will only work if the object instances are not the same, 
           cause otherwise the setattr() will result in empty data. 
           Don't know why... 
           So it is necessary to do a deepcopy of the existing object first.
        """
        return ServiceMetadataToXml(source_obj=self.source).update(
            destination_obj=deepcopy(self.xml_root.service_metadata))

For me this is the easiest way to update all fields downward the full complex structure of a complete ogc capabilitites document from the db models. Thanks.