RockefellerArchiveCenter / pisces

A service for getting and transforming data for discovery.
MIT License
1 stars 2 forks source link

Odin coerces ArchivesSpaceSubnote "items" fields to strings #191

Closed p-galligan closed 4 years ago

p-galligan commented 4 years ago

Describe the bug

Transformations are failing for AS resources in Pisces sometimes due to note/subnote mapping.

To reproduce

Steps to reproduce the behavior:

  1. Add resource ID 397 (FA193) to fixtures for testing

Expected behavior

Transformations will pass

Error message or screenshots

Full stacktrace: `====================================================================== ERROR: test_as_mappings (transformer.tests.TransformerTest)

Traceback (most recent call last): File "/code/transformer/transformers.py", line 91, in run transformed = json.loads(json_codec.dumps(mapping.apply(from_obj))) File "/usr/local/lib/python3.6/site-packages/odin/codecs/json_codec.py", line 122, in dumps return json.dumps(resource, cls=cls, kwargs) File "/usr/local/lib/python3.6/json/init.py", line 238, in dumps kw).encode(obj) File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/local/lib/python3.6/site-packages/odin/codecs/json_codec.py", line 49, in default return list(o) File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 393, in len return len(self.items) File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 389, in items list(iter(self)) File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 377, in iter for item in super(CachingMappingResult, self).iter(): File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 346, in iter yield self.mapping.apply(item, self.context, *self.mapping_options) File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 437, in apply return cls(source_obj, context).convert() File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 571, in convert values.update(self._apply_rule(mapping_rule)) File "/usr/local/lib/python3.6/site-packages/odin/mapping/init.py", line 534, in _apply_rule to_values = (list(to_values),) File "/code/transformer/mappings.py", line 167, in return (self.map_subnotes(v) for v in value) File "/code/transformer/mappings.py", line 158, in map_subnotes content = [{'label': i.get('event_date'), 'value': ', '.join(i.get('events'))} for i in value.items] File "/code/transformer/mappings.py", line 158, in content = [{'label': i.get('event_date'), 'value': ', '.join(i.get('events'))} for i in value.items] AttributeError: 'str' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/code/transformer/tests.py", line 40, in test_as_mappings transform = ArchivesSpaceDataTransformer().run(source) File "/code/transformer/transformers.py", line 98, in run raise ArchivesSpaceTransformError("Error transforming {}: {}".format(self.object_type, str(e))) transformer.transformers.ArchivesSpaceTransformError: Error transforming resource: 'str' object has no attribute 'get'`

helrond commented 4 years ago

This appears to be the offending note:

                 {
                    "jsonmodel_type": "note_chronology",
                    "title": "Biographical note",
                    "publish": true,
                    "items": [
                        {
                            "event_date": "1900",
                            "events": [
                                "Born January 25 Geneva, Switzerland"
                            ]
                        },
                        {
                            "event_date": "1924",
                            "events": [
                                "D. Sc. Chemical Engineering, University of Geneva"
                            ]
                        },
                        {
                            "event_date": "1926",
                            "events": [
                                "Ph. D. Physical Chemistry University of Geneva"
                            ]
                        },
                        {
                            "event_date": "1926-1927",
                            "events": [
                                "Volunteer in P.A.T. Levene laboratory, Rockefeller Institute for Medical Research, New York City."
                            ]
                        },
                        {
                            "event_date": "1928-1936",
                            "events": [
                                "Assistant RIMR. Physical chemist in Levene lab."
                            ]
                        },
                        {
                            "event_date": "1936-1947",
                            "events": [
                                "Associate in Physical Chemistry RIMR"
                            ]
                        },
                        {
                            "event_date": "1947-1966",
                            "events": [
                                "Associate member and Associate Professor, Rockefeller University"
                            ]
                        },
                        {
                            "event_date": "1966",
                            "events": [
                                "Prof"
                            ]
                        },
                        {
                            "event_date": "1970-",
                            "events": [
                                "Professor Emeritus (Home: 1801 Mont-Pélerin) Vaud Switzerland"
                            ]
                        },
                        {
                            "event_date": "1987",
                            "events": [
                                "Died December 12"
                            ]
                        }
                    ]
                }
helrond commented 4 years ago

in ArchivesSpaceSubnotes, items is an odin.StringField. This is fine if items is a simple list, but in cases where items is actually a list of dicts, it's less fine.

To fix this we'll need to re-convert items to a dict when necessary.