Better Support for Pathological API Patterns

[nathanhammond]

Ember Data makes working with highly nested and irregular APIs incredibly difficult. Below you'll find an example response from a real API that is very difficult to model well.

Things that are worth noting:

• Many pieces of this API are EmbeddedRecords, but those only work when promoted to the top level of what they're embedded inside.
• Some embedded properties are collections and some are one-to-one.
• Some key names are unknown and instead have to be programmatically identified.
• Other endpoints throughout the API return the same models with different data layouts.
• Some endpoints collate, while others are simply individual entities.
• I have minimal influence over the API because I'm only one of its consumers. Modifying it introduces breaking changes for all of the other consumers.

Writing imperative transformations using normalize and friends makes for incredibly tedious and error prone work with lots of indirection. (Passing between serializers.) So what could we do to make it better? How about the attrs property allowing for arbitrarily nested keys?

export default DS.RESTSerializer.extend(DS.EmbeddedRecordsMixin, {
  attrs: {
    createdTime: { key: 'value.createdTime' }, // Proposed feature #1
    value: {
      actor: { embedded: 'always' }, // Proposed feature #2
      socialDetail: {
        comments: { embedded: 'always' },
        likes: { embedded: 'always' }
      }
    }
  }
});

Moving to a pattern like this would make the behavior more declarative, easier to understand, and make the work I'm doing much less painful. I'm also open to other straw men that might help reduce the pain I encounter working with this. I'm opening this bug for conversation.

Note: this bug comes with the offer of my writing the code to enable the conclusion. :)

/cc @tomdale @wycats @chadhietala

---

{
    "updates": [{
        "id": "1234",
        "type": "activity",
        "value": {
            "com.example.feed.ShareUpdate": {
                "actor": {
                    "com.example.feed.MemberActor": {
                        "id": "5432",
                        "miniProfile": {
                            "firstName": "Johnny",
                            "lastName": "Doe",
                            "id": "5432",
                            "type": "com.example.identity.shared.MiniProfile",
                            "headline": "Thing."
                        },
                        "type": "com.example.feed.MemberActor"
                    }
                },
                "createdTime": 1429048918435,
                "content": {
                    "com.example.feed.ShareImage": {
                        "image": {
                            "string": "http://image-store.example.com/461429da-2fbd-4a8e-8c55-2ed99569d6ef-large.jpg"
                        }
                    }
                },
                "socialDetail": {
                    "comments": {
                        "elements": [],
                        "paging": {
                            "total": 0,
                            "count": 0,
                            "start": 0,
                            "links": []
                        }
                    },
                    "relevantComments": {
                        "totalComments": 0,
                        "elements": []
                    },
                    "liked": false,
                    "totalShares": 0,
                    "likes": {
                        "elements": [],
                        "paging": {
                            "total": 0,
                            "count": 0,
                            "start": 0,
                            "links": []
                        }
                    }
                }
            }
        }
    }, {
        "id": "1235",
        "type": "activity",
        "value": {
            "com.example.feed.ShareUpdate": {
                "actor": {
                    "com.example.feed.MemberActor": {
                        "id": "5432",
                        "miniProfile": {
                            "firstName": "John",
                            "lastName": "Doe",
                            "id": "5432",
                            "type": "com.example.identity.shared.MiniProfile",
                            "headline": "Thing."
                        },
                        "type": "com.example.feed.MemberActor"
                    }
                },
                "createdTime": 1429048898928,
                "content": {
                    "com.example.feed.ShareText": {
                        "text": {
                            "values": [{
                                "value": "test 1 "
                            }]
                        }
                    }
                },
                "socialDetail": {
                    "comments": {
                        "elements": [],
                        "paging": {
                            "total": 0,
                            "count": 0,
                            "start": 0,
                            "links": []
                        }
                    },
                    "relevantComments": {
                        "totalComments": 0,
                        "elements": []
                    },
                    "liked": false,
                    "totalShares": 0,
                    "likes": {
                        "elements": [],
                        "paging": {
                            "total": 0,
                            "count": 0,
                            "start": 0,
                            "links": []
                        }
                    }
                }
            }
        }
    }]
}

[igorT]

Do you know of library that implement good plucking ability like this that we could use for inspiration/or for actual code?

[nathanhammond]

Here's what I coded up today to make it a bit less painful by making it easier to dive into a model. It simply extends normalizeUsingDeclaredMapping to support key paths. For bonus points it plays nicely with EmbeddedRecord.

Usage:

export default ApplicationSerializer.extend({
  attrs: {
    actor: { key: 'value.actor', embedded: 'always' },
  }
});

Would you be interested in a PR to allow key paths?

Code:

import DS from 'ember-data';
import Ember from 'ember';
var get = Ember.get;

export default DS.RESTSerializer.extend(DS.EmbeddedRecordsMixin, {
  normalizeUsingDeclaredMapping(type, hash) {
    var attrs = get(this, 'attrs');
    var payloadKey, key, value, segments, last;

    if (attrs) {
      for (key in attrs) {
        payloadKey = this._getMappedKey(key);
        value = get(hash, payloadKey);

        if (!value) { continue; }

        if (payloadKey !== key) {
          hash[key] = value;
          if (~payloadKey.indexOf('.')) {
            segments = payloadKey.split('.');
            last = segments.pop();
            delete get(hash, segments.join('.'))[last];
          } else {
            delete hash[payloadKey];
          }
        }
      }
    }
  }
});

[fivetanley]

Any sort of big API change like this should go through the rfc process. There's a lot of code here; it would be good to flesh out the specified behavior.