Optional Included Data

[scotsimon]

Is it possible to have optional included data, similar to how attributes or relationships can be optional?

We have a request to accommodate a situation where included resource objects may or may not be returned in the response... and I'm unclear as to whether or not I can support that request with this library.

Thank you in advance for your input!

Scot

[mattpolzin]

Could you describe to me a bit more of what you would like to have be the result of include data being optional? Perhaps this is a hard question to answer so abstractly, but my gut reaction to your question is along the lines of "include data is never really guaranteed to be there in the first place by this library."

I may be misunderstanding the question, or that might actually be exactly the answer: Include data is always optional in the sense that even when include data is requested of the server, if the server does not respond with the requested include data, the response payload will still be successfully parsed by the JSONAPI library. The result in this case will be that trying to find a particular include in the array of include data will fail, but the JSONAPI library does not (currently) offer built-in ways to access those includes other than to loop over them anyway and the client may indeed discover an empty array when it goes to do so.

[scotsimon]

My experience is that if I define a document with included data, and the JSON response does not include the data, the decoding fails.

For instance, JSON decoding will fail if all of the included data defined in this document is not provided. When we force the JSON response to include all of the data, then it decodes correctly.

let document = try JSONDecoder().decode(SingleDocument<PersonResource, Include5<API.DataModel.Address.GET.AddressResource, API.DataModel.Person.GET.PersonResource, API.DataModel.Email.GET.EmailResource, API.DataModel.PhoneNumber.GET.PhoneNumberResource, API.DataModel.Organization.GET.OrganizationResource>>.self, from: data)

Based on what you're saying, that should not be the case. Is it possible that I've misconfigured something else in the library that could be causing this issue?

[mattpolzin]

I may see what I was misunderstanding -- you are probably saying the response payload does not have an include key in it at all, aren't you? whereas I was for some reason imagining it would have include: [] (an empty array) still.

[scotsimon]

I think that might be the issue... although I would have to get our API guy to reproduce that code again to confirm.

[mattpolzin]

I just checked actually, and what I was picturing is still not a problem, so now I wonder if I could get to the bottom of this faster if you could provide my an example of the response payload (even just with values kind of mocked up) so I knew the structure of the JSON being parsed.

Here's an example of parsing a payload with no included key at all into a document that stores Include1 and I see that it works when I run it in my test suite at least:

{
    "data": {
        "id": "1",
        "type": "articles",
        "relationships": {
            "author": {
                "data": {
                    "type": "authors",
                    "id": "33"
                }
            }
        }
    },
    "links": {
        "link": "https://website.com",
        "link2": {
            "href": "https://othersite.com",
            "meta": {
                "hello": "world"
            }
        }
    },
    "meta": {
        "total": 70,
        "limit": 40,
        "offset": 10
    },
    "jsonapi": {
        "version": "1.0"
    }
}

this decoding succeeds:

        let document = decoded(type: Document<SingleResourceBody<Article>, TestPageMetadata, TestLinks, Include1<Author>, TestAPIDescription, UnknownJSONAPIError>.self,
                               data: single_document_no_includes_with_metadata_with_links_with_api_description)

[scotsimon]

I will append a copy of the JSON that is causing the problem as soon as I can get my API guy to modify the endpoint. Thank you for your help!

[scotsimon]

Ok, so my developer has gotten me both the server code and the response.

The server code is here. You can see where some of the included fields have been commented out. Those relate to the included response objects in the original document I pasted above.

`class OrganizationGroupMemberSchema(BaseSchema): id = fields.Str(dump_only=True)

person = fields.Relationship(
    #self_url="/v1/person/{person_id}",
    #self_url_kwargs={"person_id": "<person.id>"},
    include_resource_linkage=True,
    type_="person",
    required=True,
    schema=PersonSchema(
        only=(
            "id",
            "first_name",
            "last_name",
            "dob",
            "gender",
            "photo_url",
            "is_managed",
            #"addresses",
            #"email_addresses",
            #"phone_numbers",
            #"phone_numbers",
            #"organizations",
            #"dependents",
            #"linked_dependents",
            #"linked_accounts"
        )),
)

join_dtm = fields.DateTime(dump_only=True, data_key="join_dtm")
leave_dtm = fields.DateTime(dump_only=True, data_key="leave_dtm")

active = fields.Bool()

class Meta(BaseSchema.Meta):
    type_ = "organization-group-member"`

And here is the response that is generated from the endpoint. This will not decode correctly with the Included data commented out.

"data": [ { "type": "organization-group-member", "attributes": { "join_dtm": "2022-07-24T03:45:52Z", "leave_dtm": null, "active": true }, "id": "10", "relationships": { "person": { "data": { "type": "person", "id": "244" } } } } ], "included": [ { "type": "person", "attributes": { "last_name": "Schenck", "is_managed": true, "date_of_birth": "2013-05-04", "first_name": "Brady", "gender": "Male", "photo_url": null }, "id": "244" } ] }

[scotsimon]

Ugh, Github has 'helpfully' formatted that JSON for me into a barely readable format. Sorry.

[scotsimon]

So... the attributes are still returned in the Included response, but something is causing it to break during decoding -- and I assumed it was because the other objects were missing because it does work when they are added back.

Thanks for any suggestions!

[mattpolzin]

Thanks for the example. I think you are saying that decoding fails when addresses, email_addresses, etc. are commented out but succeeds when they are not commented out. If that's the case, I did misunderstand you originally.

I thought decoding was failing when either the included key had an empty array value:

{
  "data": [
    "..."
  ],
  "included": []
}

Or when the included key was missing entirely:

{
  "data": [
    "..."
  ]
}

---

So, with my new understanding that decoding fails based on which attribute keys of a Person are available in the response payload, I don't think the problem is with your Include5 type.

Above, you pasted the following Document:

SingleDocument<PersonResource, Include5<API.DataModel.Address.GET.AddressResource, API.DataModel.Person.GET.PersonResource, API.DataModel.Email.GET.EmailResource, API.DataModel.PhoneNumber.GET.PhoneNumberResource, API.DataModel.Organization.GET.OrganizationResource>>

Is this definitely the document definition you are using to decode the example payload you sent me? The example payload seems to be a many-resource document where the primary resource is an organization-group-member and a person is in the included relationship data whereas the Swift code I've repasted here is a single-resource document where the primary resource is a person and organization-group-member appears to be one of the possible included entries (assuming that is what API.DataModel.Organization.GET.OrganizationResource represents).

[scotsimon]

Let me try to clarify.

The base json type is an "organization-group-member", which has a few attributes and a relationship to a "person" object. The response for the organization-group-member is returning the core attributes as well as the included object for the person. The person definition is what has the "Include5" in the document type. It defines all of the objects that might be returned (addresses, other person objects, email addresses, phone numbers, and organizations).

The GET for the "organization-group-member" works perfectly IF the server returns all of the included objects defined in the server code above. However, that information is not desirable for this record - we just want the basic attributes for the person to be returns in the included object. However, when we comment those out, the json that is generated for the GET response for the "organization-group-member" with the partial person included document is not decoding correctly.

Does that clarify, or is that just more confusing?

[mattpolzin]

It does clarify things, but it means that Include5 and Document are not designed to do quite what you are hoping they will do.

The JSONAPI library Document type always represents the whole response body; this means the data object, maybe a meta object, and maybe an included array.

{
  "data": "...",
  "meta": "...",
  "included": "..."
}

In your case, this means the document refers to a list (or batch) of organization-group-member primary resources. I'd guess based on your previous Swift snippet, this would make the type start out as:

ManyDocument<OrganizationResource, ...>

The Include5 type refers to only the included object of the document. So, instead of it defining something about the attributes or relationships of a Person in your example, it describes all the possible resource types you may need to decode in the included array of the response payload. Based on your example payload, you may actually want an Include1 like in the following document type:

ManyDocument<OrganizationResource, Include1<PersonResource>>

With this in mind, the ability to successfully parse a PersonResource that does not have all of its attributes or relationships (a sparse object) will depend on changing the definition of PersonResource; if you share how that is defined, I could probably help continue getting to the bottom of this.

---

On the topic of Include1 (or 2, 3, 4, or 5), the difference would be that ManyDocument<OrganizationResource, Include1<PersonResource>> can decode JSON like:

{
  "data": [...],
  "included": [
    {
      "type": "person",
      ...
    },
    ... (more person resources)
  ]
}

Whereas if you had ManyDocument<OrganizationResource, Include2<PersonResource, EmailResource>> then you could also decode JSON like:

{
  "data": [...],
  "included": [
    {
      "type": "person",
      ...
    },
    {
      "type": "email",
      ...
    },
    ... (more person or email resources)
  ]
}

And so on, with each resource specified in the IncludeX type being another resource that is allowed to show up in the included array of the response body.

[scotsimon]

Let me just share the code for both the Organization-Group-Member and the Person. This has been working... up until the point that we tried to limit the data coming back in the included container.

Organization-Group-Member

`extension API.DataModel.OrganizationGroupMember {

// Create a typealias for DELETE -- Identical structure to GET
typealias DELETE = GET

enum GET {

    // MARK: - OrganizationGroupMember Description for GET endpoints
    enum OrganizationGroupMemberDescription: ResourceObjectDescription {
        static let jsonType: String = "organization-group-member"

        struct Attributes: JSONAPI.Attributes {
            let joinDate: Attribute<String>
            let leaveDate: Attribute<String?>
            let isActive: Attribute<Bool>

            enum CodingKeys: String, CodingKey {
                case joinDate = "join_dtm"
                case leaveDate = "leave_dtm"
                case isActive = "active"
            }
        }

        struct Relationships: JSONAPI.Relationships {
            let person: ToOneRelationship<API.DataModel.Person.GET.PersonResource, NoIdMetadata, NoMetadata, NoLinks>
        }

        typealias Meta = NoMetadata
        typealias Links = NoLinks

    }

    // Typealias for OrganizationGroupResource for GET Endpoints
    typealias OrganizationGroupMemberResource = JSONAPI.ResourceObject<OrganizationGroupMemberDescription, NoMetadata, NoLinks, String>

    // MARK: - Decode single document for GET Endpoint
    static func decodeSingle(data: Data) throws -> API.DataModel.OrganizationGroupMember {

        typealias SingleDocument<Resource: ResourceObjectType, Include: JSONAPI.Include> = JSONAPI.Document<SingleResourceBody<Resource>, NoMetadata, NoLinks, Include, NoAPIDescription, UnknownJSONAPIError>

        do {
            let document = try JSONDecoder().decode(SingleDocument<OrganizationGroupMemberResource, Include1<API.DataModel.Person.GET.PersonResource>>.self, from: data)
            switch document.body {
                case .data(let data):
                    var response = API.DataModel.OrganizationGroupMember(from: data.primary.value)

                    let includedPerson = data.includes[API.DataModel.Person.GET.PersonResource.self]
                    if let person = includedPerson.first {
                        response.person = API.DataModel.Person(from: person)
                    }

                    return response

                case .errors(_, meta: _, links: _):
                    throw API.Error.invalidJson
            }
        } catch {
            throw API.Error.decodingFailure(API.ErrorDetails(error: error, description: error.localizedDescription))
        }

    }

    // MARK: - Decode batch document for GET Endpoint
    static func decodeBatch(data: Data) throws -> [API.DataModel.OrganizationGroupMember] {

        typealias BatchDocument<Resource: ResourceObjectType, Include: JSONAPI.Include> = JSONAPI.Document<ManyResourceBody<Resource>, NoMetadata, NoLinks, Include, NoAPIDescription, UnknownJSONAPIError>

        do {
            let document = try JSONDecoder().decode(BatchDocument<OrganizationGroupMemberResource, Include1<API.DataModel.Person.GET.PersonResource>>.self, from: data)
            switch document.body {
                case .data(let data):
                    var responses = [API.DataModel.OrganizationGroupMember]()
                    for value in data.primary.values {
                        var response = API.DataModel.OrganizationGroupMember(from: value)

                        let includedPerson = data.includes[API.DataModel.Person.GET.PersonResource.self]
                        let personObject: API.DataModel.Person.GET.PersonResource.ID? = value ~> \.person
                        for person in includedPerson {
                            if person.id.rawValue == personObject?.rawValue {
                                response.person = API.DataModel.Person(from: person)
                            }
                        }

                        responses.append(response)
                    }

                    return responses

                case .errors(_, meta: _, links: _):
                    throw API.Error.invalidJson
            }
        } catch (let error) {
            throw API.Error.decodingFailure(API.ErrorDetails(error: error, description: error.localizedDescription))
        }

    }

} // End GET

} `

Person

`extension API.DataModel.Person {

// Create a typealias for DELETE -- Identical structure to GET
typealias DELETE = GET

enum GET {

    // MARK: - Person Description for GET endpoints
    enum PersonDescription: ResourceObjectDescription {
        static let jsonType: String = "person"

        struct Attributes: JSONAPI.Attributes {
            let firstName: Attribute<String>
            let lastName: Attribute<String>
            let birthDate: Attribute<String?>
            let gender: Attribute<String?>
            let photoUrl: Attribute<String?>
            let isManaged: Attribute<Bool>

            enum CodingKeys: String, CodingKey {
                case firstName = "first_name"
                case lastName = "last_name"
                case birthDate = "date_of_birth"
                case gender = "gender"
                case photoUrl = "photo_url"
                case isManaged = "is_managed"
            }
        }

        struct Relationships: JSONAPI.Relationships {
            let addresses: ToManyRelationship<API.DataModel.Address.GET.AddressResource, NoIdMetadata, NoMetadata, NoLinks>
            let dependents: ToManyRelationship<API.DataModel.Person.GET.PersonResource, NoIdMetadata, NoMetadata, NoLinks>
            let emails: ToManyRelationship<API.DataModel.Email.GET.EmailResource, NoIdMetadata, NoMetadata, NoLinks>
            let linkedDependents: ToManyRelationship<API.DataModel.Person.GET.PersonResource, NoIdMetadata, NoMetadata, NoLinks>
            let linkedAccounts: ToManyRelationship<API.DataModel.Person.GET.PersonResource, NoIdMetadata, NoMetadata, NoLinks>
            let organizations: ToManyRelationship<API.DataModel.Organization.GET.OrganizationResource, NoIdMetadata, NoMetadata, NoLinks>
            let phoneNumbers: ToManyRelationship<API.DataModel.PhoneNumber.GET.PhoneNumberResource, NoIdMetadata, NoMetadata, NoLinks>

            enum CodingKeys: String, CodingKey {
                case addresses = "addresses"
                case dependents = "dependents"
                case emails = "email-addresses"
                case linkedDependents = "linked-dependents"
                case linkedAccounts = "linked-accounts"
                case organizations = "organizations"
                case phoneNumbers = "phone-numbers"
            }
        }

        typealias Meta = NoMetadata
        typealias Links = NoLinks

    }

    // Typealias for PersonResource for GET Endpoints
    typealias PersonResource = JSONAPI.ResourceObject<PersonDescription, NoMetadata, NoLinks, String>

    // MARK: - Decode single document for GET Endpoint
    static func decodeSingle(data: Data) throws -> API.DataModel.Person {

        typealias SingleDocument<Resource: ResourceObjectType, Include: JSONAPI.Include> = JSONAPI.Document<SingleResourceBody<Resource>, NoMetadata, NoLinks, Include, NoAPIDescription, UnknownJSONAPIError>

        do {
            let document = try JSONDecoder().decode(SingleDocument<PersonResource, Include5<API.DataModel.Address.GET.AddressResource, API.DataModel.Person.GET.PersonResource, API.DataModel.Email.GET.EmailResource, API.DataModel.PhoneNumber.GET.PhoneNumberResource, API.DataModel.Organization.GET.OrganizationResource>>.self, from: data)
            switch document.body {
                case .data(let data):
                    var person = API.DataModel.Person(from: data.primary.value)

                    let includedAddresses = data.includes[API.DataModel.Address.GET.AddressResource.self]
                    person.addresses = includedAddresses.map { API.DataModel.Address(from: $0) }

                    let includedPersons = data.includes[API.DataModel.Person.GET.PersonResource.self]
                    let dependents: [API.DataModel.Person.GET.PersonResource.ID] = data.primary.value ~> \.dependents
                    let linkedDependents: [API.DataModel.Person.GET.PersonResource.ID] = data.primary.value ~> \.linkedDependents
                    let linkedAccounts: [API.DataModel.Person.GET.PersonResource.ID] = data.primary.value ~> \.linkedAccounts

                    for p in includedPersons {
                        if dependents.contains(p.id) {
                            person.dependents.append(API.DataModel.Person(from: p))
                        }
                        if linkedDependents.contains(p.id) {
                            person.linkedDependents.append(API.DataModel.Person(from: p))
                        }
                        if linkedAccounts.contains(p.id) {
                            person.linkedAccounts.append(API.DataModel.Person(from: p))
                        }
                    }

                    let includedEmail = data.includes[API.DataModel.Email.GET.EmailResource.self]
                    person.emails = includedEmail.map { API.DataModel.Email(from: $0) }

                    let includedPhoneNumbers = data.includes[API.DataModel.PhoneNumber.GET.PhoneNumberResource.self]
                    person.phoneNumbers = includedPhoneNumbers.map { API.DataModel.PhoneNumber(from: $0) }

                    let includedOrganizations = data.includes[API.DataModel.Organization.GET.OrganizationResource.self]
                    person.organizations = includedOrganizations.map { API.DataModel.Organization(from: $0) }

                    return person

                case .errors(_, meta: _, links: _):
                    throw API.Error.invalidJson
            }
        } catch (let error) {
            throw API.Error.decodingFailure(API.ErrorDetails(error: error, description: error.localizedDescription))
        }

    }

    // MARK: - Decode batch document for GET Endpoint
    static func decodeBatch(data: Data) throws -> [API.DataModel.Person] {

        typealias BatchDocument<Resource: ResourceObjectType, Include: JSONAPI.Include> = JSONAPI.Document<ManyResourceBody<Resource>, NoMetadata, NoLinks, Include, NoAPIDescription, UnknownJSONAPIError>

        do {
            let document = try JSONDecoder().decode(BatchDocument<PersonResource, Include5<API.DataModel.Address.GET.AddressResource, API.DataModel.Person.GET.PersonResource, API.DataModel.Email.GET.EmailResource, API.DataModel.PhoneNumber.GET.PhoneNumberResource, API.DataModel.Organization.GET.OrganizationResource>>.self, from: data)
            switch document.body {
                case .data(let data):

                    var persons = [API.DataModel.Person]()

                    for value in data.primary.values {

                        var person = API.DataModel.Person(from: value)

                        let includedAddresses = data.includes[API.DataModel.Address.GET.AddressResource.self]
                        person.addresses = includedAddresses.map { API.DataModel.Address(from: $0) }

                        let includedPersons = data.includes[API.DataModel.Person.GET.PersonResource.self]
                        let dependents: [API.DataModel.Person.GET.PersonResource.ID] = value ~> \.dependents
                        let linkedDependents: [API.DataModel.Person.GET.PersonResource.ID] = value ~> \.linkedDependents
                        let linkedAccounts: [API.DataModel.Person.GET.PersonResource.ID] = value ~> \.linkedAccounts

                        for p in includedPersons {
                            if dependents.contains(p.id) {
                                person.dependents.append(API.DataModel.Person(from: p))
                            }
                            if linkedDependents.contains(p.id) {
                                person.linkedDependents.append(API.DataModel.Person(from: p))
                            }
                            if linkedAccounts.contains(p.id) {
                                person.linkedAccounts.append(API.DataModel.Person(from: p))
                            }
                        }

                        let includedEmail = data.includes[API.DataModel.Email.GET.EmailResource.self]
                        person.emails = includedEmail.map { API.DataModel.Email(from: $0) }

                        let includedPhoneNumbers = data.includes[API.DataModel.PhoneNumber.GET.PhoneNumberResource.self]
                        person.phoneNumbers = includedPhoneNumbers.map { API.DataModel.PhoneNumber(from: $0) }

                        let includedOrganizations = data.includes[API.DataModel.Organization.GET.OrganizationResource.self]
                        person.organizations = includedOrganizations.map { API.DataModel.Organization(from: $0) }

                        persons.append(person)

                    }

                    return persons

                case .errors(_, meta: _, links: _):
                    throw API.Error.invalidJson
            }
        } catch (let error) {
            throw API.Error.decodingFailure(API.ErrorDetails(error: error, description: error.localizedDescription))
        }

    }

} // End GET

} `

[scotsimon]

Well, Github mostly formatted the code correctly...

[mattpolzin]

I think you've given me enough information for me to help with the code snippets above, but one other thing that could make this easier to troubleshoot would be the error message you get when failing to decode. Hopefully it contains a hint as to what exactly was being decoded when things errored out.

[mattpolzin]

I've gone ahead and slightly modified your decodeBatch(data:) -> [API.DataModel.OrganizationGroupMember] function so that in the catch block it does print(error) and when I run that against the JSON data you provided above as an example payload, I get: "Out of the 1 includes in the document, the 1st one failed to parse: relationships object is required and missing."

That is, the JSONAPI library expected each included object to be a PersonResource and a PersonRresource has 7 required relationships (addresses, dependents, emails, linkedDependents, linkedAccounts, organizations, and phoneNumbers) but the 1st entry in the included array does not have a relationships object containing those 7 relationships.

To get a PersonDescription to parse JSON with no relationships object, you must make all 7 relationships omittable. If they can all be omitted, it is also allowed that the server omits the relationships object altogether.

That is to say, your example payload parses successfully again if you add ? to the end of each of the 7 relationships' types which marks them as ok to omit from responses (and also removes the Swift guarantee that they will be available):

let addresses: ToManyRelationship<...>?
let dependents: ToManyRelationship<...>?
let emails: ToManyRelationship<...>?
let linkedDependents: ToManyRelationship<...>?
let linkedAccounts: ToManyRelationship<...>?
let organizations: ToManyRelationship<...>?
let phoneNumbers: ToManyRelationship<...>?

---

If you wanted to guarantee those relationships exist in some response payloads but not in others, you actually need two different PersonDescription types, one for each of those situations. You might want a PersonResource and SparsePersonResource as two distinct types where in one case all relationships can be omitted and in the other they are all required.

[scotsimon]

Matt,

Thank you so much for your feedback! Essentially, my desire for an optional included object is completely dependent on making the relationship optional! I would probably never have caught that subtle distinction. You are awesome!

Thank you for taking the time to help me understand and resolve this issue.

Scot

[reshadf]

I have a question about this topic so I thought I put this here. I have a json returned from the server where the "data" key inside the relations is not always present if the relation is missing. Therefore It fails with decoding because in this case "administration" has no data key. Is there a way to work around this? I tried making the entire relationship optional but that doesn't work. @mattpolzin ( if it has a data key or the entire "administration" block is committed it works as expected )

Thank you for your time in advance!

        "administration": {
          "links": {}
        },
        "costCenter": {
          "links": {},
          "data": null
        },

[mattpolzin]

@reshadf let's move your question to a new GitHub issue if you don't mind. You can just copy your question over as you've written it well here.

"How do I specify that a relationship only has links?" Is a good question but it ends up having a pretty different answer than the above thread.