Define an initial set of methods for updating the TOB Search Database based on incoming Credentials

[nrempel]

@swcurran I'm thinking it may be difficult to define a set of possible operations to perform on incoming credentials. I'm thinking about approaching this with a configurable Django utility to transform data and map it onto a Django model.

Are you OK with this approach?

[nrempel]

Also, @ianco recommended a couple libraries we could lean on for json manipulation.

https://github.com/Onyo/jsonbender looks quite attractive. I'm going to do some more research but I'm thinking that, if this library is flexible enough, we can have each region register their own transformers for each credential type and map the data directly onto a Django model.

[swcurran]

@nrempel - I don't quite see why that is needed, but you would know better. Remember that the TOB Data Model will be locked down, and the inputs will always be a flat list of name value pairs. With those two constraints, I would think that a declarative model would be possible, which would be preferred.

[nrempel]

Based on discussions today, I will take the approach of having Issuers register their own mapping at registration-time.

[nrempel]

Currently, there is an Issuer registration step that adheres to the following format:

{
    "issuer": {
        "did": "abc123",
        "name": "bcreg",
        "abbreviation": "BCR",
        "email": "bcreg@bc.ca",
        "url": "reg.bc.ca"
    },
    "jurisdiction": {
        "name": "BC Registries",
        "abbreviation": "BCR"
    },
    "claim-types": {
        "name": "Registration",
        "schema": "registration.bc_registries",
        "version": "0.0.1",
        "endpoint": null
    }
}

I propose extending this registration payload to also include mappings to models for our search database.

{
    "issuer": {
        "did": "abc123",
        "name": "bcreg",
        "abbreviation": "BCR",
        "email": "bcreg@bc.ca",
        "url": "reg.bc.ca"
    },
    "jurisdiction": {
        "name": "BC Registries",
        "abbreviation": "BCR"
    },
    "credential-types": {
        "name": "Registration",
        "schema": "registration.bc_registries",
        "version": "0.0.1",
        "endpoint": null,
        "mapping": {
            "src-id-key": "org_registry_ID",
            "models": {
                "name": {
                    // Each field in model required (defined by TOB)
                    "name": {
                        "input": <key for name from claim>,
                        // Optional list of processors from a library of functions
                        // available in TOB
                        "processors": [
                            // Takes input and retrieves the value at key from input
                            "fromCredentialByKey"
                        ]
                    },
                    "language_code": {
                        // Since we don't include any processors,
                        // input is treated as string literal
                        "input": "en"
                    }
                },
                "address": {},
                "contact": {},
                "person": {}
            }
        }
    }
}

This mapping is registered with TOB once by the Issuer. It can change the mapping and re-register to update the mapping.

When the processor receives a credential, it looks up the mapping rules in the CredentialType table and applies them to the input and responding with errors if necessary.

Since we want the issuer to stipulate if a credential is foundational or not, we can include a "foundational" claim in credentials by convention to specify if the credential is foundational or not. This does not prevent bad actors from falsely issuing a foundational claim though so we will be trusting any issuers in the did-auth whitelist.

[andrewwhitehead]

@nrempel Maybe models should be a list to allow multiple addresses and names?

I think having the choice of something like from-claim and from-value (maybe even from-helper) would also be a lot less verbose than using a processor for those cases, although the processors sound useful in general.

[nrempel]

It could be a list, but I can't think of a use case where a single credential maps to multiple addresses. But I suppose it never hurts to be more flexible.

Yeah, I agree the format currently is pretty verbose, but I wanted the ability to have an arbitrary pipeline of processors. e.g., input("key")->getFromCredential["key"]->toISO8601->etc

Also, it's verbose but the registration is only done once, so it shouldn't be too big of a burden on any workflows

[nrempel]

Based on @cywolf's feedback, I've come up with the following format for as much flexibility as possible. Yet, the most common cases (getting value from claim or sending string literal) are less verbose. A data processing pipeline is still supported.

{
    "issuer": {
        "did": "abc123",
        "name": "bcreg",
        "abbreviation": "BCR",
        "email": "bcreg@bc.ca",
        "url": "reg.bc.ca"
    },
    "jurisdiction": {
        "name": "BC Registries",
        "abbreviation": "BCR"
    },
    "credential-types": {
        "name": "Registration",
        "schema": "registration.bc_registries",
        "version": "0.0.1",
        "endpoint": null,
        "mapping": {
            "src-id-key": "org_registry_ID",
            "models": [
                {
                    // Model names are validated
                    "model-name": "Name",
                    // Each field in model required (defined by TOB)
                    "fields": {
                        "name": {
                            "from-claim": "org_name"
                        },
                        "language_code": {
                            "from-value": "en"
                        }
                    }
                },
                {
                    "model-name": "Person",
                    "fields": {
                        "full_name": {
                            "from-claim": "owner_name"
                        },
                        "name_type": {
                            "from-value": "good"
                        },
                        "start_date": {
                            "processor": {
                                // staticly define the key to use
                                "input": "effective_date",
                                "steps": [
                                    // Look it up in the claim by key
                                    "fromClaim",
                                    // pipe to function
                                    "iso8601ToEpoch"
                                ]
                            }
                        }
                    }
                }
            ]
        }
    }
}

At registration time, TOB will enforce that a mapping can include only one of from-claim, from-value, processor.