Create Specification for Hamilton Source Format Version 1.0

[Bad-Science]

The specification for how sources should expose data needs to be formally defined. This purpose of this issue is to document the source format in an unambiguous way, with some examples included.

This specification will be used by every source for the foreseeable future, so it must be well polished. We will aim to support this Version 1.0 indefinitely, although that is not guaranteed at this time.

Some R&D has been done in creating the prototype, codenamed malg, but further refinement is needed. We have decided to go with a normalized form, instead of the nested form we used in malg. We may decide to add support for nesting or JSON pointers in a future minor version 1.X, however.

[Bad-Science]

Working on this in #5

This is what the draft looks like as of now:

Sources

A Hamilton instance can consume any number of sources. A source is meant to be an adapter between a stateful data source and the Hamilton pipeline. These sources may contain entirely disjoint data sets, or may contain overlapping or incomplete collections and records. The Hamilton pipeline will automatically resolve and amalgamate all of the collections and records from each of the sources.

To allow for this, and to simplify development, all sources must comply with the given specification (below). As long as the source complies with the specification, it can be written in any language or framework, and may interface with any external services or dependencies.

Specification

The source format borrows most of its structure from json:api, with a few modifications. The most significant difference is in the "relationship object". The Hamilton "relationship object" allows for relationships defined by arbitrary fields.

A source SHOULD represent a logical or physical source of data. A source MUST contain one or more HTTP endpoints. Each endpoint MUST respond with JSON. The JSON response MUST have the top-level objects meta and data.

The meta field MUST contain the field source. The source field containts meta-information on the source itself. It MUST be an object and MUST have the following fields:

• type: the type of records that data will contain
• name: a unique identifier for this source\ Note: Multiple endpoints of the same source SHOULD share the same name.

The data field contains the records themselves. It MUST be an array containing zero or more objects. We will call these objects Records.

Each record in MUST have the following fields:

• type: the type of the record
  • type MUST be the same value for each record in data
• id: an arbitrary identifier for the record
  • id MUST be unique within the scope of the source
  • id SHOULD have some semantic meaning and be human readable
• attributes: the attributes for the record ("attributes object")
  • attributes MUST be an object
  • attributes MAY contain any valid json values
  • attributes MUST NOT contain any member named relationships, links, removed, id, uuid
• relationships: describes how the record is related to other records
  • unlike json:api, relationships in hamilton are defined by attributes objects. This allows the programmer to specify a relationship based on any field of the related record, allowing for relationships between records that may have originiated from different or multiple sources.
  • relationships MUST be an object
  • relationships MUST contain an object for each relation ("relationship object")
  • Each "relationship object" SHOULD have the singularized type of the related object as a key
  • "relationship object"
    • A "relationship object" MUST contain one object named data
    • data MUST contain a member type, which MUST be the type of the related record
    • data MUST contain an "attributes object"
    • The "attributes object" MUST contain at least one member
    • The members of the "attributes object" SHOULD identify exactly one record of the related type

Examples

{
    "meta": {
        "source": {
            "type": "courses",
            "name": "catalog"
        }
    },
    "data": [
        {
            "type": "courses",
            "id": "courses/CSCI/1200",
            "attributes": {
                "shortname": "CSCI 1200",
                "number": "1200",
                "longname": "Data Structures",
                ...
            },
            "relationships": {
                "subject": {
                    "data": {
                        type: "subjects",
                        "attributes": { "shortname": "CSCI" }
                    }
                }
            }
        },
        {
            "type": "courses",
            "id": "courses/MATH/2400",
            "attributes": {
                "shortname": "MATH 1200",
                "number": "2400",
                "longname": "Differential Equations",
                ...
            },
            "relationships": {
                "subject": {
                    "data": {
                        type: "subjects",
                        "attributes": { "shortname": "MATH" }
                    }
                }
            }
        }
    ]
}

{
    "meta": {
        "source": {
            "type": "sessions",
            "name": "sis"
        }
    },
    "data": [
        {
            "type": "sessions",
            "id": "sessions/201809",
            "attributes": {
                "shortname": "201809",
                "longname": "Fall 2018"
            }
        }
    ]
}

{
    "meta": {
        "source": {
            "type": "sections",
            "name": "sis"
        }
    },
    "data": [
        {
            "type": "sections",
            "id": "sections/CSCI/1200/201809/01",
            "attributes": {
                "shortname": "01",
                "seats": 10,
                "seats_taken": 8,
                "status": "open"
            },
            "relationships": {
                "session": {
                    "data": {
                        "type": "sessions",
                        "attributes": { "shortname": "201809" }
                    }
                },
                "course": {
                    "data": {
                        "type": "courses",
                        "attributes": { "shortname": "CSCI 1200" }
                    }
                }
            }
        }
    ]
}