Implement data schemas and generic datahandlers

[starsinmypockets]

Description

We should define a schema which is attached to the dataset metadata and sent by the Datastore API with each API call. It should look something like this:

{
  // ... existing metadata
  resources: [
    {
      name: 'countyData',
      uuid: '1bc-123',
      fields: [
        {
            name: 'countyId',
            pk: true,
            type: 'uuid'
        },
        {
            name: 'population',
            type: 'longInt'
        },
      ]
    },
    {
      name: 'physicianData',
      uuid: '1bc-123',
      fields: [
        {
            name: 'physId',
            pk: true,
            type: 'uuid'
        },
        {
            name: 'specialty',
            type: 'int',
            fk: true
        },
        {
            name: 'WL1 County',
            type: 'int',
            fk: true
        },          
        {
            name: 'County1',
            type: 'int',
        },
        {
            name: 'race',
            type: 'enum',
            enum: ['White', 'African American', 'Native American', //...]
        },
        {
            name: 'notes',
            type: 'string'
        }
    }  
  ]
}

On the client side, we will have access to this, and can start to use our type-safety to perform data operations from generic datahandlers - rather than writing custom code

for instance, if our schema defines a field: accpeptsMedicare type: bool we could provide a datahandler definition pieChartFromBoolField: {fieldName: 'acceptsMedicare'} which returns the chart.

This concept could / should be extended to provide a well-documented DSL that is tightly coupled to the datastore api.

An imaginary dsl could look like:

pieChartSeriesFromBool
pieChartSeriesFromEnum
pieChartSeriesFromRange
barChartSeriesFromBool
barChartSeriesFromEnum

etc.

*note - these are direct mappings from known API return values (along with metadata such as type) and known data formats for chart components

Further analysis will be required to capture the useful set of generic dataHandlers

Notes

• We try to use an existing type system, maybe mysql datatypes https://dev.mysql.com/doc/refman/5.7/en/data-types.html
• This should be incorporated as an optional extension to the existing dataset metadata

[starsinmypockets]

Prob we should use JSON -schema, but I think some custom types like enum could be very useful for implementing our dsl

[starsinmypockets]

Also, adding a schema should be backwards compatible, we can create implementations that use it, but existing implementations (of dkan metadata as well as dkan-dash implementations) should continue to work

[acouch]

JSON Schema supports enum values or is that not what you were talking about?

[starsinmypockets]

@acouch yeah that's perfect