Measure metadata

[Aschen]

Description

We need another workshop about this feature first, see my comment

Each measure should have associated metadata

Example

Measure to be characterized by the metadata

const assetContent = {
  measures: {
    temperature: {
      origin: {
        model: 'Abeeway',
        reference: '48AHT21',
        // ...
      },
      value: {
        temperature: 28.4
      },
      units: {
        temperature: 'celcius',
      }
      // measures.temperature.value.temperature
      // payloadUuid and updatedAt
    },
    position: {
      origin: {
        model: 'Abeeway',
        reference: '48AHT21',
        // ...
      },
      value: {
        point: {
          lat: 42.21,
          lon: 4.84
        },
        altitude: 45,
      },
      units: {
        altitude: 'meter',
        point: 'geo_point'
      }
      // payloadUuid and updatedAt
    },
  }
}

export class DummyTempDecoder extends Decoder {
  constructor () {
    super('DummyTemp');

    this.payloadsMappings = {
      deviceEUI: { type: 'keyword' }
    };

    this.measures = {
      position: {
        altitude: 'meter',
        point: 'geo_point'
      },
      temperature: {
        degree: 'celcius degree'
      }
    }
  }
}

[Aschen]

We need to rethink how we gather and store data with the plugin:

• data can come from various sources: Device, API, computed (ax + b)
• measures should be stored in their own time-serie collection
• measures should be self-describing

Data source

Instead of having only data coming from device, we should extends the data source type:

• IoT Device: actual Device
• External API: a external API (e.g. Weather API)
• Virtual: a computed measure from other measures

Instead of the devices collection, we could have sources collection

Example

{
  type: 'device',
  model: 'abeeway-temp',
  reference: '12345',
  measures: [
    // array of the last measures (one per name)
    {
      _id: '<document-uniq-id>',
      type: 'battery',
      name: 'battery',
      unit: {
        name: 'volt',
        sign: 'V',
        type: 'number',
      },
      value: {
        battery: 98
      },
      measuredAt: 1638701749787,

      source: {
        // Only keep the payload uuids for this measure
        payloadUuids: ['4234-15266353662-53525262'],
      },
    },

    {
      _id: '<document-uniq-id>',
      type: 'temperature',
      name: 'temperature',
      unit: {
        name: 'degree',
        sign: '°',
        type: 'number',
      },
      value: {
        temperature: 23.5
      },
      measuredAt: 1638701749787,

      source: {
        payloadUuids: ['4234-15266353662-53525262'],
      },
    }
  ],
  metadata: {
    // metadata for the data source
    // coming from the referential
  },
  assetId: 'container-xlarge-TRTZ',
  engineId: 'tenant-worksite-kuzzle',
}

Measures collection

I think we need to store the measures in a separate collection for a better historization:

• historizing measures seems to be the most natural use case
• assets measures history is still available in one request (by searching measure with source.id equals to the linked device)

I suggest to have this structure (open to suggestions/improval):

{
  // Type of the measure. (e.g. "temperature")
  // The type name is also the name of the sub-property to look at 
  // in the "values" object to get the measure main value.
  type: { type: 'keyword '},

  // A data source may have different measures for the same type (e.g. measure temperature 2 times)
  name: { type: 'keyword '},

  // Measure self-description
  unit: {
    properties: {
      name: { type: 'keyword '},
      sign: { type: 'keyword '},
      type: { type: 'keyword '},  
    }
  },

  // Measure value. 
  // Each document will only have one property in this object 
  // (same name as the measure type)
  value: {
    properties: {
      // This object will contains one field per measure type to allows separate
      // type indexation.
      // When a new measure is added, this object is extended.
      temperature: { type: 'float' },
      pressure: { type: 'float' },
      position: { type: 'geo_point' },
      battery: { type: 'integer' },
    }
  },

  // Timestamp of the measure
  measuredAt: { type: 'date' },

  // Source of the measure
  source: {
    // E.g. "device"
    type: { type: 'keyword' },

    // E.g. "AbeewayTemp"
    model: { type: 'keyword' },

    // Array of payload uuids that were used to create this measure.
    payloadUuids: { type: 'keyword' },

    // ID of the data source (document _id)
    id: { type: 'keyword' },

    // Reference of the data source (e.g. a device manufacturer ID)
    reference: { type: 'keyword' },
  },
}

Battery from a device

``` { type: 'battery', name: '' unit: { name: 'volt', sign: 'V', type: 'number', }, value: { battery: 98 }, measuredAt: 1638701749787, source: { id: 'Foobar-12345', type: 'device', model: 'Foobar', reference: '12345', payloadUuids: ['4234-15266353662-53525262'], }, } ```

Position from a device

``` { type: 'position', unit: { name: 'lat_lon', sign: null, type: 'geo_point', }, value: { position: { lat: 2.4323, lon: 41.9873, } }, misc: { altitude: 12, accuracy: 125, }, measuredAt: 1638701749787, source: { id: 'Barfoo-54321', type: 'device', model: 'Barfoo', reference: '54321', payloadUuids: ['4234-15266353662-53525262'], }, } ```

Temperature from an external API

``` { type: 'temperature', unit: { name: 'degree', sign: '°', type: 'number', }, value: { temperature: 23.5 }, measuredAt: 1638701749787, source: { type: 'api', model: 'Weather', payloadUuids: ['4234-15266353662-53525262'], // optional? // If not, what do we put in the document? id: 'Weather-12345', reference: '12345', }, } ```

Pressure from a calculation

``` { type: 'pressure', unit: { name: 'pascal', sign: 'P', type: 'number', }, value: { pressure: 1.2 }, measuredAt: 1638701749787, source: { type: 'virtual', model: 'internal_pressure', payloadUuids: ['4234-15266353662-53525262', '6783-53662152663-58296126'], // optional? // If not, what do we put in the document? id: 'virtual-internal_pressure-12345', reference: '12345', }, } ```

[jpdalbi]

The measure collection historises a measure but a state of linked asset. It is the only solution to have the situation of an asset in the past. In the decoder process it should be possible to link metadata to the measures. For example the asset commun name, the asset situation (in a platform, on a parking, ...) and the name of a platform when it is inside a platform. It was possible in the former asset-histrory.

[Aschen]

@jpdalbi Entire asset historization in a separate collection is possible by listening to engine:{engine-name}:asset:measures:new event

For now it's not a default behavior because it creates a lot of overhead and it's not a systematic usecase.

I'm closing this issue since the original behavior has been implement in 2.0.0-rc1