APIs and Response body structures

[nivemaham]

Since we are working on most of the APIs, i would also like to discuss and propose a new possible APIs are response structure and discuss towards a common agreement that can comply with rest of the components, conventions, work-flows and potentially also be implemented by the dashboard. Before i start with the APIs, currently from MP perspective, we have projects, which has number of Source-types , Subjects , Sources connected. The source-type represent the kind of the source, e.g. Empatica E4 and the source is the specific instance of the source-type that is given to a subject. A source-type has number of source-data, which describes the data that are being collected by that source-type. e.g Accelerometer, Temperature etc.

Lets start with the Project view. To find all the available projects, we would use GET /api/projects , which would give a list of project objects.

To find details of a particular project, use GET /api/projects/{projectName}, which gives a single project object.

and to find all the subjects assigned to the projects, use GET /api/projects/{projectName}/subjects, which would give a list of subjects.

I would say these would be the main APIs that we would need to fill the project view. I am not sure whether we would require details of a particular source-type here.

A project response body is as of now the default response we get from ManagementPortal( MP).

{
    "id": 1,
    "projectName": "radar",
    "organization": "Radar-CNS",
    "location":   ##"London",
    "attributes": [],
    "projectStatus": "PLANNING",
    "sourceTypes": [
        {
            "id": 1,
            "producer": "Empatica",
            "model": "E4",
            "catalogVersion": "v1",
            "sourceTypeScope": "PASSIVE",
            "canRegisterDynamically": false
        }
    ]
}

Subject response body was

{
    "subjectId": sub-1,
    "active": "true",
     "effectiveTimeFrame" : {
            "firstRecordReceived" : "2018-01-10T10:08:47+00:00",
            "lastRecordReceived" : "2018-01-11T10:08:47+00:00"
     }, 
    "sources": [
        {
            "sourceId": 1,
            "type": "Empatica",
            "summary": {
                    "state": "CONNECTED",
                    "receivedMessage": "12",
                    "messageLoss": "0.2",
                    "sensor": [
                        {
                               "name": ACCELEROMETER,
                               "state": "CONNECTED", 
                               "receivedMessage": "12",
                               "messageLoss": "0.2",
                       }
                  ]
            }
       }
    ]     
}

I propose to change this to

{
    "subjectId": "sdfsdfkjlsndlskjflsdfksldfkj",
    "status": "ACTIVATED",
    "humanReadableId": "something",
    "createdDate" : "2018-01-10T10:08:47+00:00"
    "projectName" : "radar",
    "lastSeen" :    "2018-01-10T10:08:47+00:00"
    "sources": [
        {
            "sourceId": "skdjfskfjsldfk",
            "sourceName": "E4_A00017",
            "sourceTypeProducer" : "Empatice",
            "sourceTypeModel" : "Empatica_E4",
            "sourceTypeCatalogVersion" : "v0.0.1" ,     
       }
    ]     
}

The Subject-view would then require more details of the subject and sources. Here we can use other APIs to query the statistics of the data collected for each subject per source. The subject response mentioned above gives an idea of the source, source-type. To query the details of a source-type, use GET /api/source-types/{producer}/{model}/{catalogVersion}, which would give the details of a source-type,

{
  "id" : 1,
  "producer" : "Empatica",
  "model" : "E4",
  "catalogVersion" : "v1",
  "sourceTypeScope" : "PASSIVE",
  "canRegisterDynamically" : false,
  "sourceData" : [ {
    "id" : 1,
    "sourceDataType" : "ACCELEROMETER",
    "sourceDataName" : "E4-ACCELEROMETER",
    "frequency" : "somethng",
    "unit" : "jdf",
    "processingState" : "DERIVED",
    "enabled" : true,
  }, {
    "id" : 2,
    "sourceDataType" : "BATTERY",
    "sourceDataName" : "E4 BATTERY",
    "frequency" : null,
    "unit" : null,
    "processingState" : "RAW",
    "enabled" : true
  } ]
}

or a subject specific summary of a particular source could be retrieved by GET /api/projects/{projectName}/subjects/{subjectId}/sources/{sourceId} A source-summary response structure would look like

{
  "sourceId": "skdjfskfjsldfk",
  "sourceName": "E4_A00017",
  "sourceTypeProducer" : "Empatice",
  "sourceTypeModel" : "Empatica_E4",
  "sourceTypeCatalogVersion" : "v0.0.1"    
  "sourceTypeScope" : "PASSIVE",
  "canRegisterDynamically" : false,
  "sourceData" : [ {
    "id" : 1,
    "sourceDataType" : "ACCELEROMETER",
    "sourceDataName" : "E4-ACCELEROMETER",
    "frequency" : "somethng",
    "unit" : "jdf",
    "processingState" : "DERIVED",
    "enabled" : true,
    "numberOfreceivedMessage": "12",
    "approximateMessageLoss": "0.2",

  }, {
    "id" : 2,
    "sourceDataType" : "BATTERY",
    "sourceDataName" : "E4 BATTERY",
    "frequency" : null,
    "unit" : null,
    "processingState" : "RAW",
    "enabled" : true
    "numberOfreceivedMessage": "12",
    "approximateMessageLoss": "0.2",
  } ]
}

Using this response, we can then query the statistics of collected data for a particular source-data. Get aggregated data of a given source-data. Statistic could be min, max, etc. GET /api/data/{subjectId}/sources/{sourceId}/{sourceDataName}/{statistic}?timeWindow={timeWindow}&startTime={startTime}&endTime={endTime} Get the latest aggregated data of a given sensor: GET /api/data/{userId}/sources/{sourceId}/{sourceDataName}/{statistic}/latest?timeWindow={timeWindow} and other APIs to query data.

I did not mention all the APIs possible of course. This is just to give an idea on how we could format the basic response structure for different APIs, use-cases.

I request your opinion on this and discuss what is missing and what do we have more. @herkulano @kerzmaximilian @blootsvoets @yatharthranjan @dennyverbeeck

[yatharthranjan]

I agree with you on the structure as its clearer and consistent with the MP. Maybe, we should also include connected and disconnected info for each source with a subject response since the dashboard needs it. Or else it will have to make multiple api calls to get this info which could make it complicated. What do you guys think?

[nivemaham]

The connected, disconnected statuses were initially calculated by rest-api based on some number of records received and what is expected, if i am not wrong. I don't recommend to do that. If we would want to do that, i think we should also clearly define and agree on what is meant by "Connected" , "Disconnected" or other stages.

[blootsvoets]

Perhaps it would make sense to add an effectiveTimeFrame field to the subject response though? Not to iterate through the whole dataset, but at least get when data was first and last received. Based on that value, the rest-api could assess whether a subject/source is considered CONNECTED or not. We could index the key.userId, key.start combination to allow a quick lookup of those values.

[nivemaham]

{
    "subjectId": "sdfsdfkjlsndlskjflsdfksldfkj",
    "status": "ACTIVATED",
    "humanReadableId": "something",
    "createdDate" : "2018-01-10T10:08:47+00:00"
    "projectName" : "radar",
    "sources": [
        {
            "sourceId": "skdjfskfjsldfk",
            "sourceName": "E4_A00017",
            "sourceTypeProducer" : "Empatice",
            "sourceTypeModel" : "Empatica_E4",
            "sourceTypeCatalogVersion" : "v0.0.1" ,     
            "effectiveTimeFrame" : {
                    "firstRecordReceived" : "2018-01-10T10:08:47+00:00",
                    "lastRecordReceived" : "2018-01-11T10:08:47+00:00"
             },  
       }
    ]     
}

@blootsvoets somthing like this?

[yatharthranjan]

@nivemaham i think this will be sufficient from the dashboard's requirements perspective. This way the clients can decide how long before a device is considered disconnected. Also is the connectedTime here for how long the device has stayed connected ? what will this field show if the device is currently disconnected ?

[nivemaham]

@yatharthranjan No. I referred the time-stamp a source is assigned to a subject. We still don't have a way to track it from MP and the latest back-end implementation gives earliest and latest time-stamp for each source. so i have changed it back.

[nivemaham]

Closed by #89