Survey data is not uploaded or stored in CAWS

[bardram]

Survey data is not available on when downloading data from CAWS.

The only thing available is the completedTask of a survey task, like this:

        {
          "data": {
            "__type": "dk.cachet.carp.completedtask",
            "taskData": {
              "__type": "dk.cachet.carp.survey",
              "surveyResult": {
                "results": {
                  "survey.demographics.1": {
                    "end_date": "2024-10-20T20:39:57.918229",
                    "identifier": "survey.demographics.1",
                    "start_date": "2024-10-20T20:39:57.917076"
                  },
                  "survey.demographics.2": {
                    "end_date": "2024-10-20T20:39:57.918356",
                    "identifier": "survey.demographics.2",
                    "start_date": "2024-10-20T20:39:57.918324"
                  },
                  "survey.demographics.3": {
                    "end_date": "2024-10-20T20:39:57.918410",
                    "identifier": "survey.demographics.3",
                    "start_date": "2024-10-20T20:39:57.918392"
                  }
                },
                "identifier": "demo_survey",
                "start_date": "2024-10-20T20:39:57.913648"
              }
            },
            "taskName": "Task #19"
          },
          "sensorStartTime": 1729449597948697
        },

which does not contain the result of the survey either.

This can be caused by several things:

1. Data is not uploaded @bardram
2. Data is not stored @yuanchen233
3. Data is not extracted in the export file @yuanchen233

[bardram]

Seems like the survey data type is not even part of the data being collected for upload to the data stream service. Here is the list of data type being collected for upload, and survey is not part of this list:

[log] [CAMS DEBUG] CarpDataManager - Starting upload of data batches...
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.error'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.ambientlight'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.stepcount'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.freememory'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.deviceinformation'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.batterystate'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.screenevent'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.activity'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.location'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.mobility'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.weather'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.airquality'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.polar.hr'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.polar.ecg'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.currentlocation'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.weather'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.location'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.audio'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.image'.

[bardram]

Seems like the survey data type is not even part of the data being collected for upload to the data stream service. Here is the list of data type being collected for upload, and survey is not part of this list:

This is because the survey data type is not part of the list of measures to collect. And this is because it is part of an AppTask and not in the measure list.

[bardram]

There is also an issue that the JSON uploaded, is the old snake_case JSON from Research Package (and not camelCase). We need to update to the latest version of RP....

[bardram]

Survey data also seems to be stored twice in the local DB:

[log] [CAMS DEBUG] SQLiteDataManager - wrote measurement to SQLite - id: 20, type: dk.cachet.carp.survey, device role name: Primary Phone.
[log] [CAMS DEBUG] SQLiteDataManager - wrote measurement to SQLite - id: 21, type: dk.cachet.carp.survey, device role name: Primary Phone.

[bardram]

Not all survey data is part of the data downloaded from CAWS.

Note that this is using the "compressed" endpoint.

Here is what I upload as a survey data point:

SURVEY UPLOADED

``` { "sensorStartTime": 1729776116221468, "data": { "__type": "dk.cachet.carp.survey", "surveyResult": { "identifier": "demo_survey", "startDate": "2024-10-24T15:21:56.221236", "results": { "survey.demographics.1": { "identifier": "survey.demographics.1", "startDate": "2024-10-24T15:21:56.221315", "endDate": "2024-10-24T15:21:56.221363", "questionTitle": "survey.demographics.question.sex", "results": { "answer": [ { "__type": "RPChoice", "text": "survey.demographics.other", "value": 3, "isFreeText": false } ] }, "answerFormat": { "__type": "RPChoiceAnswerFormat", "questionType": "SingleChoice", "choices": [ { "__type": "RPChoice", "text": "survey.demographics.femal", "value": 1, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.male", "value": 2, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.other", "value": 3, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.prefer_not", "value": 4, "isFreeText": false } ], "answerStyle": "SingleChoice" } }, "survey.demographics.2": { "identifier": "survey.demographics.2", "startDate": "2024-10-24T15:21:56.221398", "endDate": "2024-10-24T15:21:56.221414", "questionTitle": "survey.demographics.question.age", "results": { "answer": [ { "__type": "RPChoice", "text": "70-79", "value": 7, "isFreeText": false } ] }, "answerFormat": { "__type": "RPChoiceAnswerFormat", "questionType": "SingleChoice", "choices": [ { "__type": "RPChoice", "text": "survey.demographics.under_20", "value": 1, "isFreeText": false }, { "__type": "RPChoice", "text": "20-29", "value": 2, "isFreeText": false }, { "__type": "RPChoice", "text": "30-39", "value": 3, "isFreeText": false }, { "__type": "RPChoice", "text": "40-49", "value": 4, "isFreeText": false }, { "__type": "RPChoice", "text": "50-59", "value": 5, "isFreeText": false }, { "__type": "RPChoice", "text": "60-69", "value": 6, "isFreeText": false }, { "__type": "RPChoice", "text": "70-79", "value": 7, "isFreeText": false }, { "__type": "RPChoice", "text": "80-89", "value": 8, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.90_above", "value": 9, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.prefer_not", "value": 10, "isFreeText": false } ], "answerStyle": "SingleChoice" } }, "survey.demographics.3": { "identifier": "survey.demographics.3", "startDate": "2024-10-24T15:21:56.221436", "endDate": "2024-10-24T15:21:56.221453", "questionTitle": "survey.demographics.question.smoke", "results": { "answer": [ { "__type": "RPChoice", "text": "survey.demographics.smoke.21+", "value": 6, "isFreeText": false } ] }, "answerFormat": { "__type": "RPChoiceAnswerFormat", "questionType": "SingleChoice", "choices": [ { "__type": "RPChoice", "text": "survey.demographics.smoke.never", "value": 1, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.ex", "value": 2, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.1", "value": 3, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.1-10", "value": 4, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.11-20", "value": 5, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.21+", "value": 6, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.prefer_not", "value": 7, "isFreeText": false } ], "answerStyle": "SingleChoice" } } } } } } ```

And here is the same data point (check the time stamp in sensorStartTime) when downloaded from CAWS:

SURVEY DOWNLOADED

``` { "data": { "__type": "dk.cachet.carp.survey", "surveyResult": { "results": { "survey.demographics.1": { "endDate": "2024-10-24T15:21:56.221363", "startDate": "2024-10-24T15:21:56.221315", "identifier": "survey.demographics.1" }, "survey.demographics.2": { "endDate": "2024-10-24T15:21:56.221414", "startDate": "2024-10-24T15:21:56.221398", "identifier": "survey.demographics.2" }, "survey.demographics.3": { "endDate": "2024-10-24T15:21:56.221453", "startDate": "2024-10-24T15:21:56.221436", "identifier": "survey.demographics.3" } }, "startDate": "2024-10-24T15:21:56.221236", "identifier": "demo_survey" } }, "sensorStartTime": 1729776116221468 }, ```

[bardram]

It seems like we are not uploading the full survey data - this is what is uploaded when not using the compressed end point. As can be seen, the survey data is missing.

        {
         "dataStream": {
          "studyDeploymentId": "b8b4bf38-5026-417f-bd2f-218fa704b4f7",
          "deviceRoleName": "Primary Phone",
          "dataType": "dk.cachet.carp.survey"
         },
         "firstSequenceId": 63,
         "measurements": [
          {
           "sensorStartTime": 1729778311577814,
           "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
             "identifier": "symptoms_survey",
             "startDate": "2024-10-24T15:58:31.577394",
             "results": {
              "sym_1": {
               "identifier": "sym_1",
               "startDate": "2024-10-24T15:58:31.577492",
               "endDate": "2024-10-24T15:58:31.577541"
              }
             }
            }
           }
          },
          {
           "sensorStartTime": 1729778311577814,
           "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
             "identifier": "symptoms_survey",
             "startDate": "2024-10-24T15:58:31.577394",
             "results": {
              "sym_1": {
               "identifier": "sym_1",
               "startDate": "2024-10-24T15:58:31.577492",
               "endDate": "2024-10-24T15:58:31.577541"
              }
             }
            }
           }
          }
         ],
         "triggerIds": [
          10,
          13
         ]
        }

[bardram]

It seems like we are not uploading the full survey data - this is what is uploaded when not using the compressed end point.

This was because the result objects in RP - RPResult - was not done using polymorphic serialization. Fixed in v. 1.7.2