cph-cachet / carp_studies_app

The CARP Study App
MIT License
6 stars 1 forks source link

Survey data is not uploaded or stored in CAWS #342

Open bardram opened 1 week ago

bardram commented 1 week ago

Survey data is not available on when downloading data from CAWS.

The only thing available is the completedTask of a survey task, like this:

        {
          "data": {
            "__type": "dk.cachet.carp.completedtask",
            "taskData": {
              "__type": "dk.cachet.carp.survey",
              "surveyResult": {
                "results": {
                  "survey.demographics.1": {
                    "end_date": "2024-10-20T20:39:57.918229",
                    "identifier": "survey.demographics.1",
                    "start_date": "2024-10-20T20:39:57.917076"
                  },
                  "survey.demographics.2": {
                    "end_date": "2024-10-20T20:39:57.918356",
                    "identifier": "survey.demographics.2",
                    "start_date": "2024-10-20T20:39:57.918324"
                  },
                  "survey.demographics.3": {
                    "end_date": "2024-10-20T20:39:57.918410",
                    "identifier": "survey.demographics.3",
                    "start_date": "2024-10-20T20:39:57.918392"
                  }
                },
                "identifier": "demo_survey",
                "start_date": "2024-10-20T20:39:57.913648"
              }
            },
            "taskName": "Task #19"
          },
          "sensorStartTime": 1729449597948697
        },

which does not contain the result of the survey either.

This can be caused by several things:

  1. Data is not uploaded @bardram
  2. Data is not stored @yuanchen233
  3. Data is not extracted in the export file @yuanchen233
bardram commented 1 week ago

Seems like the survey data type is not even part of the data being collected for upload to the data stream service. Here is the list of data type being collected for upload, and survey is not part of this list:

[log] [CAMS DEBUG] CarpDataManager - Starting upload of data batches...
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.error'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.ambientlight'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.stepcount'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.freememory'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.deviceinformation'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.batterystate'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.screenevent'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.activity'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.location'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Location Service' and data type 'dk.cachet.carp.mobility'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Weather Service' and data type 'dk.cachet.carp.weather'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Air Quality Service' and data type 'dk.cachet.carp.airquality'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.heartbeat'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.triggeredtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.completedtask'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.polar.hr'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Polar HR Sensor' and data type 'dk.cachet.carp.polar.ecg'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.currentlocation'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.weather'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.location'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.audio'.
[log] [CAMS DEBUG] DataStreamBuffer - getting data stream batch for device 'Primary Phone' and data type 'dk.cachet.carp.image'.
bardram commented 1 week ago

Seems like the survey data type is not even part of the data being collected for upload to the data stream service. Here is the list of data type being collected for upload, and survey is not part of this list:

This is because the survey data type is not part of the list of measures to collect. And this is because it is part of an AppTask and not in the measure list.

bardram commented 1 week ago

There is also an issue that the JSON uploaded, is the old snake_case JSON from Research Package (and not camelCase). We need to update to the latest version of RP....

bardram commented 1 week ago

Survey data also seems to be stored twice in the local DB:

[log] [CAMS DEBUG] SQLiteDataManager - wrote measurement to SQLite - id: 20, type: dk.cachet.carp.survey, device role name: Primary Phone.
[log] [CAMS DEBUG] SQLiteDataManager - wrote measurement to SQLite - id: 21, type: dk.cachet.carp.survey, device role name: Primary Phone.
bardram commented 1 week ago

Not all survey data is part of the data downloaded from CAWS.

Note that this is using the "compressed" endpoint.

Here is what I upload as a survey data point:

SURVEY UPLOADED ``` { "sensorStartTime": 1729776116221468, "data": { "__type": "dk.cachet.carp.survey", "surveyResult": { "identifier": "demo_survey", "startDate": "2024-10-24T15:21:56.221236", "results": { "survey.demographics.1": { "identifier": "survey.demographics.1", "startDate": "2024-10-24T15:21:56.221315", "endDate": "2024-10-24T15:21:56.221363", "questionTitle": "survey.demographics.question.sex", "results": { "answer": [ { "__type": "RPChoice", "text": "survey.demographics.other", "value": 3, "isFreeText": false } ] }, "answerFormat": { "__type": "RPChoiceAnswerFormat", "questionType": "SingleChoice", "choices": [ { "__type": "RPChoice", "text": "survey.demographics.femal", "value": 1, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.male", "value": 2, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.other", "value": 3, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.prefer_not", "value": 4, "isFreeText": false } ], "answerStyle": "SingleChoice" } }, "survey.demographics.2": { "identifier": "survey.demographics.2", "startDate": "2024-10-24T15:21:56.221398", "endDate": "2024-10-24T15:21:56.221414", "questionTitle": "survey.demographics.question.age", "results": { "answer": [ { "__type": "RPChoice", "text": "70-79", "value": 7, "isFreeText": false } ] }, "answerFormat": { "__type": "RPChoiceAnswerFormat", "questionType": "SingleChoice", "choices": [ { "__type": "RPChoice", "text": "survey.demographics.under_20", "value": 1, "isFreeText": false }, { "__type": "RPChoice", "text": "20-29", "value": 2, "isFreeText": false }, { "__type": "RPChoice", "text": "30-39", "value": 3, "isFreeText": false }, { "__type": "RPChoice", "text": "40-49", "value": 4, "isFreeText": false }, { "__type": "RPChoice", "text": "50-59", "value": 5, "isFreeText": false }, { "__type": "RPChoice", "text": "60-69", "value": 6, "isFreeText": false }, { "__type": "RPChoice", "text": "70-79", "value": 7, "isFreeText": false }, { "__type": "RPChoice", "text": "80-89", "value": 8, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.90_above", "value": 9, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.prefer_not", "value": 10, "isFreeText": false } ], "answerStyle": "SingleChoice" } }, "survey.demographics.3": { "identifier": "survey.demographics.3", "startDate": "2024-10-24T15:21:56.221436", "endDate": "2024-10-24T15:21:56.221453", "questionTitle": "survey.demographics.question.smoke", "results": { "answer": [ { "__type": "RPChoice", "text": "survey.demographics.smoke.21+", "value": 6, "isFreeText": false } ] }, "answerFormat": { "__type": "RPChoiceAnswerFormat", "questionType": "SingleChoice", "choices": [ { "__type": "RPChoice", "text": "survey.demographics.smoke.never", "value": 1, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.ex", "value": 2, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.1", "value": 3, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.1-10", "value": 4, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.11-20", "value": 5, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.smoke.21+", "value": 6, "isFreeText": false }, { "__type": "RPChoice", "text": "survey.demographics.prefer_not", "value": 7, "isFreeText": false } ], "answerStyle": "SingleChoice" } } } } } } ```

And here is the same data point (check the time stamp in sensorStartTime) when downloaded from CAWS:

SURVEY DOWNLOADED ``` { "data": { "__type": "dk.cachet.carp.survey", "surveyResult": { "results": { "survey.demographics.1": { "endDate": "2024-10-24T15:21:56.221363", "startDate": "2024-10-24T15:21:56.221315", "identifier": "survey.demographics.1" }, "survey.demographics.2": { "endDate": "2024-10-24T15:21:56.221414", "startDate": "2024-10-24T15:21:56.221398", "identifier": "survey.demographics.2" }, "survey.demographics.3": { "endDate": "2024-10-24T15:21:56.221453", "startDate": "2024-10-24T15:21:56.221436", "identifier": "survey.demographics.3" } }, "startDate": "2024-10-24T15:21:56.221236", "identifier": "demo_survey" } }, "sensorStartTime": 1729776116221468 }, ```
bardram commented 1 week ago

It seems like we are not uploading the full survey data - this is what is uploaded when not using the compressed end point. As can be seen, the survey data is missing.

        {
         "dataStream": {
          "studyDeploymentId": "b8b4bf38-5026-417f-bd2f-218fa704b4f7",
          "deviceRoleName": "Primary Phone",
          "dataType": "dk.cachet.carp.survey"
         },
         "firstSequenceId": 63,
         "measurements": [
          {
           "sensorStartTime": 1729778311577814,
           "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
             "identifier": "symptoms_survey",
             "startDate": "2024-10-24T15:58:31.577394",
             "results": {
              "sym_1": {
               "identifier": "sym_1",
               "startDate": "2024-10-24T15:58:31.577492",
               "endDate": "2024-10-24T15:58:31.577541"
              }
             }
            }
           }
          },
          {
           "sensorStartTime": 1729778311577814,
           "data": {
            "__type": "dk.cachet.carp.survey",
            "surveyResult": {
             "identifier": "symptoms_survey",
             "startDate": "2024-10-24T15:58:31.577394",
             "results": {
              "sym_1": {
               "identifier": "sym_1",
               "startDate": "2024-10-24T15:58:31.577492",
               "endDate": "2024-10-24T15:58:31.577541"
              }
             }
            }
           }
          }
         ],
         "triggerIds": [
          10,
          13
         ]
        }
bardram commented 6 days ago

It seems like we are not uploading the full survey data - this is what is uploaded when not using the compressed end point.

This was because the result objects in RP - RPResult - was not done using polymorphic serialization. Fixed in v. 1.7.2