search

onaio / rdt-standard

RDT Open Guidelines Application
Apache License 2.0
1 stars 5 forks source link

Create Sample Microscopy Event Form #234

Open shivenbhatt1 opened 4 years ago

shivenbhatt1 commented 4 years ago

@nateMJ Needs an Event form for extracting microscopy data to OpenSRP. This needs to be completed this week. The items below were taken from https://docs.google.com/spreadsheets/d/0B8sJOkOxfXHcc0pMRnUySEZSdWpxV1NMNHo4bkJLVGdQamdj/edit#gid=226928130

Form Needs Outlined Below

  1. PID
  2. RDT ID
  3. Collection Time
  4. Reading Date
  5. Pf "x"/200wbc (If X>0 then "Positive")
  6. Pv "x"/200wbc (If X>0 then "Positive")
  7. Pm "x"/200wbc (If X>0 then "Positive")
  8. Po "x"/200wbc (If X>0 then "Positive")
  9. Pfg "x"/200wbc (If X>0 then "Positive")
vincent-karuri commented 4 years ago

@shivenbhatt1 what is the format of the Collection Time and Reading Date?

vincent-karuri commented 4 years ago 
{
  "events": [
    {
      "obs": [
        {
          "fieldCode": "coordinates",
          "fieldDataType": "text",
          "fieldType": "formsubmissionField",
          "formSubmissionField": "gps",
          "humanReadableValues": [],
          "parentCode": "",
          "values": [
            "-1.2958468 36.7920032"
          ]
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "rdt_id",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "12202K"
          ],
          "set": [],
          "formSubmissionField": "rdt_id",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldDataType": "text",
          "fieldCode": "patient_id",
          "parentCode": "",
          "values": [
            "12345"
          ],
          "comments": null,
          "formSubmissionField": "patient_id",
          "effectiveDatetime": null,
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pv",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "positive"
          ],
          "set": [],
          "formSubmissionField": "pv",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pf",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "negative"
          ],
          "set": [],
          "formSubmissionField": "pf",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pm",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "negative"
          ],
          "set": [],
          "formSubmissionField": "pm",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "po",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "positive"
          ],
          "set": [],
          "formSubmissionField": "po",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pfg",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "positive"
          ],
          "set": [],
          "formSubmissionField": "pfg",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "collection_time",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "2019-12-04T16:48:31.357Z"
          ],
          "set": [],
          "formSubmissionField": "collection_time",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "reading_date",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "2019-12-04T16:48:31.357Z"
          ],
          "set": [],
          "formSubmissionField": "reading_date",
          "humanReadableValues": []
        }
      ],
      "teamId": "",
      "team": "",
      "dateCreated": "2019-12-04T16:48:31.357Z",
      "baseEntityId": "9af5f21c-2bde-4d4d-8980-6c46eb6e36e2",
      "entityType": "microscopy_results",
      "eventType": "microscopy_result",
      "providerId": "indtester1",
      "formSubmissionId": "f7911fd1-e713-4e85-b082-05fef68541da",
      "identifiers": {},
      "duration": 0,
      "locationId": "",
      "eventDate": "2019-12-03T21:00:00.000Z",
      "type": "Event",
      "id": "a3aefe1a-e84a-4b6c-9a7e-eb8af0e87fab"
    },
    {
      "obs": [
        {
          "fieldCode": "coordinates",
          "fieldDataType": "text",
          "fieldType": "formsubmissionField",
          "formSubmissionField": "gps",
          "humanReadableValues": [],
          "parentCode": "",
          "values": [
            "-1.2958468 36.7920032"
          ]
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "rdt_id",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "12202K"
          ],
          "set": [],
          "formSubmissionField": "rdt_id",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldDataType": "text",
          "fieldCode": "patient_id",
          "parentCode": "",
          "values": [
            "12345"
          ],
          "comments": null,
          "formSubmissionField": "patient_id",
          "effectiveDatetime": null,
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pv",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "negative"
          ],
          "set": [],
          "formSubmissionField": "pv",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pf",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "positive"
          ],
          "set": [],
          "formSubmissionField": "pf",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pm",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "negative"
          ],
          "set": [],
          "formSubmissionField": "pm",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "po",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "positive"
          ],
          "set": [],
          "formSubmissionField": "po",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "pfg",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "negative"
          ],
          "set": [],
          "formSubmissionField": "pfg",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "collection_time",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "2019-12-04T16:48:31.357Z"
          ],
          "set": [],
          "formSubmissionField": "collection_time",
          "humanReadableValues": []
        },
        {
          "fieldType": "formsubmissionField",
          "fieldCode": "reading_date",
          "fieldDataType": "text",
          "parentCode": "",
          "values": [
            "2019-12-04T16:48:31.357Z"
          ],
          "set": [],
          "formSubmissionField": "reading_date",
          "humanReadableValues": []
        }
      ],
      "teamId": "",
      "team": "",
      "dateCreated": "2019-12-04T16:48:31.357Z",
      "baseEntityId": "9af5f21c-2bde-4d4d-8980-6c46eb6e36e2",
      "entityType": "microscopy_results",
      "eventType": "microscopy_result",
      "providerId": "indtester1",
      "formSubmissionId": "e7013fd1-e783-4e85-b082-eafef68a7123",
      "identifiers": {},
      "duration": 0,
      "locationId": "",
      "eventDate": "2019-12-03T21:00:00.000Z",
      "type": "Event",
      "id": "439709af-e84a-dfac-9abe-eb8a61e87deb"
    }
  ]
}

List of microscopy events

shivenbhatt1 commented 4 years ago

@shivenbhatt1 what is the format of the Collection Time and Reading Date?

@vincent-karuri This is unknown, we can set the standard format that you prefer and I can make them implement in the lab setting.

vincent-karuri commented 4 years ago

Ok, the format is YYYY-MM-DDThh:mm:ss.sZ

shivenbhatt1 commented 4 years ago

Actually for this since it’s manual procedure and entry, let’s leave off the hours and minutes. Just DD-MM-YYYY

shivenbhatt1 commented 4 years ago

Actually for this since it’s manual procedure and entry, let’s leave off the hours and minutes. Just DD-MM-YYYY

vincent-karuri commented 4 years ago

I believe what is in the Event is the format the server expects, so maybe nifi can do the transformation and just add zeros @nateMJ ?

antonatem commented 4 years ago

@vincent-karuri Yes, that can be done

vincent-karuri commented 4 years ago

@nateMJ based on some of the queries in this doc, I think we'll also need a coordinates obs.

I've updated the events accordingly.

vincent-karuri commented 4 years ago

@nateMJ made edits to the event format, please check.

antonatem commented 4 years ago

For clarity purposes, below is the mapping of fields to create microscopy event JSON to be posted to microscopy_result. For testing, to use https://rdt-stage.smartregister.org/opensrp/ but eventually to post to https://rdt-ke.smartregister.org/opensrp/

Fields from CSV:

CSV header microscopy_result obs field Logic
SLIDES/PID patient_id
RDT ID rdt_id
P.falciparum /200wbc's pf If "x"/200wbc >0 then "Positive"
P.falciparum /500rbc's pf If x>0 then "Positive"
P.malariae /200wbc's pm If "x"/200wbc >0 then "Positive"
P.malariae /500rbc's pm If x>0 then "Positive"
P.ovale /200wbc's po If "x"/200wbc >0 then "Positive"
P.ovale /500rbc's po If x>0 then "Positive"
P.vivax /200wbc's pv If "x"/200wbc >0 then "Positive"
P.vivax /500rbc's pv If x>0 then "Positive"
Pf. gameto /200wbc's pfg If "x"/200wbc >0 then "Positive"
Pf. gameto /500rbc's pfg If x>0 then "Positive"
Reading Date (cell N6) + collection_time reading_date format YYYY-MM-DDThh:mm:ss.sZ


Fields from rdt_test event joined on rdt_id: pcr_results event field rdt_test event field
teamId teamId
team team
providerId providerId
collection_time dateCreated
baseEntityId baseEntityId


Fields generated/defaulted by NiFi: pcr_results event field value
entityType microscopy_results
eventType microscopy_result
formSubmissionId ${uuid}
eventDate ${now()} - When NiFi generates the events

Columns to Canopy DB:

vincent-karuri commented 4 years ago

We'll need to loop in QA before posting to the production server.

On Thu, Feb 13, 2020, 12:23 PM Antonate Maritim notifications@github.com wrote:

For clarity purposes, below is the mapping of fields to create microscopy event JSON to be posted to microscopy_result. For testing, to use https://rdt-stage.smartregister.org/opensrp/ but eventually to post to https://rdt-ke.smartregister.org/opensrp/

Fields from CSV from CSV: CSV header pcr_results obs field Logic SLIDES/PID patient_id RDT ID rdt_id Collection TIME collection_time format YYYY-MM-DDThh:mm:ss.sZ P.falciparum /200wbc's pf If "x"/200wbc >0 then "Positive" P.malariae /200wbc's pm If "x"/200wbc >0 then "Positive" P.ovale /200wbc's po If "x"/200wbc >0 then "Positive" P.vivax /200wbc's pv If "x"/200wbc >0 then "Positive" Pf. gameto /200wbc's pfg If "x"/200wbc >0 then "Positive" Reading Date (cell N6) reading_date format YYYY-MM-DDThh:mm:ss.sZ

Fields from rdt_test event joined on rdt_id: pcr_results event field rdt_test event field teamId teamId team team providerId providerId

Fields generated/defaulted by NiFi: pcr_results event field value entityType microscopy_results eventType microscopy_result formSubmissionId ${uuid} baseEntityId ${fragment.identifier}

Columns to Canopy DB:

  • SLIDESPID
  • RDT ID
  • SEX
  • AGE
  • Collection TIME
  • LAB Received
  • P.falciparum /200wbc's
  • P.falciparum /500rbc's
  • P.malariae /200wbc's
  • P.malariae /500rbc's
  • P.ovale /200wbc's
  • P.ovale /500rbc's
  • P.vivax /200wbc's
  • P.vivax /500rbc's
  • Pf. gameto /200wbc's
  • Pf. gameto /500rbc's
  • Comments
  • Field Site
  • REF No

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/onaio/rdt-standard/issues/234?email_source=notifications&email_token=AH7P3HSZS5FIYW7MNY7H6ZLRCUGSPA5CNFSM4JZXBRVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELUAIHQ#issuecomment-585630750, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7P3HSELOO4S54CROGLAWLRCUGSPANCNFSM4JZXBRVA .

antonatem commented 4 years ago

I've posted 12 events to microscopy_result event found here https://rdt-stage.smartregister.org/opensrp/rest/event/getAll?serverVersion=1581938808715&limit=100&eventType=microscopy_result.

@shivenbhatt1 A few more clarifications on the CSV data:

antonatem commented 4 years ago

@shivenbhatt1 is this CSV file Microscopy data.xlsx meant to be used for microscopy_result event fields and is the mapping below correct for the new template of MS data from KEMRI?

microscopy_result abs field CSV field
patient_id CareStart_RDT_ID
rdt_id ONA_RDT_ID
pf Target Name column with row value 18s plasmodium falciparum
pm Target Name column with row value 18s plasmodium malariae
po Target Name column with row value 18s plasmodium ovale
pv Target Name column with row value 18s plasmodium vivax
pfg Target Name column with row value 18s plasmodium gameto
collection_time Experiment Run End Time?
reading_date ?
shivenbhatt1 commented 4 years ago

Also include If x>0 /500 RBCs then value is positive for listed strain

shivenbhatt1 commented 4 years ago

@vincent-karuri

Could you specify what is meant by "eventdate" below. Is it the time that Nifi sends the event to OpenSRP? Or is it the time the microscopy slide was created? or the time the microscopy slide was read?

"teamId": "", "team": "", "dateCreated": "2019-12-04T16:48:31.357Z", "baseEntityId": "9af5f21c-2bde-4d4d-8980-6c46eb6e36e2", "entityType": "microscopy_results", "eventType": "microscopy_result", "providerId": "indtester1", "formSubmissionId": "e7013fd1-e783-4e85-b082-eafef68a7123", "identifiers": {}, "duration": 0, "locationId": "", "eventDate": "2019-12-03T21:00:00.000Z", "type": "Event", "id": "439709af-e84a-dfac-9abe-eb8a61e87deb"

shivenbhatt1 commented 4 years ago

@vincent-karuri Just confirming, "collection_time" is the date/time when microscopy slide was created, this. We should be able to set it as the "end" date/time from rdt_test.

"fieldType": "formsubmissionField", "fieldCode": "collection_time", "fieldDataType": "text", "parentCode": "", "values": [ "2019-12-04T16:48:31.357Z" ], "set": [], "formSubmissionField": "collection_time", "humanReadableValues": []

vincent-karuri commented 4 years ago

@vincent-karuri

Could you specify what is meant by "eventdate" below. Is it the time that Nifi sends the event to OpenSRP? Or is it the time the microscopy slide was created? or the time the microscopy slide was read?

"teamId": "", "team": "", "dateCreated": "2019-12-04T16:48:31.357Z", "baseEntityId": "9af5f21c-2bde-4d4d-8980-6c46eb6e36e2", "entityType": "microscopy_results", "eventType": "microscopy_result", "providerId": "indtester1", "formSubmissionId": "e7013fd1-e783-4e85-b082-eafef68a7123", "identifiers": {}, "duration": 0, "locationId": "", "eventDate": "2019-12-03T21:00:00.000Z", "type": "Event", "id": "439709af-e84a-dfac-9abe-eb8a61e87deb"

This is the time nifi creates the event.