LoggerEnc import from 2017-19

florianm commented 4 years ago

Problem

I’m also concerned that temperature logger info has dropped off for the 2017-18 and 2018-19 seasons. It seems weird that they were recorded fine in 2016-17 at Thev (no Del then), then no record at all for 2017-18 and 2018-19 at either Thev or Del and then when I exported the data out of ODK Central for the 2019-20 season the info was there again for both locations. I was going to wait until after the data import for the season just gone to finish to see if it dropped that data but seeing as I was writing this email I figured I’d add this on, and I suspect it’s a different import process given the new server, etc so if there was in issue before it probably wouldn’t exist now anyway.

The data import for early form versions did load loggers, the newer versions (36-54) did not. The current ETL pipeline does. Easiest to search raw submissions and backfill loggers.

florianm commented 4 years ago

Solution

Manually import the 20-odd loggerenc from ODKA downloads.

Download raw submission data from ODK Aggregate

import os; path = os.path.join(settings.MEDIA_ROOT, "odka")
from wastd.observations.utils import *
save_all_odka(path=path)

Find LoggerEnc in submissions

Search all "Track or Treat" ( = TurtleNestEncounter, TNE) submissions for logger_details.

Backfill TNE / LoggerEnc

Example submission:

{
    "submission": {
      "@xmlns": "http://opendatakit.org/submissions",
      "@xmlns:orx": "http://openrosa.org/xforms",
      "data": {
        "data": {
          "@id": "build_Track-or-Treat-0-51_1517196378",
          "@instanceID": "uuid:5717cb2e-569a-412f-9be1-7e46e2180cc5",
          "@submissionDate": "2018-01-31T11:32:53.850Z",
          "@isComplete": "true",
          "@markedAsCompleteDate": "2018-01-31T11:32:53.850Z",
          "orx:meta": {
            "orx:instanceID": "uuid:5717cb2e-569a-412f-9be1-7e46e2180cc5"
          },
          "observation_start_time": "2018-01-30T22:50:50.701Z",
          "reporter": "Karratha5",
          "device_id": "15206a105ccfb2db",
          "details": {
            "nest_age": "old",
            "species": "natator-depressus",
            "nest_type": "hatched-nest",
            "observed_at": "-20.4658400000 117.0753833000 15.8000000000 4.7000000000"
          },
          "track_photos":...,
          "nest_photos": ...,
          "nest": {
            "habitat": "beach-above-high-water",
            "disturbance": "absent",
            "nest_tagged": "yes",
            "logger_found": "yes",
            "eggs_counted": "yes",
            "hatchlings_measured": "no",
            "fan_angles_measured": "no"
          },
          "egg_count": ...
          "nest_tag": {
            "status": "resighted",
            "flipper_tag_id": "WA88343",
            "date_nest_laid": "2017-12-03",
            "tag_label": null,
            "tag_comments": null,
            "photo_tag": null
          },
          "logger_details": {
            "logger_id": "763",
            "photo_logger": "1517353298662.jpg"
          },
          "fan_angles": ...,
          "hatchling_emergence_time_group": ...,
          "emergence_climate": ...,
          "other_light_sources":...,
          "observation_end_time": "2018-01-30T23:01:44.317Z"
        }
      },
      "mediaFile": {
        "filename": "1517353298662.jpg",
        "hash": "md5:528fac01397c1070710b212de716251f",
        "downloadUrl": "https://dpaw-data.appspot.com/view/binaryData?blobKey=build_Track-or-Treat-0-51_1517196378%5B%40version%3Dnull+and+%40uiVersion%3Dnull%5D%2Fdata%5B%40key%3Duuid%3A5717cb2e-569a-412f-9be1-7e46e2180cc5%5D%2Flogger_details%5B%40ordinal%3D1%5D%2Fphoto_logger%5B%40ordinal%3D1%5D&as_attachment=true"
      }
    }
  },

Search TNE for instance_id: https://wastd.dbca.wa.gov.au/admin/observations/turtlenestencounter/?q=uuid%3A5717cb2e-569a-412f-9be1-7e46e2180cc5 (found one TNE)
DO NOT create a new LoggerEncounter - until we figure out logger asset management, loggers are handled as "nest tags", as in a uniquely marked physical item associated with a turtle nest.
Add a NestTagObservation to the TNE with the logger ID as tag. This associates the logger with the nest.
- Accuracy (m): details > observed_at > fourth number
- Nest: dist, tagged, logger, eggs, hatch, fan angles: from form > nest
- NestTagObservation: source ODK, source ID same as TNE, status re-sighted, flipper tag ID = "Lxxx" (xxx=logger id) comments "temp logger", comment "Temp logger retrieved from nest during excavation"
- If logger photo exists: download, upload file as MediaAttachment: source ODK, source_id encounter source_id plus "-logger-XXX"
- Delete any duplicated nest excavations or nest tags
- Curate record