Nest tags missing 2019 season

[RyanDou]

There are no nest tag observations that have been imported into WAStD from the 2019/2020 season (except those from Port Hedland who I assume were using the UAT form).

Reproducible example

I have no idea the steps that led to this bug however... This url is a nest encounter where an excavation was undertaken: https://wastd.dbca.wa.gov.au/admin/observations/encounter/160085/change/?_changelist_filters=when__day%3D30%26when__month%3D1%26when__year%3D2020 If you look at the photo associated with the encounter you can see that the nest tag for that encounter was JG8. Using odata I can see that there is a nest tag observation in ODK central that is associated with this encounter ( however if you go to the nest tag observations in 2020 you can see that this observation is not present: https://wastd.dbca.wa.gov.au/admin/observations/nesttagobservation/?encounter__when__year=2020

[florianm]

The ETL script discards all Null records (else it would generate lots of "empty" non-observations). All records with only a photo, but no text data in the nest tag section, would have no nest-tag-observation (but "nest tag found"). This particular record seems to have a date_nest_laid, so it should have been uploaded.

Re-running import now to diagnose shows this log:

◉ No new TurtleNestTagObservations to create
◉ Retained 326 existing and uncurated TurtleNestTagObservations

This indicates there are 326 nest tags for season 2019-20. WAStD has:

• No current season nest tags before Dec 2019
• 37 nest tags (all PtH) in Dec 2019 https://wastd.dbca.wa.gov.au/admin/observations/nesttagobservation/?encounter__when__month=12&encounter__when__year=2019
• 12 nest tags in Jan 2020 https://wastd.dbca.wa.gov.au/admin/observations/nesttagobservation/?encounter__when__month=1&encounter__when__year=2020
• No nest tags outside PtH
• PtH nest tags defs come from prod server (UAT was one week at the beginning)

There is skip logic both in the ETL script (to speed up incremental imports) and inside WAStD (to guard data against duplicates).

Sanity checks:

• Does the record exist in odkc_ex (extracted), odkc_tf (transformed) - YES, odkc_up (split by skip logic into create, update, skip)? YES
• Server Logs should update https://wastd.dbca.wa.gov.au/admin/observations/turtlenestencounter/160085/change/?_changelist_filters=when__day%3D30%26when__month%3D1%26when__year%3D2020 (TNE 160085):

  [INFO] {'source': 2, 'source_id': 'uuid:1ae8335f-01ad-48a5-81ad-39a605a98310', 'status': 'resighted', 'tag_label': 'JG8', 'date_nest_laid': '2019-12-06', 'encounter_id': 160085}

• 326 nesttag obs supposedly go in, 37+12 nest tag obs are visible through API. Why?
• TurtleNestEncounters with "nest tag present" exist without nesttagobs. Strange. Re-running import now with "update existing unchanged" on.
  • Thv: 24 in 2020 https://wastd.dbca.wa.gov.au/admin/observations/turtlenestencounter/?area__id__exact=17&nest_tagged__exact=present&when__year=2020

The solution to this riddle is the core of QAing https://github.com/dbca-wa/wastdr/milestone/1 and https://github.com/dbca-wa/wastdr/milestone/2

[florianm]

The bug lies with wastd, not wastdr.

wastd.observations.api.ObservationBatchUpsertViewSet got "what to update" wrong (contained "to create" records) and failed silently on not updating non-existing records. Currently fixed by doing a more manual update_or_create instead of the bulk_update, but should really fix the records_to_update list comprehension which requires a good night's sleep and some serious coffee.

[florianm]

Deployed to PROD and data reloaded. https://wastd.dbca.wa.gov.au/admin/observations/encounter/160085/change/?_changelist_filters=when__day%3D30%26when__month%3D1%26when__year%3D2020 for example now shows a TurtleNestTag Observation.