Tracking changed/deleted records for harvesters

[staplegun]

Trove and other data harvesters need to know when a record previously harvested should be updated or removed, so their data remains in sync with the NMA source.

• changes can be tracked by querying on the date modified field
• entity ripple changes (see #39) need notification
• deletions need notification (possibly following a weekly data reload)

Issue #41 is considering an API status - this might be usable for harvesters too.

[Conal-Tuohy]

AcsStatus = Removed

[SimmoK]

Records 47157 and 230014 have been marked as 'removed' today. Good to check them out next import

[Conal-Tuohy]

http://nma-dev.conaltuohy.com/object?status_code=410

[Conal-Tuohy]

Deployed and should be testable tomorrow. To query for all deleted objects: https://data.nma.gov.au/object?status_code=410 To query for all objects updated since September the 3rd (including deletions): https://data.nma.gov.au/object?datestamp=[2018-09-03+TO+2999] To query for objects deleted since September the 3rd: https://data.nma.gov.au/object?status_code=410&datestamp=[2018-09-03+TO+2999]

[Conal-Tuohy]

Querying for deleted records using https://data.nma.gov.au/object?status_code=410&datestamp=[2018-08-08+TO+2999] yields a response with status 200 OK:

{
  "data": [
    {
      "id": "47157",
      "type": "object",
      "_meta": {
        "modified": "2018-09-03",
        "statusCode": "410",
        "reason": "Gone"
      }
    },
    {
      "id": "230014",
      "type": "object",
      "_meta": {
        "modified": "2018-09-03",
        "statusCode": "410",
        "reason": "Gone"
      }
    }
  ],
  "meta": {
    "results": 2
  }
}

In JSON-LD it's more verbose, including the licence for the metadata that says the object is gone ;-)

{
  "context": "/context.json",
  "id": "object?status_code=410&datestamp=[2018-08-08+TO+2999]",
  "type": "Aggregation",
  "entities": 2,
  "aggregates": [
    {
      "@context": "/context.json",
      "id": "http://data.nma.gov.au/object/47157#",
      "type": "PhysicalObject",
      "identified_by": {
        "id": "http://data.nma.gov.au/object/47157#repositorynumber",
        "type": "Identifier",
        "classified_as": {
          "id": "http://vocab.getty.edu/aat/300404621",
          "type": "Type",
          "label": "repository number"
        },
        "value": "47157"
      },
      "documented_in": {
        "id": "http://data.nma.gov.au/object/47157",
        "type": "http://www.cidoc-crm.org/cidoc-crm/E31_Document",
        "modified": "2018-09-03",
        "subject_to": {
          "id": "http://data.nma.gov.au/term/metadata-rights",
          "type": "Right",
          "component": {
            "id": "https://creativecommons.org/licenses/by-nc/4.0/",
            "type": "Right",
            "label": "CC BY-NC"
          },
          "label": "Copyright National Museum of Australia / CC BY-NC"
        },
        "response": {
          "type": "Response",
          "reason_phrase": "Gone",
          "status_code_value": "410"
        }
      }
    },
    {
      "@context": "/context.json",
      "id": "http://data.nma.gov.au/object/230014#",
      "type": "PhysicalObject",
      "identified_by": {
        "id": "http://data.nma.gov.au/object/230014#repositorynumber",
        "type": "Identifier",
        "classified_as": {
          "id": "http://vocab.getty.edu/aat/300404621",
          "type": "Type",
          "label": "repository number"
        },
        "value": "230014"
      },
      "documented_in": {
        "id": "http://data.nma.gov.au/object/230014",
        "type": "http://www.cidoc-crm.org/cidoc-crm/E31_Document",
        "modified": "2018-09-03",
        "subject_to": {
          "id": "http://data.nma.gov.au/term/metadata-rights",
          "type": "Right",
          "component": {
            "id": "https://creativecommons.org/licenses/by-nc/4.0/",
            "type": "Right",
            "label": "CC BY-NC"
          },
          "label": "Copyright National Museum of Australia / CC BY-NC"
        },
        "response": {
          "type": "Response",
          "reason_phrase": "Gone",
          "status_code_value": "410"
        }
      }
    }
  ]
}

Querying for the individual deleted object http://data.nma.gov.au/object/47157# yields a response with status 410 Gone:

{
  "data": null,
  "errors": [
    {
      "status": "410",
      "title": "Gone"
    }
  ]
}

In JSON-LD:

{
  "context": "/context.json",
  "type": "Request",
  "id": "object/47157",
  "request_uri": "object/47157",
  "response": {
    "type": "Response",
    "status_code_value": 410,
    "reason_phrase": "Gone"
  }
}