Improve semantics on CollectingEvent identifiers with respect to DwC import

[mjy]

Thanks to @debpaul's testing we have some things that we should work to improve on. The basic issues are included in the image below.

@LordFlashmeow, @LocoDelAssembly please chime in after digesting this. In particular we need to re-vist how AntWeb is getting data in-out, as they have most extensively imported identifiers.

I do anticipate a couple things:

• Clarifying the concept of Identifier::Local::TripCode to more closely align with fieldNumber.
• Perhaps supporting namespacing of fieldNumber based imports
• Perhaps using fieldNumber as a ways to collate unique CEs, as we do for eventID (we'd have to prioritize what to use if both present- something like use eventID for uniquness and add multple TripCode identifiers
• Ensure, where intended, that the dwc export aligns, it's crossing concepts right now I think.
• Potentially migrate some existing data to clarify evolved semantics (e.g. change Identify types, etc.)

In general we also need to interpret eventID. To me it should reflect the digital record, i.e. eventID is roughly 1 to 1 with a single TW CE

Thoughts?

[LocoDelAssembly]

Please clarify if fieldIdentifier is actually fieldNumber? ( https://dwc.tdwg.org/list/#dwc_fieldNumber, currently mapped as verbatim_trip_identifier )

eventID is mapped as TripCode identifier and the namespace is either assigned from non-standard TW:Namespace:eventID field, or from the auto-generated per import dataset one (there was a bug with this one https://github.com/SpeciesFileGroup/taxonworks/commit/73b187deb60c1c4568904da90bfe7e40fe4a975b)

At present it is eventID that serves as a way to detect that two events are the same, scoped either to a specific TW:Namespace:eventID, or a import dataset-specific namespace. When a match is found, existing CE is used rather than creating a new one and merge data.

Proposal is new namespace, probably global (although would break stuff), to identify CEs?

[mjy]

Sorry for label confusion, yes fieldNumber

[mjy]

Making a note that TW:Namespace:eventID matches against Namespace#short_name

[mjy]

Duplicate of https://github.com/SpeciesFileGroup/taxonworks/issues/2852.

[mjy]

• [x] map eventID to new identifier type in DwC importer (Identifier::Local::Event)
  • eventID defined as proposed on paper ^
• [x] revisit existing eventID data (largely AntWeb)
• Deprecate Identifier::Local::TripCode for `Identifier::Local::FieldNumber' Migrate all Identifier::Local::TripCode to FieldNumber
• [x] db migrate TripCode to FieldNumber

Add constraints per type:

• [x] FieldNumber to CollectingEvent
• [x] RecordNumber to CollectionObject
• [x] Event to CollectingEvent
• [x] CatalogNumber to CollectionObject
• [x] verbatim_field_number must equal cached if corresponding Identifier::Local::FieldNumber is also provided

Ensure we handle DwC import for

• [ ] Identifier::Local::Event (CollectingEvent uniqueness)
• [ ] Identifier::Local::TripCode (CollectingEvent uniqueness)
• [ ] Identifier::Local::CatalogNumber (CollectionObject)
• [ ] Identifier::Local::RecordNumber (CollectionObject)

Ensure DwcOccurrenceHooks are fired

• [x] Event
• [x] FieldNumber
• [x] RecordNumber
• [x] CatalogNumber

Ensure DwcOccurrence Index is written

• [x] dwc_event_id -> Event
• [x] dwc_field_number -> FieldNumber
• [x] map verbatim_trip_identifier to dwc_field_number
• [x] dwc_record_number -> RecordNumber
• [x] dwc_catalog_number -> CatalogNumber

[mjy]

Behaviours

CollectingEvents

• Uniqueness/re-use

  • [ ] If eventID without TW:Namespace:eventID is provided
  • Then subsequent rows will re-use the same CollectingEvent regardless of differing data
  • [ ] If eventID AND TW:Namespace:eventID are provided
    • Then all rows will be checked against both subsequent rows and existing CEs for matches
    • Matches by ID will ignore values of previously provided data!!!
  • [ ] Same pattern for fieldNumber
  • [ ] If both fieldNumber and eventID provided then
    • [ ] If they match different existing CollectingEvents Raise error
    • [ ] If they are missmatched versus different prior rows, then Raise error

• fieldNumber and verbatim_trip_identifer

  • [ ] If both provided they must match (via identifier.cached)
  • [ ] Do not copy fieldNumber to verbatim_trip_identifier
  • [ ] Do not use verbatim_trip_identifier instantiate a fieldNumber

• eventID

  • [x] Providing eventID without any Namespace will generate a new per-dataset namespace

• fieldNumber

  • [ ] Providing fieldNumber without any Namespace will Raise

• verbatim_trip_identifier

  • Is only populated when explicitly referenced with a header (TW:CollectingEvent:verbatim_trip_identifier)

CollectionObjects

• catalogNumber and recordNumber

  • If both provided and they are inconsistent versus both prior rows OR existing COs then Raise

• catalogNumber

  • [ ] Keep behaviour option to virtualize when more than one dataset row references the same identifier

• recordNumber

  • [ ] If duplicated in rows or existing Records then raise, period. No virtualization.