genevieve-anderegg commented 1 year ago

Initial Request

Goal

As discussed in Taxonomy Committee today:

5814#issuecomment-1613183411

Be able to link LSIDs to catalog records (vs a classification) so that database aggregators (OBIS) can directly read the LSID (which is already stored in the classification metadata in the WoRMS via Arctos taxonomy source) instead of relying only on the scientific name, which omits taxon concept data

Context

This value will allow scientific name IDs to be directly linked to catalog records. This will allow OBIS to distinguish between homonyms #4794, which it currently cannot and is currently limiting the publishing of hundreds of DMNS:Inv records to OBIS. Currently the LSID is contained within the JSON in a catalog record, and @dustymc may be able to extract it and present it to the IPT after the next release #6518 , but by manually pushing the LSID to aggregators we can determine how the data are behaving. Additionally, this capability may become increasingly necessary with initiatives such as Ocean Genome Legacy https://github.com/ArctosDB/arctos/issues/4776#issue-1276978446

Table

(forthingcoming in next release) https://arctos-test.tacc.utexas.edu/info/ctDocumentation.cfm?table=ctidentification_attribute_code_tables

Proposed Value

scientificNameID

Proposed Definition

https://dwc.tdwg.org/list/#dwc_scientificNameID An identifier for the nomenclatural (not taxonomic) details of a scientific name.

Collection type

All

Attribute Extras

Attribute data type

Free text

Attribute units

None

Priority

A method for linking LSID (and other scientificnameIDs) to catalog records will likely become more and more necessary. Piloting methods for doing that are of medium importance. Useful!

Example Data

Requests with clarifying sample data are generally much easier to understand and prioritize. Please attach or link to any representative data, in any form or format, which might help clarify the request.

Available for Public View

Yes

Helpful Actions

[x] Add the issue to the Code Table Management Project.
[ ] Please reach out to anyone who might be affected by this change. Leave a comment or add this to the Committee agenda if you believe more focused conversation is necessary.

@ArctosDB/arctos-code-table-administrators

Approval

All of the following must be checked before this may proceed.

_The How-To Document should be followed. Pay particular attention to terminology (with emphasis on consistency) and documentation (with emphasis on functionality). No person should act in multiple roles; the submitter cannot also serve as a Code Table Administrator, for example._

[ ] Code Table Administrator[1] - check and initial, comment, or thumbs-up to indicate that the request complies with the how-to documentation and has your approval
[ ] Code Table Administrator[2] - check and initial, comment, or thumbs-up to indicate that the request complies with the how-to documentation and has your approval
[ ] DBA - The request is functionally acceptable. The term is not a functional duplicate, and is compatible with existing data and code.
[ ] DBA - Appropriate code or handlers are in place as necessary. (ID_References, Media Relationships, Encumbrances, etc. require particular attention)

Rejection

If you believe this request should not proceed, explain why here. Suggest any changes that would make the change acceptable, alternate (usually existing) paths to the same goals, etc.

Can a suitable solution be found here? If not, proceed to (2)
Can a suitable solution be found by Code Table Committee discussion? If not, proceed to (3)
Take the discussion to a monthly Arctos Working Group meeting for final resolution.

Implementation

Once all of the Approval Checklist is appropriately checked and there are no Rejection comments, or in special circumstances by decree of the Arctos Working Group, the change may be made.

[ ] Review everything one last time. Ensure the How-To has been followed. Ensure all checks have been made by appropriate personnel.
[ ] Add or revise the code table term/definition as described above. Ensure the URL of this Issue is included in the definition.

Close this Issue.

DO NOT modify Arctos Authorities in any way before all points in this Issue have been fully addressed; data loss may result.

Special Exemptions

In very specific cases and by prior approval of The Committee, the approval process may be skipped, and implementation requirements may be slightly altered. Please note here if you are proceeding under one of these use cases.

Adding an existing term to additional collection types may proceed immediately and without discussion, but doing so may also subject users to future cleanup efforts. If time allows, please review the term and definition as part of this step.
The Committee may grant special access on particular tables to particular users. This should be exercised with great caution only after several smooth test cases, and generally limited to "taxonomy-like" data such as International Commission on Stratigraphy terminology.

genevieve-anderegg commented 1 year ago

@Jegelewicz @dustymc @sharpphyl Please edit as needed

dustymc commented 1 year ago

Before proceeding (unless there's some other reason to proceed, this request seems fine by itself), see if this can be magicked out of data as a column in DWC. Next release will cache all "classification as preferred by collection for involved names" data with the record, so this will become accessible (but may require more resources than we have to extract - it won't necessarily be flat or predictable, particularly at scale).

Also worth exploring, particularly if this turns out to be nontrivial to extract as part of the all-Arctos DWC packaging:

can/will DWC consumers just get what they want from the included data?
could the collections handle this in some way ( eg download plus open refine script)?
can we somehow sustainably package DWC on a by-collection basis?

EDIT: added classification_id

Example:

https://arctos-test.tacc.utexas.edu/guid/DMNS:Inv:23162

arctosprod@arctos>> select previousidentifications from flat where guid='DMNS:Inv:23162';
[
  {
    "idby": "Candice Johnson",
    "made_date": "2017-03-23",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Potamilus alatus",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Heteroconchia, Palaeoheterodonta, Unionida, Unionoidea, Unionidae, Ambleminae, Lampsilini, Potamilus, Potamilus alatus",
          "name": "Potamilus alatus",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Heteroconchia"
            },
            {
              "psn": 7,
              "typ": "subterclass",
              "term": "Palaeoheterodonta"
            },
            {
              "psn": 8,
              "typ": "order",
              "term": "Unionida"
            },
            {
              "psn": 9,
              "typ": "superfamily",
              "term": "Unionoidea"
            },
            {
              "psn": 10,
              "typ": "family",
              "term": "Unionidae"
            },
            {
              "psn": 11,
              "typ": "subfamily",
              "term": "Ambleminae"
            },
            {
              "psn": 12,
              "typ": "tribe",
              "term": "Lampsilini"
            },
            {
              "psn": 13,
              "typ": "genus",
              "term": "Potamilus"
            },
            {
              "psn": 14,
              "typ": "species",
              "term": "Potamilus alatus"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "857376"
            },
            {
              "typ": "authority",
              "term": "(Say, 1817)"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2022). MolluscaBase. Potamilus alatus (Say, 1817). Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=857376 on 2022-02-22"
            },
            {
              "typ": "display_name",
              "term": "<i>Potamilus alatus</i>"
            },
            {
              "typ": "isBrackish",
              "term": "0"
            },
            {
              "typ": "isFreshwater",
              "term": "1"
            },
            {
              "typ": "isMarine",
              "term": "0"
            },
            {
              "typ": "isTerrestrial",
              "term": "0"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:857376"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2015-10-16T11:46:27.003Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "747094"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientific_name",
              "term": "Potamilus alatus"
            },
            {
              "typ": "scientificname",
              "term": "Potamilus alatus"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=857376"
            },
            {
              "typ": "valid_AphiaID",
              "term": "857376"
            },
            {
              "typ": "valid_authority",
              "term": "(Say, 1817)"
            },
            {
              "typ": "valid_name",
              "term": "Potamilus alatus"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Potamilus alatus#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Potamilus alatus",
        "variable": "A"
      }
    ],
    "identification_order": 1,
    "identification_agents": [
      {
        "agent_name": "Candice Johnson",
        "agent_identifier": "https://arctos.database.museum/agent/21256093",
        "identifier_order": 1
      }
    ],
    "identification_remarks": "Former nature_of_id: student.",
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "features",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  }
]

Jegelewicz commented 1 year ago

I don't think we can expect anyone to find

"typ":"lsid",
                     "term":"urn:lsid:marinespecies.org:taxname:857376"

and associate that with dwc: scientificNameID.

We are going to have to do that and present it directly as

"dwc:scientificNameID":"urn:lsid:marinespecies.org:taxname:857376"

I don't know the most efficient way to do it, but this is what we need to do. I also think that asking every publishing collection to run a script or do some OpenRefine task every month in order to publish to OBIS makes Arctos even more difficult to use. This seems like it deserves a technical solution - not a manual one.

One thing seems like it should be easy. We should be able to pass the "Arctos Identifier" for a given classification in dwc:scientificNameID. If that worked something like https://arctos.database.museum/name/Potamilus%20alatus#WoRMSviaArctos, then at least I could FIND the lsid from the information there. But this also brings to mind something that I have heard over and over again - use other's identifiers. LSID are being used by WoRMS and we should find a way to just USE those to make this stuff easier. If the "Arctos Identifier" for a given WoRMS classification was urn:lsid:marinespecies.org:taxname:857376, everyone would be clear what we are talking about. This would also mean that when multiple LSID exist for a taxon name, we need a way for the collection to pick one - a more direct connection between an identification and a classification than just the Taxonomy Source.

Sorry for rambling - this may all be pie-in-the-sky or complete nonsense, just thought I'd lay it out here for consideration.

dustymc commented 1 year ago

when multiple LSID exist for a taxon name, we need a way for the collection to pick one

That may be the compelling reason to NOT do https://github.com/ArctosDB/arctos/issues/6532#issuecomment-1641119452 (make this my problem) and DO do https://github.com/ArctosDB/arctos/issues/6532#issue-1810842111 (initial request, require collections to specify). Just considering the possibility of trying to extract a single value out of what's going to be possible as identifications (which isn't what you said, I hadn't even considered that possibility) is making my head hurt!

deserves a technical solution

It's the social aspect I'm worried about, but there's a technical bit too. I assume the technical bits would be straightforward (albeit probably resource-hungry) given a sufficiently-robust description of what's needed.

the "Arctos Identifier"

There is no "the" - these data (as I understand them, which might be just wrong) assume a very limited relationship between catalog records and taxon names. Arctos has no such limitations.

something like https://arctos.database.museum/name/Potamilus%20alatus#WoRMSviaArctos

It is possible to make identifications to names with no classifications from preferred sources (eg https://arctos.database.museum/name/Potamilus%20alatus) - I think "intentionally ambiguous" is the only possible interpretation of that.

we should find a way to just USE those

Next release includes them in the cache, not sure how that could be seen as anything but using them!

Jegelewicz commented 1 year ago

What a timely discussion we had this week - see https://github.com/tdwg/dwc-qa/issues/203 and despair.

genevieve-anderegg commented 1 year ago

What a timely discussion we had this week - see https://github.com/tdwg/dwc-qa/issues/203 and despair.

So will we still be able to implement what we discussed in this issue?

dustymc commented 1 year ago

timely discussion

Ugh, Arctos isn't that simple!

implement

Three things:

If I'm to shove the specified ID into some DWC pigeonhole, I think this should probably actually be a record attribute - identification attributes still leave a lot of complexity to wade through. (And I'm still struggling to see how this can be usefully simplified at all so maybe that's just wrong, I'm very open to whatever.)
(1) is pointless if I then shove The Chosen One into the wrong pigeonhole, so I'm not sure how to mesh this with the linked discussion, and
I'd still like some high-level feedback indicating that this is something that we want to support - @mkoo ?? (This is going to use some CPU, add a lot of complexity, maybe add yet another Attribute, and may ultimately just look like Arctos is broken - it'll be a haphazard curatorially-asserted thing that probably won't be consistent and may just not align with the data at all.)

Jegelewicz commented 1 year ago

I think this should probably actually be a record attribute

What happens when there are multiple identifications? a complex ID (A x B or A or B)?

Jegelewicz commented 1 year ago

This isn't some off-topic thing. This is an important part of the DES. Being able to unambiguously point to a taxon name AND it's classification is something we should be supporting.

dustymc commented 1 year ago

What happens when there are multiple identifications? a complex ID (A x B or A or B)?

Exactly!

I'm already including those data in the only container than can carry them. (It can be shaped differently, but it can't be simplified without losing functionality.)

You're asking me to do some impossible task and smoosh 'em all into a single container. (And the linked discussion is asking for information that I can't possibly have - how the heck would I know - or care! - why someone's involved some taxon in some identification!?)

I'm trying - not very successfully - to balance those clearly conflicting goals. The ONLY possible way I can see to do that is to have the user pick some taxon and just roll with it, and record attributes are a convenient place to store the results of that flattening.

important part of the DES

Then why is everyone refusing to deal with the actual data?! I don't think "this is important!" and "must stuff it into a spreadsheet cell" are in any way compatible....

Jegelewicz commented 1 year ago

refusing to deal with the actual data?

We are trying to use the ACTUAL DATA but there is nowhere to put it in Arctos!

dustymc commented 1 year ago

nowhere to put it in Arctos!

That is absolutely not true. https://arctos.database.museum/guid/DMNS:Inv:21694 for example:

arctosprod@arctos>> select previousidentifications from flat where guid='DMNS:Inv:21694';
[
  {
    "idby": "James Shearer",
    "made_date": "2019-10-22",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Ostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Ostrea, Ostrea megodon",
          "name": "Ostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Ostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Ostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "542159"
            },
            {
              "typ": "authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Ostrea megodon Hanley, 1846. Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Ostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:542159"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "138298"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Ostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Ostrea megodon"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Ostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Ostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 1,
    "identification_agents": [
      {
        "agent_name": "James Shearer",
        "agent_identifier": "https://arctos.database.museum/agent/21313362",
        "identifier_order": 1
      }
    ],
    "identification_remarks": null,
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "revised taxonomy",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  },
  {
    "idby": "Peter Bond",
    "made_date": "2016-03-01",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Undulostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Undulostrea, Undulostrea megodon",
          "name": "Undulostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Undulostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Undulostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "506742"
            },
            {
              "typ": "authority",
              "term": "(Hanley, 1846)"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Undulostrea megodon (Hanley, 1846). Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=506742 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Undulostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:506742"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "492149"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Undulostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Undulostrea megodon"
            },
            {
              "typ": "status",
              "term": "unaccepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=506742"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Undulostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Undulostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 0,
    "identification_agents": [
      {
        "agent_name": "Peter Bond",
        "agent_identifier": "https://arctos.database.museum/agent/21263960",
        "identifier_order": 1
      }
    ],
    "identification_remarks": "Former nature_of_id: student.",
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "features",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  },
  {
    "idby": "James Shearer",
    "made_date": "2019-10-22",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Ostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Ostrea, Ostrea megodon",
          "name": "Ostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Ostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Ostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "542159"
            },
            {
              "typ": "authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Ostrea megodon Hanley, 1846. Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Ostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:542159"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "138298"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Ostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Ostrea megodon"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Ostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Ostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 0,
    "identification_agents": [
      {
        "agent_name": "James Shearer",
        "agent_identifier": "https://arctos.database.museum/agent/21313362",
        "identifier_order": 1
      }
    ],
    "identification_remarks": null,
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "revised taxonomy",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  },
  {
    "idby": "James Shearer",
    "made_date": "2019-10-22",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Ostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Ostrea, Ostrea megodon",
          "name": "Ostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Ostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Ostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "542159"
            },
            {
              "typ": "authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Ostrea megodon Hanley, 1846. Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Ostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:542159"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "138298"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Ostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Ostrea megodon"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Ostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Ostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 0,
    "identification_agents": [
      {
        "agent_name": "James Shearer",
        "agent_identifier": "https://arctos.database.museum/agent/21313362",
        "identifier_order": 1
      }
    ],
    "identification_remarks": null,
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "revised taxonomy",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  },
  {
    "idby": "James Shearer",
    "made_date": "2019-10-22",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Ostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Ostrea, Ostrea megodon",
          "name": "Ostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Ostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Ostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "542159"
            },
            {
              "typ": "authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Ostrea megodon Hanley, 1846. Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Ostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:542159"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "138298"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Ostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Ostrea megodon"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Ostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Ostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 0,
    "identification_agents": [
      {
        "agent_name": "James Shearer",
        "agent_identifier": "https://arctos.database.museum/agent/21313362",
        "identifier_order": 1
      }
    ],
    "identification_remarks": null,
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "revised taxonomy",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  },
  {
    "idby": "James Shearer",
    "made_date": "2019-10-22",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Ostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Ostrea, Ostrea megodon",
          "name": "Ostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Ostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Ostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "542159"
            },
            {
              "typ": "authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Ostrea megodon Hanley, 1846. Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Ostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:542159"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "138298"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Ostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Ostrea megodon"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Ostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Ostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 0,
    "identification_agents": [
      {
        "agent_name": "James Shearer",
        "agent_identifier": "https://arctos.database.museum/agent/21313362",
        "identifier_order": 1
      }
    ],
    "identification_remarks": null,
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "revised taxonomy",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  },
  {
    "idby": "James Shearer",
    "made_date": "2019-10-22",
    "concept_label": null,
    "short_citation": null,
    "scientific_name": "Ostrea megodon",
    "sensu_publication": null,
    "identification_taxa": [
      {
        "taxon": {
          "ftn": "Biota, Animalia, Mollusca, Bivalvia, Autobranchia, Pteriomorphia, Ostreida, Ostreoidea, Ostreidae, Ostreinae, Ostrea, Ostrea megodon",
          "name": "Ostrea megodon",
          "ctrms": [
            {
              "psn": 1,
              "typ": "superdomain",
              "term": "Biota"
            },
            {
              "psn": 2,
              "typ": "kingdom",
              "term": "Animalia"
            },
            {
              "psn": 3,
              "typ": "phylum",
              "term": "Mollusca"
            },
            {
              "psn": 4,
              "typ": "class",
              "term": "Bivalvia"
            },
            {
              "psn": 5,
              "typ": "subclass",
              "term": "Autobranchia"
            },
            {
              "psn": 6,
              "typ": "infraclass",
              "term": "Pteriomorphia"
            },
            {
              "psn": 7,
              "typ": "order",
              "term": "Ostreida"
            },
            {
              "psn": 8,
              "typ": "superfamily",
              "term": "Ostreoidea"
            },
            {
              "psn": 9,
              "typ": "family",
              "term": "Ostreidae"
            },
            {
              "psn": 10,
              "typ": "subfamily",
              "term": "Ostreinae"
            },
            {
              "psn": 11,
              "typ": "genus",
              "term": "Ostrea"
            },
            {
              "psn": 12,
              "typ": "species",
              "term": "Ostrea megodon"
            }
          ],
          "nctrms": [
            {
              "typ": "AphiaID",
              "term": "542159"
            },
            {
              "typ": "authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "citation",
              "term": "MolluscaBase eds. (2023). MolluscaBase. Ostrea megodon Hanley, 1846. Accessed through: World Register of Marine Species at: https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159 on 2023-04-28"
            },
            {
              "typ": "display_name",
              "term": "<i>Ostrea megodon</i>"
            },
            {
              "typ": "isMarine",
              "term": "1"
            },
            {
              "typ": "lsid",
              "term": "urn:lsid:marinespecies.org:taxname:542159"
            },
            {
              "typ": "match_type",
              "term": "exact"
            },
            {
              "typ": "modified",
              "term": "2017-09-25T08:59:00.343Z"
            },
            {
              "typ": "parentNameUsageID",
              "term": "138298"
            },
            {
              "typ": "rank",
              "term": "Species"
            },
            {
              "typ": "scientificname",
              "term": "Ostrea megodon"
            },
            {
              "typ": "scientific_name",
              "term": "Ostrea megodon"
            },
            {
              "typ": "status",
              "term": "accepted"
            },
            {
              "typ": "taxonRankID",
              "term": "220"
            },
            {
              "typ": "url",
              "term": "https://www.marinespecies.org/aphia.php?p=taxdetails&id=542159"
            },
            {
              "typ": "valid_AphiaID",
              "term": "542159"
            },
            {
              "typ": "valid_authority",
              "term": "Hanley, 1846"
            },
            {
              "typ": "valid_name",
              "term": "Ostrea megodon"
            }
          ],
          "source": "WoRMS (via Arctos)",
          "classification_id": "https://arctos.database.museum/name/Ostrea megodon#WoRMSviaArctos"
        },
        "taxon_id": "https://arctos.database.museum/name/Ostrea megodon",
        "variable": "A"
      }
    ],
    "identification_order": 0,
    "identification_agents": [
      {
        "agent_name": "James Shearer",
        "agent_identifier": "https://arctos.database.museum/agent/21313362",
        "identifier_order": 1
      }
    ],
    "identification_remarks": null,
    "identification_attributes": [
      {
        "agent_name": null,
        "attribute_type": "nature of identification",
        "attribute_units": null,
        "attribute_value": "revised taxonomy",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      },
      {
        "agent_name": null,
        "attribute_type": "identification confidence",
        "attribute_units": null,
        "attribute_value": "medium",
        "determined_date": null,
        "agent_identifier": null,
        "attribute_remark": null,
        "determination_method": null
      }
    ]
  }
]

Jegelewicz commented 1 year ago

OK - so it is there - how do we get it into dwc:scientificNameID? This is all we are asking for.

dustymc commented 1 year ago

AWG discussion

add scientificNameID to flat
treat this something like genus, kingdom, etc. and cache it with changes
see what it costs, revisit

right?

dustymc commented 1 year ago

I think this is working as proposed at test, but it's near impossible to test at scale at the moment (growing pains...) - I'm relatively sure the cost isn't going to be prohibitive in prod, but that needs confirmed before this is closed.

I updated https://arctos-test.tacc.utexas.edu/guid/DMNS:Inv:11276 (one order>0 ID) and https://arctos-test.tacc.utexas.edu/guid/DMNS:Inv:10808 (>1 order>0 ID), which results in....


arctosprod@arctos>> select scientificNameID from flat where guid='DMNS:Inv:10808';
                                   scientificnameid                                   
--------------------------------------------------------------------------------------
 urn:lsid:marinespecies.org:taxname:137813; urn:lsid:marinespecies.org:taxname:215480
(1 row)

Time: 4.092 ms
arctosprod@arctos>> select scientificNameID from flat where guid='DMNS:Inv:11276';
             scientificnameid              
-------------------------------------------
 urn:lsid:marinespecies.org:taxname:225473
(1 row)

sharpphyl commented 1 year ago

I think this is working as proposed at test

Is the record at test supposed to show the LSID as a separate field or is it all behind the scenes?

And do we need to do anything except check all the homonyms that OBIS excluded and make sure WoRMS (via Arctos) has the correct URL/LSID selected?

I cleaned up https://arctos.database.museum/guid/DMNS:Inv:21694 referenced above. I have no idea how that happened but thanks for pointing it out.

dustymc commented 1 year ago

all behind the scenes?

Sorta? Like all "flattened terms" it'll be a concatenation of select taxa-stuff used in accepted identifications, and you can see that on the guid pages by clicking the eyeballs

need to do anything

It'd be nice to know if I'm doing something useful before I do it in production, but I'm also not sure what I'd do about it if I'm not. (Took us long enough to get here!)

ArctosDB / arctos

Code Table Request - make "scientificNameID" an identification attribute value #6532

Initial Request

Goal

4776 #5814 https://github.com/ArctosDB/arctos/issues/5814#issuecomment-1613183411

Context

Table

Proposed Value

Proposed Definition

Collection type

Attribute Extras

Attribute data type

Attribute units

Priority

Example Data

Available for Public View

Helpful Actions

Approval

Rejection

Implementation

Special Exemptions