Linkage of Parts to Collecting Events

campmlc commented 6 years ago

Per conversation with John Wieczorek, we need to seek funding to implement changes to the event model so that parts = material samples are linked to events = occurrences. This would resolve a deficiency in our current data model and would alleviate the problems Arctos has with serving data to GGBN. Perhaps this can be accomplished with GGBN funding.

dustymc commented 6 years ago

Sorry I missed that call!

If I'm understanding, this should work for multiple Occurrences resulting from multiple Encounters. I don't see how it would work for multiple Occurrences resulting from multiple equally-valid opinions. It absolutely will not work for a single part having been involved in lots of Occurrences (eg, most everything in any cultural collection).

What exactly are we trying to fix?

campmlc commented 6 years ago

In the Arctos model, there is no way to associate a specific material sample with an occurrence. We don't have a way to associate a part in a record that has multiple collecting events with a specific collecting event. We do this manually by adding in accession number and tissue number to Collecting Event remarks and to part attributes. But this is awkward and artificial. There needs to be a way to associate each part = material sample with a specific date and locality of collection. This is related to the inability to track more than one accession within a single cataloged record. When there are multiple collecting events, there can be/must be multiple accessions as the only means of tracking which parts were collected when and where.

dustymc commented 6 years ago

This is mostly a duplicate of https://github.com/ArctosDB/arctos/issues/602.

Specimen events are NOT Occurrences, they're just events that link to specimens. Some of them may eventually be mapped to DWC:Occurrences, but that can't/shouldn't drive our model.

must be multiple accessions as the only means of tracking which parts were collected

That does not sound like a stable (or particularly useful) pathway. You could name the events and use that or something - not elegant, but it is stable.

Option One: We could flip cataloged item and events. That will very likely drastically change how we see events, and I suspect it will increase the workload significantly. Eg, now when you get a georeference back from some external source (or add a corrected Event but want to keep the old or etc.) you just dump it in as another Event, and perhaps eventually make it accepted (or unaccepted) as time allows. Under this, I think you'd have to preemptively make that determination and move all parts, collectors, accns, otherIDs, attributes, etc. to the new event (or duplicate them or something weird). I'm not sure how you'd get at "this event, which is no longer accepted, was once accepted for THOSE parts-and-such" (esp. when there are multiple accepted and unaccepted part-producing events under a specimen).

Cultural items (parts) commonly go through a bunch of events - the one part is manufactured, used, etc. (And maybe so are the subjects of mark-recapture and similar, depending how you want to look at things - I think we often focus on the bit of blood, but the wolf is the real item of interest and was present at all of the events.) I think there would be a split of some sort between "this thing has been through multiple events" and "this thing is comprised of parts which originated at multiple events."

That still seems more or less CORRECT to me, but I don't think it's something we can approach without significant funding, and I'm not sure how usable we can make it even with funding.

Option Two: we could go all @tucotuco on this thing, embrace an actual event-based model, and make EVERYTHING into an event. (ID a specimen? Event. Add a part? Event. Agent relationship? Event.) I think this is the most powerful option - I certainly can't think of anything it's not capable of doing "correctly." AFAIK nothing remotely like this exists; development would require serious resources (I think this is a new-everything approach; eg, we'd want a couple months and/or a good consultant just to explore DB options).

Option Three: We could back up and reexamine how we're using cataloged item. Cataloged items are explicitly arbitrary, so I'm not sure there's anything inherently wrong with cataloging item-at-events rather than individuals. I think that's what every other system does, including some specimens in Arctos (eg, we can't PREVENT this even if there's another pathway - think tissues and bones in different collections), and there is comfort in numbers. There's lots of flexibility in how we present data (eg, a "one of many" cataloged item could pull from it's relatives and look a lot like it does now), although I'm not sure how far we can push that in downloads and DWC and such. This probably still requires some funding, but I think is almost certainly the lowest-impact option. It may also be the best reflection of how the data (field notes and such) are arranged.

I think Option Three may be the most approachable at the moment. It shouldn't require any back-end changes, just UI. The split is clear: "this thing has been through multiple events" is one cataloged items with multiple events, "this thing is comprised of parts which originated at multiple events" is multiple related cataloged items (perhaps with multiple events eg to reflect uncertainty). It's a complete reversal of our current approach - this would "require" (ish) un-merging all of those wolves you merged at MSB (sorry!), but should require no changes for "normal" specimens. The most concerning aspect is probably the chance of bits of one animal being treated as independent samples, but perhaps there's something we can do to clarify that (REQUIRE relationships in downloads is probably a good first step).

I'm sure there are more options - I'm not trying to constraint this, those are just all I can think of at the moment.

https://github.com/ArctosDB/arctos/issues/1357 should probably go on the back burner until there's some commitment to something, so I'm flagging this critical.

Jegelewicz commented 6 years ago

I agree that option 3 is the best at this time and I like the idea of "same as" relationships as a required part of downloads. Does this information get translated properly at the GBIF/iDigBio aggregator level?

AJLinn commented 6 years ago

As long as the changes don't impact how we're tracking the events associated with the cultural objects, I'll defer to the affected collections. It's vitally important that we are able to continue to clearly document that cultural objects are sometimes made/used/collected in 3 different places and times (or time ranges) or sometimes it's all the same; the object has a life (like a biological creature) and is involved in historical events sometimes as a result. It is that life we document thru these multiple events.

campmlc commented 6 years ago

I'm not happy with option three,and I suspect that Jon and Joe and the mammal folks at MSB who have worked so long to make the current system work will not be either. The biggest problem I see is in citations. We would go back to citing different parts of the same individuals by different catalog numbers depending on what part was loaned out. It is bad enough trying to track usage now. We need attribution for two reasons: 1) to give credit to collections for specimen use, and this would be unaffected and 2) to track all the usage and pubs back to a single individual, e.g. a wolf in an endangered species recovery program. How would option 3 be dealt with not only by Arctos but by all the other aggregators and external databases? How do we get attribution back to an individual?

On Thu, Jun 14, 2018 at 2:10 PM, Teresa Mayfield notifications@github.com wrote:

I agree that option 3 is the best at this time and I like the idea of "same as" relationships as a required part of downloads. Does this information get translated properly at the GBIF/iDigBio aggregator level?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-397423065, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hNUD4hEDNgiYhYoz4gVY0N0PZcXeks5t8sM8gaJpZM4UT5xn .

KyndallH commented 6 years ago

I also do not like option 3.

What if we did option 2ish if it means "event" as in adding a date. Pretty much when you add the occurrence, it is tied to a date. What if we added a date of collection onto parts? For most, it would simply be the date of death which is super simple in our current system. Yet, when you add to a record you can add the occurrence that has a date and it will be tied to the part via the date on the part. This would be nice too for loans - subsample a tissue, date recorded (and ideally who did it/modified that part of the record - but that may just be a personal preference). Also, subsampled this insect on this date but the DNA extraction I'm adding wasn't done until this date. This may also help the cultural collections too. The object made on this date and location. Object modified on this date and in X location.

All occurrence tied to the catalog number. Different occurrences tied together by date within the catalog number.

Would this help solve the initial problem? In GGBN, you could have Cat. No. with date. So UAM:XXX 05-05-2018, UAM:XXX 02-02-2017, etc.

campmlc commented 6 years ago

This works for Arctos, and is basically what we've been doing most recently, adding in date and Other ID and accession as part attributes for each part. I think this is what Dusty means by a "soft" linkage.

The problem is the aggregators, or more specifically, GGBN. GBIF can deal with our current system just fine. They can't figure this out.

Also, doing it part by part is a form of denormalization. I like Teresa's idea of linking the event ID to the part somehow.

On Thu, Jun 14, 2018 at 3:16 PM, Kyndall Hildebrandt < notifications@github.com> wrote:

I also do not like option 3.

What if we did option 2ish if it means "event" as in adding a date. Pretty much when you add the occurrence, it is tied to a date. What if we added a date of collection onto parts? For most, it would simply be the date of death which is super simple in our current system. Yet, when you add to a record you can add the occurrence that has a date and it will be tied to the part via the date on the part. This would be nice too for loans - subsample a tissue, date recorded (and ideally who did it/modified that part of the record - but that may just be a personal preference). Also, subsampled this insect on this date but the DNA extraction I'm adding wasn't done until this date. This may also help the cultural collections too. The object made on this date and location. Object modified on this date and in X location.

All occurrence tied to the catalog number. Different occurrences tied together by date within the catalog number.

Would this help solve the initial problem? In GGBN, you could have Cat. No. with date. So UAM:XXX 05-05-2018, UAM:XXX 02-02-2017, etc.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-397441117, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hM3MD1-aeRHdX3xhW9zIhnHNSYW6ks5t8tLCgaJpZM4UT5xn .

campmlc commented 6 years ago

Regardless of what happens with GGBN, I like Kyndall's idea for our own use.

On Thu, Jun 14, 2018 at 3:25 PM, Mariel Campbell campbell@carachupa.org wrote:

This works for Arctos, and is basically what we've been doing most recently, adding in date and Other ID and accession as part attributes for each part. I think this is what Dusty means by a "soft" linkage.

The problem is the aggregators, or more specifically, GGBN. GBIF can deal with our current system just fine. They can't figure this out.

Also, doing it part by part is a form of denormalization. I like Teresa's idea of linking the event ID to the part somehow.

On Thu, Jun 14, 2018 at 3:16 PM, Kyndall Hildebrandt < notifications@github.com> wrote:

I also do not like option 3.

What if we did option 2ish if it means "event" as in adding a date. Pretty much when you add the occurrence, it is tied to a date. What if we added a date of collection onto parts? For most, it would simply be the date of death which is super simple in our current system. Yet, when you add to a record you can add the occurrence that has a date and it will be tied to the part via the date on the part. This would be nice too for loans

subsample a tissue, date recorded (and ideally who did it/modified that part of the record - but that may just be a personal preference). Also, subsampled this insect on this date but the DNA extraction I'm adding wasn't done until this date. This may also help the cultural collections too. The object made on this date and location. Object modified on this date and in X location.

All occurrence tied to the catalog number. Different occurrences tied together by date within the catalog number.

Would this help solve the initial problem? In GGBN, you could have Cat. No. with date. So UAM:XXX 05-05-2018, UAM:XXX 02-02-2017, etc.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-397441117, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hM3MD1-aeRHdX3xhW9zIhnHNSYW6ks5t8tLCgaJpZM4UT5xn .

dustymc commented 6 years ago

I can't say I'm thrilled with it either, but I don't see a better approach within our reach.

What others do is easy: They ignore it, or maybe put something in some remarks field.

Nothing much should change at aggregators - we provide them Occurrences now, and we would with any other model that does what we need to do. They would be able to link those Occurrences to specific parts/attributes/etc. in any model we might end up in, and can't now.

The citation guidance is "cite the object of scientific interest." You don't really have one here - some folks are looking at the wolf, some are looking at the wolf when it was THERE, THEN. I think that's a wash.

We'll always have to deal with this, I think - someone looks at a skull and cites https://arctos.database.museum/guid/DMNS:Mamm:12344 instead of http://arctos.database.museum/guid/MSB:Mamm:233616. What I'm proposing is basically making this...

... look more like this....

eg, treat those two (or 50 or whatever) records more like part of the same thing (scientific viewpoint) rather than as distinct things that share some stuff (an administrative viewpoint).

date of collection onto parts?

The issue includes parts, attributes, otherIDs, collectors, media, encumbrances, and probably some other stuff. That's a lot of digital duct tape - I'd rather find a structurally-defensible solution.

I also don't see how we'd maintain referential integrity there. Event date=X, use that to link all that other junk, oops, actually the event date was Y - now what? Yes it's a "soft" linkage - it's not enforced (eg, there are not shared keys, just strings) - you can break it with a typo or by changing something or etc. (Think same data in MSB and DGR collections.)

dustymc commented 6 years ago

subsample a tissue, date recorded (and ideally who did it/modified that part of the record - but that may just be a personal preference). Also, subsampled this insect on this date but the DNA extraction I'm adding wasn't done until this date.

That's always been in the model, but it's not very exposed. Changes in the last round of GGBN brought it up a bit - you can explicitly create "subsamples" now. Who and when are pulled from user environment - that's easy enough to change, if ya'll are willing to provide those data when you create/modify parts.

UAM@ARCTOS> desc specimen_part
 Name                                  Null?    Type
 ----------------------------------------------------------------- -------- --------------------------------------------
 COLLECTION_OBJECT_ID                          NOT NULL NUMBER
 PART_NAME                             NOT NULL VARCHAR2(255)
 SAMPLED_FROM_OBJ_ID                                NUMBER
 DERIVED_FROM_CAT_ITEM                         NOT NULL NUMBER

UAM@ARCTOS> desc coll_object
 Name                                  Null?    Type
 ----------------------------------------------------------------- -------- --------------------------------------------
 COLLECTION_OBJECT_ID                          NOT NULL NUMBER
 COLL_OBJECT_TYPE                          NOT NULL CHAR(2)
 ENTERED_PERSON_ID                         NOT NULL NUMBER
 COLL_OBJECT_ENTERED_DATE                      NOT NULL DATE
 LAST_EDITED_PERSON_ID                              NUMBER
 LAST_EDIT_DATE                                 DATE
 COLL_OBJ_DISPOSITION                          NOT NULL VARCHAR2(20)
 LOT_COUNT                             NOT NULL NUMBER
 CONDITION                             NOT NULL VARCHAR2(4000)
 FLAGS                                      VARCHAR2(20)

UAM@ARCTOS>

Extractions and bits you lop off into new parts and such should have a SAMPLED_FROM_OBJ_ID pointing to the part from which they were removed. All parts (which are collection objects) have entered and edited metadata.

dustymc commented 6 years ago

The co-cataloged thing actually present another problem - we need to distinguish between "Occurrences" created by admin decisions (eg, where to catalog stuff) and those created by repeated sampling. Date is probably close enough most of the time, but I think we should be explicit via a new relationship. "Same individual as" should be split into two terms, one which means "same critter, same event" and one which means "same critter, distinct event."

dustymc commented 6 years ago

There is a VERY quick-n-dirty demo of Option Three in test - http://arctos-test.tacc.utexas.edu/SpecimenDetail_MultiOccurrence.cfm?collection_object_id=12

i_am_tester should have access to all of the collections involved. You can create more of these in the normal way, but you'll need the collection_object_id (from flat, or I can help with that) to see them in the demo page.

I created a new relationship "occurrence of" (that can change, I just think it needs to be more explicit than "same individual as") and related some (unrelated) records to each other through it

At the top of the (potential replacement) specimendetail page, Arctos checks for related occurrences - if it finds any it tries to pull data (from filtered_flat - this can't expose potentially-restricted data without explicit AWG approval), provides a link if it can't do that, and just displays the relationship/otherID data if it can't do that. (If we go here, maybe we should only allow occurrence of to link to things with resolvable identifiers - or not, zoo animals probably have bits cataloged in Arctos and in things that we can't talk to).

Formatting and details are obviously very preliminary, this is just a demonstration testing if we can create a useful representation of individuals from cataloged Occurrences. The data could be displayed differently in the "occurrence area" or mixed in with data from the cataloged item you're on, or WHATEVER. The only point of this is to make a big-picture decision regarding repeated sampling of individuals.

campmlc commented 6 years ago

I guess I don't understand this - I don't see what has changed on the display?

On Thu, Jun 14, 2018 at 3:35 PM, dustymc notifications@github.com wrote:

I can't say I'm thrilled with it either, but I don't see a better approach within our reach.

What others do is easy: They ignore it, or maybe put something in some remarks field.

Nothing much should change at aggregators - we provide them Occurrences now, and we would with any other model that does what we need to do. They would be able to link those Occurrences to specific parts/attributes/etc. in any model we might end up in, and can't now.

The citation guidance is "cite the object of scientific interest." You don't really have one here - some folks are looking at the wolf, some are looking at the wolf when it was THERE, THEN. I think that's a wash.

We'll always have to deal with this, I think - someone looks at a skull and cites https://arctos.database.museum/guid/DMNS:Mamm:12344 instead of http://arctos.database.museum/guid/MSB:Mamm:233616. What I'm proposing is basically making this...

[image: screen shot 2018-06-14 at 2 22 33 pm] https://user-images.githubusercontent.com/5720791/41439129-66f96642-6fde-11e8-9676-5b9574ac8781.png

... look more like this....

[image: screen shot 2018-06-14 at 2 22 53 pm] https://user-images.githubusercontent.com/5720791/41439148-795c5330-6fde-11e8-8d35-3457cf60b4fa.png

eg, treat those two (or 50 or whatever) records more like part of the same thing (scientific viewpoint) rather than as distinct things that share some stuff (an administrative viewpoint).

date of collection onto parts?

The issue includes parts, attributes, otherIDs, collectors, media, encumbrances, and probably some other stuff. That's a lot of digital duct tape - I'd rather find a structurally-defensible solution.

I also don't see how we'd maintain referential integrity there. Event date=X, use that to link all that other junk, oops, actually the event date was Y - now what? Yes it's a "soft" linkage - it's not enforced (eg, there are not shared keys, just strings) - you can break it with a typo or by changing something or etc. (Think same data in MSB and DGR collections.)

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-397445898, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hCZ02QJMn2iYUeobCWqtGxlRZS1Mks5t8tcFgaJpZM4UT5xn .

campmlc commented 6 years ago

I agree with distinguishing the occurrence types. Perhaps having each of these related occurrence tables as a pop-up that only shows if the link is clicked? How would this affect GBBN?

On Fri, Jun 15, 2018 at 1:12 PM, dustymc notifications@github.com wrote:

There is a VERY quick-n-dirty demo of Option Three in test - http://arctos-test.tacc.utexas.edu/SpecimenDetail_MultiOccurrence.cfm? collection_object_id=12

i_am_tester should have access to all of the collections involved. You can create more of these in the normal way, but you'll need the collection_object_id (from flat, or I can help with that) to see them in the demo page.

I created a new relationship "occurrence of" (that can change, I just think it needs to be more explicit than "same individual as") and related some (unrelated) records to each other through it

At the top of the (potential replacement) specimendetail page, Arctos checks for related occurrences - if it finds any it tries to pull data (from filtered_flat - this can't expose potentially-restricted data without explicit AWG approval), provides a link if it can't do that, and just displays the relationship/otherID data if it can't do that. (If we go here, maybe we should only allow occurrence of to link to things with resolvable identifiers - or not, zoo animals probably have bits cataloged in Arctos and in things that we can't talk to).

Formatting and details are obviously very preliminary, this is just a demonstration testing if we can create a useful representation of individuals from cataloged Occurrences. The data could be displayed differently in the "occurrence area" or mixed in with data from the cataloged item you're on, or WHATEVER. The only point of this is to make a big-picture decision regarding repeated sampling of individuals.

[image: screen shot 2018-06-15 at 11 51 29 am] https://user-images.githubusercontent.com/5720791/41485116-116430de-7094-11e8-82e1-55ff1b02e542.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-397716622, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hB3S642NBzEVqoJ8VTRTGhUfYZxcks5t9AcqgaJpZM4UT5xn .

dustymc commented 6 years ago

what has changed on the display

The new table - screenshot in https://github.com/ArctosDB/arctos/issues/1545#issuecomment-397716622 - vs. just the current red "there's another specimen that's sorta the same as this one" thing.

pop-up that only shows if the link is clicked

That's basically what the current links do. The new table thing gets the data on the same page. A popup is super-easy (eg, just stuff the related specimendetail page in a popup).

GBBN

It's basically what we're providing them now - a bit simpler to query perhaps. I don't think it would help us line up with GBIF (not-GGBN) OccurrenceIDs since we'd still have to have the "part" component for (only) GGBN.

ccicero commented 6 years ago

I, like others, don't really like option 3. That seems like it's taking us backwards. Folks have spent a lot of time recataloging and organizing so that every part of a specimen has a single catalog number. We don't want to go back to recataloging them separately, and then down the line with more funding move to an event-based model which is really what we need (and what we've been discussing for at least 10+ years). We should think about how to fund option #2 without sacrificing what we have.

I'm not really sure what I'm looking at in the test example, but it's confusing and I don't think that the general user will understand this.

We have two distinct challenges here: (1) Internal - how to make Arctos work for these kinds of data - linking parts, attributes, identifiers, etc. to a collecting event. (2) external - publishing occurrence data to aggregators including GGBN. For #1, we should work on what will be the most robust solution which seems to be pointing to an event based model that will require more funding. For #2, what if we exported the multiple occurrence data from one cataloged item as one occurrence record, with localities and dates concatenated in a way that's similar to how we concatenate parts and attributes? It won't be a clear 1:1 relationship, but anyone who comes across such a record and wants more information should be encouraged to contact the original source. for clarification.

The number of records in Arctos with multiple occurrences that represent actual multiple accepted events is probably relatively small? I'm curious what % of records have multiple accepted events across all collections.

dustymc commented 6 years ago

taking us backwards

Or we've been taking us away from the "correct" model. Nobody has ever done anything quite like this. It's not really surprising that we had to throw some real-world data at it to see a problem.

lot of time recataloging and organizing so that every part of a specimen has a single catalog number.

2 angles:

They've failed (and will continue to do so as long as administrators are involved) - https://arctos.database.museum/guid/DMNS:Mamm:12344 vs http://arctos.database.museum/guid/MSB:Mamm:233616. We WILL have "same specimen" data with multiple catalog numbers. If we have to deal with it anyway, maybe we can use that to do something clever here.
"Specimen"="cataloged item" (http://handbook.arctosdb.org/documentation/catalog.html) and "cataloged item"="whatever someone felt like cataloging, preferably the item of scientific interest." I can see nothing "wrong" about cataloging Occurrences - that's what everyone else does (from necessity, but still...), and they are the "item of scientific interest" from some perspectives/for some users (like GBIF).

The only "sacrifice" I see involves citations - given two records and identifiers, researchers WILL do weird things (including crappy science). MAYBE we can do more with relationships or something, but this is a real problem "in the wild." It's not much of a problem locally - we can force them to see the related data (demo above), but we can't force GBIF to give them that perspective or not let them delete that big messy column full of links, whatever it was, from their downloads.

In any case, I won't intentionally do anything that loses data; nothing will prevent us from funding a better model. (And I'm actually not so sure how it would fix this - ya'll are still going to want to catalog things, and you'll still pick either the wolf or the occurrence of the wolf to catalog, and here we'll be again. An event-based model might not be constrained by structure, but that won't necessarily help with "tradition.")

Under the current model, you find a wolf (with a single Occurrence/Event), cite it, then someone dumps 20 more events in and what was a good citation when you made it is now ambiguous - I'm not convinced that we're doing such a great job with citations now.

but it's confusing and I don't think that the general user will understand this.

Everything else they've ever seen, including Arctos samples in GBIF, treat every "Occurrence" as a separate THING. If we're trying to reduce confusion, joining the herd probably makes some sense. (But see above...)

Anything that works for Arctos will work for DWC. The problem is that these data are being provided - entered and stored - in a nonrecoverable format.

concatenate parts and attributes

Those are things we make up - GBIF can't tell a concatenated part from our data (neither can I!), and DWC apparently never anticipated the possibility of multiple parts. GBIF (and I) can absolutely tell a list of states from a state, and DWC dictates that the item of scientific interest is the Occurrence, not individual. I think this would be more confusing, not less.

We do have a "JSON Locality" option - I'm not sure how GBIF et al. would respond to that though.

anyone who comes across such a record and wants more information should be encouraged to contact the original source

We don't share most of our data via DWC - this is and always will be true for everything. (And since this is mostly about citations, has ANYONE successfully traced a GBIF citation back to an Arctos specimen? I can't. I'm not sure that what we catalog is our most pressing citation-related problem!)

number of records in Arctos with multiple occurrences

Around 50000.

that represent actual multiple accepted events

Who knows - this is why we need "occurrence of" and "same individual as" (whatever we call them). "Dozens, maybe hundreds" probably.

And FWIW that's still doesn't leave us with Occurrences - https://arctos.database.museum/guid/DMNS:Mamm:12344 and http://arctos.database.museum/guid/MSB:Mamm:233616 are one Occurrence, http://arctos.database.museum/guid/MVZ:Egg:10972 is at least two, etc. "Occurrences" are something we map to when we can, not really something we natively have.

THANKS!! - this is very useful.

I still don't much like Option One - I think it would have serious usability issues.

I do like Option Two, but it's a major project and I'm not sure it solves anything all by itself. What WOULD you catalog in a pure event-based model anyway?

If anyone has an Option Four, this would be a really great time to throw it out there!

If we do go with Option Three, perhaps we can do more with https://arctos.database.museum/info/ctDocumentation.cfm?table=CTCATALOGED_ITEM_TYPE. The existing data are useless - we have "observation" which basically means "something that didn't get cataloged in the main collection" but contains things that would be cataloged in SOME real collections, and "specimen" which we've explicitly defined as "something someone felt like cataloging."

https://arctos.database.museum/info/ctDocumentation.cfm?table=CTCOLL_OBJECT_TYPE also exists, although I'm not really sure what any of that stuff is or what it's supposed to do - I just use CI (cataloged item) and SP (specimen part) because it's there, not because it does anything for me.

dustymc commented 6 years ago

Also re:

Folks have spent a lot of time recataloging and organizing

I can magic some/most of that, if we do end up unraveling it. I'm looking at http://arctos.database.museum/guid/MSB:Mamm:292063. The dates in part remarks as an unambiguous format would be REALLY helpful - I can probably get from "Coll: 15 June 2014" to "Event Date: 2014-06-15" but it's also somewhat likely to be messy.

Those accession numbers can probably be resolved to transactions.

comma-space-NK-space-integer seems to lead to an otherID.

The data in part attributes is better, but seems very inconsistent (eg, doesn't exist for most parts). In general, I think I'd rather deal with messy data in one place instead of less-messy data in a bunch of places - I'm not sure this is useful unless it's consistently applied.

If I can turn strings into data objects, I should be able to handle the conversion. If I can't do that, it's because the data don't support it - eg, it's evidence that this model is insufficient.

I don't really see this component as much of a problem, or at least not as a fatal problem.

Having a half-dozen primary identifiers (catalog numbers) which all mean "that wolf" looks to me like the biggest problem with that approach. (It's also what we have in GBIF now, assuming folks generally cite Occurrences and not IndividualID, and it's significantly less confusing than what we're giving to GGNB.)

To be clear, I'm not really advocating anything at this point, I'm just trying to understand the possibilities and what they mean for everything else.

campmlc commented 6 years ago

The data are inconsistently applied because there is a historical component.

1) First we put any of the linking data between parts and events in part remarks, for lack of an alternative. The NK (OtherID assigned to the tissues for each event), collection date and accession went in with the usual problems that occur with anything in remarks. There was some attempt to standardize the format at the time, but we were dealing with open text and human error.

2) Next we implemented using part attributes. This was better, but there are still records that use both part remarks and part attributes because of the temporal nature of adding events. There never was a way to deal with linking multiple accessions.

3) Most of the initial records that were dealt with this way are approx 800 Mexican wolves in a project with USFWS endangered species recovery, and we have been working on a manuscript to describe this process. In addition, we had a museum studies graduate student spend 2 years converting individually cataloged records to single catalog records with multiple events. She only got halfway through the process, so now there are records in all stages. Note that what Dusty is proposing undoes all of her effort and also contradicts what we've been saying we are doing in public presentations and meetings. At least the manuscript hasn't been published, yet.

4) Next, we received 40,000 tissues representing about 11,000 animals from NEON over the last year. All of these were converted, at great human effort and cost, from an occurrence based database (single row, single tissue, no obvious connection to an original animal other than a (sometimes mistranscribed) field ID. Each part was given part attributes for collecting date and/or accession, and the date of collection is also included in the container/part label and as part of the OtherID. Attributes are tied to events by determined date. See below for the level of complexity involved. This method was standardly applied, and could theoretically be undone.

5) Note also that by converting from an occurrence based system, we identified many flaws and errors in this mark/recapture data, such as the individual being called one species for one capture event and then another species on the following, or one sex on one event and another sex on the following etc. It is this reason, the ability to link all derivative data together in a single record to compare data over time and examine for patterns and errors, as well as having all citations in one place, that make this method scientifically valuable.

5) Consider how much time and effort went into implementing this system over the past 5 years, and the amount of frustration and anger generated by the prospect of undoing and thereby negating all of that effort and expense that was invested in we thought was an innovative and scientifically valid solution.

On Tue, Jun 19, 2018 at 10:20 AM, dustymc notifications@github.com wrote:

Also re:

Folks have spent a lot of time recataloging and organizing

I can magic some/most of that, if we do end up unraveling it. I'm looking at http://arctos.database.museum/guid/MSB:Mamm:292063. The dates in part remarks as an unambiguous format would be REALLY helpful - I can probably get from "Coll: 15 June 2014" to "Event Date: 2014-06-15" but it's also somewhat likely to be messy.

Those accession numbers can probably be resolved to transactions.

comma-space-NK-space-integer seems to lead to an otherID.

The data in part attributes is better, but seems very inconsistent (eg, doesn't exist for most parts). In general, I think I'd rather deal with messy data in one place instead of less-messy data in a bunch of places - I'm not sure this is useful unless it's consistently applied.

If I can turn strings into data objects, I should be able to handle the conversion. If I can't do that, it's because the data don't support it - eg, it's evidence that this model is insufficient.

I don't really see this component as much of a problem, or at least not as a fatal problem.

Having a half-dozen primary identifiers (catalog numbers) which all mean "that wolf" looks to me like the biggest problem with that approach. (It's also what we have in GBIF now, assuming folks generally cite Occurrences and not IndividualID, and it's significantly less confusing than what we're giving to GGNB.)

To be clear, I'm not really advocating anything at this point, I'm just trying to understand the possibilities and what they mean for everything else.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398459459, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hFUuWh-9XDPET6t4xe11FII0T1vrks5t-STngaJpZM4UT5xn .

campmlc commented 6 years ago

Search» https://arctos.database.museum/SpecimenSearch.cfm
Enter Data» https://arctos.database.museum/guid/MSB:Mamm:299278#
Manage Data» https://arctos.database.museum/guid/MSB:Mamm:299278#
Manage Arctos» https://arctos.database.museum/guid/MSB:Mamm:299278#
Reports/Services» https://arctos.database.museum/guid/MSB:Mamm:299278#
Portals https://arctos.database.museum/home.cfm
My Stuff» https://arctos.database.museum/myArctos.cfm
About/Help» http://arctosdb.org/

MSB:Mamm:299278 NK: Peromyscus maniculatus or Peromyscus leucopus get a DOI https://arctos.database.museum/tools/doi.cfm?collection_object_id=27157834 Harvard Forest North America, United States, Massachusetts, Worcester County 07/21/14 feces (frozen); feces (frozen); feces (frozen); feces (frozen); hair; feces (frozen); hair; hair; hair; feces (frozen) Report Bad Data [0][0] MSB Mammals http://www.msb.unm.edu/mammals/index.html

Identification
Accn
Locality
Agents
Parts
Part Locn.
Attributes
Other IDs
Media
Encumbrances

IdentificationsEdit Peromyscus maniculatus https://arctos.database.museum/name/Peromyscus%20maniculatus or Peromyscus leucopus https://arctos.database.museum/name/Peromyscus%20leucopus Animalia; Chordata; Mammalia; Rodentia; Cricetidae; Neotominae; Peromyscus maniculatus, Peromyscus leucopus Ratón norteamericano; deer mouse; souris sylvestre; souris á pattes blanches; white-footed mouse Identified by National Ecological Observatory Network on 2013-09-30 Nature of ID: field Location (6 Events) [ expand ]Edit [image: BerkeleyMapper]BerkeleyMapper https://arctos.database.museum/bnhmMaps/bnhmMapData.cfm?collection_object_id=27157834 Determination Type: encounter assigned by National Ecological Observatory Network on 2013-09-30 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: first capture Collecting Source: wild caught Event Date: 2013-09-30 Verbatim Date: 9/30/13 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Verbatim Coordinates: 42.461612/-72.22859394 Datum: World Geodetic System 1984 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2013-10-02 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2013-10-02 Verbatim Date: 10/2/13 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-06-18 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-06-18 Verbatim Date: 06/18/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-08-16 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-08-16 Verbatim Date: 08/16/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-09-19 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-09-19 Verbatim Date: 09/19/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-07-21 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-07-21 Verbatim Date: 07/21/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Collector(s)Edit National Ecological Observatory Network https://arctos.database.museum/agent.cfm?agent_name=National%20Ecological%20Observatory%20Network Identifiers [ expand ]Edit NEON sample ID: HARV.20140618.R0216.H NEON sample ID: HARV.20140816.R0216.H NEON sample ID: HARV.20140919.R0216.H NEON sample ID: HARV.20130930.R0216.H NEON sample ID: HARV.20140721.R0216.F NEON sample ID: HARV.20140618.R0216.F NEON sample ID: HARV.20130930.R0216.F NEON sample ID: HARV.20131002.R0216.F NEON sample ID: HARV.20140919.R0216.F NEON sample ID: HARV.20140816.R0216.F NEON: National Ecological Observatory Network: NEON.MAM.D01.R0216 Parts [ expand ]Edit Part NameConditionDispositionQtyLabelBarcodePLPathLoanRemarks

feces (frozen) unknown in collection 1 HARV.20140618.R0216.F A6BPI https://arctos.database.museum/findContainer.cfm?barcode=A6BPI Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←1 FECAL D01 DGR18176←82←HARV.20140618.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140919.R0216.F A5LA2 https://arctos.database.museum/findContainer.cfm?barcode=A5LA2 Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←16 FECES D01 DGR18188←39←HARV.20140919.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140721.R0216.F A5KTK https://arctos.database.museum/findContainer.cfm?barcode=A5KTK Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←6 FECAL D01 DGR18180←8←HARV.20140721.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140816.R0216.F A5L3V https://arctos.database.museum/findContainer.cfm?barcode=A5L3V Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←13 FECAL D01 DGR18184←63←HARV.20140816.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20130930.R0216.F A5LF9 https://arctos.database.museum/findContainer.cfm?barcode=A5LF9 Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 3←NEON-3 DGR18139←2013 Fecal D01 Box 2←67←HARV.20130930.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20131002.R0216.F A5LOX https://arctos.database.museum/findContainer.cfm?barcode=A5LOX Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 3←NEON-3 DGR18139←2013 FECAL D01 BOX 3 DGR10053←6←HARV.20131002.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20130930.R0216.H A4TTO https://arctos.database.museum/findContainer.cfm?barcode=A4TTO HARV.20130930.R0216.H HARV.20130930.R0216.H, Accession 2017.059

hair unknown in collection 1 HARV.20140919.R0216.H A4TYW https://arctos.database.museum/findContainer.cfm?barcode=A4TYW HARV.20140919.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20140816.R0216.H A4U3T https://arctos.database.museum/findContainer.cfm?barcode=A4U3T HARV.20140816.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20140618.R0216.H A4U6P https://arctos.database.museum/findContainer.cfm?barcode=A4U6P HARV.20140618.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm Attributes [ expand ]Edit sex: female National Ecological Observatory Network, 2014-06-18 sex: female National Ecological Observatory Network, 2014-09-19 sex: female National Ecological Observatory Network, 2014-08-16 sex: female National Ecological Observatory Network, 2014-07-21 sex: female National Ecological Observatory Network, 2013-09-30 sex: female National Ecological Observatory Network, 2013-10-02 Std. Meas. total length tail length hind foot efn weight 80 mm 21 mm 16 mm 37 g age class: adult National Ecological Observatory Network, 2013-09-30 age class: adult National Ecological Observatory Network, 2013-10-02 age class: adult National Ecological Observatory Network, 2014-06-18 age class: adult National Ecological Observatory Network, 2014-07-21 age class: adult National Ecological Observatory Network, 2014-08-16 age class: adult National Ecological Observatory Network, 2014-09-19 reproductive data: pregnant National Ecological Observatory Network, 2014-09-19 Edit Entered By: Adrienne L. Raniszewski on 2017-08-29 Last Edited By: UAM on 2018-05-24 AccessionEdit Edit 2017.059.Mamm https://arctos.database.museum/editAccn.cfm?Action=edit&transaction_id=21114891 or View 2017.059.Mamm https://arctos.database.museum/viewAccn.cfm?transaction_id=21114891 Showing Media results 1 - 2 of 2 [ [ view details ] https://arctos.database.museum/MediaSearch.cfm?action=search&specimen_accn_id=27157834 Media linked to this specimen's Accession.

[image: Receipt letter and sample number summary; Mammal specimens 2017.059.Mamm] https://arctos.database.museum/media/10570050?open text (application/pdf) Media Details https://arctos.database.museum/media/10570050 Receipt letter and sample number summary [image: Shipping manifest; Mammal specimens 2017.059.Mamm] https://arctos.database.museum/media/10570049?open text (text/html) Media Details https://arctos.database.museum/media/10570049 Shipping manifest

Usage Contributed By Project: National Ecological Observatory Network (NEON) MSB Mammal BioArchive https://arctos.database.museum/ProjectDetail.cfm?src=proj&project_id=10002483

On Tue, Jun 19, 2018 at 10:53 AM, Mariel Campbell campbell@carachupa.org wrote:

The data are inconsistently applied because there is a historical component.

1) First we put any of the linking data between parts and events in part remarks, for lack of an alternative. The NK (OtherID assigned to the tissues for each event), collection date and accession went in with the usual problems that occur with anything in remarks. There was some attempt to standardize the format at the time, but we were dealing with open text and human error.

2) Next we implemented using part attributes. This was better, but there are still records that use both part remarks and part attributes because of the temporal nature of adding events. There never was a way to deal with linking multiple accessions.

3) Most of the initial records that were dealt with this way are approx 800 Mexican wolves in a project with USFWS endangered species recovery, and we have been working on a manuscript to describe this process. In addition, we had a museum studies graduate student spend 2 years converting individually cataloged records to single catalog records with multiple events. She only got halfway through the process, so now there are records in all stages. Note that what Dusty is proposing undoes all of her effort and also contradicts what we've been saying we are doing in public presentations and meetings. At least the manuscript hasn't been published, yet.

4) Next, we received 40,000 tissues representing about 11,000 animals from NEON over the last year. All of these were converted, at great human effort and cost, from an occurrence based database (single row, single tissue, no obvious connection to an original animal other than a (sometimes mistranscribed) field ID. Each part was given part attributes for collecting date and/or accession, and the date of collection is also included in the container/part label and as part of the OtherID. Attributes are tied to events by determined date. See below for the level of complexity involved. This method was standardly applied, and could theoretically be undone.

5) Note also that by converting from an occurrence based system, we identified many flaws and errors in this mark/recapture data, such as the individual being called one species for one capture event and then another species on the following, or one sex on one event and another sex on the following etc. It is this reason, the ability to link all derivative data together in a single record to compare data over time and examine for patterns and errors, as well as having all citations in one place, that make this method scientifically valuable.

5) Consider how much time and effort went into implementing this system over the past 5 years, and the amount of frustration and anger generated by the prospect of undoing and thereby negating all of that effort and expense that was invested in we thought was an innovative and scientifically valid solution.

On Tue, Jun 19, 2018 at 10:20 AM, dustymc notifications@github.com wrote:

Also re:

Folks have spent a lot of time recataloging and organizing

I can magic some/most of that, if we do end up unraveling it. I'm looking at http://arctos.database.museum/guid/MSB:Mamm:292063. The dates in part remarks as an unambiguous format would be REALLY helpful - I can probably get from "Coll: 15 June 2014" to "Event Date: 2014-06-15" but it's also somewhat likely to be messy.

Those accession numbers can probably be resolved to transactions.

comma-space-NK-space-integer seems to lead to an otherID.

The data in part attributes is better, but seems very inconsistent (eg, doesn't exist for most parts). In general, I think I'd rather deal with messy data in one place instead of less-messy data in a bunch of places - I'm not sure this is useful unless it's consistently applied.

If I can turn strings into data objects, I should be able to handle the conversion. If I can't do that, it's because the data don't support it - eg, it's evidence that this model is insufficient.

I don't really see this component as much of a problem, or at least not as a fatal problem.

Having a half-dozen primary identifiers (catalog numbers) which all mean "that wolf" looks to me like the biggest problem with that approach. (It's also what we have in GBIF now, assuming folks generally cite Occurrences and not IndividualID, and it's significantly less confusing than what we're giving to GGNB.)

To be clear, I'm not really advocating anything at this point, I'm just trying to understand the possibilities and what they mean for everything else.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398459459, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hFUuWh-9XDPET6t4xe11FII0T1vrks5t-STngaJpZM4UT5xn .

campmlc commented 6 years ago

Sorry, folks, for the large image, but in order to share the logged in view, I couldn't just give a link. Here is the link to that specimen, as an example of what we are dealing with: https://arctos.database.museum/guid/MSB:Mamm:299278

And here is a screenshot of the rest of the record, showing OtherIDs, parts, part attributes, and attributes for the multiple events:

Collector(s)Edit National Ecological Observatory Network https://arctos.database.museum/agent.cfm?agent_name=National%20Ecological%20Observatory%20Network Identifiers [ expand ]Edit NEON sample ID: HARV.20140618.R0216.H NEON sample ID: HARV.20140816.R0216.H NEON sample ID: HARV.20140919.R0216.H NEON sample ID: HARV.20130930.R0216.H NEON sample ID: HARV.20140721.R0216.F NEON sample ID: HARV.20140618.R0216.F NEON sample ID: HARV.20130930.R0216.F NEON sample ID: HARV.20131002.R0216.F NEON sample ID: HARV.20140919.R0216.F NEON sample ID: HARV.20140816.R0216.F NEON: National Ecological Observatory Network: NEON.MAM.D01.R0216 Parts [ expand ]Edit Part NameConditionDispositionQtyLabelBarcodePLPathLoanRemarks

feces (frozen) unknown in collection 1 HARV.20140618.R0216.F A6BPI https://arctos.database.museum/findContainer.cfm?barcode=A6BPI Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←1 FECAL D01 DGR18176←82←HARV.20140618.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140919.R0216.F A5LA2 https://arctos.database.museum/findContainer.cfm?barcode=A5LA2 Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←16 FECES D01 DGR18188←39←HARV.20140919.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140721.R0216.F A5KTK https://arctos.database.museum/findContainer.cfm?barcode=A5KTK Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←6 FECAL D01 DGR18180←8←HARV.20140721.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140816.R0216.F A5L3V https://arctos.database.museum/findContainer.cfm?barcode=A5L3V Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←13 FECAL D01 DGR18184←63←HARV.20140816.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20130930.R0216.F A5LF9 https://arctos.database.museum/findContainer.cfm?barcode=A5LF9 Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 3←NEON-3 DGR18139←2013 Fecal D01 Box 2←67←HARV.20130930.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20131002.R0216.F A5LOX https://arctos.database.museum/findContainer.cfm?barcode=A5LOX Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 3←NEON-3 DGR18139←2013 FECAL D01 BOX 3 DGR10053←6←HARV.20131002.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20130930.R0216.H A4TTO https://arctos.database.museum/findContainer.cfm?barcode=A4TTO HARV.20130930.R0216.H HARV.20130930.R0216.H, Accession 2017.059

hair unknown in collection 1 HARV.20140919.R0216.H A4TYW https://arctos.database.museum/findContainer.cfm?barcode=A4TYW HARV.20140919.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20140816.R0216.H A4U3T https://arctos.database.museum/findContainer.cfm?barcode=A4U3T HARV.20140816.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20140618.R0216.H A4U6P https://arctos.database.museum/findContainer.cfm?barcode=A4U6P HARV.20140618.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm Attributes [ expand ]Edit sex: female National Ecological Observatory Network, 2014-06-18 sex: female National Ecological Observatory Network, 2014-09-19 sex: female National Ecological Observatory Network, 2014-08-16 sex: female National Ecological Observatory Network, 2014-07-21 sex: female National Ecological Observatory Network, 2013-09-30 sex: female National Ecological Observatory Network, 2013-10-02 Std. Meas. total length tail length hind foot efn weight 80 mm 21 mm 16 mm 37 g age class: adult National Ecological Observatory Network, 2013-09-30 age class: adult National Ecological Observatory Network, 2013-10-02 age class: adult National Ecological Observatory Network, 2014-06-18

On Tue, Jun 19, 2018 at 10:58 AM, Mariel Campbell campbell@carachupa.org wrote:

Search» https://arctos.database.museum/SpecimenSearch.cfm

Enter Data» https://arctos.database.museum/guid/MSB:Mamm:299278#

Manage Data» https://arctos.database.museum/guid/MSB:Mamm:299278#

Manage Arctos» https://arctos.database.museum/guid/MSB:Mamm:299278#

Reports/Services» https://arctos.database.museum/guid/MSB:Mamm:299278#

Portals https://arctos.database.museum/home.cfm

My Stuff» https://arctos.database.museum/myArctos.cfm

About/Help» http://arctosdb.org/

MSB:Mamm:299278 NK: Peromyscus maniculatus or Peromyscus leucopus get a DOI https://arctos.database.museum/tools/doi.cfm?collection_object_id=27157834 Harvard Forest North America, United States, Massachusetts, Worcester County 07/21/14 feces (frozen); feces (frozen); feces (frozen); feces (frozen); hair; feces (frozen); hair; hair; hair; feces (frozen) Report Bad Data [0][0] MSB Mammals http://www.msb.unm.edu/mammals/index.html

Identification

Accn

Locality

Agents

Parts

Part Locn.

Attributes

Other IDs

Media

Encumbrances

IdentificationsEdit Peromyscus maniculatus https://arctos.database.museum/name/Peromyscus%20maniculatus or Peromyscus leucopus https://arctos.database.museum/name/Peromyscus%20leucopus Animalia; Chordata; Mammalia; Rodentia; Cricetidae; Neotominae; Peromyscus maniculatus, Peromyscus leucopus Ratón norteamericano; deer mouse; souris sylvestre; souris á pattes blanches; white-footed mouse Identified by National Ecological Observatory Network on 2013-09-30 Nature of ID: field Location (6 Events) [ expand ]Edit [image: BerkeleyMapper]BerkeleyMapper https://arctos.database.museum/bnhmMaps/bnhmMapData.cfm?collection_object_id=27157834 Determination Type: encounter assigned by National Ecological Observatory Network on 2013-09-30 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: first capture Collecting Source: wild caught Event Date: 2013-09-30 Verbatim Date: 9/30/13 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Verbatim Coordinates: 42.461612/-72.22859394 Datum: World Geodetic System 1984 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2013-10-02 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2013-10-02 Verbatim Date: 10/2/13 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-06-18 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-06-18 Verbatim Date: 06/18/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-08-16 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-08-16 Verbatim Date: 08/16/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-09-19 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-09-19 Verbatim Date: 09/19/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Determination Type: encounter assigned by National Ecological Observatory Network on 2014-07-21 Higher Geography: North America, United States, Massachusetts, Worcester County http://en.wikipedia.org/wiki/Worcester_County,_Massachusetts more https://arctos.database.museum/geography.cfm?geog_auth_rec_id=1002612 Verbatim Locality: HARV_032 Locality Nickname: NEON Domain 01, plot HARV_032 Specific Locality: Harvard Forest Specimen/Event Remarks: Accession 2017.059 Collecting Method: sample-release: recapture Collecting Source: wild caught Event Date: 2014-07-21 Verbatim Date: 07/21/14 Verification Status: unverified ⚠ Define Coordinates: 42.461612 / -72.22859394 Error: 64 m Georeference Source: not available Georeference Protocol: not recorded

https://maps.google.com/maps?ll=42.365818,-71.89623&z=7&t=m&hl=en-US&gl=US&mapclient=apiv3 Map Data Terms of Use https://www.google.com/intl/en-US_US/help/terms_maps.html Map Satellite 50 km map key/tools Collector(s)Edit National Ecological Observatory Network https://arctos.database.museum/agent.cfm?agent_name=National%20Ecological%20Observatory%20Network Identifiers [ expand ]Edit NEON sample ID: HARV.20140618.R0216.H NEON sample ID: HARV.20140816.R0216.H NEON sample ID: HARV.20140919.R0216.H NEON sample ID: HARV.20130930.R0216.H NEON sample ID: HARV.20140721.R0216.F NEON sample ID: HARV.20140618.R0216.F NEON sample ID: HARV.20130930.R0216.F NEON sample ID: HARV.20131002.R0216.F NEON sample ID: HARV.20140919.R0216.F NEON sample ID: HARV.20140816.R0216.F NEON: National Ecological Observatory Network: NEON.MAM.D01.R0216 Parts [ expand ]Edit Part NameConditionDispositionQtyLabelBarcodePLPathLoanRemarks

feces (frozen) unknown in collection 1 HARV.20140618.R0216.F A6BPI https://arctos.database.museum/findContainer.cfm?barcode=A6BPI Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←1 FECAL D01 DGR18176←82←HARV.20140618.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140919.R0216.F A5LA2 https://arctos.database.museum/findContainer.cfm?barcode=A5LA2 Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←16 FECES D01 DGR18188←39←HARV.20140919.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140721.R0216.F A5KTK https://arctos.database.museum/findContainer.cfm?barcode=A5KTK Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←6 FECAL D01 DGR18180←8←HARV.20140721.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20140816.R0216.F A5L3V https://arctos.database.museum/findContainer.cfm?barcode=A5L3V Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 2←NEON-2 DGR18138←13 FECAL D01 DGR18184←63←HARV.20140816.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20130930.R0216.F A5LF9 https://arctos.database.museum/findContainer.cfm?barcode=A5LF9 Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 3←NEON-3 DGR18139←2013 Fecal D01 Box 2←67←HARV.20130930.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm feces (frozen) unknown in collection 1 HARV.20131002.R0216.F A5LOX https://arctos.database.museum/findContainer.cfm?barcode=A5LOX Museum of Southwestern Biology←Genomic Resources←326 Freezer Room←DGR-13←Rack 3←NEON-3 DGR18139←2013 FECAL D01 BOX 3 DGR10053←6←HARV.20131002.R0216.F

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20130930.R0216.H A4TTO https://arctos.database.museum/findContainer.cfm?barcode=A4TTO HARV.20130930.R0216.H HARV.20130930.R0216.H, Accession 2017.059

hair unknown in collection 1 HARV.20140919.R0216.H A4TYW https://arctos.database.museum/findContainer.cfm?barcode=A4TYW HARV.20140919.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20140816.R0216.H A4U3T https://arctos.database.museum/findContainer.cfm?barcode=A4U3T HARV.20140816.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm hair unknown in collection 1 HARV.20140618.R0216.H A4U6P https://arctos.database.museum/findContainer.cfm?barcode=A4U6P HARV.20140618.R0216.H

Attribute https://arctos.database.museum/guid/MSB:Mamm:299278#Value https://arctos.database.museum/guid/MSB:Mamm:299278#Date https://arctos.database.museum/guid/MSB:Mamm:299278#Dtr. https://arctos.database.museum/guid/MSB:Mamm:299278#Rmk. https://arctos.database.museum/guid/MSB:Mamm:299278# Accession 2017.059.Mamm Attributes [ expand ]Edit sex: female National Ecological Observatory Network, 2014-06-18 sex: female National Ecological Observatory Network, 2014-09-19 sex: female National Ecological Observatory Network, 2014-08-16 sex: female National Ecological Observatory Network, 2014-07-21 sex: female National Ecological Observatory Network, 2013-09-30 sex: female National Ecological Observatory Network, 2013-10-02 Std. Meas. total length tail length hind foot efn weight 80 mm 21 mm 16 mm 37 g age class: adult National Ecological Observatory Network, 2013-09-30 age class: adult National Ecological Observatory Network, 2013-10-02 age class: adult National Ecological Observatory Network, 2014-06-18 age class: adult National Ecological Observatory Network, 2014-07-21 age class: adult National Ecological Observatory Network, 2014-08-16 age class: adult National Ecological Observatory Network, 2014-09-19 reproductive data: pregnant National Ecological Observatory Network, 2014-09-19 Edit Entered By: Adrienne L. Raniszewski on 2017-08-29 Last Edited By: UAM on 2018-05-24 AccessionEdit Edit 2017.059.Mamm https://arctos.database.museum/editAccn.cfm?Action=edit&transaction_id=21114891 or View 2017.059.Mamm https://arctos.database.museum/viewAccn.cfm?transaction_id=21114891 Showing Media results 1 - 2 of 2 [ [ view details ] https://arctos.database.museum/MediaSearch.cfm?action=search&specimen_accn_id=27157834 Media linked to this specimen's Accession.

[image: Receipt letter and sample number summary; Mammal specimens 2017.059.Mamm] https://arctos.database.museum/media/10570050?open text (application/pdf) Media Details https://arctos.database.museum/media/10570050 Receipt letter and sample number summary [image: Shipping manifest; Mammal specimens 2017.059.Mamm] https://arctos.database.museum/media/10570049?open text (text/html) Media Details https://arctos.database.museum/media/10570049 Shipping manifest

Usage Contributed By Project: National Ecological Observatory Network (NEON) MSB Mammal BioArchive https://arctos.database.museum/ProjectDetail.cfm?src=proj&project_id=10002483

On Tue, Jun 19, 2018 at 10:53 AM, Mariel Campbell campbell@carachupa.org wrote:

The data are inconsistently applied because there is a historical component.

1) First we put any of the linking data between parts and events in part remarks, for lack of an alternative. The NK (OtherID assigned to the tissues for each event), collection date and accession went in with the usual problems that occur with anything in remarks. There was some attempt to standardize the format at the time, but we were dealing with open text and human error.

2) Next we implemented using part attributes. This was better, but there are still records that use both part remarks and part attributes because of the temporal nature of adding events. There never was a way to deal with linking multiple accessions.

3) Most of the initial records that were dealt with this way are approx 800 Mexican wolves in a project with USFWS endangered species recovery, and we have been working on a manuscript to describe this process. In addition, we had a museum studies graduate student spend 2 years converting individually cataloged records to single catalog records with multiple events. She only got halfway through the process, so now there are records in all stages. Note that what Dusty is proposing undoes all of her effort and also contradicts what we've been saying we are doing in public presentations and meetings. At least the manuscript hasn't been published, yet.

4) Next, we received 40,000 tissues representing about 11,000 animals from NEON over the last year. All of these were converted, at great human effort and cost, from an occurrence based database (single row, single tissue, no obvious connection to an original animal other than a (sometimes mistranscribed) field ID. Each part was given part attributes for collecting date and/or accession, and the date of collection is also included in the container/part label and as part of the OtherID. Attributes are tied to events by determined date. See below for the level of complexity involved. This method was standardly applied, and could theoretically be undone.

5) Note also that by converting from an occurrence based system, we identified many flaws and errors in this mark/recapture data, such as the individual being called one species for one capture event and then another species on the following, or one sex on one event and another sex on the following etc. It is this reason, the ability to link all derivative data together in a single record to compare data over time and examine for patterns and errors, as well as having all citations in one place, that make this method scientifically valuable.

5) Consider how much time and effort went into implementing this system over the past 5 years, and the amount of frustration and anger generated by the prospect of undoing and thereby negating all of that effort and expense that was invested in we thought was an innovative and scientifically valid solution.

On Tue, Jun 19, 2018 at 10:20 AM, dustymc notifications@github.com wrote:

Also re:

Folks have spent a lot of time recataloging and organizing

I can magic some/most of that, if we do end up unraveling it. I'm looking at http://arctos.database.museum/guid/MSB:Mamm:292063. The dates in part remarks as an unambiguous format would be REALLY helpful - I can probably get from "Coll: 15 June 2014" to "Event Date: 2014-06-15" but it's also somewhat likely to be messy.

Those accession numbers can probably be resolved to transactions.

comma-space-NK-space-integer seems to lead to an otherID.

The data in part attributes is better, but seems very inconsistent (eg, doesn't exist for most parts). In general, I think I'd rather deal with messy data in one place instead of less-messy data in a bunch of places - I'm not sure this is useful unless it's consistently applied.

If I can turn strings into data objects, I should be able to handle the conversion. If I can't do that, it's because the data don't support it - eg, it's evidence that this model is insufficient.

I don't really see this component as much of a problem, or at least not as a fatal problem.

Having a half-dozen primary identifiers (catalog numbers) which all mean "that wolf" looks to me like the biggest problem with that approach. (It's also what we have in GBIF now, assuming folks generally cite Occurrences and not IndividualID, and it's significantly less confusing than what we're giving to GGNB.)

To be clear, I'm not really advocating anything at this point, I'm just trying to understand the possibilities and what they mean for everything else.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398459459, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hFUuWh-9XDPET6t4xe11FII0T1vrks5t-STngaJpZM4UT5xn .

dustymc commented 6 years ago

Again, I'm not advocating for anything, I'm just trying to respond to your needs as I see them.

Big-picture, there are three possibilities:

Do nothing-ish. What we have now sorta works, and it works just fine for most of what's in Arctos. I don't really see how you could avoid using strings as keys, which as you say comes with transcription errors and such. I don't see this as a defensible long-term solution, but that doesn't mean we HAVE to stop (yet, anyway).
Do something with our current resources - eg, we KNOW Option Three "works" because it's what we've done in the past (and what we'd continue to do with the "first cataloged Occurrence" and etc.), we just need to figure out if SOMEHOW pulling related "records" together - by creating relationships, displaying things differently, creating new kinds of data objects (eg, new classes of cataloged items and/or collection objects) to carry that information, or something of the sort - is a useful approach.
Do something big. Event-based models might do something useful, but as above I'm not sure it's a silver bullet either.

I don't think what I'm "proposing" (such a strong word for this stage!) undoes anything - if the data are clean it's just a more stable representation of them, and if they're not they're not. Eg,

This method was standardly applied, and could theoretically be undone.

is true - IF those data can be pulled apart into "Occurrences" then what I'm "proposing" just adds value (eg, explicit links between parts and place-time and identifications and etc. for individuals) If the standards were less standard than hoped, then the data aren't very accessible now and won't be very accessible in any other model.

We came up with a hypothesis and I think ya'll are telling me it's becoming unsatisfactory as we throw enough data at it for details to emerge - science!

the ability to link all derivative data together in a single ~~record~~[VIEW] to compare data over time and examine for patterns and errors

"A record" is an arbitrary thing in Arctos (and any other deeply-relational structure). There are many thousands of "records" in Arctos that have multiple IDs. Those cover

something's funky - we're not so good at identifying or transcribing or interpreting our marks or something
second opinion - looks like A to you and B to me.
taxonomic weirdness - "let's all call this ~~Myodes~~ ~~Clethrionomys~~ ~~Myodes~~ ~~Clethrionomys~~ ~~Myodes~~ ~~Clethrionomys~~ Myodes."
Probably some other stuff.

I sort of think explicitly separating the "something's funky" case (eg, linking the IDs to two data objects - cataloged items perhaps) is a useful approach - I think it's probably easier to bring those together than it is to try to separate out two "views" from one "record."

dustymc commented 6 years ago

Here's a screenshot of https://arctos.database.museum/guid/MSB:Mamm:299278

msb-mamm-299278-peromyscus maniculatus or peromyscus leucopus 2018-06-19 11-11-02

dustymc commented 6 years ago

Also:

https://arctos.database.museum/info/ctDocumentation.cfm?table=CTTAXA_FORMULA

An [I'll go fix this] determination to either of two taxa. The word "or" is inserted between the two values.

I don't think anything's WRONG with using that here, but it's DIFFERENT. The concept was created for "I can do a lot better than Genus, but I'm not sure which of these two species this individual is. I'm absolutely certain I'm dealing with one individual." This (I think) adds another component - eg, was it IDed as maniculatus because those look a lot like leucopus (eg, there's some uncertainty in the identification), or because it was really a different animal and something else went wrong (eg, there's some uncertainty in marking, reading those marks, or transcribing those data)?

campmlc commented 6 years ago

Yes, either of those two interpretations or both are possible. This is the limitation with mark/recapture data and the reason for the published arguments against this approach and for voucher based collections in NEON. In the meantime, this is what we have.

On Tue, Jun 19, 2018, 12:24 PM dustymc notifications@github.com wrote:

Also:

https://arctos.database.museum/info/ctDocumentation.cfm?table=CTTAXA_FORMULA

An [I'll go fix this] determination to either of two taxa. The word "or" is inserted between the two values.

I don't think anything's WRONG with using that here, but it's DIFFERENT. The concept was created for "I can do a lot better than Genus, but I'm not sure which of these two species this individual is. I'm absolutely certain I'm dealing with one individual." This (I think) adds another component - eg, was it IDed as maniculatus because those look a lot like leucopus (eg, there's some uncertainty in the identification), or because it was really a different animal and something else went wrong (eg, there's some uncertainty in marking, reading those marks, or transcribing those data)?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398497904, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hGkld5V5s3LA_z4-Cx1PUzXxjmEhks5t-UHNgaJpZM4UT5xn .

dustymc commented 6 years ago

this is what we have.

But it's not what came in, correct? If I'm understanding, you got

ID#--ID 123--P. manic 123--P. leucopus

which could mean they're not very good with identifications, or not very good at keeping the series unique (eg, there are multiple "123" tags out there), or someone was aiming for "223" but hit the wrong key, or someone read the tag wrong, or ... - there are a LOT of reasons that lead here, and I'd only get to one of them from the "A or B" ID you've applied.

Did they have 300 P. manic IDs and one leucopus, or were those more evenly distributed, or what? I don't see any of that in your data.

As two (or 500 or whatever) "records" you'd faithfully record what you got and it would be up to a researcher to sort it out - you could get at distributions of IDs and such.

campmlc commented 6 years ago

Yes, good question. It is certainly not obvious from the Arctos record. I will get the folks who worked on this involved to compare to the original data. In our current system we should at least add multiple identifications with determiners and dates if that is the case to distinguish from field IDs that were ambiguous. Whatever model we end up with should be able to track the differences between these scenarios. Is that what an event-based model would enable?

On Tue, Jun 19, 2018 at 1:17 PM, dustymc notifications@github.com wrote:

this is what we have.

But it's not what came in, correct? If I'm understanding, you got

ID#--ID 123--P. manic 123--P. leucopus

which could mean they're not very good with identifications, or not very good at keeping the series unique (eg, there are multiple "123" tags out there), or someone was aiming for "223" but hit the wrong key, or someone read the tag wrong, or ... - there are a LOT of reasons that lead here, and I'd only get to one of them from the "A or B" ID you've applied.

Did they have 300 P. manic IDs and one leucopus, or were those more evenly distributed, or what? I don't see any of that in your data.

As two (or 500 or whatever) "records" you'd faithfully record what you got and it would be up to a researcher to sort it out - you could get at distributions of IDs and such.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398513736, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hMTm6T8NLy3H_P6xAvIvqHczr-0yks5t-U5JgaJpZM4UT5xn .

dustymc commented 6 years ago

multiple identifications with determiners and dates

Better, I think, but you still have to pick an "accepted" in our model. (Because we've established and enforce that rule, not because here's anything structurally-special about the "accepted" ID - eg, we could treat them more like we do localities and let 20 be "we can't prove these are wrong" - and then, also like localities, we'd have to figure out how to stuff that complex data into a simple spreadsheet/table cell.)

Is that what an event-based model would enable?

Not inherently that I can see - it'd be about the same as we have now (except IDs would be events instead of in a special 'container'). It'd still be up to ya'll to decide if we "require" an accepted ID or not, and how to visualize the data in a tabular format if we allow zero or 600 IDs as "not unaccepted" and all that jazz.

Jegelewicz commented 6 years ago

The only "sacrifice" I see involves citations - given two records and identifiers, researchers WILL do weird things (including crappy science). MAYBE we can do more with relationships or something, but this is a real problem "in the wild." It's not much of a problem locally - we can force them to see the related data (demo above), but we can't force GBIF to give them that perspective or not let them delete that big messy column full of links, whatever it was, from their downloads.

Yes, this problem is much larger than Arctos! Historically, from the perspective of any individual collection, we were cataloging a thing, generally an individual animal or evidence of one. But things are complicated and parts of that animal may now be in several different institutions. Then we add the complication of parts from a single individual separately cataloged multiple times over its life. It doesn't matter if we decide to catalog the individual animal or to separately catalog each part of it collected over time when we are thinking about one collection (and right now, all of Arctos is one collection in my mind) because we can create relationships between the separately cataloged items to show they belong to a single individual. The real problem is when cataloged items from one individual reside in multiple disconnected collections or data get disrupted upon aggregation and that is something that the whole community needs to work through.

Right now neither the GBIF nor the GGBN models take into account the two different ways that institutions might decide to catalog multiple occurrences of one individual, be it parts scattered between institutions or parts of the same individual collected over time at one institution (never mind that parts collected over time from one individual may also be scattered over multiple institutions!) Ideally, we could give an individual some kind of identifier separate from the catalog number so that any part or occurrence related to the individual would be linked or at least resolvable. Of course that means the entire community has to get on board....

I view managing a biological collection as cataloging a thing or things collected by someone at a place and time and that each catalog number should reflect that relationship. If some of those cataloged things are related, we should record and publish that. I don't see catalog numbers as identifiers for an individual animal, they are meant to to keep track of objects and to keep those objects connected with their data. I think we should treat individual animals as something separate. And perhaps we also need a way to flag cataloged items as being potentially related so that when it's possible but not completely obvious we can at least raise a question for anyone who might use the data. Relationships in Arctos are great but perhaps as soon as "same individual as" is invoked, a new identifier should be required. Then, any other cataloged item linked to one with an existing individual ID by this method would get that identifier added as well. Could we use such an identifier to create a "single individual" record that displays all collections related to that identifier?

We might create a solution in Arctos, but the rest of the biological data community needs to be in agreement or else we are going to continue to have these issues at the aggregator level.

dustymc commented 6 years ago

aggregator

They don't care (well they probably do, but they can't do anything about it!), they just provide a transport mechanism for Occurrences and assume that's what we'll feed it. We don't have the resources to transform http://arctos.database.museum/guid/MVZ:Egg:10972 (and all the other weird stuff) into proper Occurrences, so we do SOMETHING. That's generally treating it as one Occurrence, but maybe it's more-correct/less-evil to withhold those data or something.

The real problem is when cataloged items from one individual reside in multiple disconnected collections or data get disrupted upon aggregation and that is something that the whole community needs to work through.

There's absolutely nothing you can do without a CMS that can both form and receive links. I know of one of those....

(never mind that parts collected over time from one individual may also be scattered over multiple institutions!)

Yea, one of those wolves MUST have a hunk cataloged somewhere outside Arctos and it would be fun to play with. I can't find it though.

some kind of identifier separate from the catalog number

That's easy enough to do, but doing something with that ID (eg, keeping it unique so it can be resolvable) is likely anything but.

I don't see catalog numbers as identifiers for an individual animal, they are meant to to keep track of objects and to keep those objects connected with their data

I think, to my great surprise, that I'm coming around to that viewpoint as well. Also of note: the word "individual" gets REALLY weird in some collections. We can probably keep ignoring that for a while....

treat individual animals as something separate.

That's my Option Three, where "something" is "cataloged items with relationships."

flag cataloged items as being potentially related

Just add something to https://arctos.database.museum/info/ctDocumentation.cfm?table=CTID_REFERENCES. That might be part of what we're trying to do with "collected with" but ???

new identifier should be required

I don't think we can or should do that. Say you add a million "whatever of" relationships pointing to my records. If I'm in Arctos I'll get an email, and reciprocating is just a couple clicks away. I have the option of not reciprocating - maybe I don't believe you, or don't want to link to your data because I see mine as "primary" (eg, many "host" specimens contain unreciprocated links for similar reasons), or WHATEVER - I should have that choice. (I think, sometimes....) If I'm not in Arctos, I can't do anything about it anyway. (It would be very easy to extend our reciprocity mechanism outside Arctos - it was built with that in mind - there's just nothing out there that can talk back.)

perhaps as soon as "same individual as" is invoked, a new identifier should be required. Then, any other cataloged item linked to one with an existing individual ID by this method would get that identifier added as well.

You lost me, and I think I lost myself along a similar pathway earlier. I think you're wanting some sort of "meta-cataloged-item" - maybe not all that different than "same lot as" from https://github.com/ArctosDB/arctos/issues/1574.

Could we use such an identifier to create a "single individual" record that displays all collections related to that identifier?

That's what my demo does, but from all/any of the records. I think that may also be more defensible scientifically.

From the "everything everywhere" viewpoint you see both specifics (the catalog number cited in publications) and the full picture (the other 20 samples of the same critter). I think this is most useful, however we get you there and whatever it looks like.
From the "meta-cataloged-item" viewpoint you'd presumably be instructed to cite the "meta-catnum" but that page would contain a bunch of material and data that you didn't use - seems confusing/non-replicable.
From the current perspective the (single) cataloged item may have a bunch of data and material that it didn't have when you did your research, and again I can't tell what you looked at.

We can use the relationship itself to form that "individual" picture - that's "just" an interface problem. (It's also a problem shared by lots of other things - prey is "part of the predator," parasites are "part of the host" etc. for some use cases. It's generally easy to go from many to one viewpoints - to merge - and generally not so easy to go the other way, the problem that spawned this Issue.)

going to continue to have these issues [outside Arctos]

Easy: bring that stuff in to Arctos. We are looking for solutions to problems that most collection don't know they have, and don't have the tools to detect. That turns out to be non-trivial in a system with highly normalized data and powerful linking mechanisms. I don't think it's remotely possible in anything else.

tucotuco commented 6 years ago

Small excerpt for a point of information...

Right now neither the GBIF nor the GGBN models take into account the two

different ways that institutions might decide to catalog multiple occurrences of one individual, be it parts scattered between institutions or parts of the same individual collected over time at one institution (never mind that parts collected over time from one individual may also be scattered over multiple institutions!) Ideally, we could give an individual some kind of identifier separate from the catalog number so that any part or occurrence related to the individual would be linked or at least resolvable. Of course that means the entire community has to get on board....

Darwin Core, and therefore GBIF and GGBN, support the identification of an individual, and that could be used to relate records. The field is called organismID (http://rs.tdwg.org/dwc/terms/index.htm#organismID). It is worth looking at the Darwin Core definition of an Organism ( http://rs.tdwg.org/dwc/terms/index.htm#Organism) to be clear on what that ID can refer to.

dustymc commented 6 years ago

https://terms.tdwg.org/wiki/dwc:individualID also exists, and we populate it with GUID. I'm not sure how that's meant to differ from Organism - @tucotuco ???

If we assign GUIDs to Occurrences, we would (do now...) need to find SOMETHING to stuff in whatever "the critter" field we should be using. I suppose we could try to force users to create something (which might have reasonable success within collections, maybe), but that's weird and awkward and we've got enough problems without creating fake identifiers to maintain.

SO....we could use something that already exists and works - here's a good "individualID" (or OrganismID or whatever it should be):

https://arctos.database.museum/SpecimenResults.cfm?guid=DMNS:Mamm:12344,MSB:Mamm:233616

That's the "normal" multi-specimen viewpoint, but it won't work unless your login has access to both collections, and it's a table so has the limitations associated with tabular views (although you can turn JSON-stuff on and cheat a bit).

We could also throw something new together, which would allow most anything - customizing the view, using a user who can cross VPD boundaries (in the public data), or WHATEVER. I think there might be some value in providing a distinctive viewpoint in which we can easily tell users exactly what they're seeing, even if it's the same data as other views. Another actionable whatever-this-should-beID:

https://arctos.database.museum/demo.cfm?dwcguid=DMNS:Mamm:12344,MSB:Mamm:233616

That "identifier" could be easily built on the fly from relationships wherever it's useful to do so.

I think that may be what I (and @Jegelewicz ??) were trying to get at above re: "meta cataloged items."

campmlc commented 6 years ago

This may be out in left field, but given the recent BCoN discussion about DOIs being assigned to all levels of the attribution hiercharchy, should we be considering something like assigning a DOI to the organism itself? Does GBIF assign an organism ID, or do they leave that to collections?

On Wed, Jun 20, 2018, 8:49 AM dustymc notifications@github.com wrote:

https://terms.tdwg.org/wiki/dwc:individualID also exists, and we populate it with GUID. I'm not sure how that's meant to differ from Organism - @tucotuco https://github.com/tucotuco ???

If we assign GUIDs to Occurrences, we would (do now...) need to find SOMETHING to stuff in whatever "the critter" field we should be using. I suppose we could try to force users to create something (which might have reasonable success within collections, maybe), but that's weird and awkward and we've got enough problems without creating fake identifiers to maintain.

SO....we could use something that already exists and works - here's a good "individualID" (or OrganismID or whatever it should be):

https://arctos.database.museum/SpecimenResults.cfm?guid=DMNS:Mamm:12344,MSB:Mamm:233616

That's the "normal" multi-specimen viewpoint, but it won't work unless your login has access to both collections, and it's a table so has the limitations associated with tabular views (although you can turn JSON-stuff on and cheat a bit).

We could also throw something new together, which would allow most anything - customizing the view, using a user who can cross VPD boundaries (in the public data), or WHATEVER. I think there might be some value in providing a distinctive viewpoint in which we can easily tell users exactly what they're seeing, even if it's the same data as other views. Another actionable whatever-this-should-beID:

https://arctos.database.museum/demo.cfm?dwcguid=DMNS:Mamm:12344,MSB:Mamm:233616

That "identifier" could be easily built on the fly from relationships wherever it's useful to do so.

I think that may be what I (and @Jegelewicz https://github.com/Jegelewicz ??) were trying to get at above re: "meta cataloged items."

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398778821, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH0hJfArXgzc5qdSy9sVuEhALxXStqoks5t-mEOgaJpZM4UT5xn .

dustymc commented 6 years ago

DOI

YES!! I think they take purchase orders and credit cards....

I'd love to throw DOIs at everything - the mechanism to do so in Arctos exists - but we'd have to buy them. EZID (where we get DOIs now) won't tell me a hard limit, but they do start making funny noises when when not-large-enough numbers come up. You can pull them individually for many things in Arctos (they're useful for citing Archives, Specimens, Media, etc., for example), but that's not a large-scale solution.

ARKs are available at scale, but it's not entirely clear to me what that would DO for us. Our homegrown IDs are functionally-equivalent, and ARK doesn't have the commercial buy-in DOI comes with.

Unrelated here, but ARKS would make really great barcodes. Someone could cite the barcode and still end up at the specimen. I made https://n2t.net/ark:/87299/x6td9vc6 for whatever was in the top of the "random" widget the last time I tried to sucker someone into that.

Jegelewicz commented 6 years ago

The field is called organismID (http://rs.tdwg.org/dwc/terms/index.htm#organismID). It is worth looking at the Darwin Core definition of an Organism ( http://rs.tdwg.org/dwc/terms/index.htm#Organism) to be clear on what that ID can refer to.

https://terms.tdwg.org/wiki/dwc:individualID also exists, and we populate it with GUID. I'm not sure how that's meant to differ from Organism - @tucotuco ???

Yes! That is what I was getting at. A view in Arctos of all related occurrences is fine, but in order to make it real to those outside of Arctos, we need to give those individual animals (organisms) their own ID. So for most collection objects, the organism ID and GUID would be the same, but those with a relationship of "same as" should share an OrganismID. Possible?

dustymc commented 6 years ago

those with a relationship of "same as" should share an OrganismID. Possible?

https://arctos.database.museum/demo.cfm?dwcguid=DMNS:Mamm:12344,MSB:Mamm:233616

tada!

Jegelewicz commented 6 years ago

Synthesize-O-Matic

Love it! It seems to me that this approach would work well - it just means that each set of specimens collected at a specific time and place will need to have its own catalog number, then we could easily provide the data to both GGBN and GBIF. Correct?

I guess the question is how does the OrganismID get generated?

Jegelewicz commented 6 years ago

Easy: bring that stuff in to Arctos. We are looking for solutions to problems that most collection don't know they have, and don't have the tools to detect. That turns out to be non-trivial in a system with highly normalized data and powerful linking mechanisms. I don't think it's remotely possible in anything else.

Yes, because we really ARE the integrated digital biorepository that iDigBio wishes it could be.....

dustymc commented 6 years ago

Yup, we'd autogen Identifiers (which also happen to be URLs that DO SOMETHING) for specimens with multiple Occurrences.

Actually it's probably useful to just do that for everything. The demo link is ONE Occurrence represented by multiple specimens. Explicitly saying "this Occurrence is also the Organism" in DWC seems moderately useful for "normal" specimens (those with 1:1 specimen:occurrence ratio).

We can discuss detail if we get there, but it's basically the same functionality as...

just looking for related specimens (in this case related in specific ways, and then concatenating the results into some sort of actionable URI).

amgunderson commented 6 years ago

I don't have time to read this thread but it looks pretty scary to me. If there is a problem serving data to GGBN maybe that is GGBNs problem and not Arctos' problem. We have enough problems of our own and do not need to solve GGBNs problems too. I am very much not in favor of cataloging every part as a unique item with it's own catalog number. I know I may not understand what is being discussed here though.

tucotuco commented 6 years ago

Just a couple of quick notes...

On Thu, Jun 21, 2018 at 2:49 AM, dustymc notifications@github.com wrote:

https://terms.tdwg.org/wiki/dwc:individualID also exists, and we populate it with GUID. I'm not sure how that's meant to differ from Organism - @tucotuco https://github.com/tucotuco ???

individualID no longer exists. It was superseded with organismID (see Decision_2014-10-26_14 on http://rs.tdwg.org/dwc/terms/history/decisions/index.htm).

If we assign GUIDs to Occurrences, we would (do now...) need to find SOMETHING to stuff in whatever "the critter" field we should be using. I suppose we could try to force users to create something (which might have reasonable success within collections, maybe), but that's weird and awkward and we've got enough problems without creating fake identifiers to maintain.

SO....we could use something that already exists and works - here's a good "individualID" (or OrganismID or whatever it should be):

https://arctos.database.museum/SpecimenResults.cfm? guid=DMNS:Mamm:12344,MSB:Mamm:233616

That's the "normal" multi-specimen viewpoint, but it won't work unless your login has access to both collections, and it's a table so has the limitations associated with tabular views (although you can turn JSON-stuff on and cheat a bit).

And it would change if someone cataloged again using some part of that individual, so it fails the persistence test.

We could also throw something new together, which would allow most anything - customizing the view, using a user who can cross VPD boundaries (in the public data), or WHATEVER. I think there might be some value in providing a distinctive viewpoint in which we can easily tell users exactly what they're seeing, even if it's the same data as other views. Another actionable whatever-this-should-beID:

https://arctos.database.museum/demo.cfm?dwcguid=DMNS: Mamm:12344,MSB:Mamm:233616

That "identifier" could be easily built on the fly from relationships wherever it's useful to do so.

I think that may be what I (and @Jegelewicz https://github.com/Jegelewicz ??) were trying to get at above re: "meta cataloged items."

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398778821, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcP64jj9t4WEhsLW-RJ8hMPJg4LTPeVks5t-mEOgaJpZM4UT5xn .

tucotuco commented 6 years ago

GBIF does not assign an organismID. They can't even detect the organism without an identifier coming in. That is up to the data publishers.

On Thu, Jun 21, 2018 at 2:59 AM, Mariel Campbell notifications@github.com wrote:

This may be out in left field, but given the recent BCoN discussion about DOIs being assigned to all levels of the attribution hiercharchy, should we be considering something like assigning a DOI to the organism itself? Does GBIF assign an organism ID, or do they leave that to collections?

On Wed, Jun 20, 2018, 8:49 AM dustymc notifications@github.com wrote:

https://terms.tdwg.org/wiki/dwc:individualID also exists, and we populate it with GUID. I'm not sure how that's meant to differ from Organism - @tucotuco https://github.com/tucotuco ???

If we assign GUIDs to Occurrences, we would (do now...) need to find SOMETHING to stuff in whatever "the critter" field we should be using. I suppose we could try to force users to create something (which might have reasonable success within collections, maybe), but that's weird and awkward and we've got enough problems without creating fake identifiers to maintain.

SO....we could use something that already exists and works - here's a good "individualID" (or OrganismID or whatever it should be):

https://arctos.database.museum/SpecimenResults.cfm? guid=DMNS:Mamm:12344,MSB:Mamm:233616

That's the "normal" multi-specimen viewpoint, but it won't work unless your login has access to both collections, and it's a table so has the limitations associated with tabular views (although you can turn JSON-stuff on and cheat a bit).

We could also throw something new together, which would allow most anything - customizing the view, using a user who can cross VPD boundaries (in the public data), or WHATEVER. I think there might be some value in providing a distinctive viewpoint in which we can easily tell users exactly what they're seeing, even if it's the same data as other views. Another actionable whatever-this-should-beID:

https://arctos.database.museum/demo.cfm?dwcguid=DMNS: Mamm:12344,MSB:Mamm:233616

That "identifier" could be easily built on the fly from relationships wherever it's useful to do so.

I think that may be what I (and @Jegelewicz https://github.com/Jegelewicz ??) were trying to get at above re: "meta cataloged items."

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398778821, or mute the thread https://github.com/notifications/unsubscribe-auth/ AOH0hJfArXgzc5qdSy9sVuEhALxXStqoks5t-mEOgaJpZM4UT5xn .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-398782542, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcP62ao43fBx_nvN-AjhFeZv-0Zz7AOks5t-mM-gaJpZM4UT5xn .

tucotuco commented 6 years ago

On Thu, Jun 21, 2018 at 3:32 AM, Teresa Mayfield notifications@github.com wrote:

The field is called organismID (http://rs.tdwg.org/dwc/terms/ index.htm#organismID). It is worth looking at the Darwin Core definition of an Organism ( http://rs.tdwg.org/dwc/terms/index.htm#Organism) to be clear on what that ID can refer to.

https://terms.tdwg.org/wiki/dwc:individualID also exists, and we populate it with GUID. I'm not sure how that's meant to differ from Organism - @tucotuco https://github.com/tucotuco ???

Yes! That is what I was getting at. A view in Arctos of all related occurrences is fine, but in order to make it real to those outside of Arctos, we need to give those individual animals (organisms) their own ID.

So for most collection objects, the organism ID and GUID would be the same, but those with a relationship of "same as" should share an OrganismID. Possible?

Careful, if you mix occurrences with organisms in the meaning of a field, you lose that meaning. If you are really serious about identifiers and what can one day be done with them using semantic technologies, you must keep the concepts separate and give the organism with its organismID and all of its occurrences occurrenceIDs (speaking Darwin Core, of course).

dustymc commented 6 years ago

fails the persistence test.

I was hoping nobody would notice....

I'm not sure there's a path around this.

I have A, you have B, I assert they're the same (so want to assign them both an OrganismID), you're in Arctos (so capable of that) but don't believe me. ("Arctos" is very reluctant to let me modify your data. I can make relationships TO your specimens, those send you email so it's easy to reciprocate, but there's no mechanism for me to directly change your specimens.)
I have A, you have B, I assert they're the same (so want to assign them both an OrganismID), you're in not-Arctos so you don't have the ability to reciprocate even if you want to.
I have A, you have B, we agree they're the same, we assign them a "persistent" OrganismID, oopsie, actually they're not the same.
I catalog A, I catalog B, they go out with individual OrganismIDs, oh, right, they're the same - now what?

Etc etc bla bla bla. I don't really see these as being persistent assertions/things/whatever, so assigning them persistent IDs quickly becomes complicated. (And FWIW the Occurrences we create for DWC don't have particularly persistent identifiers either - DWC has the wrong type of core data object to facilitate that.)

if you mix occurrences with organisms in the meaning of a field, you lose that meaning.

I don't think it would be mixed, but I suppose that's details.

For most specimens, it's the same thing (for all we know at the moment, anyway) - the record represents an Occurrence, and that single Occurrence is all we know of the Organism. These might share a value.
For some specimens, the single Occurrence is only one part of the Organism - the composite view provide a picture of the Organism. These would not share a value.
For some specimens, the Occurrence is an alleged duplicate (eg, someone cataloged bits in multiple collections) - the Organism view is weird under that. These would not share a value.
For some specimens, the Occurrence isn't an Occurrence at all (eg, someone cataloged host eggs and nest parasites) - the Organism view is weird under that too. These might share a value.

Jegelewicz commented 6 years ago

@amgunderson we aren't talking about cataloging every single part - rather one catalog number for whatever set of parts was collected at a specific time in a specific place.

campmlc commented 6 years ago

Need to schedule a call week of Aug 13 to discuss. Please respond if interested in participating.

tucotuco commented 6 years ago

Yes, I am interested.

On Mon, Aug 13, 2018 at 8:49 PM Mariel Campbell notifications@github.com wrote:

Need to schedule a call week of Aug 13 to discuss. Please respond if interested in participating.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/1545#issuecomment-412704072, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcP61FfDWF6iASl4BZbvz4-dfz5qcihks5uQhBxgaJpZM4UT5xn .

Jegelewicz commented 6 years ago

Yes

jldunnum commented 6 years ago

Yes

Jonathan L. Dunnum Ph.D. Senior Collection Manager Division of Mammals, Museum of Southwestern Biology University of New Mexico Albuquerque, NM 87131 (505) 277-9262 Fax (505) 277-1351

MSB Mammals website: http://www.msb.unm.edu/mammals/index.html Facebook: http://www.facebook.com/MSBDivisionofMammals

Shipping Address: Museum of Southwestern Biology Division of Mammals University of New Mexico CERIA Bldg 83, Room 204 Albuquerque, NM 87131

From: Teresa Mayfield notifications@github.com Sent: Tuesday, August 14, 2018 10:01:44 AM To: ArctosDB/arctos Cc: Jonathan Dunnum; Assign Subject: Re: [ArctosDB/arctos] Linkage of Parts to Collecting Events (#1545)

Yes

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/ArctosDB/arctos/issues/1545#issuecomment-412924241, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQengzDROr5bRbSZ8sbGeZAWBekcGrWyks5uQvRogaJpZM4UT5xn.

ArctosDB / arctos

Linkage of Parts to Collecting Events #1545