Closed dustymc closed 2 years ago
I greatly dislike "Verbatim Event Details" - that "node" is much more than that. The rest I could live with, I think, but it's extremely verbose.
We should also be looking at APIs - they will not share this terminology. (And the above doesn't cover all non-API usage).
Please do not make arbitrary changes; this needs planned.
what exactly does specimen/event remark mean
It is remarks regarding the intersection of specimens - "catalog records" - and the event stack.
arctosprod@arctos>> \d specimen_event
Table "core.specimen_event"
Column | Type | Collation | Nullable | Default
-----------------------+-----------------------------+-----------+----------+-------------------------------------------
specimen_event_id | integer | | not null | nextval('sq_specimen_event_id'::regclass)
collection_object_id | integer | | not null |
collecting_event_id | integer | | not null |
assigned_by_agent_id | integer | | not null |
assigned_date | timestamp without time zone | | not null |
specimen_event_remark | character varying(4000) | | |
specimen_event_type | character varying(60) | | not null |
collecting_method | character varying(4000) | | |
collecting_source | character varying(60) | | |
verificationstatus | character varying(60) | | not null |
habitat | character varying(4000) | | |
verified_by_agent_id | integer | | |
verified_date | character varying(30) | | |
OK - I have had a few nights of sleep to dream about this and here are some thoughts.
First - I think that if the data entry screen were structured in the same way the catalog record is, there would be less confusion. (and I know it sort-of is, but in a different order)
Beyond that, let's talk about the "event stack"
I'm going to start with the top of the stack - the part where we assign an event to a catalog record, currently known as the "specimen/event" (I have been trying to rid us of specimen)
Here is a comparison of all the ways we refer to sections of this part of the event stack:
Arctos Page/What it does | Assigned Event Type | Event Assigned By | Event Assigned Date | Event Verification Status | Event Verified By | Event Verification Date | Event Assignment Remark | Event Habitat | Event Source | Event Method |
---|---|---|---|---|---|---|---|---|---|---|
Edit catalog record | Specimen/Event Type | Event Determiner | Determined Date | Verification Status | Verified By | Verified Date | Specimen/Event Remark | Habitat | Collecting Source | Collecting Method |
Catalog record | Event Type | Event Type: assigned by | Event Date | Event Verification: | Event Verification: verified by | Event Verification: on | Event Details: remark | Event Details: habitat | Event Details: Collecting Source | Event Details: Collecting Method |
Data entry (old) | Specimen/Event Type | Event Determiner | Detr. Date | VerificationStatus | N/A | N/A | Specimen/Event Remark | Habitat | Coll. Src. | Coll. Meth. |
Data entry (new) | Event Type | Event Determiner | Det. Date | VerificationStatus | N/A | N/A | Event Remark | Habitat | Collecting Source | Collecting Method |
Bulkloader builder | specimen_event_type | event_assigned_by_agent | event_assigned_date | verificationstatus | N/A | N/A | specimen_event_remark | habitat | collecting_source | collecting_method |
Bulkload specimen-events | specimen_event_type | assigned_by_agent | assigned_date | verificationstatus | N/A | N/A | specimen_event_remark | habitat | collecting_source | collecting_method |
So I can see why there is confusion but before we start changing stuff, I think we need to take a hard look at what this information is meant to convey and make sure that the labels fit. I'd like to see this whole part of the stack better reflect what it is actually doing - recording information about who, when, and why the associated event/locality are assigned to the object(s) or observation(s) this catalog record describes. If that is a poor description of what is going on, please give me a better one!
FWIW documentation says this (and needs to get rid of "specimen")
Specimen Event Specimen Events link Specimens to Collecting Events.
A few other musings:
data entry screen were structured in the same way the catalog record is
It won't be - one has a fixed structure (with some "extras"), the other is purely data driven and has no limitations.
different order
The entry screen has no order at all (and maybe some day other screens will be similar).
what this information is meant to convey
https://arctos.database.museum/tblbrowse.cfm?tbl=specimen_event
make sure that the labels fit.
That's one of the original use cases for field-level documentation, but we don't follow it (nor do we have the tools to make that mostly-realistic). Maybe that should be merged with the table browser in some way. Also, context - which varies from UI to UI - matters, unless we want to be very verbose everywhere (which brings on a whole new batch of usability problems).
In any case, I'd like to keep more of the very core documentation - "this table does THAT, this field means THING" - in the database itself, where it's available to APIs (including local UI). That's probably yet another Issue....
why "habitat" falls within this section of the stack and is not a "collecting event" attribute.
That's reflective of a lack of precision in recording events; events end up as "today, within a mile of here" and habitat within that event is "under a rotting log" and "bare vertical rock" and .... I doubt we can do much for legacy data, but we can certainly make recommendations to GPS-equipped collectors going forward.
"collecting method" and "collecting source",
Same as above, but also things like host being collected in a pitfall and parasites being collected in a Number Whatever Sieve (so not "fixable" via more precise place-time data).
data entry screen were structured in the same way the catalog record is
It won't be - one has a fixed structure (with some "extras"), the other is purely data driven and has no limitations.
I just mean the same order - right now locality is where collectors and identifiers are in the catalog record. If we could keep to the same general order, this would help, I think - but this is just one person's opinion.
locality is where
... ever a user put it.
Fair enough.
OK, here is the comparison for "collecting event" which is a misnomer, because it can also represent any of the other event types in https://arctos.database.museum/info/ctDocumentation.cfm?table=ctspecimen_event_type
Arctos Page/What it does | Verbatim Event Location | Verbatim Event Date | Formatted Event Start Date | Formatted Event End Date | Event Remark | Event Name | Verbatim Coordinates | Event Attribute Type | Event Attribute Value | Event Attribute Units | Event Attribute Determiner | Event Attribute Date | Event Attribute Method | Event Attribute Remark | Locality Name | Event ID | Locality ID |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Edit catalog record | Verbatim Locality | Verbatim Date | Began Date/Time | Ended Date/Time | Collecting Event Remark | Collecting Event Nickname | As-entered Coordinates | Event Attribute Type | Event Attribute Value | Event Attribute Units | Event Attribute Determiner | Event Attribute Date | Event Attribute Method | Event Attribute Remark | locality_name | N/A | |
Catalog record | Verbatim Locality | Verbatim Date | Event Date | Event Date (to) | Remark | Event Nickname | As Entered | Attribute | Value | included in value if applicable | Determiner | Date | Method | Remark | Locality Nickname | N/A | |
Data entry (old) | Verbatim Locality | VerbatimDate | Begin | End | CollEvntRemk | Event Nickname | Coordinates (event and locality) | N/A | N/A | N/A | N/A | N/A | N/A | N/A | Locality Nickname | Existing EventID | |
Data entry (new) | Verbatim Locality | Verbatim Date | Began Date | Ended Date | Collecting Event Remark | Event Name | all the coordinate stuff that is not decimal degrees | N/A | N/A | N/A | N/A | N/A | N/A | N/A | Locality Name | Event ID | Locality ID |
Bulkloader builder | verbatim_locality | verbatim_date | began_date | ended_date | coll_event_remarks | collecting_event_name | all the coordinate stuff that is not decimal degrees | N/A | N/A | N/A | N/A | N/A | N/A | N/A | locality_name | collecting_event_id | locality_id |
Bulkload specimen-events | verbatim_locality | verbatim_date | began_date | ended_date | coll_event_remarks | collecting_event_name | all the coordinate stuff that is not decimal degrees | N/A | N/A | N/A | N/A | N/A | N/A | N/A | locality_name | N/A | N/A |
Bulkload collecting events | verbatim_locality | verbatim_date | began_date | ended_date | coll_event_remarks | collecting_event_name | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | locality_name | N/A | N/A |
Bulkload collecting event attributes | N/A | N/A | N/A | N/A | N/A | event_name | N/A | event_attribute_type | event_attribute_value | event_attribute_units | event_determiner | event_determined_date | event_determination_method | event_attribute_remark | N/A | N/A | N/A |
A few thoughts on this:
misnomer
No argument, but that's Major Surgery (and I'm not sure I have better names).
Why are we including Event and Locality ID's as pick values when these could so easily change?
@campmlc had some compelling reason, and the merge does handle the bulkloader (but not some spreadsheet on your desktop, of course).
Where in the heck does "associated species" fall in all of this?
https://arctos.database.museum/tblbrowse.cfm?tbl=coll_object_remark
Is it part of the event stack or not?
No - but it should be moved to specimen_event, I think. Just a migration miss IIRC.
I can enter locality attributes as extras
You cannot: https://arctos.database.museum/tblbrowse.cfm?tbl=bulkloader
Why can't I enter event attributes as "extras"?
Maybe plausible, certainly open for discussion, a few could be flattened as well, I think I'd prefer to pre-create named events and just use them (but IDK how that aligns with reality).
I can enter locality attributes as extras
You cannot: https://arctos.database.museum/tblbrowse.cfm?tbl=bulkloader
UMMMM
Also
"extras" - if you see it in the builder, it's not an extra.
Sorry for my poor use of "extras" - my argument is still the same, if I can do this for a locality, it seems that I should also be able to do it for an event, either directly or via "extras".
And finally - comparison for locality
Arctos Page/What it does | Higher Geography | Locality Name | Locality ID | Specific Locality | Minimum Elevation | Maximum Elevation | Elevation Units | Minimum Depth | Maximum Depth | Depth Units | Locality Remark | Coordinate Units | Decimal Latitiude | Decimal Longitude | Maximum Coordinate Error | Coordinate Error Units | Datum | Georeference Source | Georeference Protocol | WKT Media ID | Locality Attribute | Locality Attribute Value | Locality Attribute Units | Locality Attribute Determiner | Locality Attribute Determined Date | Locality Attribute Determined Method | Locality Attribute Remark |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Edit catalog record | Higher Geography | Locality Nickname | Specific Locality | Min. Elev. | Max. Elev. | Elev. Unit | Min. Depth | Max. Depth | Depth Units | Locality Remarks | N/A | Decimal Latitude | Decimal Longitude | Max Error | Max Error Units | Datum | Georeference Source | Georeference Protocol | WKTMediaID | Type | Value | Units | Determiner | Date | Method | Remark | |
Catalog record | Higher Geography | Locality Nickname | Specific Locality | Elevation | (to) Elevation | Elevation | Depth | (to) Depth | Depth | Locality Remarks | N/A | Coordinates | Coordinates | Error | Error | Datum | Georeference Source | Georeference Protocol | Attribute | Value | Units | Determiner | Date | Method | Remark | ||
Data entry (old) | Higher Geog | Locality Nickname | Existing LocalityID | Spec Locality | Elevation (min-max) between | Elevation (min-max) between | Elevation (min-max) between | Depth (min-max) between | Depth (min-max) between | Depth (min-max) between | LocalityRem | Original lat/long Units | Dec Lat | Dec Long | Max Error | Max Error | Datum | Georeference Source | Georeference Protocol | WKTMediaID | Type | Value | Unit | Determiner | Date | Method | Remark |
Data entry (new) | Higher Geography | Locality Name | Locality ID | Specific Locality | Minimum Elevation | Maximum Elevation | Elevation Units | Minimum Depth | Maximum Depth | Depth Units | Locality Remarks | Coordinate Units | Dec Lat | Dec Long | Max Error | Max Error Units | Datum | Georeference Source | Georeference Protocol | WKTMediaID | locality_attribute_type | locality_attribute_value | locality_attribute_unit | locality_attribute_determiner | locality_attribute_detr_date | locality_attribute_detr_meth | locality_attribute_remark |
Bulkloader builder | higher_geog | locality_name | locality_id | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | orig_lat_long_units | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | wkt_media_id | locality_attribute_type | locality_attribute_value | locality_attribute_units | locality_attribute_determiner | locality_attribute_detr_date | locality_attribute_detr_meth | locality_attribute_remark |
Bulkload specimen-events | locality_name | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | orig_lat_long_units | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | wkt_media_id | N/A | N/A | N/A | N/A | N/A | N/A | N/A | ||
Bulkload collecting events | higher_geog | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | |||
Bulkload locality | |||||||||||||||||||||||||||
Bulkload locality attributes | N/A | locality_name | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | attribute_type | attribute_value | attribute_units | attribute_determiner | determined_date | determination_method | attribute_remark |
I need to add in the names for bulkload locality - but the tool 404s for me right now. See #4053
A few things here.
No higher geography field in the specimen event loader
Datum is not included on the catalog record
who georeferenced and when t
File a CT request
"behind the scenes" (coordinates other than decimal degrees are entered and used to create decimal degree values), the who would be "Arctos"?
There's only one possibility, we gain nothing by recording it.
No higher geography field in the specimen event loader
Get a template - there is no higher geography in it.
Datum is not included on the catalog record
Screen Shot 2021-10-28 at 4 31 54 PM
HMMMM I swear I saw one with coordinates but no datum - is it possible that it gets hidden when it is "unknown"? Guess not, I'm blaming that on late lunch...
A revised summary
Arctos Page/What it does | Higher Geography | Locality Name | Locality ID | Specific Locality | Minimum Elevation | Maximum Elevation | Elevation Units | Minimum Depth | Maximum Depth | Depth Units | Locality Remark | Coordinate Units | Decimal Latitiude | Decimal Longitude | Maximum Coordinate Error | Coordinate Error Units | Datum | Georeference Source | Georeference Protocol | WKT Media ID | Locality Attribute | Locality Attribute Value | Locality Attribute Units | Locality Attribute Determiner | Locality Attribute Determined Date | Locality Attribute Determined Method | Locality Attribute Remark |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Edit catalog record | Higher Geography | Locality Nickname | N/A | Specific Locality | Min. Elev. | Max. Elev. | Elev. Unit | Min. Depth | Max. Depth | Depth Units | Locality Remarks | N/A | Decimal Latitude | Decimal Longitude | Max Error | Max Error Units | Datum | Georeference Source | Georeference Protocol | WKTMediaID | Type | Value | Units | Determiner | Date | Method | Remark |
Catalog record | Higher Geography | Locality Nickname | N/A | Specific Locality | Elevation | (to) Elevation | Elevation | Depth | (to) Depth | Depth | Locality Remarks | N/A | Coordinates | Coordinates | Error | Error | Georeference Source | Georeference Protocol | Attribute | Value | Units | Determiner | Date | Method | Remark | ||
Data entry (old) | Higher Geog | Locality Nickname | Existing LocalityID | Spec Locality | Elevation (min-max) between | Elevation (min-max) between | Elevation (min-max) between | Depth (min-max) between | Depth (min-max) between | Depth (min-max) between | LocalityRem | Original lat/long Units | Dec Lat | Dec Long | Max Error | Max Error | Datum | Georeference Source | Georeference Protocol | WKTMediaID | Type | Value | Unit | Determiner | Date | Method | Remark |
Data entry (new) | Higher Geography | Locality Name | Locality ID | Specific Locality | Minimum Elevation | Maximum Elevation | Elevation Units | Minimum Depth | Maximum Depth | Depth Units | Locality Remarks | Coordinate Units | Dec Lat | Dec Long | Max Error | Max Error Units | Datum | Georeference Source | Georeference Protocol | WKTMediaID | locality_attribute_type | locality_attribute_value | locality_attribute_unit | locality_attribute_determiner | locality_attribute_detr_date | locality_attribute_detr_meth | locality_attribute_remark |
Bulkloader builder | higher_geog | locality_name | locality_id | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | orig_lat_long_units | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | wkt_media_id | locality_attribute_type | locality_attribute_value | locality_attribute_units | locality_attribute_determiner | locality_attribute_detr_date | locality_attribute_detr_meth | locality_attribute_remark |
Bulkload specimen-events | higher_geog | locality_name | N/A | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | orig_lat_long_units | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | wkt_media_id | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
Bulkload collecting events | higher_geog | N/A | N/A | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | N/A | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
Bulkload locality | higher_geog | locality_name | N/A | spec_locality | minimum_elevation | maximum_elevation | orig_elev_units | min_depth | max_depth | depth_units | locality_remarks | N/A | dec_lat | dec_long | max_error_distance | max_error_units | datum | georeference_source | georeference_protocol | wkt_media_id | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
Bulkload locality attributes | N/A | locality_name | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | attribute_type | attribute_value | attribute_units | attribute_determiner | determined_date | determination_method | attribute_remark |
Search | Continent/Ocean, Sea, Country, State/Province, USGS Quad Map, County, Island Group, Island, Geographic Feature, Drainage | N/A | N/A | Specific Locality | Elevation | Elevation | Elevation | N/A | N/A | N/A | Locality Remarks | N/A | N/A | N/A | Coordinate Precision | Coordinate Precision | N/A | Georeference Source | N/A | N/A | Locality Attribute 1 | Locality Attribute 1 Value | Locality Attribute 1 Unit | Locality Attribute 1 Determiner | N/A | Locality Attribute 1 Method | Locality Attribute 1 Remark |
Arctos Page/What it does | Verbatim Event Location | Verbatim Event Date | Formatted Event Start Date | Formatted Event End Date | Event Remark | Event Name | Verbatim Coordinates | Event Attribute Type | Event Attribute Value | Event Attribute Units | Event Attribute Determiner | Event Attribute Date | Event Attribute Method | Event Attribute Remark | Locality Name | Event ID | Locality ID |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Edit catalog record | Verbatim Locality | Verbatim Date | Began Date/Time | Ended Date/Time | Collecting Event Remark | Collecting Event Nickname | As-entered Coordinates | Event Attribute Type | Event Attribute Value | Event Attribute Units | Event Attribute Determiner | Event Attribute Date | Event Attribute Method | Event Attribute Remark | locality_name | N/A | N/A |
Catalog record | Verbatim Locality | Verbatim Date | Event Date | Event Date (to) | Remark | Event Nickname | As Entered | Attribute | Value | included in value if applicable | Determiner | Date | Method | Remark | Locality Nickname | N/A | N/A |
Data entry (old) | Verbatim Locality | VerbatimDate | Begin | End | CollEvntRemk | Event Nickname | Coordinates (event and locality) | N/A | N/A | N/A | N/A | N/A | N/A | N/A | Locality Nickname | Existing EventID | N/A |
Data entry (new) | Verbatim Locality | Verbatim Date | Began Date | Ended Date | Collecting Event Remark | Event Name | all the coordinate stuff that is not decimal degrees | N/A | N/A | N/A | N/A | N/A | N/A | N/A | Locality Name | Event ID | Locality ID |
Bulkloader builder | verbatim_locality | verbatim_date | began_date | ended_date | coll_event_remarks | collecting_event_name | all the coordinate stuff that is not decimal degrees | N/A | N/A | N/A | N/A | N/A | N/A | N/A | locality_name | collecting_event_id | locality_id |
Bulkload specimen-events | verbatim_locality | verbatim_date | began_date | ended_date | coll_event_remarks | collecting_event_name | all the coordinate stuff that is not decimal degrees | N/A | N/A | N/A | N/A | N/A | N/A | N/A | locality_name | N/A | N/A |
Bulkload collecting events | verbatim_locality | verbatim_date | began_date | ended_date | coll_event_remarks | collecting_event_name | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | locality_name | N/A | N/A |
Bulkload collecting event attributes | N/A | N/A | N/A | N/A | N/A | event_name | N/A | event_attribute_type | event_attribute_value | event_attribute_units | event_determiner | event_determined_date | event_determination_method | event_attribute_remark | N/A | N/A | N/A |
Search | Verbatim Locality | Verbatim Date | Collected On or After | Collected On or Before | Collecting Event Remark | Search | N/A | Event Attributes | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
Arctos Page/What it does | Assigned Event Type | Event Assigned By | Event Assigned Date | Event Verification | Event Verified By | Event Verification Date | Event Assignment Remark | Event Habitat | Event Source | Event Method |
---|---|---|---|---|---|---|---|---|---|---|
Edit catalog record | Specimen/Event Type | Event Determiner | Determined Date | Verification Status | Verified By | Verified Date | Specimen/Event Remark | Habitat | Collecting Source | Collecting Method |
Catalog record | Event Type | Event Type: assigned by | Event Date | Event Verification: | Event Verification: verified by | Event Verification: on | Event Details: remark | Event Details: habitat | Event Details: Collecting Source | Event Details: collecting method |
Data entry (old) | Specimen/Event Type | Event Determiner | Detr. Date | VerificationStatus | N/A | N/A | Specimen/Event Remark | Habitat | Coll. Src. | Coll. Meth. |
Data entry (new) | Event Type | Event Determiner | Det. Date | VerificationStatus | N/A | N/A | Event Remark | Habitat | Collecting Source | Collecting Method |
Bulkloader builder | specimen_event_type | event_assigned_by_agent | event_assigned_date | verificationstatus | N/A | N/A | specimen_event_remark | habitat | collecting_source | collecting_method |
Bulkload specimen-events | specimen_event_type | assigned_by_agent | assigned_date | verificationstatus | N/A | N/A | specimen_event_remark | habitat | collecting_source | collecting_method |
Search | Record/Event Type | N/A | N/A | Verification Status | N/A | N/A | Record/Event Remark | Habitat * in Locality | Collecting Source | Collecting Method |
But I think this has always been a problem what exactly does specimen/event remark mean? It has never been clear.
Anyway, change this to "Collection Object Remark" to match data entry
Change this to "Verbatim Event Details" to match catalog record (but see below)
Change this to "Verbatim Event Details" to match data entry
Change this to "Catalog Record Event Remark" on the catalog record
Change this to "Catalog Record Event Remark" in data entry
And also make appropriate changes to search. I can do all of this and let everyone see it in test.
Originally posted by @Jegelewicz in https://github.com/ArctosDB/arctos/issues/3691#issuecomment-951095270