Open iDigBioBot opened 6 years ago
TestField | Value |
---|---|
GUID | 710fe118-17e1-440f-b428-88ba3f547d6d |
Label | AMENDMENT_EVENT_FROM_EVENTDATE |
Description | Propose amendment to values in any of dwc:year, dwc:month, dwc:day, dwc:startDayOfYear or dwc:endDayOfYear from the content of dwc:eventDate. |
TestType | Amendment |
Darwin Core Class | Event |
Information Elements ActedUpon | dwc:year |
dwc:month | |
dwc:day | |
dwc:startDayOfYear | |
dwc:endDayOfYear | |
Information Elements Consulted | dwc:eventDate |
Expected Response | INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN if any of (1) dwc:day from dwc:eventDate if dwc:day is EMPTY and dwc:eventDate has a precision of a day or finer and is within a single day, (2) dwc:month from dwc:eventDate if dwc:month is EMPTY and dwc:eventDate has a precision of a single month or finer and is within a single month, (3) dwc:year from dwc:eventDate if dwc:year is EMPTY and dwc:eventDate has a precision of a single year or finer and is within a single year, (4) dwc:startDayOfYear and dwc:endDayOfYear if they are EMPTY and dwc:eventDate has a precision of a day or better; otherwise NOT_AMENDED. |
Data Quality Dimension | Completeness |
Term-Actions | EVENT_FROM_EVENTDATE |
Parameter(s) | |
Source Authority | |
Specification Last Updated | 2024-07-23 |
Examples | [dwc:eventDate="2023-01-26", dwc:year="2023", dwc:month="", dwc:day="", dwc:startDayOfYear="", dwc:endDayOfYear="": Response.status=FILLED_IN, Response.result= dwc:startDayOfYear="26", dwc:month="1", dwc:day="26",dwc:endDayOfYear="26", Response.comment="dwc:month, dwc:day, dwc:startDayOfyear and dwc:endDayOfYear filled in from dwc:eventDate"] |
[dwc:eventDate="2023", dwc:year="2023", dwc:month="", dwc:day="", dwc:startDayOfYear="", dwc:endDayOfYear="": Response.status=NOT_AMENDED, Response.result=, Response.comment="No amendments possible"] | |
Source | VertNet |
References |
|
Example Implementations (Mechanisms) | Kurator:event_date_qc |
Link to Specification Source Code | FilteredPush event_date_qc DwCEventDQ.amendmentEventFromEventdate() unit test in DwcEventDQTest |
Notes | Only fields that are empty will be have changes proposed, and only if dwc:eventDate has a valid ISO 8601-1 date. The dwc:eventDate is the canonical form of the event date (it is the first trusted form). If event date does not contain a range, dwc:startDayOfYear = dwc:endDayOfYear. Time (as compared to date) is not deemed a CORE component. Note, see sequencing tests section of standards document, run this amendment after any other amendment which may affect dwc:eventDate |
Comment by Paul Morris (@chicoreus) migrated from spreadsheet: One way of simplifying the core test suite would be to identify a small set of primary fields in cases where Darwin Core allows for multiple representations of the data (Event fields are the clearest example of this), and only propose amendments that work to fill in the primary fields from secondary fields (eventDate from day, month, year, verbatimEventDate, startDayOfYear, endDayOfYear, eventTime), and not include in the core suite ammendments that fill in secondary fields from the primary fields.
Comment by Arthur Chapman (@ArthurChapman) migrated from spreadsheet: I agree with Paul ghere
Comment by Paula Zermoglio (@pzermoglio) migrated from spreadsheet: Agree too. However, you may piss some people off, eg those that prefer using YMD instead of eventDate?
@JohnW wanted to be able to back populate - at least for year - as many people want to extract just the year from the data.
The danger with backpopulating is that eventDate is capable of handling richer information than the atomic fields (date ranges and date ranges which span more than one year). A consumer who wants to simply obtain the year from dwc:year does so at their own peril if dwc:eventDate contains a date range which spans more than one year. That, however, is a let the consumer beware kind of issue. We shouldn't advocate back populating because people may want to use the data, as it potentially is unfit for their use, but I'm entirely in favor of back populating in order to make data sets consistent in their presentation - filling in all fields that can be filled in.
I propose changing the description to: One or more empty component terms of the dwc:Event class (dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear) have been filled in from a valid value in the term dwc:eventDate.
The concept reflected in in the note " and only if dwc:eventDate has a valid ISO 8601:2004(E) date" should be reflected in the description. It is an important point for implementers.
There is an inter-amendment workflow dependency to note here - this amendment should run after all other amendments that may affect the value of eventDate. (i.e. fill in the event date from the verbatimEventDate, then fill in year/month/day/startDayOfYear/endDayOfYear from the interpreted eventDate value, etc.). I've added a note to this effect to the prerequisites.
We should note that time is not included here as it is not considered core. In working with an implementation of this, I've found extracting time from eventDate to be fraught with all sorts of concerns (which aren't going to be core concerns), including handling time zone and handling times on eventDates which involve ranges. Non-trivial to specify appropriate behaviors, and those aren't core. We are much safer not having this particular test propose to fill in eventTime.
We do need to specify if endDayOfYear is expected to be filled in if the eventDate represents a single day.
@chicoreus - wrt to workflow - I have added "After #33, #49, #86, #93, #132" for test #52 - see the circulated workflow document
@chicoreus - fully agree wrt note on time.
I have reviewed the parameters and notes (all good) and believe we have a useable outcome.
See comment in #131 about whether this test should specify not filling in endDayOfYear in cases where eventDate is a range of years.
@chicoreus. In discussion in Gainesville, I think I said I saw less value in filling all these other fields from eventDate than going the other way and making sure eventDate was filled in wherever possible. So, filling in endDayOfYear etc. I see being of lower value, and I would be happy with your suggestion. However, those that advocated for this test should be better commenting on it than I.
Reading your comment on #131, I agree with your logic @chicoreus. How would you work the Expected Response?
INTERNAL_PREREQUESITES_NOT_MET if the field dwc:eventDate is EMPTY or does not contain a valid ISO 8601-1:2019 date; AMENDED if one or more EMPTY terms of the dwc:Event class (dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear) have been filled in from a valid unambiguously interpretable value in dwc:eventDate, unless dwc:eventDate spans years in which case dwc:endDayOfyear is not FILLED_IN; otherwise NOT_CHANGED?
@Tasilee. We don't have a term for "NOT_FILLED_IN so I would say
INTERNAL_PREREQUESITES_NOT_MET if the field dwc:eventDate is EMPTY or does not contain a valid ISO 8601-1:2019 date; AMENDED if one or more EMPTY terms of the dwc:Event class (dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear) have been filled in from a valid unambiguously interpretable value in dwc:eventDate and eventDate is wholly within the one year; otherwise, or if dwc:eventDate spans more than one year, NOT_CHANGED?
@ArthurChapman: Much better. Editing.
Or, perhaps even simpler - putting in into positive and being consistent
INTERNAL_PREREQUESITES_NOT_MET if the field dwc:eventDate is EMPTY or does not contain a valid ISO 8601-1:2019 date; AMENDED if one or more EMPTY terms of the dwc:Event class (dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear) have been filled in from a valid unambiguously interpretable value in dwc:eventDate and eventDate is wholly within the one year; otherwise NOT_CHANGED
@ArthurChapman: Yep
Dealing with timezones is a real issue faced when integrating and accessing data. Yes, it is difficult but clarifying expected behavior when a) data spans timezones, b) dealing with datasets holding date times with and without timezone and c) when the timezone of the consumer is not known would be helpful. It might result in recommendations to store a localDateTime
version as well as a UTC-normalized version.
This should also take into consideration the expectations of simple human observations (e.g. a naturalist record from 07:30 local time on 1st January 2019 in New Zealand being returned in a 2019 search and not a 2018 search) as well as those of machine recorders where high-frequency samples are taken and stored in UTC regardless of location even though the moving organism is crossing timezones.
@timrobertson100 discussion in TG2 call today: (1) time was decided as out of scope for the TG2 tests early on, and we'd have to add in all the complexities of time to address the timezone concern. (2) The tests, in particular this one, are asking about the representation of date in a single record, and don't involve comparisons between dates in different records. If dwc:eventTime is included in the New Zealand record you discuss, a consumer of the data is able to interpret which year to place that record into for purposes of search, we are seeing this as an independent question from the internal consistency of the terms in the record itself.
Discussion in call and @tucotuco 's observation that Darwin Core is vauge on whether dwc:endDayOfYear is tied to the end of a date range, and the thought that we could put a stake in the ground towards expectations for this case of endDayOfYear meaning the day of the year of the end of a range expressed in eventDate. Thus, @Tasilee here's a proposal for changing the specification of this test:
INTERNAL_PREREQUESITES_NOT_MET if the field dwc:eventDate is EMPTY or does not contain a valid ISO 8601-1:2019 date; AMENDED if one or more EMPTY terms of the dwc:Event class (dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear) have been filled in from a valid unambiguously interpretable value in dwc:eventDate; otherwise NOT_CHANGED
And Note: Only fields that are empty will be amended, and only if dwc:eventDate has a valid ISO 8601-1:2019 date. The dwc:eventDate is the canonical form of the event date (it is the first trusted form). If event date does not contain a range, dwc:startDayOfYear = dwc:endDayOfYear. Time (as compared to date) is not deemed a CORE component. NB Run this amendment after any other amendment which may affect dwc:eventDate. If eventDate contains a date range, dwc:startDayOfYear is to be interpreted as the day of year of the start of the date range, and dwc:endDayOfYear is to be interpreted as the day of the year of the end of the date range, thus endDayOfYear could be smaller than startDayOfYear as in 2015-12-15/2016-01-15.
Thanks @chicoreus. It is simpler but certainly needs the Note for clarification. Is everyone else happy with this before I amend?
In regards ISO 8601, we can reference it in the specs as discussed but I am tempted to add https://en.wikipedia.org/wiki/ISO_8601 to the references everywhere (in each test) that requires it. What do you think? Reasoning: We all agreed on the benefits of having our table as self-contained as possible (accepting a move from non-canonical to Notes) and access to details of ISO standards is not a simple end-point or free.
Also worth referencing EDTF https://www.loc.gov/standards/datetime/ which is incorporated into ISO:8601-2 (2019), and looking for somewhere with a good summary of 8601/8601-2 to point people at.
Agreed. I have added these refs here, but assume they need to go everywhere we have a TIME tab?
Changed "AMENDED" to "FILLED_IN" in accordance with discussions April 16.
Given recent discussions, I have changed
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to bdq:sourceAuthority; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if they can be unambiguously interpreted from values in dwc:eventDate, and dwc:eventDate is wholly within one year; otherwise NOT_AMENDED
to
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid ISO 8601-1 date; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if they can be unambiguously interpreted from values in dwc:eventDate, and dwc:eventDate is wholly within one year; otherwise NOT_AMENDED
...and I have removed the ref to bdq:sourceAuthority.
This may be too picky, but I would reword to:
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains a value that is not valid according to ISO 8601-1; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if they can be unambiguously interpreted from values in dwc:eventDate, and dwc:eventDate is wholly within one year; otherwise NOT_AMENDED
I like that wording @tucotuco
Given comments from @tucotuco, then maybe
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains a value that is not valid according to ISO 8601-1:2019; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if they can be unambiguously interpreted from values in dwc:eventDate, and dwc:eventDate is wholly within one year; otherwise NOT_AMENDED
I have changed it to this for now and will align the other occurrences if there is agreement.
To standardize the terminology, I am editing occurrences of "not valid" in the Expected Responses to "invalid"
That conforms to the Vocabulary #152 where we have "Invalid" ["Where the Data Quality Dimension: Conformance (q.v.) is not satisfied due to Information Elements (q.v.) containing non-standard values, values outside of an acceptable range, or values unable to be found in a bdq:sourceAuthority (q.v.)."] but not "not valid" (or NOT_VALID)
I've edited the Expected Response according to @tucotuco suggestion:
From
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1:2019; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if they can be unambiguously interpreted from values in dwc:eventDate, and dwc:eventDate is wholly within one year; otherwise NOT_AMENDED
to
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1 or dwc:eventDate is not wholly within one year; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if any were unambiguously interpreted from values in dwc:eventDate; otherwise NOT_AMENDED
and updated the References
I have updated the ISO reference link.
We may be able to handle consistency with notes, but the bit ", and dwc:eventDate is wholly within one year" may not hold.
Here is a way we could be explicit in the specification about expectations, changing from:
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1 or dwc:eventDate is not wholly within one year; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if any were unambiguously interpreted from values in dwc:eventDate, and dwc:eventDate is wholly within one year; otherwise NOT_AMENDED
To:
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1 or dwc:eventDate is not wholly within one year; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if any were unambiguously interpreted from values in dwc:eventDate, under the following rules: if dwc:eventDate is a range of more than one day or lacks precision to day, dwc:day is not populated; if dwc:eventDate is a range spanning than one month, or lacks precision to month, dwc;month is not populated; if dwc:eventDate is a range spanning more than one year dwc:year is not populated; dwc;startDayOfYear and endDayOfYear are to be filled in if they can be unambiguously interpreted from dwc:eventDate; otherwise NOT_AMENDED
Noting that for ranges that span a year boundary, startDayOfYear may be larger than endDayOfYear, and that startDayOfYear and endDayOfYear should be populated if eventDate has either reduced precision (e.g. 1872), or is a range (1872-04-06/1872-08-12).
ISO 8601-1 distinguishes between reduced precision dates and date ranges (thus 1984 and 1984-01-01/1984-12-31 are not the same thing), but biodiversity data is stored in a wide variety of data structures (year, month, day integer fields, pairs of date fields for start and end, structured strings that can take EDTF like strings with explicit markers of uncertainty (1872--)), and go through various transformations with various software including libraries that don't distinguish between reduced precision dates and ranges), and we have low confidence that the meaning of dwc:eventDate=1984 is different from the meaning of dwc:eventDate=1984-01-01/1984-12-31.
But, we might want to put only the following in the specification;
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1 or dwc:eventDate is not wholly within one year; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if any were unambiguously interpreted from values in dwc:eventDate, otherwise NOT_AMENDED
And then put into notes: Fill in values of empty terms under the following rules: if dwc:eventDate is a range of more than one day or lacks precision to day, dwc:day is not populated; if dwc:eventDate is a range spanning than one month, or lacks precision to month, dwc;month is not populated; if dwc:eventDate is a range spanning more than one year dwc:year is not populated; dwc;startDayOfYear and endDayOfYear are to be filled in if they can be unambiguously interpreted from dwc:eventDate;
Changed 'NOT AMENDED' example.
Also awaiting vote or comment on https://github.com/tdwg/bdq/issues/52#issuecomment-1588225146 from @ArthurChapman and @tucotuco
Can't just vote as there are two options! Also both options have some errors need - "...range spanning more than one month..."
Both options have a contradiction - if the year spans > one year, it can't be both INTERNAL_PREREQUISITES_NOTMET and FILLED_IN.
I would vote for a modification of the second option, i.e.
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN one or more EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if any were unambiguously interpreted from values in dwc:eventDate, otherwise NOT_AMENDED
And then put into notes: Fill in values of empty terms under the following rules: if dwc:eventDate is a range of more than one day or lacks precision to day, dwc:day is not populated; if dwc:eventDate is a range spanning more than one month, or lacks precision to month, dwc;month is not populated; if dwc:eventDate is a range spanning more than one year dwc:year is not populated; dwc;startDayOfYear and endDayOfYear are to be filled in if they can be unambiguously interpreted from dwc:eventDate noting that for ranges that span a year boundary, startDayOfYear may be larger than endDayOfYear.
I would prefer
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN EMPTY terms dwc:year, dwc:month, dwc:day, dwc:startDayOfYear, dwc:endDayOfYear if any were unambiguously interpreted from values in dwc:eventDate, otherwise NOT_AMENDED
but then the implementation would then be dependent on the Notes, which to me is a no no. What about something like-
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN dwc:day if it is EMPTY and dwc:eventDate can be resolved to a single day, FILLED_IN dwc:month if it is EMPTY and dwc:eventDate can be resolved to a single month, FILLED_IN dwc:year if it is EMPTY and dwc:eventDate can be resolved to a single year, FILLED_IN dwc:startDayOfYear and dwc:endDayOfYear if they are EMPTY and they can be unambiguously interpreted from dwc:eventDate; otherwise NOT_AMENDED.
?
The spanning years Note is then OK.
See also the discussion in #67 and notes from TG2 call by @ArthurChapman therin https://github.com/tdwg/bdq/issues/67#issuecomment-1588185053
Working from @Tasilee's comment and the notes in https://github.com/tdwg/bdq/issues/67#issuecomment-1588185053 how about (though we could restructure following the precision of eventDate as in https://github.com/tdwg/bdq/issues/67#issuecomment-1588185053 instead):
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN if any of the following EMPTY values are populated: 1) day from dwc:eventDate is proposed as the value for dwc:day if dwc:day is EMPTY and dwc:eventDate has a precision of a day or finer, 2) month from dwc:eventDate is proposed as the value for dwc:month if dwc:month is EMPTY and dwc:eventDate has a precision of a single month, 3) year from dwc:eventDate is proposed as the value for dwc:year if dwc:year is EMPTY and dwc:eventDate has a precision of a single year or finer and is within a single year, 4) values for dwc:startDayOfYear and dwc:endDayOfYear are proposed if they are EMPTY and dwc:eventDate has a precision of a day or better; otherwise NOT_AMENDED.
I'm not sure if the form "precision of a day or finer" should be that or "precision of a day or finer and spans a single day or less" for day and month, more confident that the later should be the case for year. @tucotuco?
Looks better - getting there
My worry here are references to "day", "month" and "year". A first? My inclination would be to the form-
FILLED_IN 1) dwc:day from dwc:eventDate if dwc:day is EMPTY and dwc:eventDate has a precision of a day or finer...overall
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN 1) dwc:day from day from dwc:eventDate if dwc:day is EMPTY and dwc:eventDate has a precision of a day or finer, 2) dwc:month from dwc:eventDate if dwc:month is EMPTY and dwc:eventDate has a precision of a single month, 3) dwc:year from dwc:eventDate if dwc:year is EMPTY and dwc:eventDate has a precision of a single year or finer and is within a single year, 4) dwc:startDayOfYear and dwc:endDayOfYear if they are EMPTY and dwc:eventDate has a precision of a day or better; otherwise NOT_AMENDED.
@Tasilee I like that phrasing.
I am happy with that @Tasilee but note minor correction (1) dwc.day from dwc:eventDate.... (was 1) dwc:day from day from dwc:eventDate...
INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is EMPTY or contains an invalid value according to ISO 8601-1; FILLED_IN 1) dwc:day from dwc:eventDate if dwc:day is EMPTY and dwc:eventDate has a precision of a day or finer, 2) dwc:month from dwc:eventDate if dwc:month is EMPTY and dwc:eventDate has a precision of a single month, 3) dwc:year from dwc:eventDate if dwc:year is EMPTY and dwc:eventDate has a precision of a single year or finer and is within a single year, 4) dwc:startDayOfYear and dwc:endDayOfYear if they are EMPTY and dwc:eventDate has a precision of a day or better; otherwise NOT_AMENDED.
I have edited the Expected response (and Specification Last Updated) accordingly. NEEDS WORK??
I think NEEDS WORK can be turned off.
Further to a post elsewhere; we have in different tests:
Updated phrasing of the note, Amendments do not amend data, they propose changes (in Response.result, with a Response.status of FILLED_IN (only FILLED_IN in this case) or AMENDED). We need to be careful that we don't use these constants as verbs.
Added missing "or finer" from clause 2 for month. Now consistent with day and year clauses, otherwise implemetors would not fill in month if the event date had a precision of one day.
See discussion in #67 and issue #204, likely needs a little bit of work to be consistent with the related tests.