TG2-AMENDMENT_GEODETICDATUM_ASSUMEDDEFAULT

[iDigBioBot]

TestField	Value
GUID	7498ca76-c4d4-42e2-8103-acacccbdffa7
Label	AMENDMENT_GEODETICDATUM_ASSUMEDDEFAULT
Description	Proposes an amendment to fill in dwc:geodeticDatum using a prameterized value if the dwc:geodeticDatum is empty.
TestType	Amendment
Darwin Core Class	dcterms:Location
Information Elements ActedUpon	dwc:geodeticDatum
	dwc:coordinateUncertaintyInMeters
	dwc:decimalLatitude
	dwc:decimalLongitude
Information Elements Consulted
Expected Response	INTERNAL_PREREQUISITES_NOT_MET if dwc:geodeticDatum is bdq:NotEmpty; FILLED_IN dwc:geodeticDatum using the value of bdq:defaultGeodeticDatum, report FILLED_IN and, if dwc:coordinateUncertaintyInMeters, dwc:decimalLatitude and dwc:decimalLongitude are bdq:NotEmpty, amend the value of dwc:coordinateUncertaintyInMeters by adding the maximum datum shift between the specified bdq:defaultGeodeticDatum and any other datum at the provided dwc:decimalLatitude and dwc:decimalLongitude and instead report AMENDED; otherwise NOT_AMENDED.
Data Quality Dimension	Completeness
Term-Actions	GEODETICDATUM_ASSUMEDDEFAULT
Parameter(s)	bdq:defaultGeodeticDatum
Source Authority	bdq:defaultGeodeticDatum = "EPSG:4326" {[https://epsg.org/crs_4326/WGS-84.html]}
Specification Last Updated	2024-08-18
Examples	[dwc:geodeticDatum="[null]", dwc:decimalLatitude="-30.00", dwc:decimalLongitude="130.00", dwc:coordinateUncertaintyInMeters="50": Response.status=AMENDED, Response.result=dwc:geodeticDatum="EPSG:4326", dwc:coordinateUncertaintyInMeters="2836", Response.comment="dwc:godeticDatum is bdq:Empty so filled in with default and dwc:coordinateUncertaintyInMeters amended to maximum possible value"]
	[dwc:geodeticDatum="WGS84", dwc:decimalLatitude="", dwc:decimalLongitude="", dwc:coordinateUncertaintyInMeters="": Response.status=NOT_AMENDED, Response.result="", Response.comment="dwc:geodeticDatum contains an interpretable value"]
Source	ALA, GBIF
References	Maptiler (2019) EPSG.io. https://epsg.io/ EPSG (2024) About the EPSG Dataset. https://epsg.org/ Spatial Reference (2024) What is SpatialReference.org. https://spatialreference.org/ Geomatic Solutions (2018) Georepository. Version 9.0.0.1062. https://georepository.com/ Chapman AD and Wieczorek JR (2020) Georeferencing Best Practices. Copenhagen: GBIF Secretariat. https://doi.org/10.15468/doc-gg7h-s853 Wieczorek C and Wieczorek J (2021) Georeferencing Calculator. http://georeferencing.org/georefcalculator/gc.html
Example Implementations (Mechanisms)
Link to Specification Source Code
Notes	If the dwc:coordinateUncertaintyInMeters is bdq:Empty, not interpretable, or not valid, this amendment should not provide a dwc:coordinateUncertaintyInMeters. If the dwc:coordinateUncertaintyInMeters is bdqNotEmpty and is valid, this amendment should add to the dwc:coordinateUncertaintyInMeters the uncertainty contributed by the maximum datum shift at the given coordinates. Since different systems have differing requirements for what the default datum should be, it is left unspecified, but should match whatever the target datum is in AMENDMENT_COORDINATES_CONVERTED (620749b9-7d9c-4890-97d2-be3d1cde6da8). After the amendment is performed, the dwc:geodeticDatum field should be the assumed default datum as parameterized. An example implementation to determine the uncertainty added by asserting a default datum (datum shift) where a known datum is not declared can be found in datumshiftproj.py in the source code for the Georeferencing Calculator (Wieczorek & Wieczorek 2021). Included in the source code is a 5-degree grid of datum shifts from an unknown datum to WGS84.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet: See 7e0c0418-fe16-4a39-98bd-80e19d95b9d1 GEODETIC_DATUM_INVALID (VALIDATION_GEODETICDATUM_NOTSTANDARD) for the Validation method and MEASURE_GEODETICDATUM_SINGLE_COMPLETENESS for the measurement method. Potential ammendment ordering dependency, should run after an amendment that proposes a value for coordinateUncertantyInMeters

[chicoreus]

Parameter, as @ArthurChapman pointed out in #178 is which geodetic datum to use, not which vocabulary of geodetic datums to use.

bdq:sourceAuthority (default = http://epsg.io/)

Can only be interpreted as dwc:geodeticDatum is assumed to be "http://epsg.io" when not specified. This is clearly nonsensical, the default value should be WGS84, or the appropriate EPSG code, probably EPSG:4326.

Parameter is definitely needed but should be:

bdq:defaultGeodeticDatum (default=EPSG:4326)

or

bdq:defaultGeodeticDatum (default=https://epsg.io/4326)

With the notes specifying that the geodetic datum is taken from the http://epsg.io/ vocabulary....

[ArthurChapman]

@chicoreus This is the default sourceAuthority not the default value - so the value should be a value within that default source authority - whether it is ESPG:4326 (WGS84), EPSG:4674 (SIRGAS2000) or EPSG:7844 (GDA2020), etc. What you are testing for is a value in the field that is consistent with the a value in http://epsg.io/ - the same as you look for a country code in the ISO standard. On thinking about it - probably doesn't need to be Paramaterized and just accept that authority as THE authority.

[Tasilee]

@ArthurChapman - you raise a nice distinction: A source (in this case again unique), but then the default value, of which there can be plenty. I can see therefore two scenarios for Parameterised?

1. A choice of "specified source authority" and/or
2. A choice of a default value from a "specified source authority"

[ArthurChapman]

@Tasilee This is the case for all tests (especially NOTSTANDARD and STANDARDIZED tests) that use an external Standard - ISO, DCMI, in this case EPSG, of any Vocabulary. The vocabulary, standard, etc. is the bdq:sourceAuthority and you are checking to see if the value in the record is a valid record in the bdq:sourceAuthority (in the case of Validations) or can be amended to conform with a value in the bdq:sourceAuthority (in the case of Amendments). In nearly all cases, there is only one sourceAuthority (except as @chicoreus mentions with Taxon names), so there is no choice of sourceAuthority needed, only the choice of a value from that sourceAuthority. Those few cases where there is a choice of sourceAuthority (taxon names) brings in both your 1) and 2) above. Thus, I agree with @chicoreus that we don't need as many Paramaterized tests as we have previously so tagged. Unless @tucotuco has justifications for them that we have not thought of.

[chicoreus]

Phrasing of Amended text is unclear. Don't know how to interpret the comma:

AMENDED if the field dwc:geodeticDatum was EMPTY or was uninterpretable, the value of dwc:geodeticDatum was set to a predefined default value;

Perhaps

AMENDED if the field dwc:geodeticDatum was EMPTY or was uninterpretable and the value of dwc:geodeticDatum was set to a predefined default value;

[Tasilee]

I commented in @chicoreus email of September 1. We do need to standardize the phrasing.

[ArthurChapman]

I just checked the editing history - back in August last year the wording had a "therefore" after the comma AMENDED if the field dwc:geodeticDatum was EMPTY or was uninterpretable, therefore the value of dwc:geodeticDatum was set to a predefined default value;

[Tasilee]

Sounds good to me. I've edited the ER as issues to discuss/fix are building.

[ArthurChapman]

Needs discussing in conjunction with #60. I have not altered to conform with #59 as it is complicated. Parameterization should be removed I think in line with #59.

[Tasilee]

@ArthurChapman: I think this IS parameterised as it requires a default value for dwc:geodeticDatum? The Parameter isn't EPSG as a sourceAuthority, but which EPSG code is the default?

The Example should have the assumption, e.g., "dwc:geodeticDatum is NULL and has been assumed EPSG:4326"?

[Tasilee]

In line with recent discussions, there is ONE dq:sourceAuthority for the GDs, epsg.io, therefore that isn't a Parameter. In this case, why are we even including "specified source authority" other than than in References and possibly Notes to recommend use of EPSG codes? There is no need to access the source authority in the Expected response, simply to look up the Parameter value, e.g., EPSG:4326 as this is implementation dependent. There is no need for "EXTERNAL_PREREQUISITES_NOT_MET".

I would also suggest amending the example to "dwc:geodeticDatum is NULL, so amended to dwc:geodetciDatum EPSG:4326".

[chicoreus]

@Tasilee Parameterized as different users might want different defaults. This one is appropriate to have a parameter.

[ArthurChapman]

No, @Tasilee - you still have to go to an external source to look up the EPSG Code - so if the EPSG Server is down, then the EXTERNAL_PREREQUISITES_NOT_MET because the bdq:sourceAuthority (i.e. epsg.io) could not be found.

@chicoreus - it is not Paramaterized, because there is only one bdq:sourceAuthority, therefore there is no other Parameter that one can chose - we are only using the EPSG codes

[chicoreus]

@ArthurChapman needs a parameter, not bdq:sourceAuthority, but defaultCRS, with epsg:4326 as the default.

[ArthurChapman]

OK @chicoreus - I can agree - so if you are in Brazil, and the geodeticDatum is empty - you can set it to EPSG:4674 (i.e. the SIRGAS 2000 geodetic datum). I am happy with that. So with what @Tasilee said - you don't need to go to an External source - you set it as EPSG:4326 or some other value you add in the Parameter (for example EPSG:4674)

[Tasilee]

I disagree @ArthurChapman. When running the test, you don't need to look up anything at epsg.io. The Parameter is a specific GD!

[chicoreus]

@Tasilee I concur.

[Tasilee]

I am going to edit expected response, parameter, example and notes so please check.

[ArthurChapman]

I still have a few problems with this one (my suggested changes in bold

1. In Expected Response: Now "therefore the value of dwc:geodeticDatum was set to a default value; otherwise..." - I think this would better read "therefore the value of dwc:geodeticDatum was set to the parameterized EPSG value; otherwise..." It is not a default value - someone (Brazil could use EPSG:4674 for example) - it is a parameterized value - which if not set defaults to the default value (EPSG:4326). I think we should mentioon EPSG in this somewhere.
2. In the Notes curreently read "Since different systems have differing requirements for what the default datum should be, it is left unspecified, but should match whatever the target datum is in #43 . After the amendment is performed, the dwc:geodeticDatum field should be the assumed default datum as parameterized." I think should be changed to "Since different systems have differing requirements for what the parameterized datum should be, it is left unspecified, but should match whatever the target datum is in #43 . After the amendment is performed, the dwc:geodeticDatum field should be the assumed parameterized datum." It is not the default datum - it is the parameterized datum.

[Tasilee]

Thanks @ArthurChapman. It isn't a 'parameterized EPSG value" or a "parameterized datum". I think it is a parameter which takes an EPSG code/value. I'd prefer

Expected response:

INTERNAL_PREREQUISITES_NOT_MET if the Parameter is not set; AMENDED to the Parameter value if dwc:geodeticDatum was EMPTY; otherwise NOT_CHANGED

Notes:

The test is Parameterised as different installations will have different defaults for dwc:geodeticDatum. If dwc:geodeticDatum is EMPTY, the amendment will result in dwc:geodeticDatum taking the Parameter value.

In reviewing this test, some things become apparent. EXTERNAL_PREREQUISITES_NOT_MET is misspelled EXTERNAL_PREREQUESITES_NOT_MET in at least 10 tests. I'll fix those next.

Second, if dwc:geodeticDatum is NOT_EMPTY, we do nothing, regardless if it is interpretable or not.

[Tasilee]

Also...I don't think we need to use "the field...." more simply, for example, "dwc:..."

[ArthurChapman]

I agree with @Tasilee - this test needs rewording - I've looked at it in several ways but not yet come up with an answer.

Basically, if it is not EMPTY we do nothing so couldn't we say something like (a little redundancy)

INTERNAL_PREREQUISITES_NOT_MET if dwc:geodeticDatum is not EMPTY, or is an uninterpretable value or the Parameter is not set; AMENDED to the Parameter value if dwc:geodeticDatum was EMPTY or is not an interpretable value; otherwise NOT_CHANGED

Still not really happy with this.

[chicoreus]

I'm thinking perhaps as simple as:

Parameter: bdq:defaultGeodeticDatum, default value = 'EPSG:4326'

CHANGED to the value of bdq:defaultGeodeticDatum if dwc:geodeticDatum is EMPTY, otherwise NOT_CHANGED.

If we specify the default value for the parameter, then we don't have to worry about EXTERNAL_PREREQUISITES_NOT_MET, as the parameter will be available even if not specified, and there isn't an internal data value that would cause us to specify INTERNAL_PREREQUISITES_NOT_MET.

[ArthurChapman]

@chicoreus - that would seem OK

We need to allow the setting of a different parameter for bdq:defaultGeodeticDatum. We allowed for this to be parameterized because some countries legislate a datum for that country - e.g. Brazil and the South American Datum (SAD69(96) epsg:5534 from memory)

[Tasilee]

I like @chicoreus's comment. I like simple. @ArthurChapman: All the suggested Parameter is implying is that if it isn't set, use EPSG:4326. I've edited the Specifications accordingly.

[ArthurChapman]

Happy with that

[Tasilee]

Changed "AMENDED" to "FILLED_IN" in accordance with discussions April 16.

[chicoreus]

coordinateUncertaintyInMeters is listed as an information element and discussed in the notes, but not mentioned in the expected response.

[chicoreus]

The logic of the notes also results in a conflicting response.status - dwc:geodeticDatum would be FILLED_IN, but dwc:coordinateUncertaintyInMeters could be AMENDED, and the response structure does not allow for multiple status values.

[ArthurChapman]

Some issues - @tucotuco

1. If dwc:coordinateUncertainyInMeters is empty should we add something (5.2km) - but see notes
2. If dwc:coordinateUncertainyInMeters and we add a value should there be a default
3. Do we need to add dwc:decimalLatitude and dwc:decimalLongitude so we can determined dwc:coordinateUncertainyInMeters?
4. @Tasilee suggests taking dwc:coordinateUncertainyInMeters out of elements - I am reluctant.
5. remove this test - @chicoreus and I wish to retain

Needs further discussion at next ZOOM

[chicoreus]

Notes on discussion: AMENDED rather than FILLED_IN, as this test adds an assumption that downstream consumers need to understand the implications of. Likewise, should propose a change to the coordinateUncertaintyinMeters from the resulting implication of going from an unknown datum to the assumed default.

[tucotuco]

1. If dwc:coordinateUncertainyInMeters is empty should we add something (5.2km) - but see notes

No. We can not calculate the coordinateUncertaintyInMeters automatically and 5359 m would only be the minimum.

2. If dwc:coordinateUncertainyInMeters and we add a value should there be a default

The reasonable alternatives I see are a) to provide the default maximum contribution from an unknown datum (5359 m), b) do a lookup (example) for the uncertainty from an unknown datum based on the coordinates, or c) do a calculation (example) for the uncertainty from an unknown datum based on the coordinates. Option a) is simple, fast, and readily recognizable. Option b) is an estimate, but can bring down the average uncertainty from an unknown datum to 3200 m and the best case uncertainty for an unknown datum to 1554 m. Option c) is similar to option b) but without being estimated.

3. Do we need to add dwc:decimalLatitude and dwc:decimalLongitude so we can determined dwc:coordinateUncertainyInMeters?

For case a) above, no. For cases b) and c) above, yes.

4. @Tasilee suggests taking dwc:coordinateUncertainyInMeters out of elements - I am reluctant.

I think it is a mistake to take it out. If we are amending the datum, we have to account for the consequence if we can.

5. remove this test - @chicoreus and I wish to retain

The implications of making this amendment are bothersome indeed. The reality is that aggregators are already doing it and are unlikely to be convinced otherwise. Not doing it would make the number of mappable records, and thus visualizations, much less impressive. For example, from the 2022-07-14 snapshot of GBIF with 2,094,473,711 (93.8% of) Occurrence records having coordinates, 294,700,318 (13.2%) would be left out.

What will people look at before trying to use the data? The most likely scenario is that the coordinates will be used and WGS84 assumed without further inspection. This amendment will make no difference in this case.

For those slightly more demanding, having a datum where there really ought not to be one might make the record look like it has higher quality than it really has, and thus make a certain class of people who are unaware of uncertainty or who blithely ignore it more likely to use records that do not have sufficient quality for their purpose. This is bad, but since the aggregators are going to provide all coordinates either transformed to WGS84, or as if they already were in WGS84, there is no preventing it by not having this amendment.

For those who try to be diligent, the augmentation of the uncertainty would be a helpful indicator to either throw the records out or inspect them further (actually georeference them with the locality description) to see if they can come up with a refined uncertainty specific enough to be of use.

Since the test (with uncertainty augmentation where possible) can help those who are trying to do rigorous science and won't make any difference to those who aren't, I think it is worth keeping.

[Tasilee]

Thanks @tucotuco but would you prefer your (2) or (3)? Either way, we add dwc:latitude and dwc:longitude to the Information Elements, and the Expected Response becomes something like

If dwc:geodeticDatum is EMPTY, FILLED_IN the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum; otherwise NOT_AMENDED. If dwc:coordinateUncertaintyInMeters is EMPTY and dwc:latitude and dwc:longitude are NOT_EMPTY, FILLED_IN the value of dwc:coordinateUncertaintyInMeters with an estimate based on the values of bdq:defaultGeodeticDatum, dwc:latitude and dwc:longitude; otherwise NOT_AMENDED

This is a bit odd, but I guess having two parts makes it obvious what the intent is. Note that I changed the ordering to "If...then..." as it seems simpler. I suspect this applies to other AMENDMENTs.

[tucotuco]

If I were doing the implementation, I would want to do the best possible, which would be c). That would also avoid the issue I just realized a) and b) have, which is that the maximum uncertainty depends on the bdq:defaultGeodeticDatum. The value 5359 m applies to WGS84 (and others that are essentially indistinguishable from WGS84, but with different names). The estimate method b) would have to have an entirely different lookup table from the WGS84 one I provided. The calculation method would just work, always.

[ArthurChapman]

I full agree @tucotuco that 3) is the only one that makes sense to me. Another reason, is if you are not using WGS84 as your default but another datum (South American Datum in Brazil etc.), then that maximum would be quite different.

[Tasilee]

@ArthurChapman 's comment occurred to me as well, but I did wonder at the time about how common would be an empty geodeticDatum with a default of a regional datum? I guess if you were intentionally transforming say EPSG:4326 say to AGD2020, you would want DEC:geodeticDatum to not be empty.

So what would the Expected Response look like?

[tucotuco]

@ArthurChapman Right, different maximum AND different values as a function of location.

[chicoreus]

(1) If dwc:geodeticDatum is EMPTY, fill in the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum, report AMENDED and continue to (2); otherwise NOT_AMENDED. (2) If dwc:coordinateUncertaintyInMeters is NOT_EMPTY and dwc:latitude and dwc:longitude are NOT_EMPTY, amend the value of dwc:coordinateUncertaintyInMeters by adding to the existing value the maximum possible difference between an arbitrary datum and the specified bdq:defaultGeodeticDatum for the provided dwc:latitude and dwc:longitude.

[ArthurChapman]

If dwc:geodeticDatum is EMPTY, fill in the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum, report AMENDED and If dwc:coordinateUncertaintyInMeters is NOT_EMPTY and dwc:latitude and dwc:longitude are NOT_EMPTY, amend the value of dwc:coordinateUncertaintyInMeters by adding to the existing value the maximum possible difference between an arbitrary datum and the specified bdq:defaultGeodeticDatum for the provided dwc:latitude and dwc:longitude; otherwise NOT_AMENDED.

[tucotuco]

If dwc:geodeticDatum is EMPTY, fill in the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum, report AMENDED and, if dwc:coordinateUncertaintyInMeters, dwc:decimalLatitude and dwc:decimalLongitude are NOT_EMPTY, amend the value of dwc:coordinateUncertaintyInMeters by adding the maximum datum shift between the specified bdq:defaultGeodeticDatum and any other datum at the provided dwc:decimalLatitude and dwc:decimalLongitude; otherwise NOT_AMENDED.

[chicoreus]

Alternative that allows for either amended or filled in:

If dwc:geodeticDatum is EMPTY, fill in the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum, report FILLED_IN and, if dwc:coordinateUncertaintyInMeters, dwc:decimalLatitude and dwc:decimalLongitude are NOT_EMPTY, amend the value of dwc:coordinateUncertaintyInMeters by adding the maximum datum shift between the specified bdq:defaultGeodeticDatum and any other datum at the provided dwc:decimalLatitude and dwc:decimalLongitude and report AMENDED; otherwise NOT_AMENDED.

[chicoreus]

Or:

If dwc:geodeticDatum is EMPTY, fill in the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum, report FILLED_IN and, if dwc:coordinateUncertaintyInMeters, dwc:decimalLatitude and dwc:decimalLongitude are NOT_EMPTY, amend the value of dwc:coordinateUncertaintyInMeters by adding the maximum datum shift between the specified bdq:defaultGeodeticDatum and any other datum at the provided dwc:decimalLatitude and dwc:decimalLongitude and instead report AMENDED; otherwise NOT_AMENDED.

[tucotuco]

I have updated the Notes to give implementors some guidance about how to go about the datum shift calculation.

[Tasilee]

Added to Notes: "This test will fail if there are leading or trailing white space or non-printing characters." and amended Description to address a not empty dwc:coordinateUncertaintyInMeters.

[Tasilee]

I have been checking over additions to the Notes on Validations dependent on a Vocabulary adding "This test will fail if there are leading or trailing white space or non-printing characters." BUT, I cannot see that this AMENDMENT requires either the VOCABULARY tag or the new Notes text added.

[Tasilee]

The Expected Response here does not conform to our usual phrasing. Could I suggest a change from

If dwc:geodeticDatum is EMPTY, fill in the value of dwc:geodeticDatum with the value of bdq:defaultGeodeticDatum, report FILLED_IN and, if dwc:coordinateUncertaintyInMeters, dwc:decimalLatitude and dwc:decimalLongitude are NOT_EMPTY, amend the value of dwc:coordinateUncertaintyInMeters by adding the maximum datum shift between the specified bdq:defaultGeodeticDatum and any other datum at the provided dwc:decimalLatitude and dwc:decimalLongitude and instead report AMENDED; otherwise NOT_AMENDED.

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:geodeticDatum is EMPTY; FILLED_IN the value of dwc:geodeticDatum using the value of bdq:defaultGeodeticDatum; AMEND the value of dwc:coordinateUncertaintyInMeters by adding the maximum datum shift between the specified bdq:defaultGeodeticDatum and any other datum at the provided dwc:decimalLatitude and dwc:decimalLongitude if dwc:coordinateUncertaintyInMeters, dwc:decimalLatitude and dwc:decimalLongitude are NOT_EMPTY; otherwise NOT_AMENDED.

I realize the "report' bit is missing but we are assuming for other tests the activation of the phrase "FILLED-IN" or "AMENDED" or "NOT_AMENDED" etc apply as a report. So, if the FILLED-IN and AMENDED (or NOT_AMENDED) phrases are activated, they are reported?

[chicoreus]

On Thu, 02 Feb 2023 14:33:24 -0800 Lee Belbin @.***> wrote:

INTERNAL_PREREQUISITES_NOT_MET if dwc:geodeticDatum is EMPTY;

This clause doesn't belong. The purpose of this amendment is to supply a value for geodeticDatum when none is provided.

Probably this is why the current phrasing is unfamiliar, the case is unusual.

We likely want to keep the current phrasing, it did take some thinking through to get it refined into that state.

[tucotuco]

I think we want to keep the current phrasing. The recommended fix is not correct.

[ArthurChapman]

Agree with @chicoreus. Suggestion by @Tasilee it is not correct that INTERNAL_PREREQUISITES_NOT_MET if the dwc:geodeticDatum is EMPTY - instead, you fill it in with the default. I looked at trying to reword it to make it a bit clearer, but can't do better than what we have.