search

tdwg / bdq

Biodiversity Data Quality (BDQ) Interest Group
https://github.com/tdwg/bdq
43 stars 7 forks source link

TG2-AMENDMENT_COUNTRYCODE_FROM_COORDINATES #73

Open iDigBioBot opened 6 years ago

iDigBioBot commented 6 years ago
TestField Value
GUID 8c5fe9c9-4ba9-49ef-b15a-9ccd0424e6ae
Label AMENDMENT_COUNTRYCODE_FROM_COORDINATES
Description Proposes an amendment to the value of dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code.
TestType Amendment
Darwin Core Class dcterms:Location
Information Elements ActedUpon dwc:countryCode
Information Elements Consulted dwc:decimalLatitude
dwc:decimalLongitude
Expected Response EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is bdq:Empty, or if dwc:countryCode is bdq:NotEmpty; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary in the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_AMENDED.
Data Quality Dimension Completeness
Term-Actions COUNTRYCODE_FROM_COORDINATES
Parameter(s) bdq:sourceAuthority
Source Authority bdq:sourceAuthority default = "10m-admin-1 boundaries UNION with Exclusive Economic Zones" {[https://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-admin-1-states-provinces/] spatial UNION [https://www.marineregions.org/downloads.php#marbound]}
Specification Last Updated 2024-08-18
Examples [dwc:decimalLatitude="-25.23", dwc:decimalLongitude="135.43", dwc:countryCode="": Response.status=FILLED_IN, Response.result=dwc:countryCode="AU", Response.comment="dwc:decimalLatitude and dwc:decimalLongitude contain interpretable values"]
[dwc:decimalLatitude="-38.280937", dwc:decimalLongitude="72.047790", dwc:countryCode="": Response.status=NOT_AMENDED, Response.result="", Response.comment="Coordinates do not fall in the boundary of any country"]
Source ALA, GBIF, iDigBio
References
Example Implementations (Mechanisms)
Link to Specification Source Code
Notes This amendment simply fills dwc:countryCode from a lookup of dwc:decimalLatitude and dwc:decimalLongitude. dwc:coordinateUncertaintyInMeters and dwc:coordinatePrecicision (if present) imply a buffer around the provided coordinates. Likewise, country polygons cannot be 100% accurate at all scales (Dooley 2005), so a spatial buffer of the country boundaries is also justified. Taking spatial buffers into account does however greatly complicate the logic and the implementation of this and related tests. In this test, a detection of multiple country codes by sampling within the buffer while possible, is not considered.
Tasilee commented 4 years ago

Has everyone had a look at the Expected response and the Notes?

tucotuco commented 4 years ago

I agree with @chicoreus that the EEZ is still relevant and a point in it should return the appropriate country code. The EE zone confers sovereign rights (if not sovereignty) that are pertinent to biodiversity questions, including exploitation rights and customs. Other than that, I am OK with the Expected Response.

For the Notes, I would replace "dwc:coordinatePrecicision and dwc:coordinateUncertaintyInMeters (if present) imply a potential displacement of the provided coordinates" with "The value of dwc:coordinateUncertaintyInMeters, and in special cases dwc:coordinatePrecision (if present) represent a spatial extent around the provided coordinates".

ArthurChapman commented 4 years ago

Yes - I am convinced that EEZ has to go back in. There still seems to be some suggestion that the Buffer should remain. I prefer a buffer if possible just because of the terrible resolution of country borders - and especially EEZ boundaries. And as @tucotuco is suggesting that point in polygon shoudl use a GIS anyway, then the buffer is not a difficult addition.

Tasilee commented 4 years ago

OK, enough sleep time I guess: @tucotuco - could you have another go at the Expected response and Notes. (Your turn again :). Once all happy and contented, I'll examine and edit the related issues.

tucotuco commented 4 years ago

Expected Response was:

"EXTERNAL_PREREQUISITES_NOT_MET if the data from bdq:sourceAuthority was not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:decimalLatitude and dwc:decimalLongitude are EMPTY or dwc:countryCode was not EMPTY; FILLED_IN if dwc:decimalLatitude and dwc:decimalLongitude resolve within a country boundary within bdq:sourceAuthority; otherwise NOT_CHANGED"

changed to

"EXTERNAL_PREREQUISITES_NOT_MET if the data from bdq:sourceAuthority were not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:decimalLatitude and dwc:decimalLongitude are EMPTY or dwc:countryCode is not EMPTY; FILLED_IN if dwc:decimalLatitude and dwc:decimalLongitude are inside a boundary defined by the union of the boundary for the country code plus it's Exclusive Economic Zone, if any, plus an exterior buffer given by bdq:spatialBufferMeters; otherwise NOT_CHANGED"

Tasilee commented 4 years ago

Thanks @tucotuco. That is an elegant phrasing of complex logic. I've slightly updated the Notes to reflect the Expected response but I need help.

tucotuco commented 4 years ago

Notes were:

"[bdq:sourceAuthority default = country shapes = xxxxxxxxxxxxxxx, bdq:spatialBufferMetres default = 3km].

dwc:coordinatePrecicision and dwc:coordinateUncertaintyInMeters (if present) imply a potential displacement of the provided coordinates. These two terms can be considered spatial buffers. Likewise, country polygons cannot be 100% accurate at all scales, so a spatial buffer of the country boundaries is justified. Taking the spatial buffers into account does however greatly complicate both the logic and the implementation of such tests. The same applies to potential conversion of the Spatial Reference System (SRS) of dwc:decimalLatitude and dwc:decimalLongitude to the SRS used in the bdq:sourceAuthority."

I suggest: "[bdq:sourceAuthority default for country shapes = spatial UNION of terrestrial boundaries from gadm.org and EEZs from marineregions.org, bdq:spatialBufferMeters default = 3000].

The justification for the combination of terrestral spatial boundaries with the boundaries of Exclusive Economic Zones (EEZs) is that the EEZ confers sovereign rights (if not sovereignty) that are pertinent to biodiversity questions, and depending on the taxon, it may be justified for a Location to be within such a zone. The justification for a spatial buffer outside of the boundaries is that the accuracy of spatial boundaries in source authorities depends on scale. Taking the spatial buffer into account lowers the probability of incorrectly assigning a country code based on the coordinates. With a buffer, there will be a zone on every shared boundary that will return more than one country code. In these cases, the country code should not be amended. The coordinates used for the test must use the same Coordinate Reference System (CRS) as the bdq:sourceAuthority, otherwise the datum shift could result in an error of up to more than 5.5 km, depending on the location, and thus an incorrect assignment of a country code.

Thinking it over one last time as we hone in on the best solution, it occurs to me that the Expected Response has to mention the case in which more than one country code would be return in the areas of overlapping buffers. Thus, I suggest to replace the previous proposal with:

"EXTERNAL_PREREQUISITES_NOT_MET if the data from bdq:sourceAuthority were not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:decimalLatitude and dwc:decimalLongitude are EMPTY or dwc:countryCode is not EMPTY; FILLED_IN if dwc:decimalLatitude and dwc:decimalLongitude are inside one and only one boundary defined by the union of the boundary for the country code plus it's Exclusive Economic Zone, if any, plus an exterior buffer given by bdq:spatialBufferMeters; otherwise NOT_CHANGED".

Tasilee commented 4 years ago

Very well considered @tucotuco. Many thanks. It makes sense to me, but I'm in no position to argue.

ArthurChapman commented 4 years ago

Looks good - an improvement

chicoreus commented 4 years ago

dwc:countryCode is not EMPTY should be NOT_RUN, rather than INTERNAL_PREFRQUISITES_NOT_MET.

chicoreus commented 4 years ago

See: https://github.com/kurator-org/ffdq-api/blob/master/src/main/java/org/datakurator/ffdq/model/ResultState.java

chicoreus commented 4 years ago

Otherwise looks good.

Tasilee commented 3 years ago

Maybe Expected Response should be

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority service was not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:decimalLatitude or dwc:decimalLongitude are EMPTY or dwc:countryCode is not EMPTY; AMENDED if dwc:decimalLatitude and dwc:decimalLongitude are inside a boundary defined by the union of the boundary for the country code plus it's Exclusive Economic Zone, if any, plus an exterior buffer given by bdq:spatialBufferMeters; otherwise NOT_CHANGED

?

ArthurChapman commented 3 years ago

Also makes sense - can't find crountry code if either one is missing

tucotuco commented 3 years ago

I wonder if we might add one more condition. What if the countryCode is NOT EMPTY, but it also is not a valid countryCode? Couldn't we supply the countryCode from the coordinates then?

On Mon, Nov 1, 2021 at 12:51 AM Arthur Chapman @.***> wrote:

Also makes sense - can't find crountry code if either one is missing

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tdwg/bdq/issues/73#issuecomment-955904000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADQ72573THTY4Y4F5HIPL3UJYFD3ANCNFSM4EKSNA7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Tasilee commented 3 years ago

@tucotuco's point raises again the links with our concept of paired required VALIDATIONs and also in this case, with 'partner' AMENDMENTs.

For this AMENDMENT for example, we do have two associated VALIDATIONs

20 (VALIDATION_COUNTRYCODE_NOTSTANDARD)

98 (VALIDATION_COUNTRYCODE_EMPTY)

and also the AMENDMENT

48 (AMENDMENT_COUNTRYCODE_STANDARDIZED)

However, @chicoreu ' s recent reminder that AMENDMENTs should be 'stand-alone', which I guess implies that we need to test for all possibilities designated by the Expected Response for each test.

In this Test case, @tucotuco 's speculation seems warranted. In creating basic test data to a template, I have for example, been testing scenarios such as these-

dwc:decimalLatitude="x", dwc:decimalLongitude="135.87" dwc:decimalLatitude="-25.23", dwc:decimalLongitude="135.43", dwc:countryCode="AX" dwc:decimalLatitude="-25.23", dwc:decimalLongitude="" dwc:decimalLatitude="-25.23", dwc:decimalLongitude="185.43"

Line 2 above currently produces, 'INTERNAL_PREREQUISITES_NOTMET' while it is pretty obvious that dwc:countryCode="AU" and we anticipate that #20 will pick that up.

As you can see, we also don't specifically handle 'invalid' lat/long so the test currently will run and 'NOT_CHANGED' will be the outcome. Or should we be aiming for 'INTERNAL_PREREQUISITES_NOTMET' in those cases?

Should the ER be

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority service was not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:decimalLatitude or dwc:decimalLongitude are EMPTY or are uninterpretable; AMENDED if dwc:decimalLatitude and dwc:decimalLongitude are inside a boundary defined by the union of the boundary for the country code plus it's Exclusive Economic Zone, if any, plus an exterior buffer given by bdq:spatialBufferMeters; otherwise NOT_CHANGED.

What this says is that dwc:decimalLatitude and dwc:decimalLongitude will amend dwc:countryCode regardless if it is EMPTY or if some value is there. You can figure out what occurred in the log of #20, #98 and #48.

Sigh

albenson-usgs commented 3 years ago

I'm curious what would happen in this one for areas that are outside of national jurisdiction? https://github.com/tdwg/dwc/issues/315#issuecomment-766844055

tucotuco commented 3 years ago

@albenson-usgs If the coordinates fall in international waters, no country code would be provided because there isn't one in the ISO standard - at least according to the specification. In implementation, an agreed upon user defined code such as XZ (https://unece.org/trade/cefact/unlocode-code-list-country-and-territory) could be returned.

tucotuco commented 3 years ago

This is going to sound radical, but I think there remain several issues that are confounding this discussion.

First, I don't think an amendment should be made if valid coordinates and a valid country code are provided. Georeferencing best practices argue that we have no basis for promoting one over another. Either could be in error, or both could be correct but not detectable as such from the source authority. This condition would result in NOT_CHANGED.

Second, the amendment must provide one country code or none. Because of this, buffers to the shapes for the country codes are irrelevant. They should NOT be used lest two answers be returned.

Third, I would think that if valid coordinates are provided, the internal prerequisites to be able to amend the country code are met, even it isn't amended.

Fourth, whether to use EE Zones seems to me to be up to the implementation, the test can be silent on that.

Thus, I'd say the Expected Result would be:

"EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority was not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:decimalLatitude or dwc:decimalLongitude are EMPTY or are uninterpretable; AMENDED if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary in the bdq:sourceAuthority attributable to a single valid country code; otherwise NOT_CHANGED, including if countryCode is already populated with a valid country code."

ArthurChapman commented 3 years ago

I remember we have had lots of discussion on this. I don't think we would ever propose an amendment if the coordinates were present and a country code was present. This would be picked up with the VALIDATION #50. WRT that test we also had a lot of discussion on a buffered boundary (to cater for uncertainty etc.) - so that test says "plus an exterior buffer given by bdq:spatialBufferInMeters" I can understand the need for a buffer in the VALIDATION test (#50).

As I read the current wording "INTERNAL_PREREQUISITES_NOT_MET if ..... dwc:countryCode is not EMPTY" - so @tucotuco the Amendment just reports INTERNAL_PREREQUISITES_NOT_MET if countryCode is already populated

With respect to EEZ, I see no reason to change the current wording (fits also with #50.

With buffering which fits with the VALIDATION test #50 - I am not so convinced of its need in the Amendment - I can see its need in the VALIDATION to cater for Uncertainty, etc. - I can't see how buffering works with the Amendment - as the wording is currently, the buffering seems to only apply to the EEZ.

In summary - I don't see the need for your suggested change @tuco - except for possibly removing the buffering

Tasilee commented 3 years ago

From @tucotuco version of the Expected Response - if dwc:countryCode is NOT_EMPTY and we have 'valid coordinates', then we could either be correcting the country code or overwriting it (= not changing it). But @tucotuco said "First, I don't think an amendment should be made if valid coordinates and a valid country code are provided". So there is some ambiguity in John's version of the Expected Response in that we are not making it obvious to programmers how the test would be implemented?

I tend to agree with @ArthurChapman in having the Expected Response as (do no harm :)

"EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority was not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:countryCode is NOT_EMPTY, dwc:decimalLatitude or dwc:decimalLongitude are EMPTY or are uninterpretable; AMENDED if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary in the bdq:sourceAuthority attributable to a single valid country code; otherwise NOT_CHANGED, including if countryCode is already populated with a valid country code."

This aligns with the many other similar Expected Responses. Unless you are suggesting we change them all to allow for 'overwriting' NOT_EMPTY term values? I'm not comfy with that at the moment.

tucotuco commented 3 years ago

@ArthurChapman I retract my suggestion to make an amendment if the countryCode is populated. That would be contrary to consistency in our approach. It arose from thinking that we COULD do it, but that doesn't mean we should, so sorry about that mini-circle. I think it is from having been away from the BDQ stuff too long.

The rest is about being able to determine why something is not changed. One way is to say the INTERNAL_PREREQUISTES_NOT_MET if the dwc:countryCode is populated. That makes sense to me, even though there is no change, because it is a specific internal condition of the data that results in there not being a change. It could also be done the way I did above in a cause for a NOT_CHANGED result. With the retraction of being able to make a change if the country code is not valid, I prefer the expression of the result in INTERNAL_PREREQUISITES_NOT_MET. Here's what my modified Expected Result would look like:

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority was not available; INTERNAL_PREREQUISITES_NOT_MET if either or both of dwc:decimalLatitude and dwc:decimalLongitude is EMPTY or uninterpretable, or if dwc:countryCode is NOT_EMPTY; AMENDED if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_CHANGED."

I think this is compatible with what Lee has distilled.

ArthurChapman commented 3 years ago

@tucotuco I am happy with that suggestion

tucotuco commented 3 years ago

Happy Arthurs make happy tucos.

On Wed, Nov 3, 2021 at 7:01 PM Arthur Chapman @.***> wrote:

@tucotuco https://github.com/tucotuco I am happy with that suggestion

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tdwg/bdq/issues/73#issuecomment-960187273, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADQ723BM7GCNIKOBQV7ANTUKGWKPANCNFSM4EKSNA7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Tasilee commented 3 years ago

It is redundant to include "INTERNAL_PREREQUISITES_NOT_MET if either or both of dwc:decimalLatitude and dwc:decimalLongitude is EMPTY or uninterpretable"

Surely it is sufficient to say "INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude are EMPTY or uninterpretable..."?

tucotuco commented 2 years ago

Yes.

On Sat, Nov 6, 2021 at 11:57 PM Lee Belbin @.***> wrote:

It is redundant to include "INTERNAL_PREREQUISITES_NOT_MET if either or both of dwc:decimalLatitude and dwc:decimalLongitude is EMPTY or uninterpretable"

Surely it is sufficient to say "INTERNAL_PREREQUISITES_NOT_MET if eitherdwc:decimalLatitude or dwc:decimalLongitude are EMPTY or uninterpretable..."?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tdwg/bdq/issues/73#issuecomment-962543542, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADQ724PXN3X3WXRSG6COO3UKXTK7ANCNFSM4EKSNA7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ArthurChapman commented 2 years ago

Agree

Tasilee commented 2 years ago

ER edited accordingly

ArthurChapman commented 2 years ago

We don't say what is AMENDED - suggest changing to:

| Expected Response | EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY or uninterpretable, or if dwc:countryCode is NOT_EMPTY; AMENDED dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_AMENDED. |

ArthurChapman commented 2 years ago

Further to my last comment - I notice that several AMENDMENTS don't say within the AMEND part of the Expected Response what is actually being amended (see also #60 and others) . We appear to be consistent, but are we correct.

Tasilee commented 2 years ago

I'd stick with present tense for everything possible..."AMEND dwc:xxx if ..."

ArthurChapman commented 2 years ago

Agree - that is one thing needing correcting - but that wasn't what I was raising - we don't say what is amended in most of the AMENDMENT tests - should we?

e.g. CURRENT (with AMEND changed

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY or uninterpretable, or if dwc:countryCode is NOT_EMPTY; AMEND if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_AMENDED.

or should it be EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY or uninterpretable, or if dwc:countryCode is NOT_EMPTY; AMEND dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_AMENDED.

i.e. adding what is being amended ( in this case "dwc:countryCode" within the AMEND area. If we do this, it would mean a change to most of the AMENDMENT tests.

Tasilee commented 2 years ago

My comment did suggest that we did need AMEND {target} if ....

ArthurChapman commented 2 years ago

I have checked a few AMENDMENTS and we don't have {target} in most of them. Perhaps we need to go through all the AMENDMENTS and add.

Tasilee commented 2 years ago

I'll make a start on Monday.

Tasilee commented 2 years ago

Looking at this AMENDMENT, do we need two bdq:sourceAuthorities? gadm and country codes?

ArthurChapman commented 2 years ago

Don't need a second bdq:sourceAuthority as country codes come from "ISO 3166-1-alpha-2 country code". Probably should change to

...AMENDED if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:sourceAuthority that is attributable to a single valid ISO 3166-1-alpha-2 country code ...

This would make consistent with other ISO type tests.

Tasilee commented 2 years ago

OK, done...but now I note that the proposed new form "AMEND xxx if ..., otherwise NOT_AMENDED" mixes tenses. Grrr....every time we lift a rock, there is a mess underneath it. Should we be using "AMEND ....; otherwise DON'T_AMEND"?!

ArthurChapman commented 2 years ago

I would stay with AMENDED and NOT_AMENDED

Tasilee commented 2 years ago

Changed "AMENDED" to "FILLED_IN" in accordance with discussions April 16.

Tasilee commented 2 years ago

@ArthurChapman and I have discussed my recent changes to

Also, this test no longer uses bdq:spatialBufferInMeters so the question needs to be asked if #50, #51 and #56. To be discussed at next Zoom. Also use of SRS? We need to be consistent.

ArthurChapman commented 1 year ago

We have dwc:coordinatePrecision and dwc:geodeticDatum in the Information Elements but they do not occur in the Expected Response?

Should these be deleted from the Information Elements or is our Expected Response deficient?

Note that dwc:coordinatePrecicision and dwc:coordinateUncertaintyInMeters occur in the Notes

ArthurChapman commented 1 year ago

in Expected Response bdq:sourceAuthority[countryShapes] changed to bdq:sourceAuthority[countryshapes] to conform with other similar terms under bdq:sourceAuthority

Tasilee commented 1 year ago

If we believe the Expected response is correct, then dwc:coordinatePrecision and dwc:geodeticDatum should not be information elements. Were we thinking a buffer?

chicoreus commented 1 year ago

The notes suggest using dwc:coordinatePrecision and dwc:geodeticDatum to specify a buffer, but also invoke the a buffer associated with the country boundary representation. But, we don't list a buffer as an element in the specification. So, in essence, we say both that a buffer should be used and that a buffer is not used...

Tasilee commented 1 year ago

Here are the tests that include buffer

50

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if one or more of dwc:decimalLatitude, dwc:decimalLongitude, or dwc:countryCode are EMPTY or invalid; COMPLIANT if the geographic coordinates fall on or within the boundary defined by the union of the boundary of the country from dwc:countryCode plus it's Exclusive Economic Zone, if any, plus an exterior buffer given by bdq:spatialBufferInMeters; otherwise NOT_COMPLIANT.

51

EXTERNAL_PREREQUISITES_NOT_MET if either bdq:sourceAuthority[taxonomyismarine] or bdq:sourceAuthority[geospatialland] are not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:dcientificName was EMPTY or the marine/non-marine status of the taxon is not interpretable from bdq:sourceAuthority[taxonomyismarine] or the values of dwc:decimalLatitude or dwc:decimalLongitude are EMPTY; COMPLIANT if the taxon marine/non-marine status from bdq:sourceAuthority[taxonomyismarine] matches the marine/non-marine status of dwc:decimalLatitude and dwc:decimalLongitude on the boundaries given by bdq:sourceAuthority[geospatialland] plus an exterior buffer given by bdq:spatialBufferInMeters; otherwise NOT_COMPLIANT.

56

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority was not available; INTERNAL_PREREQUISITES_NOT_MET if the values of dwc:decimalLatitude, dwc:decimalLongitude, or dwc:stateProvince are EMPTY or invalid; COMPLIANT if the geographic coordinates fall on or within the boundary from the bdq:sourceAuthority for the given dwc:stateProvince (after coordinate reference system transformations, if any, have been accounted for), or within the distance given by bdq:spatialBufferInMeters outside that boundary; otherwise NOT_COMPLIANT.

BUT, I don't think a buffer is justified in this AMENDMENT. We are not comparing two things. We are looking up dwc:countryCode from coordinates. Using dwc:coordinateUncertaintyInMeters or dwc:coordinatePrecision or dwc:geodeticDatum wouldn't help, unless the test matches to country - which it doesn't of course.

So, I am going to amend the specs accordingly, and the rest of you can argue the case...

Tasilee commented 1 year ago

Was this the intent?

Notes

This amendment simply fills dwc:countryCode from a lookup of dwc:decimalLatitude and dwc:decimalLongitude. dwc:coordinateUncertaintyInMeters and dwc:coordinatePrecicision (if present) imply a buffer around the provided coordinates. Likewise, country polygons cannot be 100% accurate at all scales (Dooley 2005), so a spatial buffer of the country boundaries is also justified. Taking spatial buffers into account does however greatly complicate the logic and the implementation of this and related tests. In this test, a detection of multiple country codes by sampling within the buffer while possible, is not considered.

ArthurChapman commented 1 year ago

Another issue with this test (and the others where we have bdq:sourceAuthority[xxxx]). If these are separated into different terms in the Vocabulary as suggested as a possibility in https://github.com/tdwg/bdq/issues/152#issuecomment-1610418820 - then we would need to change Parameter(s) to bdq:sourceAuthority to "bdq:sourceAuthority[countryshapes]" and Source Authority to "bdq:sourceAuthority default = "ADM1 boundaries" [https://gadm.org] UNION with "EEZs" [https://marineregions.org]"

ALSO - we have in bdq:souceAuthority "bdq:sourceAuthority[countryCode]" but this term doesn't occur anywhere in the test (or any other test) and is not in the Vocabulary - should this be bdq:sourceAuthority[countryshapes]???

Tasilee commented 1 year ago

I have updated the Source authority and description

ArthurChapman commented 1 year ago

I don't think there is anything outstanding wrt NEEDS WORK - other than a decision on how we treat bdq:sourceAuthority[countryshapes]