Implement record cleaner rule checking

[JimBacon]

The service will have to implement the following validation rules

• [x] Date
• [x] Spatial reference
• [x] Species name
• [x] Vice county

The service will have to implement the following verification rules

• [x] Period (Checks records are within the introduction and extinction date range)
• [x] Period within year (Checks species records are within correct time of year for its life stage)
• [x] Ancillary species list (Checks species are within a given list)
• [x] Without polygon (Checks the location of the record is within a list of grid squares or vice counties)

The service should implement the following verification rules for compatibility with the existing record cleaner

• [ ] Checklist (Checks values from any given field are within a given list)
• [ ] Polygon (Checks the location of a record is within a polygon)
• [ ] Regular expression (Checks values from any given field conform to a regular expression)

All the verification rules can reversed to perform the inverse function.

[JimBacon]

To date records have only been flagged when they fail a rule check. It has not been possible to distinguish between records which have passed tests and records for which no tests exist. @kitenetter, I recall you expressing a desire for this distinction to be made. How would you like this to be done?

We can anticipate that only a few species will have period and ancillary rules so an absence of these rules is not remarkable. Period within year (phenology) and Without polygon (ten km distribution) rules are more likely to be created for everything.

As a suggestion, we could change from pass/fail to pass/warn/fail as an overall result where 'warn' indicates either the phenology or tenkm rules are missing.

An accompanying list of messages can detail the reason for a warn or fail.

What do you think?

[kitenetter]

From our discussion today, here is a proposed way forward:

1. We work on the basis that for any taxon group, the list of taxa included is defined by the list of taxa that have ID difficulty rules.
2. When importing data into record cleaner, we run a validation check to find out of the imported data contains taxa for which no ID difficulty rules exist.
3. If no ID difficulty rule exists, the record is flagged as "No rule checks available" or similar.
4. If an ID difficulty rule does exist, the record is taken on to the next stage of checking against all rules that are available for that taxon.
5. Our guidance to rule creators should make it clear that ID difficulty rules need to exist for all taxa of interest.

[JimBacon]

@kitenetter I've just come back to the above, rediscovering it as I moved ID Difficulty checks in to the verification stage. Can you clarify the following:

• In step 3, when no ID Difficulty rule exists, is the record considered as passing or failing verification?
• In step 4, do you want any indication of missing rules?

[kitenetter]

@JimBacon

• step 3: definitely should not be counted as "passing", and ideally it's not "failing" either, it is a third category "no rule available". But if we need to stick to just two categories then this has to be considered a fail.
• step 4: it would be a nice-to-have to be able to indicate which rules are available or not, but this gets complex doesn't it? As well as ID difficulty, there could potentially be rules for distribution, phenology within year, year range, and 'rarity' (can't remember what this is actually called but it does exist for a few taxa). If you can see a way of communicating that it would be great, but otherwise a simple "no other rules available" if none exist?