Enable the use of local record cleaner rules

[kitenetter]

The request is for a verifier to be able to upload a set of locally relevant rules that can be used to filter records, e.g. where local phenology is different to the phenology defined nationally.

@johnvanbreda has suggested 3 possibilities:

1. Modify the existing time of year rules so that they are linked to a geographic region, e.g. England, or a VC boundary.
2. Allow the user to upload a set of rules in a spreadsheet. The rules are then parsed to create a long, complex Elasticsearch filter which returns the records that pass or fail as required.
3. Allow the user to upload a set of rules in a spreadsheet. Then, run a script within Elasticsearch that finds the records of each given taxon within the verifier’s VC boundary, and sets a custom flag in the ES data according to the pass/fail result. This can then be used in Elasticsearch filters, or added as a flag column to the data grid. This gives the closest end-result to simply allowing verifiers to define their own rules.

The above objects are in order of flexibility, with 1 being the least flexible and simplest to implement and 3 being the most flexible and hardest to implement. The main issue with both #2 and #3 will be the impact on Elasticsearch performance.

A benefit of options 2 and 3 is that they allow county-based verifiers to be independent and use local rules or not as they choose.

Initial testing to be carried out on the dev server to further explore options 2 and 3 and to see what impact they might have on performance.

[johnvanbreda]

Coding for this done, will set up on the test environment...

[johnvanbreda]

This is now available to test on the main verification page on the test iRecord install.

[kitenetter]

@johnvanbreda great to see this. Have tried it out this evening.

1. Setting up new rules: Geographic limits When choosing a named place, such as a vice-county, it would be good if the relevant boundary was displayed on the map.

2. Setting up the spreadsheet I have a number of questions about the spreadsheet, probably easiest to talk these through over zoom.

3. Importing a spreadsheet I set up a simple spreadsheet to list rare species, that looked like this:

On trying to import it I got an error message:

Why is it not recognising the taxon id and rule type columns?

[johnvanbreda]

@kitenetter I've implemented the location boundaries on the map for point 1. For point 3 can you send me the spreadsheet?

Would tomorrow be a good day for a Zoom?

[kitenetter]

@johnvanbreda here is the spreadsheet: Berks rare moths rules.xlsx

I can do any time tomorrow, let me know when you prefer and I'll send a zoom link.

[johnvanbreda]

• I've fixed the issues uploading the spreadsheet.
• Note that the rule type should now be "species recorded" not "species_recorded".
• The format for reverse rule is now "yes".
• Answering a question about how "limit to" fields interact when specified in both a rule and a ruleset. The answer is that both checks must pass. So, if a rule has limit to grid ref SY1234 and a ruleset has limit to grid ref SY2345 then the rule will never actually run.

[johnvanbreda]

@kitenetter whenever you are ready I can add the Custom Rules button to the live iRecord verification pages, assuming you are happy that it's been tested enough.

[kitenetter]

No, it needs testing and guidance. That is in progress but will probably take a while yet.

[kitenetter]

@johnvanbreda last time I looked I could no longer find the test implementation of this, is it still available?