Validation issues

[chrisvanswaay]

@JimBacon @johnvanbreda @Gary-van-Breda @DavidRoy @CrisSevilleja Not sure who works on this, but there are some strange things happening with validation. This sample contains a record of C minimus, which is not possible here:

Now I went into the validation and rejected this observation (and the others of that species in this place). When I search in the validation I cannot find it anymore:

This is good, it means the status of the species has changed.

But it is still in the downloads via https://butterfly-monitoring.net/downloads and keeps coming back to us via the api. So I removed them yesterday from our database, today they are back via the api.

Two possible solutions:

• add the verification status to all records in download (and api?), so we can filter on them here.
• Or even better: remove rejected records from the api and download.

Hope this can be done, as I am now working my way through all records via our autovalidation rules, and then these come up. Ideally I want to change the status on the source (which is the eBMS) and that then the api takes care it get adapted at our place, and then both databases are fine.

[chrisvanswaay]

PS Alternatively I can validate at our end, would be fine for me, but then the error would remain in the eBMS, and best is to correct at the source database and all sync via the api.

[DavidRoy]

@johnvanbreda I think this is one for you and the requirement is to add record status to the API so that Dutch BC can filter records accordingly.

@chrisvanswaay by default, the verificatoin page only shows 'pending' records (i.e. those needing review). You can switch to other record status using the Status drop-down list and select 'Not accepted' or 'All records' etc

[chrisvanswaay]

thanks @DavidRoy, never too old to learn.

[chrisvanswaay]

@johnvanbreda Any idea when you will have time for this? I have to use the data soon for a project in NL, so I have to validate to get a good dataset. Best is to do this on the eBMS and that it then gets updated via the api on our side (then all database improve), but if repairing takes you too long I will do the validation on our side (I already track the records with mistakes using our validation system), but that means that the mistakes stay at the eBMS side, and I won't be able to find them back later. So although it is more work for me to validate at eBMS, I would still prefer that, but I have to start soon. Please let me know when you think you can do this, so I can decide how to move on.

[johnvanbreda]

@chrisvanswaay if you are using the Elasticsearch API still, then each document contains a section called identification, with values for verification_status and verification_substatus. Verification status can be: C - not accepted or rejected V - verified (accepted) R - rejected (not accepted).

Verification_substatus gives more detail - combine the 2 and you get: C0 - pending verification C3 - plausible V1 - accepted as correct V2 - accepted as considered correct R4 - rejected as unable to verify R5 - rejected as incorrect.

@DavidRoy should the default occurrences downloads exclude rejected records, or include the verification status in a column?

[chrisvanswaay]

Thanks @johnvanbreda. If the verification_status and substatus is now in the api, then please also include the rejected records (which we will filter out at our end). Monday my colleague will deal with this.

[johnvanbreda]

They should have always been present, but perhaps their purpose was not clear. There is some documentation on all the fields in the Elasticsearch documents returned by the API at https://github.com/Indicia-Team/support_files/blob/master/Elasticsearch/docs/occurrences-document-structure.md.

[chrisvanswaay]

Thanks, also for the documentation, I will check Monday with my colleague.

[chrisvanswaay]

@johnvanbreda @DavidRoy Found out it is more complicated than I thought (hoped?), sorry for not fully understanding all ins-and-outs. I can validate (and use the Dutch system for that, so more precise than the eBMS validation rules for NL), and this validation status is automatically updated in our own Vlinderstichting-database. But how to get it to eBMS? I presume your api works only one way, so the validations status cannot be update automatically? Any other way? In worst case I can send over a file with observation_id and a status once a year or so.

[chrisvanswaay]

@johnvanbreda Is there a way to get from https://butterfly-monitoring.net/mydata/samples/details?sample_id=23638350 (where the C palaemon must be wrong) to the verification page without having to somehow copy the species, date and recorder manually in https://butterfly-monitoring.net/verification ? I have to validate more than 2000 records with problems, and it too time consuming to do this by copying details, then I just start to validate only our side of the system (which feels like a pity). If e.g. the species names in https://butterfly-monitoring.net/mydata/samples/details?sample_id=23638350 would be linked directly to the validation place (so if I clicked the species name C palaemon I would immediately be taken to the right place where I could validate on the eBMS in one click) then I would do this (and the validationresult of eBMS would be tranformed to our database). I really have to start validating very soon, and this is the last chance I see to do that directly on the eBMS website, which I would prefer, but it should be able fast.

[DavidRoy]

@chrisvanswaay if you know the occurrence of sample ID you can enter this in the verification grid and it will take you to the record(s). Do you want a quick zoom to look at the best way of doing this

[chrisvanswaay]

@DavidRoy The sample id is there (23638350), but I don't see the option to enter that on the verification page. And I don't know the observation_id (which you can enter). I tried entering the sampleid in the but that gives no hits:

[DavidRoy]

@chrisvanswaay click on the spanner icon to change the columns you see. One option is sampleID

[DavidRoy]

@johnvanbreda although this won't work as the species occurrence as linked to sub-samples?

[chrisvanswaay]

@DavidRoy Ah, didn't know that. At least that helps a bit, as I can change the columns and include sample id, but it still gives no hits (as @johnvanbreda already feared).

[DavidRoy]

@chrisvanswaay do you have a list of the occurrence_ids where you want to change the status?

[chrisvanswaay]

@DavidRoy Are they in the api? If yes, then I could ask a colleage to make them visible to me, at the moment I can only see the sampleid in our screen.

[DavidRoy]

@chrisvanswaay I'd be very surpised if they are not

[chrisvanswaay]

@DavidRoy OK, then I will ask my colleague to get them into my validation screen. Thanks!

[johnvanbreda]

A few extra possibilities:

1. If you enter a sample ID (or a parent sample ID) in to the search box then press return, it will search for all records that contain that ID in any of the fields (including the ID, sample ID and parent sample ID). So you will find the records OK, but you may in some cases find other records as well.
2. To ensure that the search only searches the parent sample ID field you can type something like this into the search box:

   event.parent_event_id:23638350

[chrisvanswaay]

@johnvanbreda The first one already worked, thanks.

[CrisSevilleja]

Can this issue be closed @chrisvanswaay ?

[chrisvanswaay]

yes, sorry.

[chrisvanswaay]

closed