Checking reports for verification

[andrewvanbreda]

3.1 FIT Count target flower checking report – not yet done, Priority: high, but not needed until end of season

3.2 FIT Count insect group checking report – not yet done, Priority: high, but not needed until end of season

3.3 1km square survey flower checking report – not yet done, Priority: high, but not needed until end of season

[andrewvanbreda]

@kitenetter I have a version of 3.1 working on my machine now. This is a report grid. Towards the right of each grid row is a free textbox, with a submit link . You just fill this in and click Submit and it is saved (with a confirmation, no page reload required)

I think looking at 3.2, 3.3 these are slightly more tricky. Am I right in thinking these corrections must be a selection from a species list only? Are you expecting this to be an Indicia style autocomplete box? (as drop-down would be too long) For 3.2 is that the same insect list that is displayed on the page? For 3.3 I think that is to be from the UK Master List. Although this is described as a "1km square survey flower checking report" I assume that is referring to pan-trap, not 1km FIT.

[andrewvanbreda]

@kitenetter For 3.2, 3.3 the requirement is to have the correction selection from a species list. For my current code (currently on my machine) I am making the assumption that the verifier is required to select a name from the species list, but this name is saved as basic text to the database.

I want to check this is ok as there are some advantages/disadvantages of this.

• Easier to implement, for instance I can just load the text out of the database to display on screen when viewing an existing correction.
• The alternative would be to save the ID number of the taxon in the database. So this means we would be relying on the data in UK Master List to interpret that number. So a change to the UK master list would mean the correction also changes. Whereas text storage, the correction will always remain as the verifier entered it.
• As the ID number is saved in an attribute, it is not hard-linked to the UK Master Link, so there are no enforcement rules. So if a species was deleted from the UK Master List, then our ID number that originally pointed to that would have no meaning at all (other than pointing to a deleted item). A hard text save would not have that issue.

Disadvantages of saving the text.

• We can't query to find synonyms very well. Perhaps the main relevance of this would be if we needed to display Welsh names.
• We don't pick up changes to the UK Master List

Can you let me know your thoughts on this. If the corrections need to be shown in Welsh, then I guess that is a big question about this. Thanks

[kitenetter]

Just having a look at this, but I can say straight away that we don't need Welsh names.

[kitenetter]

Lists: For 3.2, the box that is filled in if a correction is needed should provide a dropdown, using the ten names given is species list ID 221. Doesn't necessarily need to be an auto-complete dropdown.

For 3.3, it is the plant occurrences from the pan-trap samples that we are talking about, and these are ideally to be selected from the UK Master List.

For both, I think it is preferable to store the ID number for the relevant entry in the species dictionary. Am I correct in thinking that what actually gets stored is the taxa_taxon_list_id?

Names are never deleted from the UK Master List, and if a name changes it is linked as the current recommended name, so the old name can be displayed either as the name that was entered or the name that is now the recommended one. I'd prefer to keep that functionality for 3.3 if we can.

For 3.2 it is probably less important, as the short list of ten insect groups is specific to PoMS and we don't intend to change it (and synonyms etc. are not an issue). But for consistency I would still prefer to use the taxa_taxon_list_id rather than just storing the text.

[andrewvanbreda]

@kitenetter Ok, I will have a look at implementing these changes. For the UK master list item it will need to be an autocomplete else too many entries.

[andrewvanbreda]

@kitenetter I have implemented the ID saved to the database on my machine. How do you want this to work one the correction is made. At the moment on my machine the correction remains editable on the grid so it can be changed again if needed.

[kitenetter]

That sounds fine Andy. I don't think we often need to make an edit more than once but it's good to have the option to do so.

[andrewvanbreda]

@kitenetter This is what the pages currently look like. Don't click on the Submit buttons at the moment, that is not working on the site at the minute (I am not sure why, but is working on my system). Same also with the autocompletes on the last 2 forms, working on my machine, but not on that site yet. But anyway, let me know if you have any thoughts on the visual side of this at the moment

https://test-ceh-ukpoms.pantheonsite.io/fit-count-target-flower-checking https://test-ceh-ukpoms.pantheonsite.io/fit-count-insect-group-checking-report https://test-ceh-ukpoms.pantheonsite.io/1km-square-survey-flower-checking-report

[andrewvanbreda]

@BirenRathod Hi Biren, I am very confused about something I am seeing. Could you send me these two results please

select sample_id,survey_id from indicia.cache_occurrences_functional where id = 15619068;

select id,survey_id from indicia.samples where id in (select sample_id from indicia.occurrences where id = 15619068);

Cheers

[BirenRathod]

@andrewvanbreda Both queries are returned with this 9707805 64 result.

[andrewvanbreda]

@BirenRathod Thanks Biren, very interesting result (and confusing).

[andrewvanbreda]

@kitenetter The Target Flower Checking report is working if you needed me to put that live? (subject to you checking that it is what you expect).

The other two would be ready but I there is a bug I am trying to fix where sorting a column or moving grid page breaks it. (the reason for the breakage is related to these two reports using an autocomplete)

[andrewvanbreda]

@BirenRathod Hi Biren,

Could you pull this new report please whenever you get a chance. No in use yet, so is safe pull. projects/PoMS/taxon_list.xml

Thanks

[BirenRathod]

@andrewvanbreda Gone live now.

[andrewvanbreda]

@BirenRathod Thanks Biren :)

[andrewvanbreda]

@kitenetter This are ready (from my point of view) to go live whenever you are ready, you can check the test site versions here

https://test-ceh-ukpoms.pantheonsite.io/fit-count-target-flower-checking https://test-ceh-ukpoms.pantheonsite.io/fit-count-insect-group-checking-report https://test-ceh-ukpoms.pantheonsite.io/1km-square-survey-flower-checking-report

[andrewvanbreda]

@kitenetter

Actually two other considerations in regards to this,

1. Whenever you are ready, let me know how the roles need setting up in regards to these pages.

2. Also for 1km flower checking, the request with to limit this to plants. You will need to let me know how this should be filtered exactly. To do this I am currently limiting to the following taxon groups from the UK Master List (I did this as this simply matches NPMS)

clubmoss conifer fern horsetail, flowering plant ginkgo, quillwort stonewort

[andrewvanbreda]

@kitenetter I think there was a mistake in the setup of the "FIT Count insect group checking report" and "FIT Count target flower checking". That mistake was those pages were note getting data from the app. I have now added that survey id to the configuration for those two pages.

[kitenetter]

Responses to the above:

Roles These three pages need to be available for the UKCEH staff role.

"Taxon groups" Your list is correct for the flowers (in fact I would only expect PoMS to produce "flowering plant" records, but it won't do any harm to include the other categories).

General query Is it correct that the way these pages work is that you first add a corrected name, and then click the "Submit" link to save it? (Not sure if is is linked to live data, so I haven't tested this.)

When you click "Submit", does the row get removed from the grid? (Not sure whether that would be a good thing or not.)

If no correcton is needed, presumably there is no need to click "Submit"?

FIT Count target flower checking This needs to be filtered to include FIT Counts from the website public survey, FIT Counts from the website 1 km survey, and the FIT Count app.

It should only show samples that actually have target flower photos attached to the sample.

It should only show samples that fall within the UK + Isle of Man + Channel Islands.

Under normal circumstances, we will only need to see the "current year" of samples, but it is likely that samples from 2021, for instance, will be checked in early 2022. If it is easier to use the same filter box as on other pages, that would allow us to choose the year we wish to work on, that would be an acceptable solution, and would allow for the hypothetical situation where we might be carrying out corrections more than a year after the samples were added.

The number of samples shown as the total at the bottom of the grid (8736) seems unrealistically high, at least if it is only including the current year - is it just miscounting things?

FIT Count insect group checking This needs to be filtered to include FIT Counts from the website public survey, the website 1 km survey, and the FIT Count app.

It should only show occurrences that actually have insect group photos attached to the occurrence.

It should only show samples that fall within the UK + Isle of Man + Channel Islands.

I don't think we need to display the columns "Other names for entered species" or "Other names for corrected species".

The dropdown for adding a corrected name should only include the UK FIT Count taxon groups.

As above, under normal circumstances, we will only need to see the "current year" of occurrences, but it is likely that samples from 2021, for instance, will be checked in early 2022.

The number of samples shown as the total at the bottom of the grid (28437) seems unrealistically high, especailly if it is only including the current year - is it just miscounting things?

1 km survey flower checking This needs to be filtered to just the website 1 km survey.

It is currently showing insect occurrences, and needs to be filtered to the plant taxon groups listed in your previous comment.

It should only show occurrences that actually have flower photos attached to the occurrence.

It should only show samples that fall within the UK + Isle of Man + Channel Islands.

I don't think we need to display the columns "Other names for entered species" or "Other names for corrected species".

The dropdown for adding a corrected name should ideally link to the plant names from the UK master species dictionary, but if that is complicated to achieve we can stick with a free text box.

As above, under normal circumstances, we will only need to see the "current year" of occurrences, but it is likely that samples from 2021, for instance, will be checked in early 2022.

The number of samples shown as the total at the bottom of the grid (7692) seems unrealistically high, especailly if it is only including the current year - is it just miscounting things?

[andrewvanbreda]

Hi @kitenetter

Firstly in answer to your questions

1. If no correction is needed, there is no need to click Submit.
2. Yes, just add the correction and then click Submit on that row (a confirmation message is displayed). Each page is slightly different as how the correction is added (one is a free text, another is a drop-down, the other is an autocomplete). I did check these work a few days bck, but I will check again before live no point keep rechecking saving every time I tinker.

Firstl,y you mentioned some comments about things I don't even think needed changing, these are

• FIT Count target flower checking is already pointing at public survey, FIT Counts from the website 1 km survey, and the FIT Count app.
• FIT Count insect group checking is already pointing at public survey, FIT Counts from the website 1 km survey, and the FIT Count app.
• 1 km survey flower checking is already an autocomplete with drop-down not freetext. I don't think it is limited to those taxon groups though (although it should, not sure why that isn't working at moment), but I need to talk to you about this report (see bottom of this post), so perhaps we should clarify the exact requirement before I make further changes If you don't think any of these are the case let me know, but I will need an example to investigate.

Note the changes I have made are as follows:

For all reports

• Defaults to current year (with filter). This wasn't working properly and was the thing causing too many records to appear.
• Now only show rows with a photo to check
• Limited to UK, Isle of Man, Channel Islands (same as other UKPoMS pages, Warehouse location IDS 1605,1614,1606,1612,214127,203170,203169)

FIT Count insect group checking

• Removed Other Names columns
• Drop-down change to only UK taxon groups. Due to the way this change needs to work, I have now made this a drop-down only. I have charged this change to the EU PoMS project, as country specific taxon groups were introduced as part of that project rather being part of the original UKPoMS contract.

1 km survey flower checking

• Removed Other Names columns

@kitenetter I do have one question though, but I think will will need a Zoom call to resolve this. You said the 1 km survey flower checking report needs to point to the "the website 1 km survey." But you also said "It should only show occurrences that actually have flower photos attached to the occurrence". However this survey does not have flower photos attached to occurrences, it has insect photos (that might have flowers in the background). So I think I need to talk to you about what is going on here as I don't understand. A taxon group limiter wouldn't work as these are insect occurrences.

P.S: I have not changed the roles on these pages until you recheck them and they go live. That bit is easy to do.

[kitenetter]

Hi @andrewvanbreda I need to pick up this issue - sorry not to have got back to you before, and let me know if we need to discuss over zoom.

On the test site, things seem to be working as expected for the two FIT Count checking reports.

Three requests (these are desirable rather than essential):

1. When a correction is necessary, it is not obvious whether or not the "Submit" button has been clicked. E.g. you might correct a name, but then forget to click submit. Is there anything we can do to make this clearer, e.g. have the box or text in red until it has been submitted?
2. For the two FIT Count reports, can date filtering be added so that we only include samples that fall within the period 1 April to 30 September each year.
3. For the FIT Count insect report, is it possible to make the dropdown list open up with all categories diaplsyed, to avoid having to scroll down within the dropdown (the list is just slightly longer than the dropdoan display at the moment)?

In answer to the final question in your previous comment about the occurrences for the 1 km survey report, I am puzzled. The only photos that can be added to the 1 km survey data are for occurrences of plants, which are added via this form: https://ukpoms.org.uk/add-1km-samples

Insect occurrences are subsequently added to this survey, but none of the insects will have photos. There may not be many plant photos but I would expect there to be a few (although I don't know this for certain).

It would be good to get these reports onto the live site as soon as we can.

[andrewvanbreda]

@kitenetter Hi Martin,

Here are my thoughts.

1. There is no harm pressing the submit button twice. There is also a confirmation if you click submit. I will have a see if there is something quick I can do, but can't promise.

2. Ok I will try to add this, shouldn't be hard.

3. This is a site theme issue I think, maybe @BirenRathod could help?

4. I think the confusion here is the spec says this report should be "Needed for public FIT Count data and 1km square FIT Count data" That is currently how it is setup. Would you be able to clarify what surveys this should be pointing at?

[kitenetter]

@andrewvanbreda points 1 to 3 are understood and are fine.

In the document "PoMS summary specification 2020_02_08_ESTIMATES_ADDED - priorities.docx" we specified three checking reports, which are numbered 3.1, 3.2 and 3.3 in that document. Under 3.3 we have:

3.3 1km square survey flower checking report Display photos of flowers (from occurrence media) and allow addition of corrected taxon names (using plant names from UK Master List) as a new attribute alongside the user-supplied insect group name. Needed for 1km square samples; only available to agreed role/s.

So this report should be displaying occurrences of plants from survey ID 638 that have photos attached. Although we probably only need to filter for photos, not for plants as well, since any photos added will be of plants.

[andrewvanbreda]

@kitenetter Ok, I think I have fixed this extra point 4. Please check this report again

I think all that needed to happen was to change this from 637 to 638

I think the confusion is originating with the terminology, as there are two 1km square surveys, and 2 FIT Count surveys (and they overlap) To add to my confusion, the Pan Trap survey on EU PoMS is called fixed square.

Hi @kazlauskis ,

I do not know if you have seen these reports, but we have grids where experts see occurrences and then make a correction for the species which is held in an attribute. However I am just realising the attribute that holds the correction is not currently assigned to the 641 App survey. do not know if I should switch this on? (perhaps it is covered by sharing agreement). If I switch it on, can you confirm the app won't display the attribute automatically on its data entry forms, we don't want users to see this attribute. On websites this can happen depending on the way they are setup.

Cheers

[kitenetter]

@andrewvanbreda the report seesm to be showing the correct set of occurrences and photos, thanks. However, I'm not sure how the auto-complete field is set up - it doesn't seem to be offering plant names.

[andrewvanbreda]

@kitenetter Acknowledged. Will have to investigate further, the taxon group filter has been supplied, but is being ignored.

[kazlauskis]

@andrewvanbreda yes, the app will not automatically display any new attributes added to the warehouse survey.

[kazlauskis]

Just don't make it as a 'required' attribute as this would then prevent app records.

[andrewvanbreda]

@kazlauskis Thanks Karolis. Yes definitely isn't a required one, as that would also break the website's forms for the non-expert users also as you say.

[kitenetter]

@andrewvanbreda we'd like to make the FIT Count plant form available as soon as we can - this one: https://test-ceh-ukpoms.pantheonsite.io/fit-count-target-flower-checking

The only person who needs to use it at the moment is Nadine. Is it okay for her to start using it on the test site? If so that reduces the pressure to transfer to the live site.

[andrewvanbreda]

@kitenetter I would not be recommend to use on test site. For instance ,it may be used for development tasks, which may interfere with a result from it. Although unlikely to happen here, in theory best to avoid. I will transfer it to live this after I get back from lunch if no issues.

[andrewvanbreda]

@kitenetter Hi,

This is now live and I have successfully tested it on live masquerading as Nadine.

Have put as a sub-menu item in a menu called "Species correction reports" (which I can change name of if you wish)

[kitenetter]

Thanks Andy, looks good to me, will try it out with Nadine in the morning.

[andrewvanbreda]

@kitenetter Insect group checking now live

[andrewvanbreda]

@kitenetter Looks like I already had coded a fix for 1km flower checking taxon groups but hadn't added the configuration to the test site (which surprises me, maybe I got distracted). Could you check it and see if you want me to put it live. Thanks

[andrewvanbreda]

@andrewvanbreda Note to myself: 1km Flower Checking is now on live site, but not in menu until Martin checks it.

[kitenetter]

Thanks @andrewvanbreda - I've had a quick check on the live site version and it looks fine to me.

[BirenRathod]

@kitenetter Hi Martin,

Here are my thoughts.

1. There is no harm pressing the submit button twice. There is also a confirmation if you click submit. I will have a see if there is something quick I can do, but can't promise.
2. Ok I will try to add this, shouldn't be hard.
3. This is a site theme issue I think, maybe @BirenRathod could help?
4. I think the confusion here is the spec says this report should be "Needed for public FIT Count data and 1km square FIT Count data" That is currently how it is setup. Would you be able to clarify what surveys this should be pointing at?

Do I still need me to look into this?

[andrewvanbreda]

@kitenetter Did you check the 1km Flower Checking as well?

[kitenetter]

@andrewvanbreda yes, it was the 1 km flowers that I was referring to, so I think all three are fine now, thanks.

@BirenRathod the only issue is for this page: https://ukpoms.org.uk/fit-count-insect-group-checking

In the "Correction" column there is a dropdown list, which displays nine rows, but there are ten to choose from. Is it possible to have all ten rows displayed in the dropdown, to avoid having to scroll the list?

[andrewvanbreda]

@BirenRathod @kitenetter Hi Biren, Looks like Martin does need help with that problem. I think the CSS needs adjusting in the theme.

[andrewvanbreda]

@kitenetter 1km flower checking now live in menu

[BirenRathod]

@kitenetter The dropdown list size has increased now, so no need to scroll.

[andrewvanbreda]

@BirenRathod Hi Biren, Could you pull these two reports whenever you get a chance please (they are not the live versions so is safe to do so at anytime) reports/projects/PoMS/occurrence_media_name_checking_report_2.xml reports/projects/PoMS/sample_media_name_checking_report_2.xml

Thanks

[BirenRathod]

@andrewvanbreda Those reports gone live now.

[andrewvanbreda]

@BirenRathod Thanks Biren, will have a look in a bit

[andrewvanbreda]

@kitenetter Date limit now set on the 2 FIT Count report. Limit is 1st April to 30th September for the year set in the filter.

In practice, at the moment it means that 2 records where removed from the target flower checking report.

[andrewvanbreda]

Hi @kitenetter, I have had a think about the submit change. I think this wouldn't be very quick change so would need to charge to EU PoMS. Part of the reason is we have two types of report anyway (one with text, and two with autocompletes), so to some extent the change needs doing twice. Also the way the code works doesn't lend to this kind of charge particularly easily. Also there is the testing time. So I would recommend leaving it as is for now, and if you find it is a problem raise it separately, and then see if EU PoMS budget can be diverted to it.

I think also that when users make their changes and save, they should refresh the report to make sure there changes are all there as they expect anyway. I would always recommend that for any system, really always to check result after save. There is no harm is clicking submit twice also.

I would even perhaps say putting a note on the screen to remind users to check the result of what they have done would be a an effective way to approach this particular issue. Please close this issue if you are happy with this conclusion, or reply with your thoughts.

Andy

[kitenetter]

Happy to use it as it is for this year and revisit if any problems arise. I will add help text to the page if needed. Closing.