ES download: replicate functions from existing download page

[kitenetter]

We need to enable an ES download (EDP) that replicates, as far as possible, the functionality of the current download page (CDP), and outputs a csv file containing the same attributes as the current csv file. The following compares CDP (https://www.brc.ac.uk/irecord/downloads) with EDP (https://www.brc.ac.uk/irecord/elastic/download).

• CDP: Download type EDP has a "Within" dropdown, but the choices are more limited, and include download of all records (not provided on the CDP). EDP needs to have equivalent function to CDP, e.g. ability to download records for user's verification role, and for user's activity-related records.

• CDP: Filter to apply This provides an option for users to apply one of their saved custom filters. This is missing from EDP, although EDP does have a more generalised Search box that would allow (at least some) equivalent filters to be applied on the fly.

• CDP: Survey to include In principle this can be replicated by using an appropriate search term in the EDP Search box, but in order to do that one has to know which surveys exist. CDP provides a dropdown.

• CDP: Date field/Start date/End date In principle this can be replicated by using an appropriate search term in the EDP Search box, but the syntax required is much less intuitive than the dropdown and data boxes on CDP.

• CDP: Select a format to download Not yet available of EDP, which outputs a csv file. csv is undoubtedly the most widely used option, and I don't know how much demand there is for the other options on CDP. CDP's "NBN Exchange" is outdated and ideally needs to be replaced by a Darwin Core format similar to the one used to export to NBN Atlas.

• CDP csv file structure CDP provides a csv with 47 columns. There have been requests for renaming, re-ordering and adding to these columns, but to start with it makes sense to provide a csv download structure on EDP that is an exact match for the structure on CDP, in order to maintain compatability with existing user workflows, and to get something up and running while we review enhancements to the download structure.

@burkmarr at out last review telecon I think we agreed that you would look at this issue, is that what you were expecting? I'm not sure what is needed to take this forward; it seems to be a mix of setting up filtering functions on an ES download page plus linking it to an equivalent of the current csv file structure (but generated from the ES table). Is there enough here for you to work on or do we need to do further preliminary work?

I'm adding this to the February milestone but with the recognition that there isn't very much of February left, and this may need more time.

[MNSmith]

If we were able to select and save which fields we actually were able to download and in what order as a named download option that would be very useful. This faciltiy would also halp remove the differences in fields and fild orders between the CSV and other file formats.

[kitenetter]

@MNSmith we hope to do something along those lines, but the initial step is to get an equivalent of the current download output functioning, and then look at enhancements.

[burkmarr]

@kitenetter @johnvanbreda @DavidRoy. Further to our meeting on Monday and in respect of the issue as stated above, you can see where I got on this by looking at this working form on dev: http://test.brc.ac.uk/irecord_dev/es-easy-download.

This is using the dev ES index. I've replicated the interface from the standard download (easy_download_2 prebuilt form), except there is no ability to specify anything other than CSV download (and no ability to specify the fields to be downloaded).

However we need to decide whether I should continue with this development, adding in those two missing elements, or whether to switch tack and look at enhancing the explore/download combined interface that John has already implemented. I've replicated the explore/download page John mentioned in the meeting on dev using the dev ES index: http://test.brc.ac.uk/irecord_dev/es-explore-download.

On this dev form I've remove the configuration that restricted it to 'my records' so this is an 'all records' explore/download page. (As a matter of interest, I did a download of the entire dev DB - 9 million records - which took around about 90 minutes to build, resulting in a CSV of about 0.5 GB.)

I'm looking for views from any of you on which way to go with this. If we switch tack to developing the combine explore/download functionality, rather than trying to replicated what we've already got, what do we need to add to it that isn't already there?

The ES data grid provides the ability to select which ES fields are included and the order they are displayed in the explore grid. @johnvanbreda - were you suggesting that we could reflect this selection in the data download part? It seems like that should be possible. But we might need a way for different selections to be save and reapplied.

To be comparable to current download functionality we have to add the ability to pick/specify download formats from the ES rest server (as we would if sticking with the original approach).

Do we need to add some sort of numerical limit on the download size? It's a bit easy to inadvertently kick off a very large and time consuming download. It would be good to add the ability to cancel a download too if that is doable.

Your thoughts would be welcome.

[DavidRoy]

Thanks Rich - great progress. My initial thoughts:

1. We should move to the new implementation where the Download is part of the explore page.
2. To make the user interface more intuitive, could the Download link be on the same tab as the explore map/data grid - so the Download matches the same filter applied at the time, i.e. download is for what's in the data grid? Is that workable?
3. For users with specific Download roles, I suggest this is a separate menu/page just for this purpose and only displayed for those with the role
4. For large file downloads, or maybe all downloads, is an alternative to send an email with a link, to the user when the file is ready to access? This is how EIDC used to provide this function. Related to this, we've had an issue with the file download - https://github.com/BiologicalRecordsCentre/ABLE/issues/220

[johnvanbreda]

2. It could, but the reason for separation was the need to keep the UI tidy on the first tab. Rich - would a solution be to code the Download control so that the output div selector can be specified as an optional property, allowing the button and the outputs to be separated. Then the button goes under the grid (or in the grid footer?) and the outputs go on their own tab. When you click the button the tab could auto-show.

3. Do we want to become a facility for large downloads? I appreciate they are occasionally required by some of our key users, but for general users, downloading a custom filter with millions of records in it could end up putting a lot of load on the ES server.

@burkmarr yes, I was wondering if the column selection could be synced with the download somehow. At the moment the dataGrid config for a column and download config for a column are the same for directly mapped field values, but for special calculated field values there may be some differences which will need to be harmonised (which is a good idea anyway). As you say, an ability to provide several prebuilt templates, plus save your own, would be a very nice enhancement. One more thing - the ability to add custom attribute columns is supported in the code but the column picker doesn't include this facility.

[kitenetter]

Here is my summary of where we are with this, and what I think are the next steps, along with some questions:

ES download proposals 2020_04_24.docx

@DavidRoy @johnvanbreda @burkmarr (plus anyone else with an interest) please can you let me have any comments, and reponses to the questions (maybe via a tracked-changes document?), and then we can start breaking things down into individual issues to take forwards.

[burkmarr]

@kitenetter - thanks for this analysis; extremely useful. I've been struggling to find time to work on it because of a deadline for another project, but perhaps it's just as well I didn't forge ahead! I think @johnvanbreda's initial comments are going to have greatest values, especially in light of the work on ES verification. I'm not sure how best to provide comments - we would each have to download the document to do tracked changes and then we'd likely end up with several copies. What about a Google doc or something?

[johnvanbreda]

@kitenetter in answer to the questions in your document:

1. In the data, there is now a field called tracking (metadata.tracking in ES) which is auto-incremented every time cache_occurrences_functional is written to. So, for incremental updates, this is a very reliable method. Note though that it will track all changes, e.g. if Rich indexes a new locations layer, that will increment tracking if cache_occurrences_functional is updated, so it doesn't just track deliberate changes made by the recorder or verifier.

Single survey - all attributes issue - the attribute values are all stored in the ES document, so we could potentially have a structure occurrence and sample attributes download field (containing JSON they would have to parse). Or, we need to add code to the ES download to auto-add the columns for the attributes for the selected survey.

Regarding the lack of quick data filtering if the download tool is added to Explore as suggested, we could perhaps have a "tracking" filter field in the toolbar (like the way the record status filter works), or only show this for some user groups?

Question 2 - My/All records separate pages vs an option. Note that merging the 2 might make it more tricky if you want different views on each, for example My Records might show full precision if set up to do so.

It would be interesting to know which of your saved filters don't work for ES? Perhaps raise as an issue?

Question 3 - in the longer term, I think being able to configure the data grid with additional attribute columns, then these would automatically appear in downloads on the same page would be a good aim. The ES technology should support this - we just need the UI for picking the attribute columns.

Question 4 - yes, I think the current CSV output could be achieved. Best approach would be to try!

Question 5 - draft query would be useful as less ambiguity.

[burkmarr]

Tracking date of edits/verifications & question 1 What changes does the metadata.updated_on field reflect? If that reflects when records/samples or verification was last modified, then couldn't that be used? The most recent value of that field could be extracted after a download and the next download only select those where metadata.updated_on field is greater than that value. Given what John says how metadata.tracking is updated when a record is re-indexed spatially, then adding or modifying large spatially indexed layers such as VCs (a not infrequent occurrence recently) would trigger huge downloads. It would make us very wary the consequences of changing any large spatially indexed polygons on people that use download workflows.

Custom attributes & question 3 Following on from John's comment, if we made custom occurrence and sample attributes downloadable as JSON objects which customers could parse – which would allow a consistent download format from mixed surveys – it would be much better for users if we enriched JSON with the names of the attributes. So for example instead of this (which we currently have):

"attributes": [
  {
    "id": "127",
    "value": "Burkmar, Rich"
  },
  {
    "id": "1468",
    "value": "5"
  },
  {
    "id": "1469",
    "value": "123"
  },
  {
    "id": "549",
    "value": "Sherlock (2018) Earthworm AIDGAP 2nd edition"
  },
  {
    "id": "551",
    "value": "044 garden"
  }
]

We would have this:

"attributes": [
  {
    "id": "127",
    "caption": "Recorder Name"
    "value": "Burkmar, Rich"
  },
  {
    "id": "1468",
    "caption": "Number of pits"
    "value": "5"
  },
  {
    "id": "1469",
    "caption": "Unidentified"
    "value": "123"
  },
  {
    "id": "549",
    "caption": "Identification guide used"
    "value": "Sherlock (2018) Earthworm AIDGAP 2nd edition"
  },
  {
    "id": "551",
    "caption": "Habitat"
    "value": "044 garden"
  }
]

@johnvanbreda - I guess that this would require an update to the cache_builder module and a complete rebuild of the occurrences and sample caches. Is it feasible? It would make the JSON representation of the custom attributes so much more useful.

I don't see why would shouldn't add a separate single-survey download facility to parse these on our side and add as columns (see next section).

Explore page downloads vs dedicated download page As long as it is possible for people to implement the download workflow they need from the explore page then I don't think that it matters if it is a little involved. I think that kind of users we are talking about – LRCs, RS&Ss etc – will be prepared to learn what they need to do. I think that they will value flexibility and reliability over ease of use. That said I don't see why we shouldn't have more than one download facility – it's more a question of prioritizing than deciding one or the other.

Scope of downloads & question 2 Currently you can configure the page in the form structure to do either my or all records so it is easiest to do a separate page for each. That way it also becomes very easy to restrict the downloads to the my records explore page.

What needs to be developed
I don't have any questions/comments on the stated requirements at the moment.

What should the zip file contain? Perhaps it would be possible to add one or more buttons under the 'column configuration' for the ES grid to quickly configure for the standard CSV and perhaps other, like MapMate, downloads.

We can talk about the required fields for the MapMate download when the time comes. There shouldn't be any problems discerning the current CSV download format.

Nice to haves The first of these – ability to defined columns to download - is implicit in the ES Grid approach (apart from the custom attribute problem) .

I think adding other file formats is higher priority than nice to have because it is implicated elsewhere - https://github.com/BiologicalRecordsCentre/ABLE/issues/151. I should be possible to cope with any text formats, including those that encode geometry (GPX, KML). Shapefile would be a tougher nut to crack but converting between GIS formats is so easy nowadays, that I think that we shouldn't even bother with it.

[kitenetter]

Have added comments from the above to a google doc (I should have used that to start with!): https://docs.google.com/document/d/1PNNH1YW0j17xnucY1y1oEQT16NBFN4FfL4JqxOGDdPU/edit?usp=sharing

[kitenetter]

To proceed following refactoring work in ES.

[johnvanbreda]

@burkmarr I've now refactored lots of ES client code and updated the documentation at https://indicia-docs.readthedocs.io/en/latest/site-building/iform/helpers/elasticsearch-report-helper.html. All the new code is in the develop branch (client_helpers, media and warehouse which is now v4 due to a small API change). You should find there are some examples to help you get going. Here are a few hints on areas to look at:

1) The source control now has a mode option. Combine the termAggregation or compositeAggregation with this with the uniqueField, fields and aggregation options should make it easy to build data grids that are not simple lists of documents (like the species tab etc). 2) The download control now has a linkToDataGrid option. 3) When configuring the columns in a download, it can either use the dataGrid you linked it to, or a source with the addColumns option. The options available are now the same as those for a dataGrid.

[burkmarr]

Brilliant – thanks John. I have spent the day doing a bit more background learning on ES so this is great timing. I shall have a good look tomorrow.

[burkmarr]

@johnvanbreda - I've been digging into the new code this week. One of the main tasks for this issue is to provide a permissions context control equivalent to the one at the start of easy_download_2. I was looking at providing a new control in the ElasticsearchReportHelper.php. However there is a control there - [permissionFilters] - which does some of this.

It currently provides options of my records, all records and records within location (the collation location defined on the host website). So I think that what is missing, when compared against that dropdown in easy_download_2, are permission filters defined in the warehouse and taxonomic and the profile expertise stuff (taxon, location and verification) defined on the host website.

I'm thinking that the right approach would be to add these contexts to the current permissionFilters - do you agree?

I also have a question about the approach we take with allowing access via any of these filters - both via easy_download_2 and the new permissionsFilters ES control. Any given permission context is only allowed if the user has permission as defined by their permission filters in the warehouse or hostside expertise AND their hostsite role meets the hostsite permission defined for that context in the iForm settings. I'm not clear on why we add the permissions layer in the form configuration?

[johnvanbreda]

@burkmarr doesn't the [userFilters] control already give you access to the filters saved on the warehouse? So I think the only question is whether you need a single control for both of these or 2 separate controls?

Also, note that the profile expertise stuff is very much legacy code and all verifiers should be using a saved filter with defines_permissions set to true to define the data they can verify. I wouldn't bother writing new code to support it.

Using role permissions to control access to different types of downloads is necessary because if you don't have a filter with defines_permissions=true set, then the default is to allow access to all records.

[kitenetter]

We still have a small number of verifiers that have not yet been given a saved filter, but I'll try to switch them over in the next day or two.

[burkmarr]

@johnvanbreda - the userFilters only accommodates a single sharingCode option so as it stands we'd have to include several of the controls - one for each sharing code (plus the permissions filter control) which looks a bit clumsy compared to the single integrated control on the easy_download page that Martin refers to in the requirements doc.

(Also currently if any filters are found and the drop-down is displayed, the control doesn't allow the selection of no filter - you have to pick a filter, but that should be easy enough to fix.)

What about if I update the userFilters control to accommodate the specification of several sharing codes in a single dropdown? If @kitenetter is happy to have the two dropdowns - general permissions (all/my/location collation) and permission filters (verification, data flow etc) we could go with that?

[burkmarr]

@kitenetter - the second point on the 'additional functionality required' concerns an apparent bug in the status drop-down:

Status dropdown to be fully-functioning (currently does not seem to allow “Not accepted” records to be displayed).

This turns out to be due to the interaction with filters specified in the 'standard parameters' (filters) toolbar. The default value for 'quality' in that is 'exclude not accepted records'. The separate status drop-down and the 'quality' part of the standard params do essentially the same thing. To me it doesn't make sense to include both on the same page - it is potentially confusing for users and would certainly complicate the codebase to make them play nicely.

My recommendation is to do away with the separate status drop-down list on this form. I know it's less convenient for users who want to quickly filter on status and requires them to know, or find out, that it is available in the standard parameters toolbar, but users should be prepared to learn how to use a page that gives so much flexibiliy. And at the end of the day it's only a click or two more.

What do you think?

[kitenetter]

@burkmarr on the ES verification page, the default value for 'quality' in the standard parameters is to include all records, so that verifiers can see "Not accepted" records via the top-level dropdown. So a possible solution would be to change the standard parameters settings so that the quality default is "all records".

However, that could be seen as putting undue emphasis on rejected records, when most people using the Explore page are likely to want to exclude rejected records.

Is it possible to retain the top-level status dropdown, but to exclude "Not accepted" from the terms offered in the dropdown list? That way people could still find their way to the rejected records if they really wanted to, and they could also retain the convenience of the dropdown list for the main choices.

However, this isn't something I feel strongly about, so if the above suggestion is not simple to implement I agree with your proposal to remove the top-level dropdown on the Explore pages.

[burkmarr]

@kitenetter - either of those is very easy to implement through the form definition itself. So there are three possible 'solutions' which can be implemented simply through configuring the form differently:

1. Remove the 'not accepted' option from the status dropdown
2. Change the default quality value to 'all records' in standard params config
3. Remove the status dropdown entirely

Changes to the codebase would only be required if we wanted to tie the two controls together so that changing one affected the other. It doesn't sound like you are advocating that.

[kitenetter]

@burkmarr 1: all verifiers that were set up under the old system have now been removed or transferred to the new system.

2: for the filtering, I'm happy with option 1 in your previous comment (although now that you've suggested a codebase change to harmonise the two controls I do like the sound of that for a longer-term solution, both on Explore and on Verify).

[burkmarr]

@kitenetter @BirenRathod @DavidRoy @johnvanbreda Following up from the team meeting on 2nd June, here is what I took out of it with respect to moving this issue (in it's wider sense) forward. I've compiled this list from notes and my recollection of the meeting as well as the google doc (https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-624571350). I said at the meeting I would update the document, but it is proving cumbersome to maintain and cross-reference information on both the document and on GitHub so I've done it here instead. It may be best for us to close this overarching issue and make separate issues along the lines of those outline below after we've discussed them.

Please comment on the points below - especially if I have missed something or misunderstood anything that was discussed...

1. For general download capability, we will use and develop the download facility on the new ES explore pages which are implemented on the ES outputs (customisable) form (currently https://www.brc.ac.uk/irecord/elastic/explore/my).
2. We will also provide a convenience form for LERCs, RSS admins and verifiers where they can quickly perform downloads based on their permissions. This will also be based on the ES outputs (customisable) form but will not use a grid, map or any of the explore-type controls. It will use the [permissionsFilter] control and any additions needed to give us the functionality currently available from the 'download type' dropdown on the current download page.
3. Provide a new record status helper control, [recordStatus], to replace the HTML configured control on the current ES explore/download page. It needs to be possible to link this new [recordStatus] control to the existing [userFilters] control so that altering one affects the other (link them through an attribute that points to the id attribute of the other).
4. Add a new helper control [filterSummary] that summarises the filters that are active on a page as set through any of the above controls or a combination of them.
5. Add a download formats control [downloadFormats] which can appear next to the download button. It will be a dropdown including, for example: -- Current grid -- MapMate -- Etc
6. Add the ability to save and reload grid column configurations as templates. Since the download control can already be linked to the grid, this will enable people to save different download templates themselves.

Subsequent edit: the [recordStatus] control referred to in point 3 has subsequently be renamed to [statusFilters]. Also the reference to [userFilters] in point three should have been [standardParams] - i.e. the filter builder control.

[burkmarr]

Can anyone tell me where the location type is set that determines the type of location used for the 'Location of collation' dropdown for users with the regional collator's role?

[burkmarr]

@johnvanbreda - where is the location type is set that determines the type of location used for the 'Location of collation' dropdown for users with the regional collator's role?

[johnvanbreda]

@burkmarr In the user profile there is a field called field_location_expertise.

[johnvanbreda]

@burkmarr Looking through your list of points a few posts up, I had originally assumed the download formats control would be just a sub-option within the download control. Having thought about it though, I think your version is better as for ABLE and other projects we are creating download pages that have multiple download buttons for different datasets. So having one place to choose the format rather than multiple might be less messy. So I think your list of points is fine.

[burkmarr]

@johnvanbreda - but what determines the location type of the list that admins select from when assigning an area to a ueser. In iRecord this is set LRC boundaries - but is it configurable. I'm trying to work out how to set this up on my localhost for testing. (Thanks for the other comment.)

[johnvanbreda]

There is a Drupal variable called indicia_profile_collation_type_id. I thought at one point this was on the settings form but it seems not, so you'd need to call variable_set to change the value. We could add it to the settings form using hook_variable_info with addToIndiciaSettingsPage set to true.

[burkmarr]

Thanks @johnvanbreda

[burkmarr]

Created a pull-request for a new recordContext control to support point 2 (https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-638868012). Pull request: https://github.com/Indicia-Team/client_helpers/pull/99

[burkmarr]

I've created a draft new control in line with item 3 in the list: https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-638868012. I've called it statusFilters rather than recordStatus to be more consistent with other controls that do filtering. I have some questions about 1) the functionality and 2) the coding. I'll put the questions about functionality here - in particular for the attention of @kitenetter - and I'll put the coding questions- in particular for @johnvanbreda - into a draft pull request which I will make shortly.

As a reminder, this is to provide a control to filter on record status which in the current ES explore form is provided by inline HTML coding in the form definition. There is a requirement to link it to the similar quality part of the standard parameters (user filters) control in some way. Below is a description of what I've done - I would like to know if this is sufficient.

The control offers these options in line with the HTML version in the current ES explore form:

• Any status
• Any status except not accepted
• Pending
• Accepted
• Not accepted
• Plausible
• Queried
• Answered

There is a slight change in terminoloy (compared to the HTML version) in that I've used 'any status' rather than 'all records' since I think it better reflects what the control is doing. In the 'quality' part of the standard parameters control, the list is much longer:

• Accepted as correct records only
• Accepted records only
• Reviewer agreed at least plausible
• Plausible records only
• Recorder was certain
• Recorder thought the record was at least likely
• Not reviewed
• Not reviewed but trusted recorder
• Exclude not accepted records
• Exclude queried or not accepted records
• All records
• Queried records only
• Answered records only
• Not accepted records only
• Not accepted as reviewer unable to verify records only
• Queried or not accepted records

I don't see a problem with that - the standard parameters control is desgined to give more comprehensive functionality. This control is more of a convenience control for the more commonly required options.

In terms of linking these, I've chosen what I belive is the most pragmatic way. When both of these controls are on a form, if the user sets the value of the new statusFilters control to anything other than 'Any status' then the value of the quality dropdown on the standard parameters control is set to 'all records' and disabled.

The drop-down is only enabled if the user selects a value of 'Any status' on the statusFilters control (or if that control is not present on the form of course). I think that this is a sufficient link to avoid unintended combinations of status filters over the two controls. A more sophisticated link, e.g. trying to mirror selections in the two controls, would add a lot of complexity to the codebase which I don't think is warranted, but your opinions are welcome.

[burkmarr]

@kitenetter and @DavidRoy - @johnvanbreda has suggested elsewhere (https://github.com/Indicia-Team/media/pull/28) that it might be better for the standalone record status filter described in previous post to reflect the same options as available in the quality part of the standard params filter and tie them together - so that changing one affects the other when both are used on the same form. Are you happy for me to proceed on that basis?

[burkmarr]

@johnvanbreda - can you help me understand something that has tied me up for much of the day? On my test setup, the standardParams control falls over when I select a filter from the drop-down list and click the apply button. The console shows this JS error:

Cannot read property 'MyRecords' of undefined

It's generated by this line of JS in reportFilters.js:

return indiciaData.lang.reportFilters.MyRecords;

And when I look at indiciaData.lang in the console it looks like this:

There's no sign of the reportFilters object. Tracing through the code I can see lines in report_filters.php (https://github.com/Indicia-Team/client_helpers/blob/master/prebuilt_forms/includes/report_filters.php) from line 1351 which populate indiciaData.lang, summarised below:

report_helper::$javascript .= "indiciaData.lang={pleaseSelect:\"" . lang::get('Please select') . "\"};\n";
[...]
foreach ($filterModules as $category => $list) {
  foreach ($list as $moduleName => $module) {
    [...]
    report_helper::$javascript .= "indiciaData.lang.NoDescription$shortName='" . lang::get('Click to Filter ' . ucfirst($shortName)) . "';\n";
  }
}
[...]
report_helper::addLanguageStringsToJs('reportFilters', [
    'CreateAFilter' => 'Create a filter',
    'ModifyFilter' => 'Modify filter',
    'FilterSaved' => 'The filter has been saved',
    [...]

The iform.module (https://github.com/Indicia-Team/drupal-7-module-iform/blob/master/iform.module) from line 1039 first adds JS set by report_helper::addLanguageStringsToJs('reportFilters'... , and then sets JS set by report_helper::$javascript .= "indiciaData.lang={pleaseSelect:.... So indiciaData.lang={pleaseSelect:... overwrites the indiciaData.lang.reportFilters object previously created. At first sight it looks like indiciaData.lang={pleaseSelect:... is a mistake, but it isn't new code and all is working on iRecord. Can you throw any light on what is happening here?

[burkmarr]

@johnvanbreda - I've just realised that this error isn't actually stopping the filters from loading - that was another problem.

[burkmarr]

The pull requests above are for a new [filterSummary] control as referenced by point 4 here: https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-638868012

[johnvanbreda]

Thanks for the new control. Your analysis of the problem in language strings was correct so I've pushed a couple of updates to fix this (develop branch, client_helpers and media).

[burkmarr]

Thanks @johnvanbreda.

[kitenetter]

@burkmarr @johnvanbreda can I run this past you to check whether it is worth pursuing or whether it is technically impractical.

One of the difficulties in setting up the lists of terms as shown in https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-651035393 is that the wording gets quite tortuous, e.g. "Not accepted as reviewer unable to verify records only".

Would it be feasible to offer a set of multi-choice tick boxes so that users can make their own selection from the different status terms?

For the verifier view I guess we may also need to add:

• Record flagged as "certain"
• Record flagged as "likely"
• Record flagged as "uncertain"

Could we apply that multi-selection to both the top-level status filter and the "create a filter" parameters?

If this is a non-starter let me know and I'll go back to the wording of the lists we currently use and harmonise those.

[DavidRoy]

Even if we don't allow users to select their preferences, this is a good summary for the default setup (and 'convenience controls') for the three main uses of the explore report - verification, all records, my records.

[johnvanbreda]

I don't see why not - I think David Hepper asked for something similar a while back. These would need to be implemented as a new set of separate parameters (rather than the single Quality parameter) in the filter back-end code on the warehouse. I would suggest we leave the existing Quality parameter in place in the back-end code even if we remove it from the UI.

Changes required to implement the above:

• Changes to the filter builder UI.
• Changes to the helper class on the warehouse which convert filters to queries (application/helpers/report_standard_params_occurrences.php, possibly also report_standard_params_samples.php).
• Configuration options for the report filter panel to allow the list of available options to be controlled.
• Changes to the ElasticsearchProxyHelper.php file to add support for these filter options for conversion to Elasticsearch filters.

You may need to think about whether there is a need to separate record status 'V' with record substatus not set (i.e. tier 1 verification decision) from record status 'V' with any record substatus (i.e. any accepted record). The proposed UI does not make this clear.

Should we also combine into this UI any other quality related filters? -

• record cleaner no warnings
• record cleaner warnings
• identification difficulty
• possibly verification decision source (machine/human)

[burkmarr]

It seems to me that this should be the subject of a separate issue? Subsequent to our meeting yesterday, I reworked the new [statusFilter] control so that it mirrors the quality-status part of the general parameters (filter builder) control and changing one affects the other and submitted pull requests (https://github.com/Indicia-Team/client_helpers/pull/103 and https://github.com/Indicia-Team/media/pull/30). I think that we should go ahead with that and treat this new requirement as a separate issue. I'm guessing it is probably lower priority than the remaining points 5 and 6 in the list - https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-638868012? Do you agree?

[burkmarr]

I should have taggged. Question in previous comment was for all of you - @kitenetter, @DavidRoy & @johnvanbreda.

[kitenetter]

Yes, I could feel an element of mission-creep sneaking in! So do you need me to suggest any changes to the wording we use in the current two filter settings @burkmarr (as per discussion in https://github.com/BiologicalRecordsCentre/iRecord/issues/800#issuecomment-651035393)?

Point 5 will be needed but I won't be able to work on defining the download formats until later this month. For now replicating the current download format is the priority.

Point 6 will be a very nice 'nice-to-have' but I would say it is not a requirement to do ahead of replicating the current download format.

[burkmarr]

@kitenetter - yes that would be fine with me. To be clear - the first bullet list in that comment is no longer relevant since the new separater statusFilters control now uses the same list as available in the user filter control - i.e. those in the second list. If you want to make changes to the wording of items in that list, that is fine by me and shouldn't cause any issues (excpet perhaps with one or two users looking for options which have changed wording).

[burkmarr]

@johnvanbreda @andrewvanbreda @Gary-van-Breda @BirenRathod - are there any objections to me pulling the latest develop branch into the dev warehouse?

An ES download linked to a datasource works on the dev warehouse ES index, but not an ES download linked to a grid. I think that rest.php needs updating on the dev warehouse to make it work.

[BirenRathod]

@burkmarr - No problem from me.

[johnvanbreda]

@burkmarr no problem for me either.

[andrewvanbreda]

@burkmarr Apologies for slow response, yes this is fine for me.

[BirenRathod]

@johnvanbreda In response to restore data on dev warehouse, I'm thinking to use PostgreSQL 12 as dev warehouse been long pending for the upgrade. Any issue or concern?

[johnvanbreda]

No, I've been using PG12 myself and it seems fine.