Importer v2: queries re field matching

[kitenetter]

1. Blurred grid refs We currently have fields available for "Sensitivity precision" (included in the Standard fields), and "Privacy precision" (included in the Advanced fields). I think this should be the other way round - users with records to import may need to blur locations on the grounds of privacy, but we don't wish to encourage further non-standard species sensitivity blurring.

2. EUNIS habitats This is currently included in the Standard fields. However, it is not clear what terms can be imported into this field, given its hierarchical nature. Is it possible to define what terms are accepted for this field? If not I think it should be moved into the Advanced fields (leaving the free text Habitat in the Standard fields).

3. Sample method (lookup) Can we move this into the Advanced fields, to reduce confusion and duplication with "Method (free text)" and "Observation type" in the Standard fields.

4. Vice county (lookup using location code) Again, I would recommend that we remove it from view altogether, but if that is not possible it should be moved into the Advanced fields (leaving "Vice-county number" in the Standard fields).

5. Confidential This is an Advanced field, but in the past I think you said that it was a legacy field that we no longer used, so can it be removed from the import options altogether?

6. Import guid I don't know what this is for, and I can just mark it in the guidance as being for iRecord admin use only, but if there is a wider use for it it would be good to know.

7. Record status and Record substatus This would appear to allow users to import records as already verified, which I don't think we want to do, and it also raises questions about how this could be linked to a verifier name and date. Is it possible to remove this option from the importer? If not I will mark it in the guidance as being for iRecord admin use only.

8. Sample verification status Would prefer this to be removed from the options, but if that can't be done I will mark it in the guidance as being for iRecord admin use only.

9. Vice county (using database location ID) Would prefer this to be removed from the options, but if that can't be done I will mark it in the guidance as being for iRecord admin use only.

[DavidHepper]

This looks like quite a menu of changes. Please let me know when Importer v2 is sufficiently mature to be promoted to my verifiers.

[sacrevert]

4. Vice county (lookup using location code) Again, I would recommend that we remove it from view altogether, but if that is not >possible it should be moved into the Advanced fields (leaving "Vice-county >number" in the Standard fields).

Given that the BBS specifically paid for this to allow them (and potentially other botanical schemes) to transition to using iRecord for all of their data collection purposes, I'm struggling to understand how "removing this from view altogether" would square with that, although I may be missing something.

7. Record status and Record substatus This would appear to allow users to import records as already verified, which I >don't think we want to do, and it also raises questions about how this could be >linked to a verifier name and date. Is it possible to remove this option from the >importer? If not I will mark it in the guidance as being for iRecord admin use >only.

I don't know if this is why this is offered, but we did discuss that this functionality is highly desirable for the BBS transition to iRecord. Perhaps it requires some development, but it seems to me that the actual fact of entering the uploader's details into the verifier name and date fields doesn't seem very complex (at least to my naive view).

9. Vice county (using database location ID) Would prefer this to be removed from the options, but if that can't be done I will >mark it in the guidance as being for iRecord admin use only.

I agree that this could be removed, as I don't really see why anyone would need to do this when they could just use the actual VC number (I suppose it must be marginally faster, but I doubt that's enough of advantage for the potential complication/confusion to users).

[kitenetter]

@DavidHepper the items listed above apply only to the iRecord Import survey, and do not affect the DRN dragonfly imports survey.

@sacrevert as discussed via email, I think we need to clarify the VC options with @johnvanbreda. On the verification status, we cannot allow all users to import records as verified, as that would bypass the verifier system. This is something that needs to be implemented for verifiers only, not for all users. There are other options for the BBS transition.

[DavidHepper]

My personal opinion is that even verifiers shouldn't be allowed to import records as pre-accepted. Seeing them in the system after import gives an opportunity to check that they've imported correctly before acceptance. It's easy to select only the ones just imported and your verify page controls make it quick to bulk-accept the whole import.

[johnvanbreda]

My thoughts:

1. Blurred grid refs, whilst I agree with the premise here, shouldn't we wait till the recently discussed updates to sensitivity handling are implemented? Occurrence sensitivity is after all available on the casual records form.
2. Eunis habitats - I've move this to advanced fields. The correct terms to use would be the exact term at whatever level (no need to include the parent, so either "Marine" or "Rock and other hard surfaces in the intertidal and splash zone".
3. Sample method (lookup) - moved to advanced fields.
4. As Oli mentioned - this was a recently funded development for BBS. I've put it under advanced fields for now.
5. Confidential - can be used to mark a record as hidden from reporting. It's used heavily for Asian Hornet data but rarely elsewhere.
6. The import guid is automatically assigned when you do an import. On the old importer it was necessary to make it available for mapping as it was used to ensure that errors uploaded after a main upload were associated with the same import - that's no longer relevant so it could be removed altogether for v2.
7. Record status - note we don't currently have code in place for completely removing fields, just marking them as advanced. But see below. 8 & 9. As for 7.

There are several fields which may be best removed altogether - this needs a bit of new code (maybe an hour or two) so @kitenetter please confirm you think that's worthwhile.

[kitenetter]

Thanks @johnvanbreda. Proposals:

1. I hadn't realised that the casual records form was using "Sensitivity precision" - "Privacy precision" seems like the better option here as well, although I suppose the effect is the same either way. Let's leave as is, and we can review whether change is needed when we look at the overall species sensitivity work.
2. Thanks - fine to have it as an advanced field, and I can add the termlist to the Help pages.
3. Thanks.
4. That's fine for now, but I think it would be useful to know more about how the various VC-related fields function. Will move this to a separate issue.
5. Fine to leave in advanced for now.
6. Ideally should be removed but fine to leave as is for now.
7. This one and ...
8. ... are the ones I'm most worried about, and I think we need to put more thought into whether/how we enable verifiers to bring in already verified data (personally I agree with @DavidHepper view on this but realise others may take a different view). But can live with these fields being in advanced for now.
9. See 4.

I will update the Help pages in line with the above decisions.

[kitenetter]

Help page list of standard and advanced fields now updated. Still to do: add EUNIS termlist and Sample method termlist info, plus any further info needed for the VC fields.

[Sam-Amy]

Just an observation that where there is more than one suggestion to map to an attribute, the advanced fields are shown here (though not in the dropdown) even when the 'standard and advanced fields' button is not selected at the top of the page. Although clicking them does nothing, this might be a bit confusing so I wonder if it is possible to hide suggestions of advanced fields until 'standard and advanced fields' has been selected?