Closed PipBrewer closed 9 months ago
The purpose of the test is to follow the Digi App output through the post processing and then check how that behaves in the Specify workbench tool (WB)
Workbench (WB) validation complaints because the encoding makes a mess of non-ASCII characters. The files must be exported from OpenRefine as encoded with "Windows 1252: Western European"
If done this way, the mock-up file for NHMD Vascular Plants passed the test.
Could not complete the test, because the WB validation would not accept "containertype" with our values ['Multiple specimens on one object', 'One specimen on multiple objects']
Could not complete the test, because the validation dropped a novel error : { "uploaderstatus": { "operation": "validating", "taskid": "db7f4b1c-9d22-40aa-b578-6f1387a2b89e" }, "taskstatus": "FAILURE", "taskinfo": "DoesNotExist('Spdataset matching query does not exist.',)" }
A solution to the NHMA issue is currently not obvious.
The Digi App output test procedure is described here: https://github.com/NHMDenmark/Mass-Digitizer/blob/main/documentation/Pre_release%20testing%20Digi%20App%20output%20file.docx
Testing mock up records from NHMD Vascular Plants:
Workbench (WB) validation complaints because the encoding makes a mess of non-ASCII characters. The files must be exported from OpenRefine as encoded with "Windows 1252: Western European"
The upcoming upgrade of Specify7 allows for specifying encoding upon upload, which preferably should be UTF-8. Until then, stick to ANSI/win1252
Testing mock up records from NHMD Entomology:
Could not complete the test, because the WB validation would not accept "containertype" with our values ['Multiple specimens on one object', 'One specimen on multiple objects']
New container types have now been added to NHMD Entomololgy, so please try again. The test database is pointing again at NHMDtest, so please try there.
Testing mock up records from NHMA Entomology:
Could not complete the test, because the validation dropped a novel error : { "uploaderstatus": { "operation": "validating", "taskid": "db7f4b1c-9d22-40aa-b578-6f1387a2b89e" }, "taskstatus": "FAILURE", "taskinfo": "DoesNotExist('Spdataset matching query does not exist.',)" }
A new ticket has been raised here: https://github.com/NHMDenmark/Mass-Digitizer/issues/436
Redoing the test & review may take a few hours, but let's give it a day.
I realized the taxon number is not duplicated for each taxon rank in the processed file. For NHMA this is necessary for workbench matching the correct taxa.
The script results appear to check out. @jlegind will have to adapt the documentation to match
What is the issue ?
We need to make sure that the post processing works as intended and preceding this the Digi App output.
Detailed description of the issue.
The question is: Can the Digi App data export be transformed sufficiently to be successfully imported into Specify via workbench?
Why is it needed/relevant ?
This will be a measure of the post processing maturity.
Estimate level of effort required.
easy
What is the expected acceptable result.
A full validation of the post processing output in Specify workbench.
How to approach it?
What could be the challenges ?
There will be multiple versions of the GREL script corresponding to a version and a date range. There needs to be a decision on which GREL script to use ad hoc. Path to GREL scripts: N:\SCI-SNM-DigitalCollections\DaSSCo\Digi App\Data\GREL scripts