Multiple Survey Correction feature

[nspool]

Description

Support loading more than one survey into a corrections sheet and performing validation and updates for all loaded surveys.

Backlog Items

1. https://github.com/aodn/backlog/issues/4139

[LizziOh]

If a survey's observations were ingested seperately - eg. M3 ATRC observations were ingested seperately after the M1 and M2 observations, and then this feature is used to correct those surveys from one of the jobs, then are the observations ingested from the separate job unaffected and remain attached to this same survey in all ways?

[LizziOh]

Is a potential caveat of this feature that if one makes a correction using the single survey correction feature, then opens the related job and uses the multiple survey correction feature then the previous edits are lost because the observations are re-ingested from the pre-edit ingested state?

[nspool]

If a survey's observations were ingested seperately - eg. M3 ATRC observations were ingested seperately after the M1 and M2 observations, and then this feature is used to correct those surveys from one of the jobs, then are the observations ingested from the separate job unaffected and remain attached to this same survey in all ways?

The M3 observations will be attached to the existing survey. When that survey is corrected all observations will appear in the sheet regardless of when they were ingested.

Is a potential caveat of this feature that if one makes a correction using the single survey correction feature, then opens the related job and uses the multiple survey correction feature then the previous edits are lost because the observations are re-ingested from the pre-edit ingested state?

So long as the correction job is submitted first before opening the job then subsequent corrrections will always be based off the corrected data. But if there are multiple tabs open or different users correcting the same surveys then there is the possiblity that edits can be lost.

[LizziOh]

Thank you for the clarification. I have tested the work flow of both queries now and can confirm its is working as desired.

[nspool]

Just updated SysTest to version 11.0 (694) with the ability to create multiple survey corrections from the survey list.

The “🛠” symbol has been replace with a checkbox allowing up to 25 surveys to be selected. Then press "Correct Selected Surveys" to load them all into a corrections sheet.

As usual, please leave a note with any problems or feedback.

[LizziOh]

This is great!

One thing I am struggling with is corrections for the site of an ATRC survey group. This cannot be achieved either individually through the existing survey information edit page (returns a blank white screen when submitting the site code edit) or by selecting the whole survey group using this new multiple survey edit feature (as site code is not an editable field). The blank screen error when trying to a edit single ATRC survey site code occurs in systest and production. For example, all surveys in the ATRC survey group of 5.1-5.4 m from 2015 from Arch Island NIN-S5 (survey_ids 812326581 to 812326584) need their site changed to Arch Rock NIN-S4. When opening the survey edit page for an individual survey a new site code can be selected but the blank screen appears on saving. I'm assuming this is because it then fails to match the site code of the rest of the survey group, so perhaps failing a validation. Changing a site code using this feature works fine for RLS surveys in production and systest, but curiously also the latest ATRC ingested surveys in systest as I was able to change the site for ATRC survey 923401214. It would be nice to change the site code of all surveys in a survey group at once (without affecting the survey_ids) using this new multiple survey correction feature. I'm unsure if Toni has any use case for this for RLS surveys. If not possible, the ability to make the change in the survey info edit page would be good - but would need a warning explaining that the rest of the survey group need to be changed as well?

[nspool]

When opening the survey edit page for an individual survey a new site code can be selected but the blank screen appears on saving.

There is currently a bug in production the causes the screen to go blank instead of displaying the validation error message. If you do this same edit in SysTest, does it still to display a blank page?

[bpasquer]

@LizziOh the issue with the blank screen when editing survey 812326584 was a bug when survey_time is missing Nathan's put up a fix , and will be deployed to systest soon.

[nspool]

When opening the survey edit page for an individual survey a new site code can be selected but the blank screen appears on saving.

Updated SysTest to version 11.0 (695) which should fix the blank screen when editing the site code of survey 812326584 and any other surveys that are missing survey time. The Export Template fields should be also be fixed.

[LizziOh]

Thank you, I've tested this fix and it is working well.

[nspool]

Updated SysTest to version 11.0 (697) which fills the diver on SND rows with another diver from the same survey and also improves the correction summary screen to display more information.

[LizziOh]

Updated SysTest to version 11.0 (697) which fills the diver on SND rows with another diver from the same survey and also improves the correction summary screen to display more information.

These fixes work well thanks.

I did just get a blank screen when trying to test a correction job. https://nrmn-systest.aodn.org.au/data/survey/923400745,923400743,923400742,923400741,923400740,923400736,923400730,923400729,923400728,923400727,923400726,923400724,923400716,923400715,923400713,923400712,923400711,923400705,923400703,923400701,923400698,923400696,923400695,923400694,923400693/correct

I had changed several number observations and totals and it passed validations.

One thing I have also noticed is that it appears that a few errors have been slipping through ingest with the "quadrats not summing to 50 pts" validation working on the corrections page - but it was recently ingested data. Could you please also check that this vaildation is working in production ingest? eg. Quadrats do not sum to at least 50 in transect [NIN-S2, 2021-03-22, 5.3] (survey 923400695, Quadrat 1)

[LizziOh]

Also ctrl z is does not seem to be working well in the corrections page for me.

[nspool]

I did just get a blank screen when trying to test a correction job.

@LizziOh it looks like this was caused by the Job log message being too long to store in the database. I'll have a fix out shortly.

[nspool]

I did just get a blank screen when trying to test a correction job.

SysTest has been updated to 11.0 (701) and should now handle this correction job. The job title will no longer contain all of the survey numbers if the job corrects more than 4 surveys. They will all still appear in the job description however.

Also ctrl z is does not seem to be working well in the corrections page for me.

Does the Undo button continue to work or is it just the keyboard combo that isn't working?

[bpasquer]

One thing I have also noticed is that it appears that a few errors have been slipping through ingest with the "quadrats not summing to 50 pts" validation working on the corrections page - but it was recently ingested data. Could you please also check that this vaildation is working in production ingest? eg. Quadrats do not sum to at least 50 in transect [NIN-S2, 2021-03-22, 5.3] (survey 923400695, Quadrat 1)

I confirm the data has been ingested with Quadrat data on Q1 not summing to 50 for this survey. The problem also exists for 2 other surveys from the same ingested datasheet: [NIN-S1,2021-03-27,5.4] and [NIN-S3,2021-03-27,5.3]. The "Quadrat not summing to 50 points" pick up this error in the current version of the software.

I shall make an inventory of surveys containing Quadrat data not summing to 50.

[LizziOh]

I did just get a blank screen when trying to test a correction job.

SysTest has been updated to 11.0 (701) and should now handle this correction job. The job title will no longer contain all of the survey numbers if the job corrects more than 4 surveys. They will all still appear in the job description however.

Thank you

Also ctrl z is does not seem to be working well in the corrections page for me.

Does the Undo button continue to work or is it just the keyboard combo that isn't working? As far as I can tell the undo button is also broken

[nspool]

SysTest has been updated to 11.0 (711) which should fix the Undo function when editing observations. I've also added the site code to the survey view page.

[LizziOh]

SysTest has been updated to 11.0 (711) which should fix the Undo function when editing observations. I've also added the site code to the survey view page.

These fixes work well on testing, thanks

[LizziOh]

@nspool following today's discussion about the display of size validations in the corrections mechanism and ingest mechanism I can confirm they behave differently.

In ingest, LMax warnings give the size of the observation in the feedback in the format: "row" : "size"(cm) "total". For the example of 2 incorrect entries of Haliotis roei (one with invert sized set to "Yes" and one set to "No") in the largest size column:

four warnings generate:

In corrections it generates 3 warnings:

The 2 issues here are the failure of the corrections page to differentiate the invert_sized "Yes" and "No" animals in the feedback and give the size, and that also the corrections feature doesn't re-calculate the invert-sized "yes" value upon submitting the correction (in this example it incorrectly gave 1000cm as the size for row 16) - which was raised as being a bug in today's meeting:

@atcooper1

[utas-raymondng]

Use the main branch to do the testing

1. Change the last 1000/30 to 1 for few species, the "Invert measurements outside" shows in INFO instead of WARN after clicking validation

2. The wording become isInvertSizing instead of "Use InvertSizing" in the submit correction page

3. Now click cancel and go back to previous screen, change species from Hailotis rubra to Haliotis roei, submit correction again, then you can see duplicate line.