Survey group incomplete error confusing

[ccmoloney]

Software version: 0.0.344 Environment: systest

From user:

Not sure how to interpret this. I think it means there is a duplicate row of depth and species combination within a survey? or is it a forgotten "survey not done"?

[LizziOh]

Related to this is that the staging results suggested 0 incomplete surveys but the blocking errors suggest the "survey group incomplete"

[LizziOh]

The only thing I can think of is that the Survey group validation check for M3 (but not M1 and M2) is ignoring the "survey not done" entry, but the initial staging ones are not? @ccmoloney

[bpasquer]

Not sure what you mean by " the initial staging ones" But the Survey group validation check should be method-dependent. Is it @nspool ?

[LizziOh]

@bpasquer I mean in the "Summary" (top of picture in my last comment) it states there are 0 incomplete surveys. But in the blocking errors that check whther a survey group is complete it has not recognised the SND for 2 of the M3 sites. (jmp-s42 and jmp-s5 in the last file i staged). If you filter for those sites you'll see a SND entry for M3 for 10.1m and 5.1m respectively.

[ccmoloney]

@LizziOh What sheet is this related to?

[bpasquer]

Looking at the file, it looks like the M3 SND entries for 10.1 and 5.1 are not done on the same date which explains why the warning is raised. For example jmp-s5: there is no M3 transect done at depth 5.1 on the 05/04/2021 , but there is one done JMP-S5 on the 05/05/2021 ( a month later). Same problem for jmp-s42.

[LizziOh]

Ah that would explain it, however the initial 48 distinct surveys found then in the "summary" are not complete surveys? If the dates were correct there should have been 44 complete surveys found, and if the dates were wrong it should have found multiple incomplete surveys? (for example as the RLS assistant does).

[bpasquer]

I think the summary list "distinct surveys", which is irrespective of their completeness, but @nspool can confirm

[nspool]

Can confirm that the Distinct Surveys count also included incomplete surveys.

[LizziOh]

I think there still remains issues with this, even in the newly released version 0.0.361 Firstly the summary warning and the blocking validations are inconsistent with each other, as the summary picks up 45 distinct surveys including 0 incomplete surveys. The blocking validations pick up 2 errors of incomplete survey groups under method 3 for sites JMP-S42 and JMP-S5, but not the missing survey groups for the ones with the different dates. In addition the UI now picks up only 45 distinct surveys when it previously picked up 48 (?). Can I please review the logic behind these validations as they seem wrong. Please have a look at this file, in the pivot table. Highlighted are the surveys that should be incomplete ie. for ATRC do not have complete records for M1,M2, M3. There correct number of real surveys is 48. If the UI stages the ones with incorrect dates as distinct surveys then it should have recorded 58 distinct, with 28 incomplete. Even if it disregards missing M3 methods there would be 10 incomplete surveys without M2 and M1 (from the M3 mis-labelled with the wrong dates from 3 sites). 2021-JurienBay-MethodsComparisonDataATRC_PBD-BKJ-ESOfinal2.xlsx

@nspool @atcooper1 @bpasquer

[LizziOh]

Edit: Distinct surveys in the summary list does not include those from new sites as it changed from 45 to 48 in the summary list once the new site was added.

[bpasquer]

The implemented "ATRC survey complete" and the "ATRC survey group complete" checks are described in the validation checks specification document

The specifications for "ATRC survey complete" are incorrect cause they don't check for the presence of M3 data.

[bpasquer]

@LizziOh The specification for the "survey group incomplete" check is also incorrect cause the validation is done at the survey level, not at the survey-method level.

[LizziOh]

Thanks @bpasquer for the clarification. I have adjusted the 2 relevant validations (Survey complete -ATRC and Both M1 & M2 &M3 present -ATRC) in that shared file and highlighted the specific parts that require changing.

[bpasquer]

@ccmoloney ATRC rules "Survey complete" and "M1,M2 and M3 present" should be corrected as per amended specifications

[bpasquer]

Both M1 & M2 &M3 present	Both	Implemented	ATRC	Warning	Check that surveys have both M1, M3 and M2 present,	Each survey, as identified by the site no/date/depth combination, must have at least one row of ‘1’ and one row of ‘2’, and one row of ‘3’ in the Method column.
Survey complete	Both	Implemented	ATRC	Warning	Check that survey data recorded for Block 1 and Block 2 for both M1 And M2, and Block 0 for M3.	Each survey (as identified by the combination of site, date and depth must have at least one row with a value of ‘1’ and one row with a value of ‘2’ for the Block column for records of ‘1’ and “2” for the Method column AND must have at least one row with a value of ‘0’ for the Block column for records of ‘3’ in the Method column

[LizziOh]

tested, working. thanks!