Update of the Validation Process Step 3

[geoffj-FUG]

This is part of the Epic #1783. Please read the document linked to #1783 when considering this story.

Step 1 of this Epic introduces Pre-validation to the validation process. Step 2 of this epic amends the validation screen actions Step 3 of this Epic will amend the propagation process so that the pre-validation data is built.

There are two particular propagation actions available on the validation screen. Propagate POB Propagate Notes

When these actions are selected there should be a Propagation process initiated: If the Chapman Code associated with the Alternative Place of Birth is not the same as the Chapman Code of the Piece and the Chapman Code is not UNK and the Chapman Code is not OVF then the current actions take place when either Propagation Action is selected. This restricts any propagation to the piece. If only the propagate Notes action is selected then the Notes are propagated throughout the piece as happens at the moment.

When the Propagate POB Action is selected and the Chapman Code is the same as the Chapman Code for the piece or is OVF or one of the Country codes (ENG, SCT, IRL, WLS, CHI) then the propagation screen used in revalidation of vld files is presented. The validator will be given the option of propagating the POB, or Both the Alternative POB and the Notes ED – The alternative POB will be propagated throughout the ED and the POB warning flag will be set to true for the records amended. Piece - The alternative POB will be propagated throughout the piece and the POB warning flag will be set to true for the records amended. Whole Collection - The alternative POB will be propagated throughout the piece and the POB warning flag will be set to true for the records amended. The verbatim POB and the alternative POB will be added to the pre-validation data set so that they can be used to pre-validate any future pieces in any county. No propagation– The POB warning flag will be set to true for the record. This is a fail safe in case propagate is selected incorrectly. Both – The Notes will be propagated together with the alternative POB and if whole collection is selected then the Notes will be added to the pre-validation data set with the Alternative POB information so that they can be used to pre-validate any future pieces. The alternative POB change part of this choice will trigger the setting of the POB warning flag to True.. Once the propagation choice has been actioned the validator will be returned to the previous validation screen.

Once this story is completed and deployed the epic will have been completed.

Geoff

[geoffj-FUG]

Note that I have removed UNK from the above list. If somebody propagates an UNK for an abbreviation that could exist in another county, it will cause problems. I have therefore removed UNK from the set of Chapman Codes.

Geoff

[AnneV-Learn]

@geoffj-FUG Sorry but I am rather confused with the valid chapman codes in the above spec. (Alternative POB Chapman Code/Verbatim POB Chapman Code/Chapman code for the Piece), it would be really helpful if you could give some examples of valid and invalid combinations for the decision logic.

[geoffj-FUG]

Examples: The county that the piece belongs to is DEV. If the verbatim place of birth has a Chapman Code of DEV the alternative POB can be added to the pre-validation data set,. If the verbatim place of birth has a Chapman Code of OVF the alternative POB can be added to the pre-validation data set. If the verbatim place of birth has a Chapman Code that is one of the country codes (ENG, SCT, IRL, WLS, CHI) the alternative POB can be added to the pre-validation data set. If any other Chapman code is recorded against the verbatim place of birth then the alternative POB cannot be added to the pre-validation data set. Propagation can only happen at the piece level as happens now, but it can be accepted for all entries in the piece.

Geoff

[AnneV-Learn]

Thanks @geoffj-FUG

[AnneV-Learn]

@geoffj-FUG ready for testing on Test3

[geoffj-FUG]

Anne

Anne Have we taken into account the HAM and YKS scenarios? If the piece is in HAM the IOW chapman codes can be propagated as well as the HAM. If the piece is in YKS then any Riding can be propagated. If the piece is CHI all four Islands (JSY, GSY, ALD, SRK) can be propagated. They seem to be the only ones we need to worry about.

It came to mind because I have just tested the Gazetteer issue.

Geoff

[AnneV-Learn]

@geoffj-FUG Oh no I didn’t think of those. Can you continue testing anyway I’ll include that in any updates I have to make as a result of your testing (or on its own if there are none 😀)

[geoffj-FUG]

Testing Story #1787 This test also covers the previous two steps in case changes have affected them. 6 SOM files that have been transcribed loaded into test3 as testing files. – RG14_14250 to RG14_14255. All set as ready for validation. Downloaded RG14_14250. Replaced it. All OK. Changed POB valid column of 6 entries to show True instead of False but did not change record valid. Reloaded. All entries retested and POB valid reset to False.Correct. Edited file again and changed same 6 record valid from false to true. All three entries on record now show true. Replaced file. Entries accepted as valid and Warning count decreased. Correct. Edited file and changed age to 999. Valid settins set to True, True False. No Warning for 999 entry and Replaced file. All three valid fields tested as True. Changed age to 150 and Replaced file. 150 came up as an Error. Correct.

Recommendation: On Replace or Re-process the Record should be retested if any one valid field is false. Some validators download the spreadsheet and then adjust the POBs and change the True/False flag when this has been done. They do not know whether there is an invalid entry elsewhere and this habit nullifies the previous test. By making this change this habit is eliminated where the record valid field has been changed to True, but the other related field has not been changed. It does not stop editing the spreadsheet, but it does provide a measure of control. This is not an issue where validators use the validation program.

[geoffj-FUG]

Started validation: Edited records. Entered alternative POB. Submit. Screen shows Edit Entry or Accept Entry. No propagation choice until Accept actioned. Then get ED or Piece (Correct as these places were not in SOM). Is there a reason that the propagate was moved to after the Accept? Is it because we are accepting them all?

Logic is still sound. I am just wondering about the change.

When validating a SOM POB whole collection correctly displayed.

Geoff

[AnneV-Learn]

@geoffj-FUG Re: Recommendation: On Replace or Re-process the Record should be retested if any one valid field is false

With the 2 new invalid flags won’t they realise that a non-POB field is invalid and so deal with that too?

[AnneV-Learn]

@geoffj-FUG Re: Is there a reason that the propagate was moved to after the Accept? Is it because we are accepting them all?

I coded it (rightly or wrongly) that Propagation is only offered when the record_valid is true thinking that otherwise they could re-edit the POB and potentially get into a situation of propagating an invalid POB. It made the process flow logic more straightforward too!

[geoffj-FUG]

I am happy with that. The end user logic is quite plain. It will be able to be charted.

Geoff

From: Anne Vandervord @.> Sent: Friday, August 2, 2024 6:18 PM To: FreeUKGen/FreeCENMigration @.> Cc: Geoff J @.>; Mention @.> Subject: Re: [FreeUKGen/FreeCENMigration] Update of the Validation Process Step 3 (Issue #1787)

@geoffj-FUG https://github.com/geoffj-FUG Re: Is there a reason that the propagate was moved to after the Accept? Is it because we are accepting them all?

I coded it (rightly or wrongly) that Propagation is only offered when the record_valid is true thinking that otherwise they could re-edit the POB and potentially and into a situation of propagating an invalid POB. It made the process flow logic more straightforward too!

— Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/1787#issuecomment-2264832830 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFL7IIUTRR42QRNIIWDZPM6DRAVCNFSM6AAAAABI74PE36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRUHAZTEOBTGA . You are receiving this because you were mentioned. https://github.com/notifications/beacon/AKCPIFIEGFYGMCTMS7S436DZPM6DRA5CNFSM6AAAAABI74PE36WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUG72NT4.gif Message ID: @. @.> >

[geoffj-FUG]

Anne I have just started testing propagation. RG14_14251. Selected North Petherton which is not in the test3 Gazetteer. Entered an alternative POB as South Petherton (which is wrong but is in the Gazetteer). Accepted entry. Offered Propagation. Chose Whole collection. Selected next warning and it moved to the next North Petherton. South Petherton had been entered by the system but the POB-valid flag had not been amended to True. As a reult the non_POB_valid flag was not true either. Under the new system when we propagate the propagated POBs should be flagged as true thoughout the piece. If both the pob valid and non pob valid flags are true the record valid should change to true.

I have no way of confirming that the alternative for North Petherton has ended up in the pre-validation data collection as I have no access to it. I have entered 2 places today - SOM Bristol Bedminster (alternative SOM Bedminster) and SOM North Petherton (Alternative SOM South Petherton). Can you please confirm that they are there. I will leave it for now while you catch up! Geoff

[AnneV-Learn]

@geoffj-FUG The records are there: { "_id" : ObjectId("66ac918bbd30f25cfa6157e8"), "scope_year" : "ALL", "scope_county" : "ALL", "match_verbatim_birth_county" : "SOM", "match_verbatim_birth_place" : "Bristol Bedminster", "new_birth_county" : "SOM", "new_birth_place" : "Bedminster", "new_notes" : "", "propagate_pob" : true, "propagate_notes" : false, "created_by" : "GeoffJ", "u_at" : ISODate("2024-08-02T07:58:03.185Z"), "c_at" : ISODate("2024-08-02T07:58:03.185Z") } { "_id" : ObjectId("66acbe5abd30f25cfa6157ed"), "scope_year" : "ALL", "scope_county" : "ALL", "match_verbatim_birth_county" : "SOM", "match_verbatim_birth_place" : "North Petherton", "new_birth_county" : "SOM", "new_birth_place" : "South Petherton", "new_notes" : "", "propagate_pob" : true, "propagate_notes" : false, "created_by" : "GeoffJ", "u_at" : ISODate("2024-08-02T11:09:14.112Z"), "c_at" : ISODate("2024-08-02T11:09:14.112Z") }

[AnneV-Learn]

@geoffj-FUG I’ll get on to the other issue in the next few days.

[geoffj-FUG]

RG14_14252 set to validate and pre-validated. Entry previously added to re-validation data set has appeared correctly in the spreadsheet. Once the previous amendments are made the POB part of this should be OK. I will now test the ppropagation of notes and POB plus notes. Geoff

[geoffj-FUG]

Entry made to RG14_14252 against Kingston SOM. Entry was alternative POB Kingston St Mary. Note was POB - or Kingston Seymour. Propagation was offered prior to Accepting the record (the old way of doing it). When propagation actioned I was only given the option to propagate Notes. I cancelled the process at that point. Can we offer Propagation after acceptance as we did for POBs, please? Can we offer POB, Notes or Both as per the re-validation process please?

(Or have I got ahead of you? If so sorry).

Geoff

[AnneV-Learn]

@geoffj-FUG Probably best to wait until I’ve corrected the issue you raised about ‘propagated POBs should be flagged as true thoughout the piece’ before continuing testing. That should be ready on Monday.

[AnneV-Learn]

@Geoffj-FUG A question. Currently the code only uses the propagation collection during pre-validation. So no look up is done in Reload, Re-process or when validating an edit. Is that what you want?

[AnneV-Learn]

@geoffj-FUG Updated code (to handle the issues you raised) now ready for testing on Test3.

[geoffj-FUG]

RG14_2353 set for validation. Prevaidation done. Entry for SOM Kingston validated - alt POB SOM Kingston St Maty, Notes or Kingston Seymour? Entry accepted. Propagated whole collection. Spreadsheet downloaded. All entries for Kinston have been updated. First entry has test result of TRUE, TRUE, TRUE. Other entries are still false.

Within the piece all matching entries should have their alternative POB accepted when the alternative POB is propagated (hence the extra test fields). Notes cannot be set to True because we do not know what other problems there are.

Northh Petherton had an entry of South Petherton applied by prevalidation as an alternative. I accepted the first entry that occurred. We had not thought of this before but because the alternative option was applied at prevalidation it does not need to be offered propagation. We only need to accept the entry and the restof the matching POBs should be accepted as well.

Geoff

[AnneV-Learn]

@geoffj-FUG Ok I thought I had fixed the first issue there but clearly not. I’ll have to look at it (and the second issue you have raised) when I get back from my holiday in first week of Sept.

[AnneV-Learn]

@geoffj-FUG I have made some updates which are now ready for testing on Test3. It will be best to delete and reload the CSV file if you are trying to re-test an existing file.

[DeniseColbert]

@geoffj-FUG to test

[geoffj-FUG]

Test of new Validation on test3 Edited RG10_2242 so that it was not being validated. Removed all alternative POBs. Uploaded file. 67 Warnings. 12 related to non POB warnings. The rest are POB warnings. Set to start validation. Yes for prevalidation. Report received - Alternate POB / Notes were updated on the following lines [173, 1473, 1591]. Downloaded spreadsheet. Examined False entries. All flags correct. Examined alternative POBs. All conform to test dataset. Commenced validation. Entered alternative POB for entry. Accepted it. Offered propagation at up to piece level. Correct. Entered alternative POB for CON place. Offered propagation to Whole level. Correct. POB propagated. Entered alternative POB for WLS place. Offered propagation to Whole level. Correct. POB propagated. Reprocessed file. Downloaded spreadsheet. Looks OK. Added note to POB. Offered propagation correctly at file level. Propagated. Edited alternative POB for OVF place. Offered propagation at Whole level. Propagated. Correct. Propagated CON place. Downloaded spreadsheet. Checked all flags had changed. Correct. Cleared all Warnings. Reprocessed file. Incorporate – warning the vld exists. Correct. Deleted vld file. Incorporated file. Reported successful. Deleted incorporated file. Uploaded original file. Set for validation. Prevalidation initiated. Downloaded as spreadsheet. Prevalidation checked. POBs added on first validation have been included by prevalidation. All looks OK. These three stages can be deployed. Geoff

[AnneV-Learn]

@Vino-S FYI branch fc_1785_86_87_93_av is the branch that needs to be deployed. I have just merged in the latest Master to that branch (and pushed to origin) to make sure that there are no merge issues when you come to deploy to Production.

[DeniseColbert]

Done, closing