Review and update as required the Scotland Spreadsheets

[Captainkirkdawson]

@rhodamackenzie has reviewed the draft Scotland Spreadsheets. Those proposed changes need to be reviewed and incorporated as appropriate both into the spreadsheets and the record displays.

[Captainkirkdawson]

Comments are stored at https://drive.google.com/drive/u/1/folders/150Uh4r_lvGT9_nVaxf3R4wvvub3EoyDI

[geoffj-FUG]

A Draft handbook has been written for Scotland so that we do not lose the knowledge. It is currently being reviewed by Margaret Singleton (Scotland Coordinator). Geoff

[geoffj-FUG]

Story #1443 is critical to this story. The T names need to be sorted out before the spreadsheets and handbook are released. Geoff

[geoffj-FUG]

Margaret Singleton is reconciling the civil parishes and PARMS for Scotland. That means that some civil parishes will exceed the fctools parameters and she will not be able to make amendments. I have examined the 1841 and 1851 Scotland spreadsheets and all of the fields are in the England and Wales spreadsheets and have therefore been tested. The 1851 England and Wales and Scotland Spreadsheets are identical. The 1841 Scotland spreadsheet is identical to the 1851 Scotland spreadsheet except it has the Nationality field missing. (It s not identical to the England and Wales 1841 Spreadsheet). Can we therefore please create an 1851 Upload/ Replace and Download as Csvpro function for Scotland. This can be achieved by using the same code as for 1851 England and Wales. Can we therefore please create an 1841 Upload/ Replace and Download as Csvpro function for Scotland. This can be achieved by using the same code as for 1851 Scotland but with the Nationality field omitted. These are the only two functions required at this point. I will review the situation once Margaret has completed the civil parish reconciliation. If no more years are affected the story can go back on ice. Geoff

[AnneV-Learn]

@geoffj-FUG Created 1841 & 1851 Upload/ Replace and Download as Csvpro function for Scotland. Now ready for testing on Test3.

[geoffj-FUG]

Anne

I have downloaded 2 files and reloaded them – HS4_576 and HS5_576.

The 1851 file converted to csvpro OK and uploaded OK. All errors reported were as expected.

The 1841 file converted to csvpro but no schedule numbers downloaded. All schedule numbers in 1841 are 0. We had this problem when we developed the 1841 download with Kirk. There should be a condition in the code that allows for that 0 to be included in the download. It was a couple of years ago and I cannot remember how we did it in the end.

The downloaded file loaded OK but has 800+ Errors because of the missing schedule number of 0.

Geoff

From: Anne Vandervord @.> Sent: Friday, October 20, 2023 8:34 PM To: FreeUKGen/FreeCENMigration @.> Cc: Geoff J @.>; Mention @.> Subject: Re: [FreeUKGen/FreeCENMigration] Review and update as required the Scotland Spreadsheets (#927)

@geoffj-FUG https://github.com/geoffj-FUG Created 1841 & 1851 Upload/ Replace and Download as Csvpro function for Scotland. Now ready for testing on Test3.

— Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/927#issuecomment-1772497696 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFIQG7OGECQ5XQF3TRDYAJHTHAVCNFSM4QAKIM4KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXGI2DSNZWHE3A . You are receiving this because you were mentioned. https://github.com/notifications/beacon/AKCPIFNWYFYF2C6LRG2WUA3YAJHTHA5CNFSM4QAKIM4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGONGTCWIA.gif Message ID: @. @.> >

[AnneV-Learn]

@geoff-FUG I believe the issue/story relating to 0 schedule numbers was #1432. It is pretty complicated 😬, but I think we determined that there were some 1841 VLDs that had a particular problem and they would have to be adjusted manually by the coordinator. Could you please review #1432 and let me know.

[geoffj-FUG]

Anne

We seem to have an issue in the data.

I have downloaded the listing for the vld data for HS410576 and the schedule number is not populated. This is what is causing the issue of no schedule number of 0.

The dwelling number field is there and the position in household number is there.

You are correct in that this seems to be the same as #1432. This seems to be yet another iteration of the problem.

We need something in the Scotland download such as:

If year = 1941 and dwelling_number <> previous dwelling number

Schedule number = 0

Endif

I have cc’d you into this so that you get a copy of the listing.

Scotland used exactly the same spreadsheet as England and Wales in FC1. However when we looked at the schedule 0 issue we do not seem to have extended the fix to Scotland as we had paused further work on Scotland.

Geoff

From: Anne Vandervord @.> Sent: Sunday, October 22, 2023 10:01 PM To: FreeUKGen/FreeCENMigration @.> Cc: Geoff J @.>; Mention @.> Subject: Re: [FreeUKGen/FreeCENMigration] Review and update as required the Scotland Spreadsheets (#927)

@geoff-FUG I believe the issue/story relating to 0 schedule numbers was #1432 https://github.com/FreeUKGen/FreeCENMigration/issues/1432 . It is pretty complicated 😬, but I think we determined that there were some 1841 VLDs that had a particular problem and they would have to be adjusted manually by the coordinator. Could you please review #1432 https://github.com/FreeUKGen/FreeCENMigration/issues/1432 and let me know.

— Reply to this email directly, https://github.com/FreeUKGen/FreeCENMigration/issues/927#issuecomment-1774075047 view it on GitHub, or https://github.com/notifications/unsubscribe-auth/AKCPIFN7BLX6ZWJMQWWKX6DYAUDIRAVCNFSM4QAKIM4KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXGQYDONJQGQ3Q unsubscribe. You are receiving this because you were mentioned. https://github.com/notifications/beacon/AKCPIFPVAK7ZCPQ2WWYEMPLYAUDIRA5CNFSM4QAKIM4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGONG7DZJY.gif Message ID: < @.> @.>

[AnneV-Learn]

@geoffj-FUG - Ok I have updated the code with what I hope is a fix. It is ready for re-testing on Test3.

[geoffj-FUG]

Anne

perfect!

This can now be deployed but the story itself will then need to go back on Ice as there is still more work to be done.

(1841, 1851 completed. 1861 - 1911 still to develop and test)

Geoff

[AnneV-Learn]

Deployment for 1841 and 1851 done 25/10

[geoffj-FUG]

Anne

Thank you

I have removed the ready to deploy label and moved this back to on ice.

Geoff

From: Anne Vandervord @.> Sent: Wednesday, October 25, 2023 9:33 PM To: FreeUKGen/FreeCENMigration @.> Cc: Geoff J @.>; Mention @.> Subject: Re: [FreeUKGen/FreeCENMigration] Review and update as required the Scotland Spreadsheets (#927)

Deployment for 1841 and 1851 done 25/10

— Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/927#issuecomment-1779066383 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFJGAYDJ4UN7NIIA3ULYBD2GTAVCNFSM4QAKIM4KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXHEYDMNRTHAZQ . You are receiving this because you were mentioned. https://github.com/notifications/beacon/AKCPIFIYKRIGKF2HOWN3OF3YBD2GTA5CNFSM4QAKIM4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGONIFGMDY.gif Message ID: @. @.> >

[geoffj-FUG]

I am advised by Margaret Singleton that there are many 1871 Scotland pieces plus some 1861, 1881 and 1891 Scotland pieces. These four years also require the spreadsheets activating. Geoff

[geoffj-FUG]

Using the 1861 England and Wales spreadsheet there are two changes that need to be made and tested. They also need to be made in 1871.

Address section - add rooms_with_windows. Rooms_with_windows will need testing. It is defined as: Field name rooms_with_windows Column Contains Rooms with Windows Year/s Used 1861 – 1911 What Can Be Entered Number or blank Description and Rules If there is nothing recorded leave it blank. Only enter it once for each schedule on the same row as the schedule number. Note: This is NOT the same as ‘number of rooms’ in the England and Wales censuses.

Individual section: Add school_children. School children will need to be tested. It is defined as: Field name school_children Column Contains Children at School Year/s Used 1861 and 1871 only What Can Be Entered Number or blank Description and Rules Enter the number of children at school when shown on the census. Enter this only once for each schedule on the same row as the schedule number. If there is nothing recorded leave it blank.

I will test the uploading as there are only two new fields involved.

Using the 1871 England and Wales spreadsheet as a template there are a few changes that need to be made and tested:

ED Section - Remove sanitary_district. Add police_district. Police_district will need testing. It is defined as: Field name police_district Column Contains Police District Year/s Used 1871, 1881 and 1901 What Can Be Entered Narrow Text Description and Rules You can find this at the top of each enumerated page census. This was also called Police Burgh. If the information has not been recorded enter a – (hyphen) at the start of each ED. Only enter this information once at the beginning of the ED unless it changes. Do not punctuate. If the content of any of the fields in this section changes it will also need to be re-entered.

Address section - add rooms_with_windows. Rooms_with_windows will have been tested in 1861. It is defined as: Field name rooms_with_windows Column Contains Rooms with Windows Year/s Used 1861 – 1911 What Can Be Entered Number or blank Description and Rules If there is nothing recorded leave it blank. Only enter it once for each schedule on the same row as the schedule number. Note: This is NOT the same as ‘number of rooms’ in the England and Wales censuses.

Individual section: Add school_children. School children will have been tested in 1861. It is defined as: Field name school_children Column Contains Children at School Year/s Used 1861 and 1871 only What Can Be Entered Number or blank Description and Rules Enter the number of children at school when shown on the census. Enter this only once for each schedule on the same row as the schedule number. If there is nothing recorded leave it blank.

I will test the uploading as there is only one new field involved.

Geoff

[AnneV-Learn]

@geoffj-FUG 1861,1871,1881,1891 Scotland. Now ready for testing on Test3.