Revisit places for Chester le Street

[benwbrum]

This appears in FreeCEN2 as both "Chester le Street" and "Chester-le-Street". If we track this down we might finally get places correct.

[dougkdev]

Chester le Street and Chester-le-Street both exist in freecen1 (they are in different sets of years). This is not due to a problem with how places are treated in FC2, but due to differences in how the places themselves were specified for different years in FC1. The purpose of this story is to get the places importing properly. Functionality for editing places is in the roadmap in a future version and Chester le Street can be corrected once that functionality is introduced.

I do think we should consider opening a story to add the years with data into the places selection list, like it is in FC1. That would reduce confusion when users see things like Chester le Street / Chester-le-Street. It is confusing in FC2 to have both places because nothing indicates what the difference is, but it is obvious at a glance in FC1 that they are from different census years. Listing the years also indicates to the user whether data is available for a place during a given year without having to look through the database coverage pages and without unnecessary searches on places with no data.

Regarding importing places from FreeCen1, I believe Ben's original approach was the correct approach and I have added his code back in to the freecen_parsing codebase. I fixed a minor bug in how piece numbers were being treated, which was causing several .vld files to fail validations when being imported. The SCS chapman code was also missing in FC2, preventing Scottish Shipping pieces from validating. It is now added. We will push the fixes to test2 and rebuild the database before the upcoming scrum.

I discovered data problems in several of the "fixed" PARMS.DAT files, which I was able to correct manually, and noticed that 1881 previously had only two PARMS.DAT files but 1881/SOM was added just this past August (at least that's what it looks like in the set of files I had access to). That leads to a few questions about how/when those fixed PARMS.DAT files are generated/added/corrected, and whether I have access to the most recent files. I will discuss that offline with Brenda and Ben.

On Wed, Dec 16, 2015 at 11:27 AM, sean-wood notifications@github.com wrote:

Assigned #75 https://github.com/FreeUKGen/FreeCENMigration/issues/75 to @dougkdev https://github.com/dougkdev.

— Reply to this email directly or view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/75#event-493801081.

[dougkdev]

Wait for #80 to be done before testing places.

[PatReynolds]

Having a Census Place only existing at a Time looks to me to be a good way forwards, although it would mean in searching, we would need a work-around rather than simply presenting the list of all the Census Places that exist (i.e. only one Chester le Street would appear).

As Other Place (e.g. place of birth) is going to be treated differently, it would not need an associated Time.

Best wishes,

Pat

- -

Dr Pat Reynolds Executive Director Free UK Genealogy http://www.freeukgenealogy.org.uk/ A charity registered in England and Wales, number 1096940

​+44 ​1904 541411​ +44 7943 145387

36 Albemarle Road, York, YO23 1ER, UK

[dougkdev]

I agree one place name makes more sense. It would be better to correct duplicates like Chester le Street in the data than to create a search work-around for them, though. It is just a matter of editing the inconsistent place name in the .csv file that lists the place for the corresponding piece, and then uploading the corrected PARMS.DAT file created from the corrected csv.

On Fri, Jan 22, 2016 at 8:47 AM, PatReynolds notifications@github.com wrote:

Having a Census Place only existing at a Time looks to me to be a good way forwards, although it would mean in searching, we would need a work-around rather than simply presenting the list of all the Census Places that exist (i.e. only one Chester le Street would appear).

As Other Place (e.g. place of birth) is going to be treated differently, it would not need an associated Time.

Best wishes,

Pat

- -

Dr Pat Reynolds Executive Director Free UK Genealogy http://www.freeukgenealogy.org.uk/ A charity registered in England and Wales, number 1096940

​+44 ​1904 541411​ +44 7943 145387

36 Albemarle Road, York, YO23 1ER, UK

— Reply to this email directly or view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/75#issuecomment-173957628 .

[FreecenBren]

Correcting the PARMS file will still not stop the inconsistences coming through.

A PARMS file is only a list of Registration Districts (RD’s) within that County relevant to each PIECE number for that Census Year.

When we look at the 1841 Census for Durham for instance, which we have not as yet done the PARMS file we have WARDS (Called RD’s in later Census) for:-

Chester HO107296 to HO107302 (Yes just Chester)

Islandshire HO107318

Norhamshire HO107319

Durham City HO107320

Sunderland City HO107321

All of which are not as yet included in the DURHAM list we have on the testing page.

We then have to ask the question, “do we then need to get all the PARMS done of all the CENSUS years not as yet done from 1841 to 1891 (excluding 1881) to ensure that we do not have any more failing in the future?” Just a thought!

While on the subject of RD’s it has been asked by the Checkers that we also have an option to search by POB.

The RD’s lists are only part of the POB’s.

The other column in the PARMS called ‘Comprising’, is a list of ‘Parishes’ according to the Census details provided by the National Archives from where we produce the PARMS files.

They do not include the Villages/Hamlets’ etc., that are also listed in the National Archives Piece details.

We use the list in VALDREV to validate a POB, but this was created many years ago and is not a full list.

The Validator can add any POB to a ‘supplementary list’ in their own VALDREV and it will be accepted.

This will then in turn be used in the final VLD it creates that is uploaded.

Each Validator can move his new POB list that includes his own ‘Supplementary list’ to the next PIECE they validate. That saves them having to check any additions again that they added before, this also includes any variations in spelling.

Also the list in VALDREV does not include the YKS issue.

I have converted the basic list of POB’s from VALDREV to a CSV and have attached herewith.

I have included in the list the YKS POB’s that were added at a later date for ERY/WRY/NRY (Yorkshire Ridings)

This list however does not mean that it contains all the POB’s. We can deal with this when we get to decide how we are going to deal with the Front End of how the Transcribing, Checking and Validating process and which POB database we are going to use.

But it does give you a list that we can work from if need be.

Sorry if it a bit long winded but as Pat mentioned POB’s I thought it was all relevant.

Brenda

From: dougkdev [mailto:notifications@github.com] Sent: Saturday, January 23, 2016 7:24 PM To: FreeUKGen/FreeCENMigration FreeCENMigration@noreply.github.com Subject: Re: [FreeCENMigration] Revisit places for Chester le Street (#75)

I agree one place name makes more sense. It would be better to correct duplicates like Chester le Street in the data than to create a search work-around for them, though. It is just a matter of editing the inconsistent place name in the .csv file that lists the place for the corresponding piece, and then uploading the corrected PARMS.DAT file created from the corrected csv.

On Fri, Jan 22, 2016 at 8:47 AM, PatReynolds notifications@github.com wrote:

Having a Census Place only existing at a Time looks to me to be a good way forwards, although it would mean in searching, we would need a work-around rather than simply presenting the list of all the Census Places that exist (i.e. only one Chester le Street would appear).

As Other Place (e.g. place of birth) is going to be treated differently, it would not need an associated Time.

Best wishes,

Pat

- -

Dr Pat Reynolds Executive Director Free UK Genealogy http://www.freeukgenealogy.org.uk/ A charity registered in England and Wales, number 1096940

​+44 ​1904 541411​ +44 7943 145387

36 Albemarle Road, York, YO23 1ER, UK

— Reply to this email directly or view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/75#issuecomment-173957628 .

— Reply to this email directly or view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/75#issuecomment-174214049 .

[dougkdev]

Thanks, Brenda. I don't think the attachment made it through the github reply address.

Regarding the places that do not show up on the places list even though they are in the PARMS files, we only display a place in the search list if there are census records in the database for that place. So it is fine for the place/piece combinations in the PARMS files to be there in advance, those places just won't be displayed in the search list until after records have been transcribed for those places and added to the database. For the duplicate place names like Chester-le-Street / Chester le Street, making the place names consistent in the PARMS files will correct the issue of duplicates places in the list on the search page. As you indicated, POB will be more complicated, but the census places are pulled straight from those PARMS files.

Thanks again.

On Sun, Jan 24, 2016 at 5:52 AM, Brenda notifications@github.com wrote:

Correcting the PARMS file will still not stop the inconsistences coming through.

A PARMS file is only a list of Registration Districts (RD’s) within that County relevant to each PIECE number for that Census Year.

When we look at the 1841 Census for Durham for instance, which we have not as yet done the PARMS file we have WARDS (Called RD’s in later Census) for:-

Chester HO107296 to HO107302 (Yes just Chester)

Islandshire HO107318

Norhamshire HO107319

Durham City HO107320

Sunderland City HO107321

All of which are not as yet included in the DURHAM list we have on the testing page.

We then have to ask the question, “do we then need to get all the PARMS done of all the CENSUS years not as yet done from 1841 to 1891 (excluding 1881) to ensure that we do not have any more failing in the future?” Just a thought!

While on the subject of RD’s it has been asked by the Checkers that we also have an option to search by POB.

The RD’s lists are only part of the POB’s.

The other column in the PARMS called ‘Comprising’, is a list of ‘Parishes’ according to the Census details provided by the National Archives from where we produce the PARMS files.

They do not include the Villages/Hamlets’ etc., that are also listed in the National Archives Piece details.

We use the list in VALDREV to validate a POB, but this was created many years ago and is not a full list.

The Validator can add any POB to a ‘supplementary list’ in their own VALDREV and it will be accepted.

This will then in turn be used in the final VLD it creates that is uploaded.

Each Validator can move his new POB list that includes his own ‘Supplementary list’ to the next PIECE they validate. That saves them having to check any additions again that they added before, this also includes any variations in spelling.

Also the list in VALDREV does not include the YKS issue.

I have converted the basic list of POB’s from VALDREV to a CSV and have attached herewith.

I have included in the list the YKS POB’s that were added at a later date for ERY/WRY/NRY (Yorkshire Ridings)

This list however does not mean that it contains all the POB’s. We can deal with this when we get to decide how we are going to deal with the Front End of how the Transcribing, Checking and Validating process and which POB database we are going to use.

But it does give you a list that we can work from if need be.

Sorry if it a bit long winded but as Pat mentioned POB’s I thought it was all relevant.

Brenda

From: dougkdev [mailto:notifications@github.com] Sent: Saturday, January 23, 2016 7:24 PM To: FreeUKGen/FreeCENMigration FreeCENMigration@noreply.github.com Subject: Re: [FreeCENMigration] Revisit places for Chester le Street (#75)

I agree one place name makes more sense. It would be better to correct duplicates like Chester le Street in the data than to create a search work-around for them, though. It is just a matter of editing the inconsistent place name in the .csv file that lists the place for the corresponding piece, and then uploading the corrected PARMS.DAT file created from the corrected csv.

On Fri, Jan 22, 2016 at 8:47 AM, PatReynolds notifications@github.com wrote:

Having a Census Place only existing at a Time looks to me to be a good way forwards, although it would mean in searching, we would need a work-around rather than simply presenting the list of all the Census Places that exist (i.e. only one Chester le Street would appear).

As Other Place (e.g. place of birth) is going to be treated differently, it would not need an associated Time.

Best wishes,

Pat

- -

Dr Pat Reynolds Executive Director Free UK Genealogy http://www.freeukgenealogy.org.uk/ A charity registered in England and Wales, number 1096940

​+44 ​1904 541411​ +44 7943 145387

36 Albemarle Road, York, YO23 1ER, UK

— Reply to this email directly or view it on GitHub < https://github.com/FreeUKGen/FreeCENMigration/issues/75#issuecomment-173957628

.

— Reply to this email directly or view it on GitHub < https://github.com/FreeUKGen/FreeCENMigration/issues/75#issuecomment-174214049> .

— Reply to this email directly or view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/75#issuecomment-174293717 .

[PatReynolds]

I am afraid we didn't get any additional applicants, so won't be interviewing on Thursday.

I will see if it is possible/practicable to widen the internship to recent graduates and/or those who are not from the University of York.

With best wishes,

Pat - -

Dr Pat Reynolds Executive Director Free UK Genealogy http://www.freeukgenealogy.org.uk/ A charity registered in England and Wales, number 1096940

​+44 ​1904 541411​ +44 7943 145387

36 Albemarle Road, York, YO23 1ER, UK