Enhance the check_names function because of "not a character" entities in the results

[tommasobattisti]

This issue is meant to host suggestions in terms of words and regex patterns to consider when the results have to be cleaned.

[eliarizzetto]

Output file of run.py: results of the function for characters extraction and gender recognition.

output_01-08_characters_and_gender.txt

[tommasobattisti]

Leave a comment in form of:

• The occurrence you have found in the results.
• The type of operation you want to perform on that occurrence (or similar ones); E.g. substring removal; substring replacement; avoid including strings that contain certain words; etc.

[tommasobattisti]

• "naval heroes"
• remove from the set the strings like /\w+\sheros?/

[tommasobattisti]

• "christmas eve"
• remove from the set the strings like /christmas\s\w+/

[tommasobattisti]

Book 1857_browne-grannys-wonderful-chair:

• “Majesty”: do not include
• “Court”: do not include
• “Royal Prince” and “Crown Prince”: do not include (do not include /w+\sprince(ss)?/ but include strings such as “Princess Jane” or “Prince Mark” )
• “Michaelmas”: day of San Michael --> do not include
• “Frostyface” retrieved only, while it's “Dame Frostyface” and the “Dame” before the name is actually useful to put the string in the list of feminine names
• The same occurs for: “Greendalind” (Princess Greendalind); “Wantall” (Queen Wantall); “Winwealth” (King Winwealth); “Greensleeves” (Lady Greensleeves); "Wisewit" (Prince Wisewit).

[tommasobattisti]

Results at the current state of the branch (August 3, 2022):

output_01-08_characters_and_gender.txt

[tommasobattisti]

Book 1857_browne-grannys-wonderful-chair:

* “Majesty”: do not include

* “Court”: do not include

* “Royal Prince” and “Crown Prince”: do not include (do not include /w+\sprince(ss)?/ but include strings such as “Princess Jane” or “Prince Mark” )

* “Michaelmas”: day of San Michael --> do not include

* “Frostyface” retrieved only, while it's “Dame Frostyface” and the “Dame” before the name is actually useful to put the string in the list of feminine names

* The same occurs for: “Greendalind”  (Princess Greendalind); “Wantall” (Queen Wantall); “Winwealth” (King Winwealth); “Greensleeves” (Lady Greensleeves); "Wisewit" (Prince Wisewit).

---

SOLVED

[tommasobattisti]

Book: 1888_wilde-the-happy-prince-and-other-tales.txt:

• “Logic”: do not include since it's not a character
• "Huzza”: do not include since it's an exclamation
• “Climb”: do not include
• “Cornish”: do not include (maybe with /w+ish/ ? so to automatically exclude also English, Scottish, etc.)
• “Wild Basil”: do not include
• “Gilly”: do not include
• “Crocuses”: do not include
• “Pylotechnic”: do not include
• “Echo”?

[tommasobattisti]

Book: 1872_de-la-ramee-a-dog-of-flanders.txt:

• "Mechlin": do not include (from Mechlin lace)
• "Antwerp": do not include (it's a town)
• "Alvan": do not include
• "Christmas Eve": do not include
• "Daas", "Jehan" and "Jehan Daas" are the same character
• "Calvary", "Heaven" and "Hell": do not include
• "Kermesse", "Kyrie Eleison" and "Agni Dei": do not include
• "Ardennois": do not include and "Ardennoise Patrasche", only Patrasche is the name
• "Stavelot": do not include (it's a city)

[tommasobattisti]

Book 1865_carroll-alices-adventures-in-wonderland.txt:

• "Grief": do not include
• "Hare" is a character. However "March hare" should be retrieved instead (this is not a real problem, just a caveat)
• "Geography": do not include
• "Latitude" and "Longitude": do not include
• "Collar": do not include
• "Footman" is retrieved but it appears many times in the text also as a lowercase word. It should be retrieved as "Frog Footman" or "Fish Footman" instead.
• "Beau": do not include (comes from "Beau-ootiful!")

  ---

  TO BE COMPLETED

[eliarizzetto]

CONTINUES: Book 1865_carroll-alices-adventures-in-wonderland.txt:

• "Grief": do not include
• "Hare" is a character. However "March hare" should be retrieved instead (this is not a real problem, just a caveat)
• "Geography": do not include
• "Latitude" and "Longitude": do not include
• "Collar": do not include
• "Footman" is retrieved but it appears many times in the text also as a lowercase word. It should be retrieved as "Frog Footman" or "Fish Footman" instead.
• "Beau": do not include (comes from "Beau-ootiful!")

---

Do NOT include:

• "Kings" (common noun)
• "Pray" (verb)
• "Shy" (exclamation)
• "Mystery" (common noun)
• "Ahem" (exclamation)
• "Number One" (common noun)
• "Latin Grammar" (common noun)
• "Beautiful Soup" (common noun)
• "Pennyworth" (adjective)
• "Seaography" (common noun)

Other operations:

• "Mock Turtle Soup" ---> just "Mock Turtle"

• "Cheshire Cat" = "Cheshire Puss" --> unify!

• "elsie lacie" comes from "Elsie, Lacie, and Tillie" --> do not include OR separate them (just 1 occurence)

• MISSED DETECTION: "Bill" (15+ occurences)

[eliarizzetto]

1877_molesworth-the-cuckoo-clock

Either there's been some kind of error in pre-processing this text or it has been badly OCRed originally: words like "sothat", "hewas", "forit" originate from whitespaces being incorrectly eliminated.

Warning Attention:MAKE SURE THE FILE YOU HAVE LOADED IN THE CORPUS YOU ARE USING IS THE CORRECT ONE, I.E. THE ONE BELOW: The Cuckoo Clock

I have just changed it in my corpus, so you probably need to change it too!

[eliarizzetto]

1877_sewell-black-beauty

Do NOT include:

• "Gee" (exclamation)
• "Wells" (city)
• "Lansdown" (city)
• "Butcher" (common name)
• "Hard Times My" (Chapter title + first word of paragraph)
• "Bishopsgate" (city)
• "Earlshall" (city)
• "Fairstowe" (city)
• "Tattersall" (city)
• "Birtwick" (city)

Other remarks:

• "Jerry Barker" = Jeremiah Barker
• "joe green joe green" --> Joe Green
• "Langley" is not a female character, he's mentioned twice as "Captain Langley" and in the sentence "you are a military man, Langley"
• "Ginger One" = Ginger
• "Merrylegs" is a male horse
• "Filcher" is a man

[eliarizzetto]

Note New output file to check results against

This is the output of run.py (character extraction + gender recognition) with a filtering threshold applied: the charEx function only returns words occurring uppercase only and more than 2 times.

output_run_threshold2_090822.txt

[tommasobattisti]

Note New output file to check results against

This is the output of run.py (character extraction + gender recognition) with a filtering threshold applied: the charEx function only returns words occurring uppercase only and more than once.

---

Updated version:

output_run_threshold2_160822.txt

[tommasobattisti]

Related to the last version of the results: (Part I)

Book: 1872_de-la-ramee-a-dog-of-flanders:

Do not include:

• “Antwerp”: City in Belgium
• "Mechlin”: (from Mechlin lace)
• "Christmas eve”
• “Calvary”
• “Ardennois”
• "Kermesse”

  ---

Book: 1857_browne-grannys-wonderful-chair:

• [x] Good

---

Book: 1877_molesworth-the-cuckoo-clock:

Do not include:

• "Mandarine Land”: do not include strings having form “/\w+\sland/“
• “Merrybrow”: from "Merrybrow Hall”

Other operations:

• 'grizzel', 'aunt grizzel', 'sister grizzel', 'miss grizzel': the same character
• 'tabitha', 'sister tabitha', miss tabitha', 'aunt tabitha': the same character
• “Mwyne” is actually “M'Swyne”, the 's is deleted (maybe add something like “delete 's only when after the 's' a space occurr)

  ---

Book: 1902_potter-the-tale-of-peter-rabbit:

• [x] Good

  ---

Book: 1888_wilde-the-happy-prince-and-other-tales:

Do not include:

• “Huzza”: exclamation
• “Pylotechnic”

  ---

Book: 1865_carroll-alices-adventures-in-wonderland

Do not include:

• “Beau”: (comes from “Beau-ootiful!”)
• “Latitude” and “Longitude”
• “Pray” (verb)
• “Mystery” (common noun)
• "Soo”: (from Soo-oop!)
• “Beautiful Soup” (common noun)

Other:

• MISSED DETECTION: “Bill” (15+ occurences)

---

Book: 1877_sewell-black-beauty

Do not include:

• "Duchess" and "Duke" alone (but consider them in the words that can determine the gender of the names)
• "Earlshall" (do not include strings like /\w+\s?hall/)
• "Birtwick" (city)

Other remarks:

• "Ginger One" should be "Ginger"

---

Other considerations not related to specific books:

It could be useful to not consider words that finish in "ish” so to avoid including “English”, “Scottish”, etc.

[tommasobattisti]

Related to the last version of the results: (Part II)

Book: 1899_nesbit-the-story-of-the-treasure-seekers

Do not include:

• "Blackheath" (village)
• "Malabar" (region)
• "Holloway" (city)
• "Balliol" (college)

Other remarks:

• "Ellis" should be female

---

Book: 1869_ewing-mrs-overtheways-remembrances

Do not include:

• "Bible"
• "Bath" (city)
• "Reka Dom" (place)

Other remarks:

• "Metcalfe" should be female (Mrs. Metcalfe)
• "Irishman" in this case is a character

  ---

Book: 1886_hodgson-burnett-little-lord-fauntleroy

Do not include:

• "Hooray" (exclamation)
• "Dorincourt" (do not include strings in form of /\w+\s?court/)
• "Erleboro" (village)

Other remarks:

• "Cedric", "Cedric Errol", "Captain Cedric Errol", "Captain Cedric", "Captain Errol" are the same character

---

Book: 1871_macdonald-at-the-back-of-the-north-wind

Do not include:

• "London" (it shouldn't be there since we addd the check on the cities' names)
• "Apostles"
• "Chiswick" (place)
• "How"
• "Sandwich" (city)

Other remarks:

• "Little Boy Blue" should be male: add "boy" and "girl" to the words that can determine the gender

  ---

Book: 1894_kipling-the-jungle-book

Do not include:

• "Bandar" from "Bandar-log" to indicate the population of monkeys
• "Waingunga"
• "Brumby"
• "Oho"
• "Bah"
• "Nay"
• "Whoof"
• "Somalo"
• "Jungle Law"
• "Master Words"
• "Walrus Islet" (do not include /\w+\sislet/) (island|isle|islet)
• "Poison People" (remove everything having form /\w+\speople/)
• "Novastoshnah" (place)
• "Kedda" and "Keddahs" are elephant traps
• "Seeonee" and "Wolf Pack" from "Seeonee Wolf Pack", the wolves that raised Moowgli
• "Segowlee" (place
• "Lukannon" and "Lukannon Beach"

Other remarks:

• "Nathoo" should be male
• "Little Brother", "Black Panther", "Black Snake" have also a proper name, but it could happen that strings with a similar form actually denote a character name in other cases

---

Not related to specific books:

• Consider: "Mr Higgs", "Higgs", "Mrs Higgs". If we remove "Higgs" we should not lose any information and our results should be better.

[tommasobattisti]

Related to the last version of the results: (Part III)

Book: 1876_twain-the-adventures-of-tom-sawyer

Do not include:

• "Sabbath"
• "Hallo"
• "Missouri"
• "Texas"
• "Spaniard"
• "Sho"
• "Tom Sawyer's Gang"
• "St Petersburg"
• "Ting" from "Ting-a-ling-ling!"
• "Saturdays"

Other remarks:

• "Widow Douglas" should be female (add widow to the words that determine the gender)

  ---

Book: 1883_stevenson-treasure-island

Do not include:

• "Bravo"
• "Scatter"
• "Avast"
• "Fontenoy"
• "Execution Dock"
• "Haylbowline Head" (place)
• "Lillibullero" (is a march)

Other remarks:

• "Long Johns" is actually "Long John's" and "Long John has been already retrieved
• "OBrien" is retrieved. In these cases, we should leave O'Brien

  ---

Book: 1857_hughes-tom-browns-school-days

Do not include:

• "Fa" from "Fa-a-a-ag!"
• "Berks" (county)
• "Gradus" (dictionary)
• "Wessex" (kingdom)
• "Christian"
• "Manger"
• "Saxon"
• "Farringdon" (place)
• "Hullo"
• "Lawford" (path)
• "East" "West" etc.
• "Doctor"
• "Ashdown" (place)
• "Crew"
• "Peacock" (place?)
• "Triste"
• "Latin"
• "Bible"
• "School"
• "Huzza"
• "Bilton Grange" (place)
• "Great Western" (railway)
• "Avon" (river)
• "Pebbly Brook" (brook-->little stream)
• "Wellesburn" (village)
• "Wantage" (village)
• "Hazeldown" (place)
• "Balliol" (college)
• "Martinmas" (st Martin's day)
• "Himalayas"
• "Harkaway" (?)
• "Blawing Stwun (place)

---

Further considerations:

• It could be useful to include only the words that appear at least once in uppercase not after a full stop, double quotes and ‘

[tommasobattisti]

Related to the last version of the results: (Part IV)

Book: 1869_dickens-david-copperfield

Do not include:

• "Ahem"
• "Aha"
• "Hark"
• "Humph"
• "Umph"
• "Thankee"
• "London" (It shouldn't be here)
• "Gravesen" (place)
• "Master"
• "Chatham" (place)
• "Doctor"
• "Memorial" and everything ending with it
• "Latin"
• "Blackheath" (area in London)
• "Lowestoft" (place)
• "Putney" (district in London)
• "Salem House"
• "Port Middlebay"
• "Covent Garden"
• "Good God"
• "Good Heaven"
• "Windsor Terrace"
• "Suffolk Punches" (equine race)
• "Camden Town"
• "Dear Agnes" (do not include /dear\s.+/)
• "Westminister Abbey"
• "Won't" (it should be Won't)
• "Sundays"
• "Beein" (?)
• "Cookery Book"
• "Yarmouth" (place)
• "Blunderstone Rookery" (place)

Other remarks:

• 'mr copperfield', 'master copperfield', "mr david copperfield" etc. are the same character
• "Twenty Seven" = "Number Twenty Seven"
• "Larkins" should be "Miss Larkins" or "Mr. Larkins"

[tommasobattisti]

Done:

• [x] 1872_de-la-ramee-a-dog-of-flanders
• [x] 1857_browne-grannys-wonderful-chair
• [x] 1902_potter-the-tale-of-peter-rabbit
• [x] 1877_molesworth-the-cuckoo-clock
• [x] 1888_wilde-the-happy-prince-and-other-tales
• [x] 1877_sewell-black-beauty
• [x] 1899_nesbit-the-story-of-the-treasure-seekers
• [x] 1869_ewing-mrs-overtheways-remembrances
• [x] 1886_hodgson-burnett-little-lord-fauntleroy
• [x] 1871_macdonald-at-the-back-of-the-north-wind
• [x] 1876_twain-the-adventures-of-tom-sawyer
• [ ] 1869_dickens-david-copperfield: "Blunderstone Rookery", "Suffolk Punches" and "Beein" are too specific.
• [ ] 1894_kipling-the-jungle-book: "Brumby" (horse race); "Kedda" and "Keddahs" (elephant traps); "Seeonee", "Wolf Pack" from "Seeone Wolf Pack" (wolk population); "Bandar" from "Bandar-log" (monkey population); are too specific.
• [ ] 1865_carroll-alices-adventures-in-wonderland: "Beautiful Soup" is too specific.
• [ ] 1883_stevenson-treasure-island: "Avast" and "Scatter" appear every time at the first position of the sentence.
• [ ] 1857_hughes-tom-browns-school-days: "Triste" always appear at the beginning of the sentence; "Great Western", "Wellesburn", "Harkaway", Hazeldown" and "Blawing Stwun" are too specific.

[tommasobattisti]

Minor common problems:

1877_molesworth-the-cuckoo-clock

• 'grizzel', 'aunt grizzel', 'sister grizzel', 'miss grizzel': the same character
• 'tabitha', 'sister tabitha', miss tabitha', 'aunt tabitha': the same character

1865_carroll-alices-adventures-in-wonderland

• "Bill": missed detection

1886_hodgson-burnett-little-lord-fauntleroy

• "Cedric", "Cedric Errol", "Captain Cedric Errol", "Captain Cedric", "Captain Errol" are the same character

Book: 1894_kipling-the-jungle-book

• "Little Brother", "Black Panther", "Black Snake" have also a proper name, but it could happen that strings with a similar form actually denote a character name in other cases

Book: 1869_dickens-david-copperfield

• 'mr copperfield', 'master copperfield', "mr david copperfield" etc. are the same character
• "Twenty Seven" = "Number Twenty Seven"

[tommasobattisti]

General imporovements:

• Consider: "Mr Higgs", "Higgs", "Mrs Higgs". If we remove "Higgs" we should not lose any information and our results should be better.
• It could be useful to include only the words that appear at least once in uppercase not after a full stop, double quotes and ‘