solrmarc / custom_methods

This repo contain standalone custom indexing methods for use in SolrMarc, as well as examples of how those methods can be used
1 stars 3 forks source link

Duplicate removal #3

Open haschart opened 5 years ago

haschart commented 5 years ago

Frequently the routine will return two (or more) directors, where they are merely slightly different forms of the same persons name. Duplicate detection and removal should be improved. (or at least a warning should be thrown)

examples are below:

name with and without accent marks:
Edouard Molinaro Édouard Molinaro (u3790259) Shohei Imamura Shōhei Imamura (u3790272) Frantisek Vlacil František Vláčil (u3835628) Krzysztof Kieslowski Krzysztof Kieślowski (u3989919) Alī Badrakhān Ali Badrakhan (u4635858) Hagi Levi Ḥagai Leṿi (u4681240) Lucia Puenzo Lucía Puenzo (u4751032) François Truffaut Franc̜ois Truffaut (u2121798)

capitalization differences: Rolf de Heer Rolf De Heer (u4224170) Iain B. MacDonald Iain B. Macdonald (u4789806)

different spellings of the same name (typo?) (alternate spelling?): Mahesh Bhatt Mukesh Bhatt (u3890529) Luis Mankoki Luis Mandoki (u3971927) Postman Paget Portman Paget (u4006919) Daiy Von Scherler Mayer Daisy von Scherler Mayer (u4037749) Bob Balaban Bob Bababan (u4174493) Pavel Lounguine Pavel Lungin (u4244653) Mikhaila Kalatozova Mikhail Kalatozov (u4296661) Vincent Minnelli Vincente Minnelli (u4444344) Alexandre Sokourov Aleksandr Sokurov (u4613680) Ernst Lubtisch Ernst Lubitsch (u4616285) Tony Adzinikolov Tony Adznikolov (u4696097) Basu Chatterji Basu Chaterjee (u4708554) Satyan Bose Satyen Bose (u4708568) Narainder Bedi Narinder Bedi (u4708576) Pankaj Prasar Pankuj Parashar (u4711727) Arun Ruth Arun Rath (u4716652) Ellen Goosenburg Kent Ellen Goosenberg Kent (u4741116) Josef Leytes Josef Lejtes (u4752990) Ėdmonda Keosai͡ana Ėdmond Keosai͡an (u4760207) Harry Bawaja Harry Baweja (u4819424) Punit Isarr Punit Issar (u4822663) Manoj Agarwal Manoj Agrawal (u4822666) Franco Zefferelli Franco Zeffirelli (u1776030) Ngangura Mweze Ngangura Mwenze (u1959577) both spelling and accent marks: Mohammad Fadel Muḥammad Fāḍil (u4018018) Rakhshan Bani Etemad Rakhshān Banī Iʻtimād (u4319628) Salah Abou Seif Ṣalāḥ Abū Sayf (u4545527) Victor Okountsov Viktor Okunt͡sov (u4614918) Yevgenia Popova Evgenii͡a Popova (u4614918) Gulzar Gulazāra (u4711700) Saman Moghadam Sāmān Muqaddam (u4735245) Iraj Tahmasb Īraj Ṭahmāsbī (u4735247) Alexei Balabanov Alekseĭ Balabanov (u4760295) Marka Donskogo Mark Donskoĭ (u4760299) Andrū Mārtūn Andrew Marton (u4781312) Vijay Śarmā Vijay Sharma (u4819037) Walther Ruttmann Walter Ruttman (u2840725) Walther Ruttmann Walter Ruttmann (u2840725)

initials vs full name:
Jean-Paul Rappeneau J.-P. Rappeneau (u3957179) V. Shantaram Vhī Śāntārāma (u4708407) H.S. Rawail Harnam Singh Rawail (u4711697)

nickname vs full name: Theodore Bogosian Ted Bogosian (u4292280) Donald Siegel Don Siegel (u4611401) Mike Dibb Michael Dibb (u4681428) Bill Guttentag William Guttentag (u4720496) Thomas McCarthy Tom McCarthy (u4801439) Brad Silberling Bradley Silberling (u2971454)

punctuation difference: Jean-Luc Godard Jean Luc Godard (u4507415)

missing name part: Said Hamed Said (u4017957) Jeremiah S. Chechik Jeremiah Chechik (u4004603) John Little John R. Little (u4373278) Heidi Ewing Heidi E. Ewing (u4385254) Rebecca Foster Rebecca Feldman Foster (u4386732) Mikhail Romm Mikhail Ilʹich Romm (u4410122) William Wellman William Augustus Wellman (u4469528) Susan Vogel Susan Mullin Vogel (u4469650) Kamāl al-Shaykh Kamāl Shaykh (u4513813) Michael Watkins Michael W. Watkins (u4607653) Shaad Ali Sahgal Shaad Ali (u4687951) David Edwards David B. Edwards (u4729545) Ernesto Cabellos Damián Ernesto Cabellos (u4734787) Grigori Kozintsev Grigoriĭ Mikhaĭlovich Kozint͡sev (u4810744) Mukul S. Anand Mukul Anand (u4819030) Siddharth Anand Siddharth Raj Anand (u4819053) Lisa Kois Lisa M. Kois (u4819392) George Tillman, Jr. George Tillman (u4819793) Thomas Sigel Newton Thomas Sigel (u4822529) Gus Van Sant, Jr. Gus Van Sant (u2432363) Oscar "Budd" Boeticher Budd Boetticher (u3554186)

additional spaces: A.J. Antoon A. J. Antoon (u4006660) Manfred Neuwirth Manfred Neuwirth (u4245492) D.W. Griffith D. W. Griffith (u4377407) Thibaut d'Oiron Thibaut d' Oiron (u4754162) M.V. Raman M. V. Raman (u4819423)

name part ordering: Ng See Yuen See Yuen Ng (u4373277) Pong Chun-ho Chun-ho Pong (u4391142) Kim Ki-yŏng Ki-yŏng Kim (u4391148) Pyŏn Yŏng-ju Yŏng-ju Pyŏn (u4391149) Yi Man-hŭi Man-hŭi Yi (u4469859) Im Kwŏn-tʻaek Kwŏn-t'aek Im (u4469861) Kurosawa Akira Akira Kurosawa (u4473212) Kwak Kyung-Taek Kyung-Taek Kwak (u4512624) Hà Thúc Cà̂n Thúc Cà̂n. Hà (u4600066) Sembene Ousmane Ousmane Sembène (u4731028) Lou Ye Ye Lou (u4743425) Kawamoto Kihachirō Kihachirō Kawamoto (u4756579) Sōda Kazuhiro Kazuhiro Soda (u4813646) Satō Junya Junya Satō (u4826759)

name part ordering and spacing and accents: Rintaro Tarō Rin (u4624623)