improved term normalization

psolin / cleanco

Company Name Processor written in Python

MIT License

326 stars 98 forks source link

improved term normalization #56

Closed twalen closed 4 years ago

twalen commented 4 years ago

fixed remove_accents (now it also handles characters like ł)
removing punctuation from terms in normalize_terms
added variations of "LTD" in polish
added tests

petri commented 4 years ago

Thanks for a nice contribution. If I got it right, this PR (among other things), overcomes deficiencies in out-of-the-box unicode normalization by handling troublesome characters manually?

Regarding removing dots, commas and dashes from terms, why is that necessary? Are we removing those from names but not from terms? Or...?

twalen commented 4 years ago

Yes, this handles some characters manually (according to the map in NON_NFKD_MAP).

Dots, commas, etc:

in my data tit happens a lot that the punctuation for abbreviations is not very strict so if cleanco is able to handle this well it would be super-useful
the punctuation is removed from both terms and from normalized parts (nnparts) so this will not affect the resulting name but only removed terms

petri commented 4 years ago

Seems good to me.