Create unit test to detect marc unicode encoding issues

[cdrini]

Here is a recent import from IA into OL:

• IA Record: https://archive.org/details/b30530921_0001
• OL Record: https://openlibrary.org/books/OL50976356M/Abreg%C2%A9%E2%99%AD_de_toute_la_medecine_pratique_...

The long-withstanding issue ( #135 ) of mysterious characters like ©♭ appearing in the Open Library record!

The purpose of this issue is to create a unit test of the smallest possible piece that is breaking. Likely, that is the piece that takes in the MARC record. That way this error should never resurface!

Stakeholders

@hornc

[hornc]

@cdrini The source of this particular character issue is that the source record has an incorrect encoding flag in the MARC binary. It claims to be MARC-8 encoded, but the data is UTF-8 encoded... treating a UTF-8 é as if it were MARC-8 produces ©♭

For some reason, the MARC XML shows the correct UTF-8 encoding in the leader and content.

I think OL is doing the correct operations with bad data. Looking into why archive.org got incorrect data for the MARC binary (but not MARC XML) would be useful. It seems all actions on this item are recent.

[hornc]

A similar item scanned around the same time has accents displayed correctly: https://openlibrary.org/books/OL50976370M/Suppl%C3%A9ment_de_l'Abreg%C3%A9_de_toute_la_m%C3%A9decine_pratique_ou_tome_VI_de_cet_ouvrage_..._premiere_partie

[cdrini]

Here are some other recent ones:

• https://openlibrary.org/books/OL50539598M/Syst%C2%A9%C2%B7me_physique_et_moral_de_la_femme_suivi_du_syst%C2%A9%C2%B7me_physique_et_moral_de_l%27homme_et_d%27un_fragme
• https://openlibrary.org/books/OL50973604M/Trait%C2%A9%E2%99%AD_des_fi%C2%A9%C2%B7vres_pernicieuses_intermittentes
• https://openlibrary.org/books/OL50541054M/De_la_saign%C2%A9%E2%99%ADe_et_de_son_usage_dans_la_plupart_des_maladies
• https://openlibrary.org/books/OL50973620M/Vers_solitaires_et_autres_de_diverses_esp%C2%A9%C2%B7ces_dont_il_est_trait%C2%A9%E2%99%AD_dans_le_livre_de_la_generation_de
• https://openlibrary.org/books/OL50973701M/Uebersicht_der_merkw%C2%A9%C6%A1rdigen_Eigenschaften_der_veredelten_Wolverlei-Blume_%28Fl._Arnicae%29_insbesondere

I'm not sure what the pattern is that caused these to regress, but can we perhaps sniff the file and look for certain characters? Or use the marc xml instead of the binary?

[tfmorris]

The file that imported correctly https://openlibrary.org/books/OL50976370M doesn't have a binary MRC file, just a MARC XML file. I'm not sure what conditions cause that in the processing pipeline.

The additional 5 files identified by @cdrini follow the same pattern as identified by @hornc for the first example.

For some reason, the MARC XML shows the correct UTF-8 encoding in the leader and content.

Where are these files being sourced/derived from? Clearly something in the pipeline is broken. Interestingly, the two MARC files are the oldest files in the directory https://archive.org/download/b30530921_0001

Having said that MARCedit displays all the "broken" binary MARC files without any problem, so it must have some heuristic to override the encoding flag.

[LeadSongDog]

Further, the same is appearing in author names, such as: https://openlibrary.org/search?q=author%3A©+AND+ia%3A*&mode=everything or simply https://openlibrary.org/search/authors?q=©

[tfmorris]

Having said that MARCedit displays all the "broken" binary MARC files without any problem, so it must have some heuristic to override the encoding flag.

I confirmed with the author of MARCedit that he uses a heuristic for encoding detection because MARC encoding flag isn't reliable.