search

freelawproject / doctor

A microservice for document conversion at scale
https://free.law/projects/doctor
BSD 2-Clause "Simplified" License
54 stars 14 forks source link

UTF-8 Encoding Latin-1 Bug in HTML files #137

Open flooie opened 2 years ago

flooie commented 2 years ago

Unfortunately, we are experiencing occasional crashes on CL from the inability to extract HTML from certain HTML files.

These occur mostly around HTML files downloads from the NY lower courts, with messages like

*Error: 'utf-8' codec can't decode byte 0xe9 in position 437. Having gone thru a number of them I think this is a latin-1 utf8 encoding bug.

Soo...

I've reviewed the HTML text extraction and found atleast one bug in the unicode/decode encoding part of the loop. If utf8 fails it doesnt bother looping thru the other encodings. But I'm not sure that is the source of the error. So, for now I've updated the error messages, fixed that bug and I expect to wait and see if my improved error messages can catch the bug more directly.

flooie commented 2 years ago

Sentry issue: COURTLISTENER-2JJ

sentry-io[bot] commented 2 years ago

Sentry issue: COURTLISTENER-2JJ