Closed andrewngo closed 2 years ago
Henkei is very much a thin wrapper around Apache Tika. It would seem that v1.21 of Tika might be returning the results with different encoding to v2.3.0 😉
Have you checked what the encoding of the two different strings are? ie '....'.encoding
It would likely also come down to what your system default encoding is. ie for me it's UTF-8 so:
> "<p>ü\xC5\x87Í\xE6\x96\x87\xE5\xAD\x97\xE5\x90\x8D ü\xC5\x87Í"
=> "<p>üŇÍ文字名 üŇÍ"
You can change/force the encoding with encode('UTF-8')
or possibly force_encoding('UTF-8')
(the difference being that the first will try transcode characters from the different sets, vs just ignoring the differences and trying to render as is.. but again.. that's really going to come down to how your system is configured
Thanks @abrom. Using .force_encoding('UTF-8')
works as a workaround for me.
Worth noting that line breaks on Windows are now adding a carriage return (\r\n
vs \n
) on this version, which is also unexpected but easily solved. Not sure if there are any configurations to control this.
I have a PDF with some unicode text (e.g.
[Text üŇÍ文字]
) but the output is different with the new version. Looks like it's printing bytes. Any ideas?v1.21.0
v2.3.0.1
System info (Mac)