search

sambitdash / PDFIO.jl

PDF Reader Library for Native Julia.
Other
127 stars 13 forks source link

Add test cases for AGL. #101

Open dannywinrow opened 2 years ago

dannywinrow commented 2 years ago

I tried to run pdPageExtractText on the pdf located: https://www.gov.im/media/1360682/isle-of-man-inflation-report-november-2021.pdf

However, every character of the text was being interpreted as "\0"

After much pain and effort trailing through the PDFIO code, I have identified the problem as being what is returned by the fum function in PDFont. In particular when the cn"Encoding" object contains a /Differences object with values such as /uni0047 which just represent the unicode character U+47 ('p'). Since the AGL_Glyph_To_Unicode dictionary (not sure where this comes from) doesn't contain the simple unicode mappings then the zero(Char) is returned instead.

One solution might be to just compare the /uni0047 to the base encoding dictionary and if the 0x0047 part exists then add a dictionary entry. Another solution would be to add all of the standard unicode characters that already exist in your base encoding such as /uni0047 to the AGL_Glyph_To_Unicode dictionary.

I have made the assumption, when suggesting this solution, that the cn"Encoding" object is taken directly from the pdf file and not further processed.

If you'd like me to try to create a pull request, I'd be happy to, but I thought I'd ask first in case your more holistic view of the project leads to a more effective solution.

dannywinrow commented 2 years ago

I have updated this issue, since I think I have found the crux of the problem which is that PDFIO is missing part of the AGL specification which states that you first match to AGL, and if no match then you test whether it is a unicode character of the form uniXXXX or uXXXX (see specification for general case and restrictions)

sambitdash commented 1 year ago

https://github.com/sambitdash/PDFIO.jl/commit/6367aa667fa37b1cb653a165e3957bd5e1b1b6d9 Fixes it but no test cases are added as the file is no longer accessible.

sambitdash commented 1 year ago

Add test cases for AGL.

sambitdash commented 1 year ago

isle-of-man-inflation-report-november-2021.pdf Adding a copy of the file which I got by Googling. But, this version does not have an AGL code. The suggested file is no longer on the site.We need to look for a better test file.