search

pdfminer / pdfminer.six

Community maintained fork of pdfminer - we fathom PDF
https://pdfminersix.readthedocs.io
MIT License
5.84k stars 925 forks source link

PSBaseParser can't handle tokens split across end of buffer #1025

Open jbarlow83 opened 2 months ago

jbarlow83 commented 2 months ago

If a parsed token in a PSParser subclass is split across the boundary between buffers, a keyword token will be incorrect split into two separate tokens, causing the wrong keyword to be produced and destroying all subsequent parsing. The BUFSIZ is just 4096, so any data stream longer than 4096 will potentially suffer from this issue.

A simple solution is to increase the buffer size to a much larger value (gigabytes) - in practice the impact on performance will be negligible since most PDFs fit within available RAM anyway. Alternate, (it seems to me) all _parse* functions would need to be adjusted to handle the case where we hit the end of buffer and the subsequent bytes in the buffer might contain the rest of a toekn.

The attached PDF demonstrates the issue when trying to parse its cmaps, some of which are longer than 4096 bytes.

Here is log output from parsing the attached PDF's fonts -- note that the token beginbfchar is incorrectly split into two tokens, beg and inbfchar since it the token happens to be split by the end of the buffer. This cause the incorrect interpretation of all subsequent tokens. Increasing BUFSIZ mitigates the issue.

  DEBUG pdfminer.psparser - nexttoken: (4079, /b'endbfchar')
  DEBUG pdfminer.psparser - do_keyword: pos=4079, token=/b'endbfchar', stack=[(2151, b'\x00d'), (2158, b'\x00e'), (2166, b'\x00e'), (2173, b'\x00e\x00,'), (2185, b'\x00f'), (2192, b'\x00e\x00,\x00e\x00o\x00.\x00.'), (2220, b'\x00g'), (2227, b'\x00e\x00o\x00.\x00.'), (2247, b'\x00h'), (2254, b'\x00e\x00p\x00o'), (2270, b'\x00i'), (2277, b'\x00e\x00r'), (2289, b'\x00j'), (2296, b'\x00e\x00r'), (2308, b'\x00k'), (2315, b'\x00e\x00t'), (2327, b'\x00l'), (2334, b'\x00f\x00u'), (2346, b'\x00m'), (2353, b'\x00f\x00u\x00e'), (2369, b'\x00n'), (2376, b'\x00g'), (2384, b'\x00o'), (2391, b'\x00h'), (2399, b'\x00p'), (2406, b'\x00h'), (2414, b'\x00q'), (2421, b'\x00h'), (2429, b'\x00r'), (2436, b'\x00h\x00_\x00:\x00)'), (2456, b'\x00s'), (2463, b'\x00h\x00e'), (2475, b'\x00t'), (2482, b'\x00h\x00e\x00o\x00.\x00d'), (2506, b'\x00u'), (2513, b'\x00h\x00o\x00c\x00.\x00K'), (2537, b'\x00v'), (2544, b'\x00i'), (2552, b'\x00w'), (2559, b'\x00i'), (2567, b'\x00x'), (2574, b'\x00i'), (2582, b'\x00y'), (2589, b'\x00i'), (2597, b'\x00z'), (2604, b'\x00i'), (2612, b'\x00{'), (2619, b'\x00i'), (2627, b'\x00|'), (2634, b'\x00i'), (2642, b'\x00}'), (2649, b'\x00i'), (2657, b'\x00~'), (2664, b'\x00i\x00o'), (2676, b'\x00\x7f'), (2683, b'\x00i\x00o\x00.\x00.'), (2703, b'\x00\x80'), (2710, b'\x00i\x00t'), (2722, b'\x00\x81'), (2729, b'\x00l'), (2737, b'\x00\x82'), (2744, b'\x00l'), (2752, b'\x00\x83'), (2759, b'\x00l'), (2767, b'\x00\x84'), (2774, b'\x00l\x00e'), (2786, b'\x00\x85'), (2793, b'\x00m'), (2801, b'\x00\x86'), (2808, b'\x00m'), (2816, b'\x00\x87'), (2823, b'\x00m'), (2831, b'\x00\x88'), (2838, b'\x00m\x00a\x00r'), (2854, b'\x00\x89'), (2861, b'\x00m\x00e\x00t'), (2877, b'\x00\x8a'), (2884, b'\x00n'), (2892, b'\x00\x8b'), (2899, b'\x00n'), (2907, b'\x00\x8c'), (2914, b'\x00n\x00.'), (2926, b'\x00\x8d'), (2933, b'\x00n\x00_\x009'), (2949, b'\x00\x8e'), (2956, b'\x00n\x00e'), (2968, b'\x00\x8f'), (2975, b'\x00n\x00k\x00<\x00>'), (2995, b'\x00\x90'), (3002, b'\x00n\x00o\x00.\x00d\x00.'), (3026, b'\x00\x91'), (3033, b'\x00n\x00s\x00t'), (3049, b'\x00\x92'), (3056, b'\x00o'), (3064, b'\x00\x93'), (3071, b'\x00o'), (3079, b'\x00\x94'), (3086, b'\x00o'), (3094, b'\x00\x95'), (3101, b'\x00o'), (3109, b'\x00\x96'), (3116, b'\x00o'), (3124, b'\x00\x97'), (3131, b'\x00o'), (3139, b'\x00\x98'), (3146, b'\x00o'), (3154, b'\x00\x99'), (3161, b'\x00o\x00.\x00.'), (3177, b'\x00\x9a'), (3184, b'\x00o\x00.\x00.'), (3200, b'\x00\x9b'), (3207, b'\x00o\x00.\x00.'), (3223, b'\x00\x9c'), (3230, b'\x00o\x00.\x00d'), (3246, b'\x00\x9d'), (3253, b'\x00o\x00.\x00e'), (3269, b'\x00\x9e'), (3276, b'\x00o\x00.\x00n'), (3292, b'\x00\x9f'), (3299, b'\x00o\x00.\x00n'), (3315, b'\x00\xa0'), (3322, b'\x00o\x00.\x00n'), (3338, b'\x00\xa1'), (3345, b'\x00o\x00.\x00n\x00t'), (3365, b'\x00\xa2'), (3372, b'\x00o\x00.\x00n\x00u\x00s'), (3396, b'\x00\xa3'), (3403, b'\x00o\x00.\x00p'), (3419, b'\x00\xa4'), (3426, b'\x00o\x00.\x00r'), (3442, b'\x00\xa5'), (3449, b'\x00o\x00.\x00r'), (3465, b'\x00\xa6'), (3472, b'\x00o\x00.\x00r\x00r'), (3492, b'\x00\xa7'), (3499, b'\x00o\x00.\x00w'), (3515, b'\x00\xa8'), (3522, b'\x00o\x00J'), (3534, b'\x00\xa9'), (3541, b'\x00o\x00J'), (3553, b'\x00\xaa'), (3560, b'\x00o\x00J'), (3572, b'\x00\xab'), (3579, b'\x00o\x00d'), (3591, b'\x00\xac'), (3598, b'\x00o\x00e'), (3610, b'\x00\xad'), (3617, b'\x00o\x00l'), (3629, b'\x00\xae'), (3636, b'\x00o\x00s'), (3648, b'\x00\xaf'), (3655, b'\x00o\x00s'), (3667, b'\x00\xb0'), (3674, b'\x00o\x00t'), (3686, b'\x00\xb1'), (3693, b'\x00o\x00\xa5'), (3705, b'\x00\xb2'), (3712, b'\x00o\x00\xa5'), (3724, b'\x00\xb3'), (3731, b'\x00p'), (3739, b'\x00\xb4'), (3746, b'\x00p'), (3754, b'\x00\xb5'), (3761, b'\x00p\x00<\x00>'), (3777, b'\x00\xb6'), (3784, b'\x00p\x00<\x00>'), (3800, b'\x00\xb7'), (3807, b'\x00p\x00e'), (3819, b'\x00\xb8'), (3826, b'\x00p\x00r'), (3838, b'\x00\xb9'), (3845, b'\x00r'), (3853, b'\x00\xba'), (3860, b'\x00r'), (3868, b'\x00\xbb'), (3875, b'\x00r'), (3883, b'\x00\xbc'), (3890, b'\x00r'), (3898, b'\x00\xbd'), (3905, b'\x00r'), (3913, b'\x00\xbe'), (3920, b'\x00r'), (3928, b'\x00\xbf'), (3935, b'\x00r'), (3943, b'\x00\xc0'), (3950, b'\x00r'), (3958, b'\x00\xc1'), (3965, b'\x00r'), (3973, b'\x00\xc2'), (3980, b'\x00r'), (3988, b'\x00\xc3'), (3995, b'\x00r'), (4003, b'\x00\xc4'), (4010, b'\x00r'), (4018, b'\x00\xc5'), (4025, b"\x00r\x00'\x009"), (4041, b'\x00\xc6'), (4048, b'\x00r\x00.'), (4060, b'\x00\xc7'), (4067, b'\x00r\x00\\')]
  DEBUG pdfminer.psparser - nexttoken: (4090, 49)
  DEBUG pdfminer.psparser - nexttoken: (4093, /b'beg')   <--- WRONG!!!
  DEBUG pdfminer.psparser - do_keyword: pos=4093, token=/b'beg', stack=[(4090, 49)]
  DEBUG pdfminer.psparser - nexttoken: (4096, /b'inbfchar')   <--- SECOND HALF OF TOKEN
  DEBUG pdfminer.psparser - do_keyword: pos=4096, token=/b'inbfchar', stack=[(4090, 49), (4093, /b'beg')]
  DEBUG pdfminer.psparser - nexttoken: (4106, b'\x00\xc8')
  DEBUG pdfminer.psparser - nexttoken: (4113, b'\x00r\x00o\x00d')
  DEBUG pdfminer.psparser - nexttoken: (4129, b'\x00\xc9')
  DEBUG pdfminer.psparser - nexttoken: (4136, b'\x00r\x00r\x00m')
  DEBUG pdfminer.psparser - nexttoken: (4152, b'\x00\xca')
  DEBUG pdfminer.psparser - nexttoken: (4159, b'\x00r\x00r\x00m\x00e')

1361.pdf

Originating issue: https://github.com/ocrmypdf/OCRmyPDF/issues/1361

dhdaines commented 1 month ago

Indeed ... the buffering code in the parser is brittle and opaque and I'm not surprised it contains an error like this!

dhdaines commented 1 month ago

This change is the culprit: https://github.com/pdfminer/pdfminer.six/pull/885 as it doesn't distinguish between the end of the stream and the end of the buffer.

Ideally the PSParser code should be replaced with a lexer based on a more robust and well-tested codebase, but for the moment we can simply fix that fix, which I'll do in a second.