Some input PDFs with Tesseract-OCR throw error in PyPDF2.utils.PdfReadError(Unexpected escaped string: b'{')' raised in ...

Issue by Wikinaut Mon Sep 7 18:56:47 2015 Originally opened as https://github.com/fritz-hh/OCRmyPDF/issues/120

A few PDF input files (which were already processed by tesseract-ocr pdf mode) throw an error in OCRmyPDF even in the --force-ocr mode. At the moment, I have no idea what exactly happens, but the problem appears to be in PyPDF2 (I use PyPDF2 1.25.1).

The error message is only shown when OCRmyPDF is used with the -v option.

Do you have an idea what went wrong in these cases, or what else can be done to let OCRmyPDF apply another OCR run to such a PDF?

Full output:

Tasks which will be run:
Task enters queue = 'ocrmypdf.main.repair_pdf' 
Original exception:
    Exception #1
      'PyPDF2.utils.PdfReadError(Unexpected escaped string: b'{')' raised in ...
       Task = def ocrmypdf.main.repair_pdf(...):
       Job  = [ARCHIV.pdf -> .../com.github.ocrmypdf.49q2h1fj/ARCHIV.repaired.pdf, <ocrmypdf.main.WrappedLogger>, [], <_thread.lock>]

    Traceback (most recent call last):
      File "/usr/lib/python3.4/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/usr/lib/python3.4/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/usr/local/src/OCRmyPDF/ocrmypdf/main.py", line 372, in repair_pdf
        pdfinfo.extend(pdf_get_all_pageinfo(output_file))
      File "/usr/local/src/OCRmyPDF/ocrmypdf/pageinfo.py", line 145, in pdf_get_all_pageinfo
        return [_pdf_get_pageinfo(infile, n) for n in range(pdf.numPages)]
      File "/usr/local/src/OCRmyPDF/ocrmypdf/pageinfo.py", line 145, in <listcomp>
        return [_pdf_get_pageinfo(infile, n) for n in range(pdf.numPages)]
      File "/usr/local/src/OCRmyPDF/ocrmypdf/pageinfo.py", line 115, in _pdf_get_pageinfo
        text = page.extractText()
      File "/usr/lib/python3.4/site-packages/PyPDF2/pdf.py", line 2566, in extractText
        content = ContentStream(content, self.pdf)
      File "/usr/lib/python3.4/site-packages/PyPDF2/pdf.py", line 2645, in __init__
        self.__parseContentStream(stream)
      File "/usr/lib/python3.4/site-packages/PyPDF2/pdf.py", line 2677, in __parseContentStream
        operands.append(readObject(stream, None))
      File "/usr/lib/python3.4/site-packages/PyPDF2/generic.py", line 71, in readObject
        return ArrayObject.readFromStream(stream, pdf)
      File "/usr/lib/python3.4/site-packages/PyPDF2/generic.py", line 166, in readFromStream
        arr.append(readObject(stream, pdf))
      File "/usr/lib/python3.4/site-packages/PyPDF2/generic.py", line 77, in readObject
        return readStringFromStream(stream)
      File "/usr/lib/python3.4/site-packages/PyPDF2/generic.py", line 386, in readStringFromStream
        raise utils.PdfReadError(r"Unexpected escaped string: %s" % tok)
    PyPDF2.utils.PdfReadError: Unexpected escaped string: b'{'

Comment by jbarlow83 Mon Sep 7 20:14:52 2015

There's an error in the PDF file. I run qpdf to try to fix errors before anything else, but it can't fix everything.

You could "re-fry" the PDF with ghostscript, which rewrites and hopefully fixes the error. This has other side effects. gs -o "$outpdf" -sDEVICE=pdfwrite "$inpdf"

pdftk also has a feature to fix damaged PDFs.

Can you provide a link to a sample? (Dropbox, etc.)

Comment by Wikinaut Mon Sep 7 21:26:44 2015

@jbarlow83 great, because I wanted (but did not) to add some lines from my own project - but did not want to puzzle you with just another information.

Give me some time, I will try to find what went wrong (the scanned PDF was produced with gscan2pdf and tesseract. Perhaps I used already pdftk to merge some parts, and this step introduced the problem).

My own project (not published yet) creates single pages from a multi-page input pdf, converts them to single lossless-compressed images (png, not using convert but ghostscript), tesseract-ocrs (pdf rendering mode) page-by-page, and then uses pdftk to concatenate (re-merge) the single pages to the multi-page, double-layer output pdf. In that way, pdf errors are almost never coming up.

Comment by jbarlow83 Mon Sep 7 22:13:55 2015

I'd like the PDF because any errors get added to the test suite. I'm less interested in what produced the problem so much as trying to make sure all readable PDFs are accepted even if they are invalid.

Your project is quite similar to what this one does right now - just so you're aware. ocrmypdf converts pages to PNG uses ghostscript (btw: ImageMagick's convert also uses ghostscript, it just uses it badly), then uses tesseract for PDF rendering (optional with --pdf-renderer tesseract), and then uses ghostscript to merge. The error you ran into is in the first step, not later on in the pipeline.

Tesseract's PDF rendering has some issues (at least as of 3.03). It gets DPI wrong, it produces PDFs with syntax errors, and it can inflate file size in some situations. For those reasons --pdf-renderer tesseract is not the default. I'll be adding code to repair those problems or (more likely) extract the OCR layer from Tesseract's output and graft it into the final PDF, so that I can insert the text layer without transcoding any images.

Comment by Wikinaut Mon Sep 7 22:35:32 2015

re. ghostscript vs. convert, yes, you are fully right. See http://bertanguven.com/faster-conversions-from-pdf-to-pngjpeg-imagemagick-vs-ghostscript/ .

And thanks for all the other infos.

BTW, I helped the tesseract people to fix a problem with the re-compression of the image (and I am proud, because it was difficult for some days). For a certain period, they lossy-re-compressed the input image (because the design goal was then to create minimal file sizes, which is a wrong goal), which introduced coding artefacts.

This is fixed since about one year, and current tesseract version are coding very well. When you now say, that Tesseract introduces PDF syntax errors, this explains my current issue. Thanks for info. Please, if you have further information why or when Tesseracts creates wrong PDFs, pls. report your observations upstream. They (user zdenop) will fix the issue.

Just for your information:

From my script the line where a pdf page is losslessy converted to an image (which is then input to tesseract, and rendered to pdf):

density = 400
image = yyy.png
file = zzz.pdf
## see http://bertanguven.com/faster-conversions-from-pdf-to-pngjpeg-imagemagick-vs-ghostscript/
gs -dNOPAUSE -sDEVICE=png16m -sOutputFile=$image -r$density -q $file -c quit

ocrmypdf / OCRmyPDF

Some input PDFs with Tesseract-OCR throw error in PyPDF2.utils.PdfReadError(Unexpected escaped string: b'{')' raised in ... #1