Text is missing a lot of text in the PDF (non-Acro form)

[gwc4github]

Bug report

• A description of the bug When I convert my pdf to text a lot of the contents are completely missing from the output. All of the missing text seems to be in the form cells. Also, the missing "text" seems to all be numeric. (wages, SSN, Income Tax, etc.) Note that this is not an Acro form according to the test 'AcroForm' not in res The form is a "2018 W-2 and EARNINGS SUMMARY". Unfortunately I cannot share the file. I am hoping someone can give me some suggestions anyway.

• Steps to reproduce the bug. Try to minimize the number of steps needed. Include the command and/or script that you use. Also include the PDF that you use. To reproduce the error, run the following code after changing the path to where you saved the file:

# pdfTextMiner.py
# Python 2.7.6
# For Python 3.x use pdfminer3k module
# This link has useful information on components of the program
# https://euske.github.io/pdfminer/programming.html
# http://denis.papathanasiou.org/posts/2010.08.04.post.html

''' Important classes to remember
PDFParser - fetches data from pdf file
PDFDocument - stores data parsed by PDFParser
PDFPageInterpreter - processes page contents from PDFDocument
PDFDevice - translates processed information from PDFPageInterpreter to whatever you need
PDFResourceManager - Stores shared resources such as fonts or images used by both PDFPageInterpreter and PDFDevice
LAParams - A layout analyzer returns a LTPage object for each page in the PDF document
PDFPageAggregator - Extract the decive to page aggregator to get LT object elements
'''

import os
from pdfminer.pdfparser import PDFParser
from pdfminer.pdfdocument import PDFDocument
from pdfminer.pdfpage import PDFPage
# From PDFInterpreter import both PDFResourceManager and PDFPageInterpreter
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.pdfdevice import PDFDevice
# Import this to raise exception whenever text extraction from PDF is not allowed
from pdfminer.pdfpage import PDFTextExtractionNotAllowed
from pdfminer.layout import LAParams, LTTextBox, LTTextLine
from pdfminer.converter import PDFPageAggregator

''' This is what we are trying to do:
1) Transfer information from PDF file to PDF document object. This is done using parser
2) Open the PDF file
3) Parse the file using PDFParser object
4) Assign the parsed content to PDFDocument object
5) Now the information in this PDFDocumet object has to be processed. For this we need
   PDFPageInterpreter, PDFDevice and PDFResourceManager
 6) Finally process the file page by page 
'''

filePath = '/Users/caseygre/PycharmProjects/ADPPDF2Text/pdfs'

pdfFile = os.path.join(filePath + "/" + "DSSFRAN-W2s-NFM.file 000163 - 1 page.pdf")
log_file = os.path.join(filePath + "/" + "pdf_log.txt")

password = ""
extracted_text = ""

# Open and read the pdf file in binary mode
fp = open(pdfFile, "rb")

# Create parser object to parse the pdf content
parser = PDFParser(fp)

# Store the parsed content in PDFDocument object
document = PDFDocument(parser, password)

# Check if document is extractable, if not abort
if not document.is_extractable:
    raise PDFTextExtractionNotAllowed

# Create PDFResourceManager object that stores shared resources such as fonts or images
rsrcmgr = PDFResourceManager()

# set parameters for analysis
laparams = LAParams()

# Create a PDFDevice object which translates interpreted information into desired format
# Device needs to be connected to resource manager to store shared resources
# device = PDFDevice(rsrcmgr)
# Extract the decive to page aggregator to get LT object elements
device = PDFPageAggregator(rsrcmgr, laparams=laparams)

# Create interpreter object to process page content from PDFDocument
# Interpreter needs to be connected to resource manager for shared resources and device
interpreter = PDFPageInterpreter(rsrcmgr, device)

# Ok now that we have everything to process a pdf document, lets process it page by page
for page in PDFPage.create_pages(document):
    # As the interpreter processes the page stored in PDFDocument object
    interpreter.process_page(page)
    # The device renders the layout from interpreter
    layout = device.get_result()
    # Out of the many LT objects within layout, we are interested in LTTextBox and LTTextLine
    for lt_obj in layout:
        if isinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine):
            extracted_text += lt_obj.get_text()

# close the pdf file
fp.close()

# print (extracted_text.encode("utf-8"))

with open(log_file, "w") as my_log:
    # my_log.write(extracted_text.encode("utf-8"))
    my_log.write(extracted_text)
print("Done !!")

• If relevant, include the output and/or error stacktrace. No error given.

[b-gran]

I just ran into this issue as well, so posting a solution for posterity :)

The text within a Form XObject is nested in a LTFigure. In order to extract the text from Form XObjects, you need to do the following:

1. add all_texts=True to your LAParams, e.g. laparams = LAParams(alltexts=True)
2. iterate through the objects within any LTFigure to find text. see snippet below

for page in PDFPage.create_pages(document):
    interpreter.process_page(page)
    layout = device.get_result()
    for lt_obj in layout:
        if isinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine):
            extracted_text += lt_obj.get_text()
        elif isinstance(lt_obj, LTFigure):
            for lt_obj_inner in lt_obj:
                if isinstance(lt_obj_inner, LTTextBox) or isinstance(lt_obj_inner, LTTextLine):
                    extracted_text += lt_obj_inner.get_text()