Open rajathsalegame opened 6 months ago
Hi @rajathsalegame, and thanks for the suggestion. I agree that this could be a useful feature. Unfortunately for your multiprocessing
use-case, a lot of the heavy processing load currently is handled by the pdfminer.six
dependency, which I don't believe supports pickling.
To better understand the request, could you provide a bit more detail about your goals with multiprocessing and at what specific stage in your pipeline you'd be pickling/unpickling?
Hi, in my case, I need to apply page.dedupe_chars().extract_words(keep_blank_chars=True)
on every pages in a PDF file. For one file with 20 pages, it takes about 16 sec, taking up most of the time consumtion in my pipline. It will be great for pdfplumber to support pickle and multiprocessing.
BTW, if I do something like:
import pandas as pf
from joblib import Parallel, delayed
def parse_page(page):
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
with pdfplumber.open(pdf_file_or_path) as pdf:
result_parse = Parallel(-1, 'threading')(delayed(parse_page)(page) for page in pdf.pages)
It seems the error info is not about page being unpickable, but:
File "<ipython-input-50-192c80c0ccbc>", line 207, in parse_page
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 403, in dedupe_chars
p._objects = {kind: objs for kind, objs in self.objects.items()}
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 215, in objects
self._objects: Dict[str, T_obj_list] = self.parse_objects()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 275, in parse_objects
for obj in self.iter_layout_objects(self.layout._objs):
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 161, in layout
interpreter.process_page(self.page_obj)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 997, in process_page
self.render_contents(page.resources, page.contents, ctm=ctm)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 1016, in render_contents
self.execute(list_value(streams))
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 1021, in execute
parser = PDFContentParser(streams)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 251, in __init__
PSStackParser.__init__(self, None) # type: ignore[arg-type]
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 545, in __init__
PSBaseParser.__init__(self, fp)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 193, in __init__
self.seek(0)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 263, in seek
self.fillfp()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 256, in fillfp
strm = stream_value(self.streams[self.istream])
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 217, in stream_value
x = resolve1(x)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 118, in resolve1
x = x.resolve(default=default)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 106, in resolve
return self.doc.getobj(self.objid)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 866, in getobj
obj = self._getobj_parse(index, objid)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 840, in _getobj_parse
(_, obj) = self._parser.nextobject()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 656, in nextobject
self.do_keyword(pos, token)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfparser.py", line 79, in do_keyword
(objid, genno) = (int(objid), int(genno)) # type: ignore[arg-type]
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'PSKeyword'
"""
Does this make multiprocess with pdfplumber a bit easier?
For anyone who's interested, a work-around for multiprocessing
for pdfplumber
is to start multiprocessing before opening the pdf file with pdfplumber
:
from joblib import Parallel, delayed
import pdfplumber
import pandas as pd
def plumber_parsepage(pdf_path: str,
idx_page: int = 0,
) -> pd.DataFrame:
with pdfplumber.open(pdf_path) as pdf:
page = pdf.pages[idx_page]
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
return df_page
with pdfplumber.open(pdf_file_or_path) as pdf:
num_page = len(pdf.pages)
result = Parallel(-1)(delayed(plumber_parsepage)(pdf_file_or_path, i) for i in range(num_page))
For anyone who's interested, a work-around for
multiprocessing
forpdfplumber
is to start multiprocessing before opening the pdf file withpdfplumber
:from joblib import Parallel, delayed import pdfplumber import pandas as pd def plumber_parsepage(pdf_path: str, idx_page: int = 0, ) -> pd.DataFrame: with pdfplumber.open(pdf_path) as pdf: page = pdf.pages[idx_page] df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True)) return df_page with pdfplumber.open(pdf_file_or_path) as pdf: num_page = len(pdf.pages) result = Parallel(-1)(delayed(plumber_parsepage)(pdf_file_or_path, i) for i in range(num_page))
However, pdfplumber.open() itself costs too much time if there are 100 or more pages in the pdf file.
So the pickle implementation is really useful!
Would love it if there was a way a pickling implementation could be implemented for
Page
orPDF
objects. Currently none exists and makes working with things likemultiprocessing
a bit harder if one is interested in speeding up pdf processing over pages. Any advice or workaround from the community would be much appreciated :)