I have a test.pdf that has some pages that only have #s on them. When calling process_tesseract from pawls.commands.preprocess, I was getting an error I'd not seen before (despite processing hundreds of docs with your preprocessor):
/home/jman/PycharmProjects/test_pawls_parser/venv/bin/python /home/jman/PycharmProjects/test_pawls_parser/main.py
Traceback (most recent call last):
File "/home/jman/PycharmProjects/test_pawls_parser/main.py", line 10, in <module>
annotations: list = process_tesseract(file.name)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pawls/preprocessors/tesseract.py", line 101, in process_tesseract
annotations = parse_annotations(pdf_file)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pawls/preprocessors/tesseract.py", line 80, in parse_annotations
tokens = extract_page_tokens(pdf_image, pdf_size)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pawls/preprocessors/tesseract.py", line 35, in extract_page_tokens
.apply(
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1567, in apply
result = self._python_apply_general(f, self._selected_obj)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1629, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 839, in apply
res = f(group)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pawls/preprocessors/tesseract.py", line 43, in <lambda>
gp["text"].str.cat(sep=" "),
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/generic.py", line 5902, in __getattr__
return object.__getattribute__(self, name)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/accessor.py", line 182, in __get__
accessor_obj = self._accessor(obj)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/strings/accessor.py", line 181, in __init__
self._inferred_dtype = self._validate(data)
File "/home/jman/PycharmProjects/test_pawls_parser/venv/lib/python3.10/site-packages/pandas/core/strings/accessor.py", line 235, in _validate
raise AttributeError("Can only use .str accessor with string values!")
AttributeError: Can only use .str accessor with string values!. Did you mean: 'std'?
Looking at my PDF and the offending pd.Series tripping the error in line 36 of cli.pawls.preprocessors.tesseract in extract_page_tokens(), pretty sure I know what's happening here. It appears some of the tokens being extracted from the attached pdf are treated as floats in the dataframe, and line 43 of extract_page_tokens() fails when trying to concat this token as the pandas df type is float64 yet .str.cat can only be called on cells containing string type.
See the offending, dataframe-related code here in tesseract.py:
I assume my issue - where extracted pdf token data type is not string - is rare but this is bound to happen in large PDF collections, particularly as the pdf formatting is dependent on the original source PDF authoring tool.
I propose adding .astype(str) to line 43 to force conversion to of the text cell contents to string type, which should cover these kinds of corner cases where our data type could be converted to a string.
I have a test.pdf that has some pages that only have #s on them. When calling
process_tesseract
frompawls.commands.preprocess
, I was getting an error I'd not seen before (despite processing hundreds of docs with your preprocessor):Looking at my PDF and the offending pd.Series tripping the error in line 36 of
cli.pawls.preprocessors.tesseract
inextract_page_tokens()
, pretty sure I know what's happening here. It appears some of the tokens being extracted from the attached pdf are treated as floats in the dataframe, and line 43 ofextract_page_tokens()
fails when trying to concat this token as the pandas df type isfloat64
yet.str.cat
can only be called on cells containing string type.See the offending, dataframe-related code here in tesseract.py:
https://github.com/allenai/pawls/blob/1225660ccb4f3b9877bf45c04baecc2798d183ee/cli/pawls/preprocessors/tesseract.py#L20-L46
I assume my issue - where extracted pdf token data type is not string - is rare but this is bound to happen in large PDF collections, particularly as the pdf formatting is dependent on the original source PDF authoring tool.
I propose adding .astype(str) to line 43 to force conversion to of the text cell contents to string type, which should cover these kinds of corner cases where our data type could be converted to a string.
I've opened PR #199 with a fix.