Open DinoLiww opened 4 months ago
The default processing pipeline skips small blocks.
You must adjust the processing pipeline, see here - adjust max_area_pct
for RemoveFullPageStubs
.
I'm using this pipeline with good results:
class MyIngestionPipeline(processing.IngestionPipeline):
def __init__(self):
self.transformations = [
processing.RemoveTextInsideTables(),
processing.CombineNodesSpatially(criteria="either_stub"),
processing.CombineBullets(),
processing.CombineHeadingsWithClosestText(),
processing.RemoveFullPageStubs(max_area_pct=0.20),
]
from openparse import processing
class MyIngestionPipeline(processing.IngestionPipeline): def init(self): self.transformations = [ processing.RemoveTextInsideTables(), processing.CombineNodesSpatially(criteria="either_stub"), processing.CombineBullets(), processing.CombineHeadingsWithClosestText(), processing.RemoveFullPageStubs(max_area_pct=0.20), ]
parser = openparse.DocumentParser( processing_pipeline=MyIngestionPipeline(), table_args={"parsing_algorithm": "pymupdf"} )
basic_doc_path = ".\ASE#1_KR_B.PDF" parsed_basic_doc = parser.parse(basic_doc_path)
and now it can present all of info I need to handle.
But 2 questions left here, hope can get your feedbacks
@DinoLiww max area controls the maximum size of an element. In this case it's based on a percentage of page size. If you don't have elements that are getting dropped, then it's because there's no elements that take up more than 10% (0.1) of a page. This is mostly used to filter out pages that have massive text (like a title) that aren't helpful in a RAG pipeline.
As for your second point, can you eleborate?
Initial Checks
Description
Hi there,
Thanks for your open parse 1st and it looks cool in most of the time. But when I try to bring my real world tasks into OP and it seems some problems come up. when I run openparse_quickstart.ipynb to parse some pdf files as attached, PO actually. it seems open parse missing some blocks within the pdf files. Please kindly let me know how to move.
Thanks! Dino
ASE.PDF
Amkor.PDF
Example Code
Python, open-parse & OS Version