Easily chunk complex documents the same way a human would.
Chunking documents is a challenging task that underpins any RAG system. High quality results are critical to a sucessful AI application, yet most open-source libraries are limited in their ability to handle complex documents.
Open Parse is designed to fill this gap by providing a flexible, easy-to-use library capable of visually discerning document layouts and chunking them effectively.
🔍 Visually-Driven: Open-Parse visually analyzes documents for superior LLM input, going beyond naive text splitting.
✍️ Markdown Support: Basic markdown support for parsing headings, bold and italics.
📊 High-Precision Table Support: Extract tables into clean Markdown formats with accuracy that surpasses traditional tools.
🛠️ Extensible: Easily implement your own post-processing steps.
💡Intuitive: Great editor support. Completion everywhere. Less time debugging.
🎯 Easy: Designed to be easy to use and learn. Less time reading docs.
import openparse
basic_doc_path = "./sample-docs/mobile-home-manual.pdf"
parser = openparse.DocumentParser()
parsed_basic_doc = parser.parse(basic_doc_path)
for node in parsed_basic_doc.nodes:
print(node)
📓 Try the sample notebook here
Chunking documents is fundamentally about grouping similar semantic nodes together. By embedding the text of each node, we can then cluster them together based on their similarity.
from openparse import processing, DocumentParser
semantic_pipeline = processing.SemanticIngestionPipeline(
openai_api_key=OPEN_AI_KEY,
model="text-embedding-3-large",
min_tokens=64,
max_tokens=1024,
)
parser = DocumentParser(
processing_pipeline=semantic_pipeline,
)
parsed_content = parser.parse(basic_doc_path)
📓 Sample notebook here
Uses pydantic under the hood so you can serialize results with
parsed_content.dict()
# or to convert to a valid json dict
parsed_content.json()
Python 3.8+
Dealing with PDF's:
Extracting Tables:
pip install openparse
Enabling OCR Support:
PyMuPDF will already contain all the logic to support OCR functions. But it additionally does need Tesseract’s language support data, so installation of Tesseract-OCR is still required.
The language support folder location must be communicated either via storing it in the environment variable "TESSDATA_PREFIX", or as a parameter in the applicable functions.
So for a working OCR functionality, make sure to complete this checklist:
Install Tesseract.
Locate Tesseract’s language support folder. Typically you will find it here:
Windows: C:/Program Files/Tesseract-OCR/tessdata
Unix systems: /usr/share/tesseract-ocr/5/tessdata
macOS (installed via Homebrew):
/opt/homebrew/share/tessdata
/opt/homebrew/Cellar/tesseract/<version>/share/tessdata/
Set the environment variable TESSDATA_PREFIX
Windows: setx TESSDATA_PREFIX "C:/Program Files/Tesseract-OCR/tessdata"
Unix systems: declare -x TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
macOS (installed via Homebrew): export TESSDATA_PREFIX=$(brew --prefix tesseract)/share/tessdata
Note: On Windows systems, this must happen outside Python – before starting your script. Just manipulating os.environ will not work!
This repository provides an optional feature to parse content from tables using a variety of deep learning models.
pip install "openparse[ml]"
Then download the model weights with
openparse-download
You can run the parsing with the following.
parser = openparse.DocumentParser(
table_args={
"parsing_algorithm": "unitable",
"min_table_confidence": 0.8,
},
)
parsed_nodes = parser.parse(pdf_path)
Note we currently use table-transformers for all table detection and we find its performance to be subpar. This negatively affects the downstream results of unitable. If you're aware of a better model please open an Issue - the unitable team mentioned they might add this soon too.
https://github.com/Filimoa/open-parse/tree/main/src/cookbooks
https://filimoa.github.io/open-parse/
Does your use case need something special? Reach out.