search

databricks-industry-solutions / pixels

Facilitates simple large scale processing of HLS Medical images, documents, zip files. Previously at https://github.com/dmoore247/pixels
https://databricks-industry-solutions.github.io/pixels/
Other
25 stars 15 forks source link

ENH: Support PDFs #45

Open dmoore247 opened 9 months ago

dmoore247 commented 9 months ago

Is your feature request related to a problem? Please describe. Add ability to chunk out the text in a PDF into a dataframe / delta lake table to enable building RAG applications / semantic search applications. Have controls for overlap and chunk size. Add capability to Pixels to index and collect header metadata from PDFs. Add capability to run embedding on chunked out text.

The value add is minimal lines of code to scale up processing 100k+ PDF files.

Sample python splitting code take from this volumes blog

@udf('array<string>')
def gen_chunks(path: str) -> list[str]:
   from pdfminer.high_level import extract_text
   from langchain.text_splitter import TokenTextSplitter
   text = extract_text(path)
   splitter = TokenTextSplitter(chunk_size = 500, chunk_overlap  = 50)
   return [doc.page_content for doc in splitter.create_documents([text])]

The solution should be a pluggable SparkML Transformer