implement full paper RAG to assist extraction

[shababo]

so far we've only looked at a PNG of the figure and caption to extract data. in the main example we've been testing with, it does not properly estimate the wavelength values that were used. however, they are explicitly enumerated in the methods section.

we already implemented a MultimodalDocumentHandler, but when I just tested parsing the PDF on llama cloud, it did not do a great job.

• [x] verify/optimize pdf parsing
• [x] verify/optimize query engine over pdf
• [x] incorporate query engine into data extraction

for now, we'll keep working primarily in notebooks. once this part is up, we should be able to create some classes and API which should facilitate further testing.

[shababo]

did a bunch of experimentation and learned more about the components of RAG pipelines. the new plan is to parse the PDF is three ways, and then transform those nodes/documents into some kind of composed index/store. have some ideas on the latter, but feel like i have a good beat on the initial ingestion pipelines:

ingest PDF to doc/nodes that

• [ ] capture the text and section structure of the paper
• [ ] contain images of each figure with associated information
• [ ] contains images and text for each full page of the PDF