Update PDF Parser to be used

[ChakshuGautam]

• [ ] Store

  • [ ] Structure: Parent <> Child relationship
  • [ ] Images
  • [ ] Tables
  • [ ] Summary of chunks

• [ ] Search for the following fields

  • [ ] Fuzzy Search on everything text - list with a bunch of rows (relevancy score)
  • [ ] Vector Em Search - list with a bunch of rows (relevancy score)
  • [ ] Images - Start with child
  • [ ] Tables - Start with child

• [ ] Retrieval

  • [ ] parents wherever needed (figure out how to define need)

[GautamR-Samagra]

@ksgr5566. The pdf parser work will be tracked here

[GautamR-Samagra]

Ouput should look like this - link @ksgr5566

[ChakshuGautam]

@GautamR-Samagra to have hardcoded rules to figure out the relative hierarchy of chunks.

[GautamR-Samagra]

Created csv : here

Required columns :

ID | Doc ID | contentString | summaryString | summaryEmbedding | text | textEmbedding | page | Tags | semanticVersion(section index) | sectionTitle | sectionString | sectionImages | isTable | isImage | isQuoted | parentString | imageBase64 | titleOfCurrentSection | siblingChunkUp | siblingChunkDown | meta | linkedChunks

The csv above for Samagra docs cover all except : isQuoted Sibling Chunk up Sibling Chunk down linked chunks

@prtkjakhar @aashutosh-samagra use the csv as it also contains the image base64 which doesnt fit on the google sheet