POC1: Core chatbot workflow

[lelouvincx]

As a backend member, I want to conduct on the first POC about a full basic LLM workflow. So that I can integrate to the base backend.

The flow

• Read and parse PDF document
• Chunk documents and metadata extraction
• Proper index
• Proper retrieving, node postprocessing and response synthesing

Guides

• Use jupyter notebooks
• Use LlamaIndex to speed up development
• To overcome tech dept, you can use SimpleIndexStore and SimpleVectorStore in LlamaIndex.
• Raw documents: https://drive.google.com/drive/folders/1lYhvNN2sCaN8w8igBEdJucZJOD_OLVGZ?usp=drive_link

Success criteria

• Work well with Vietnamese
• Extract information as much as possible
• 80% unit test passed

@lelouvincx will provide OpenAI's api key.

[lelouvincx]

To overcome tech dept, you can use SimpleIndexStore and SimpleVectorStore in LlamaIndex.

[lelouvincx]

Raw documents: https://drive.google.com/drive/folders/1lYhvNN2sCaN8w8igBEdJucZJOD_OLVGZ?usp=drive_link

[lelouvincx]

@vn01122003 please checkout this doc: https://docs.llamaindex.ai/en/stable/module_guides/querying/retriever/

Focus on BM25 Retriever.

[vn01122003]

For now, we are focusing on this Colab workspace: https://colab.research.google.com/drive/1Nafvyr5pctN4dPM25BIrrmwdiuTop2QV?usp=sharing

[vn01122003]

Updated 'Read and parse PDF document' section to Colab work space Can be reviewed for early revision