RAG0001: Preprocess Documents (2)

OpenPecha / rag_prep_tool

MIT License

0 stars 0 forks source link

RAG0001: Preprocess Documents (2) #1

Closed tenzin3 closed 4 months ago

tenzin3 commented 4 months ago

Description:

Extracting meta data of the clean extracted text of books (Dalai Lama's).

Meta Data list:

book title
page no
chapter
start char
end char
content

Expected Output:

json file with format given below.

Implementation Plan

Work Items

[x] script to extract pdf from book
[x] script to transfer page annotation to clean text
[x] script to clean the text
[x] script to build metadata
[x] run on Art of Happiness at Work book
[x] run on Ethics for the New Millennium

tenzin3 commented 4 months ago

We need to select one more book with following requirements

has around 200 pages.
has a pdf version
context of book similar to "Art of Happiness at work by Dalai Lama"
has a clean text version (.txt) .

Most of the pdf version and clean text file are presented in the github/MonlamAI. Below is a list of dalail lama books that has a clean version text.(Not sure if pdf version is also available or not.) list of dalai lama books

tenzin3 commented 4 months ago

chose Ethics for the New Millennium for the second book.