Performance Tracking KPI's - 1

[rahmansahinler1]

Description Performance of the individual processes like indexing, embedding, etc. needs to be tracked down for optimizing. Every run KPI's will be saved under a dictionary. Then, if yes creates .csv output.

Example For Embedding KPI's:

1. Sentence Amount
2. Total time spent on embedding - total_emb_time
3. Batch size
4. Average time spend on embedding (total time / total sentence) - avg_emb_time

Workflow

1. Create a logic for every KPI
2. Save KPI's to the dictionary
3. Create a logic for generating .csv
4. Add a control parameter to the globals.py (generate_csv)
5. Create 10 sample .csv KPI files into the assets folder

Acceptance Criteria 10 .csv file for different runs for embedding and indexing

[ozgurnsahin]

Dictionary is created every runs recods are logged into a .csv file and tracked

[ozgurnsahin]

Iterated KPI tracking functions to only record unique batch and sentence amount pairs. Added linear regression model to optimize batch size of the embedding function and to speed up overall processes.