search

gptscript-ai / knowledge

Knowledge for GPTScript
https://gptscript-ai.github.io/knowledge/
Apache License 2.0
29 stars 14 forks source link

feat: add progress indication for embedding generation process #127

Closed iwilltry42 closed 1 month ago

iwilltry42 commented 1 month ago

This gives some progress indication for the embeddings generation. Unfortunately, without modifying the chromem-go code for this, we cannot get the "progress": "1/3" style output here, so maybe in Otto we can just do a counter based on the "status": "starting" with "num_documents": 3 being the total number of times it should appear.

{"time":"2024-09-19T14:56:15.374927781+02:00","level":"INFO","msg":"Adding documents to collection (generating embeddings)","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"vectorstore","vectorstore":"chromem-go","status":"starting"}
{"time":"2024-09-19T14:56:15.374966246+02:00","level":"INFO","msg":"Creating embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"starting"}
{"time":"2024-09-19T14:56:15.374975889+02:00","level":"INFO","msg":"Creating embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"starting"}
{"time":"2024-09-19T14:56:15.375093968+02:00","level":"INFO","msg":"Creating embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"starting"}
{"time":"2024-09-19T14:56:15.609623657+02:00","level":"INFO","msg":"Created embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"completed"}
{"time":"2024-09-19T14:56:15.748787889+02:00","level":"INFO","msg":"Created embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"completed"}
{"time":"2024-09-19T14:56:16.205515467+02:00","level":"INFO","msg":"Created embedding","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"embedding","status":"completed"}
{"time":"2024-09-19T14:56:16.205883188+02:00","level":"INFO","msg":"Added documents to collection (generated embeddings)","flow":"ingestion","rootPath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","filepath":".local/testdata/metadata/Reunion-Under-The-Stars.pdf","phase":"store","num_documents":3,"stage":"vectorstore","vectorstore":"chromem-go","status":"completed"}