I added a new 'index-dataset' command which works in a similar fashion to the 'describe-dataset' one.
It writes the embedding to the vector db (tested with pinecone) and stores the dataset description path parts as the 'document'.
I've also modified the frontend code a little so that:
When fetching from the frontend db it will fetch the descriptions from storage.
Context now only uses the generated descriptions and not the datasets, so it's smaller and simpler to generate.
@OriHoch
I added a new 'index-dataset' command which works in a similar fashion to the 'describe-dataset' one. It writes the embedding to the vector db (tested with pinecone) and stores the dataset description path parts as the 'document'. I've also modified the frontend code a little so that: