Open jonathanpv opened 11 months ago
Related project: https://github.com/Anush008/fastembed-rs
Value prop:
Users will want to send logs closer from the edge to catch errors quickly eg, if something is a security flaw worthy or not
Vector embeddings can be supported through VRL and on edge using ONXX so its possible to demo this within vrlplayground as well
I imagine someone's VRL function to handle an event, if the event is a certain error to add label "error": "security flaw from hot path library" which will have an accompanying "error_embedding": [vector[78]]
and then downstream someone's UI or search can be like
"do we have any errors recently in the hotpath?"
Vector embeddings support
Vector should support an embedding-transform through VRL
The only thing we need would be a configuration of the embedding endpoints to use and the place to store the output
Why? Semantic search through data can be powered cheaply using vector embeddings, in order to be a step towards AI-powered monitoring we should support translating logs to vectors and adding it as a field.
Sample feature:
Use Cases
as a user I can use natural language to search through my logs
the end result will allow users to have intelligent search through logs with natural language.
as a developer I can implement semantic search quickly with vector.dev
a better developer experience.
for example:
a search query like:
"give me the failures from amazon in the last three hours"
can output the most relevant logsAttempted Solutions
No response
Proposal
I propose an investigation between the referenced services and see if this is a quick implementation or if it is not worth the investment / already supported but with a different customization.
References
This is an example endpoint that generates matrices based on text input, there are other one's but openAI is the most prevalent solution at the moment
This is a service specialized in vector search
Pinecone is also a popular vector store
User's can implement vector search using just a SQL database as well
Version
No response