[agents] Add support for AstraDB Collections (Astra Vector DB, using Stargate)

[eolivelli]

Summary:

• add support for Astra Collections: asserts, query (similarity search. regular query and execute-statement) and vector-db-writer sink
• add examples

With this patch there is a new vector-database type: "astra-vector-db".

If you use "service=astra" than you are going to use the classic APIs with CQL, if you use "service=astra-vector-db" then you use the JSON API, based on Stargate (https://gitub.com/stargate).

The sample application and the integration tests contains examples about how to use it. We will follow up on the documents repository with detailed reference for all the commands.

Quick overview

configuration.yaml:

configuration:
  resources:
  - type: "datasource"
    name: "AstraDatasource"
    configuration:
      service: "astra-vector-db"
      token: "${secrets.astra-vector-db.token}"
      endpoint: "${secrets.astra-vector-db.endpoint}"

Similarity search:

  - name: "lookup-related-documents"
    type: "query-vector-db"
    configuration:
      datasource: "AstraDatasource"
      query: |
          {
              "collection-name": "documents",
              "limit": 20,
              "vector": ?
          }
      fields:
        - "value.question_embeddings"
      output-field: "value.related_documents"

Vector DB Sink:

  - name: "Write to Astra"
    type: "vector-db-sink"
    input: "chunks-topic"
    configuration:
      datasource: "AstraDatasource"
      collection-name: "documents"
      fields:
        - name: "id"
          expression: "fn:concat(value.filename, '-', value.chunk_id)"
        - name: "vector"
          expression: "value.embeddings_vector"
        - name: "text"
          expression: "value.text"
        - name: "filename"
          expression: "value.filename"
        - name: "chunk_id"
          expression: "value.chunk_id"