datastax / pulsar-transformations

Apache License 2.0
10 stars 8 forks source link

[query] Add new Query Step to perform queries to external databases (with AstraDB Vector Search) #76

Closed eolivelli closed 1 year ago

eolivelli commented 1 year ago

Sample configuration:

{
  "datasource": {
    "service": "astra",
    "username": "........",
    "password": "....",
    "secureBundle": "UEsDBBQACAAIANRT1VYAAAAA....."
  },
  "steps": [
    {
      "type": "query",
      "query": "SELECT * FROM vsearch.products where id=?",
      "output-field": "results",
      "fields": ["id"]
    }
  ]
}

Sample configuration to compute a similarity search on a Vector Database (AstraDB)

{
  "datasource": {
    "service": "astra",
    "username": ".....",
    "password": "......t",
    "secureBundle": "......"
  },
  "openai": {"url": "https://datastax-openai-dev.openai.azure.com/", "access-key": "......", "provider": "azure"},
  "steps": [
    {
      "type": "compute-ai-embeddings",
      "model": "text-embedding-ada-002",
      "embeddings-field": "embeddings",
      "fields": ["name", "description"]
    },
    {
      "type": "query",
      "query": "SELECT * FROM vsearch.products ORDER BY item_vector ANN OF ? LIMIT 1;",
      "output-field": "results",
      "fields": ["embeddings"]
    }
  ]
}