search

microsoft / chat-copilot

MIT License
1.92k stars 655 forks source link

What's the default queue name for AzureQueue? #958

Open glenq opened 2 months ago

glenq commented 2 months ago

What's the default queue name for Azure Queue for Chat-Copilot? Alternatively, how can I set the azure queue name for the Chat-Copilot App?

I'm attempting to enable the memory pipeline for Chat-Copilot App and I'm in the process of setting up the environment accroding to two documents: webapi README and memory pipeline README

However, I'm unable to locate any configuration or code related to the queue name. I'm curious about how Azure Queue works in Chat-Copilot. I haven't come across any code related to sending queue messages in Chat-Copilot. Therefore, I'm unsure about how to add a queue in the storage account. Alternatively, does Chat-Copilot automatically create it in the code?

This might be a simple question, but it has been troubling me for a long time. I hope you can help me clarify it.

image

jourdant commented 2 months ago

If you set KernelMemory__DataIngestion__OrchestrationType to Distributed and set your KernelMemory__DataIngestion__DistributedOrchestration__QueueType to AzureQueue, the following queues are either created or need to be created (in the case where you're using Managed Identity with a data only role):

glenq commented 1 month ago

If you set KernelMemory__DataIngestion__OrchestrationType to Distributed and set your KernelMemory__DataIngestion__DistributedOrchestration__QueueType to AzureQueue, the following queues are either created or need to be created (in the case where you're using Managed Identity with a data only role):

  • extract
  • extract-poison
  • gen-embeddings
  • gen-embeddings-poison
  • partition
  • partition-poison
  • private-delete-document
  • private-delete-document-poison
  • private-delete-index
  • private-delete-index-poison
  • save-embeddings
  • save-embeddings-poison
  • summarize
  • summarize-poison

@jourdant Thank you for your reply. May I ask where I can find these queue names (Documentation or code)? Because I also need to create an Azure AI Search index, but I'm unsure about which fields I need to create for the index.

image

jourdant commented 1 month ago

Honestly I have no idea! I suspect the SK SDK must be creating these automatically. The only reason I know these exist is that they've popped up in my environment.

I've been working on a more production-ready template based on this template. Part of that is locking down the permissions which means I also have to create a lot of these resources at deployment time like you. For the Azure AI Search Indexes, here are the json definitions you need to create the two indexes:

{
  "name": "chatmemory",
  "defaultScoringProfile": null,
  "fields": [
    {
      "name": "id",
      "type": "Edm.String",
      "searchable": true,
      "filterable": true,
      "retrievable": true,
      "sortable": false,
      "facetable": false,
      "key": true,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "tags",
      "type": "Collection(Edm.String)",
      "searchable": false,
      "filterable": true,
      "retrievable": true,
      "sortable": false,
      "facetable": false,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "payload",
      "type": "Edm.String",
      "searchable": true,
      "filterable": false,
      "retrievable": true,
      "sortable": false,
      "facetable": false,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "embedding",
      "type": "Collection(Edm.Single)",
      "searchable": true,
      "filterable": false,
      "retrievable": true,
      "sortable": false,
      "facetable": false,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": 1536,
      "vectorSearchProfile": "KMDefaultProfile",
      "synonymMaps": []
    }
  ],
  "scoringProfiles": [],
  "corsOptions": null,
  "suggesters": [],
  "analyzers": [],
  "normalizers": [],
  "tokenizers": [],
  "tokenFilters": [],
  "charFilters": [],
  "encryptionKey": null,
  "similarity": {
    "@odata.type": "#Microsoft.Azure.Search.BM25Similarity",
    "k1": null,
    "b": null
  },
  "semantic": null,
  "vectorSearch": {
    "algorithms": [
      {
        "name": "KMDefaultAlgorithm",
        "kind": "hnsw",
        "hnswParameters": {
          "metric": "cosine",
          "m": 4,
          "efConstruction": 400,
          "efSearch": 500
        },
        "exhaustiveKnnParameters": null
      }
    ],
    "profiles": [
      {
        "name": "KMDefaultProfile",
        "algorithm": "KMDefaultAlgorithm",
        "vectorizer": null
      }
    ],
    "vectorizers": []
  }
}

and

{
  "name": "global-documents",
  "defaultScoringProfile": null,
  "fields": [
    {
      "name": "Id",
      "type": "Edm.String",
      "searchable": false,
      "filterable": false,
      "retrievable": true,
      "sortable": false,
      "facetable": false,
      "key": true,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "Embedding",
      "type": "Collection(Edm.Single)",
      "searchable": true,
      "filterable": false,
      "retrievable": true,
      "sortable": false,
      "facetable": false,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": 1536,
      "vectorSearchProfile": "searchProfile",
      "synonymMaps": []
    },
    {
      "name": "Text",
      "type": "Edm.String",
      "searchable": true,
      "filterable": true,
      "retrievable": true,
      "sortable": true,
      "facetable": true,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "Description",
      "type": "Edm.String",
      "searchable": false,
      "filterable": true,
      "retrievable": true,
      "sortable": false,
      "facetable": true,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "AdditionalMetadata",
      "type": "Edm.String",
      "searchable": false,
      "filterable": true,
      "retrievable": true,
      "sortable": false,
      "facetable": true,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "ExternalSourceName",
      "type": "Edm.String",
      "searchable": false,
      "filterable": true,
      "retrievable": true,
      "sortable": false,
      "facetable": true,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    },
    {
      "name": "IsReference",
      "type": "Edm.Boolean",
      "searchable": false,
      "filterable": true,
      "retrievable": true,
      "sortable": false,
      "facetable": true,
      "key": false,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "synonymMaps": []
    }
  ],
  "scoringProfiles": [],
  "corsOptions": null,
  "suggesters": [],
  "analyzers": [],
  "normalizers": [],
  "tokenizers": [],
  "tokenFilters": [],
  "charFilters": [],
  "encryptionKey": null,
  "similarity": {
    "@odata.type": "#Microsoft.Azure.Search.BM25Similarity",
    "k1": null,
    "b": null
  },
  "semantic": null,
  "vectorSearch": {
    "algorithms": [
      {
        "name": "searchAlgorithm",
        "kind": "hnsw",
        "hnswParameters": {
          "metric": "cosine",
          "m": 4,
          "efConstruction": 400,
          "efSearch": 500
        },
        "exhaustiveKnnParameters": null
      }
    ],
    "profiles": [
      {
        "name": "searchProfile",
        "algorithm": "searchAlgorithm",
        "vectorizer": null
      }
    ],
    "vectorizers": []
  }
}

To import those definitions manually, go to the Indexes settings on your AI Search Service, click the Add Index > Add Index (JSON) button and paste one at a time over the top of the default text.

I have had success creating these indexes manually and then setting the *DbType settings to AzureAISearch resulting in the UI showing AzureAISearch in the UI and also seeing chunked docs in the search index:

image

Hope that works for you too.

glenq commented 1 month ago

@jourdant Thank you very much. I will try it.

aaronba commented 5 days ago

Hello, since this was asked in this thread. I wanted to share that I believe that the queue names and the -poison suffix from come from the Kernel Memory repo

Specifically, this reference file. Constants.cs from Kernel Memory