Azure-Samples / graphrag-accelerator

One-click deploy of a Knowledge Graph powered RAG (GraphRAG) in Azure
https://github.com/microsoft/graphrag
MIT License
1.92k stars 316 forks source link

[BUG] graphrag pods cannot reach CosmosDB when accelerator is deployed in Azure Government #71

Open timothymeyers opened 4 months ago

timothymeyers commented 4 months ago

Describe the bug

Deploying the accelerator to Azure Government results in the following CrashLoopBackoff error for both the -index and -query pods.

azure.cosmos.exceptions.CosmosHttpResponseError: (Forbidden) Request originated from IP 52.XXX.XXX.XXX through public internet. This is blocked by your Cosmos DB account firewall settings. More info:
https://aka.ms/cosmosdb-tsg-forbidden
ActivityId: XXXX, Microsoft.Azure.Documents.Common/2.14.0
Code: Forbidden
Message: Request originated from IP 52.XXX.XXX.XXX through public internet. This is blocked by your Cosmos DB account firewall settings. More info:
https://aka.ms/cosmosdb-tsg-forbidden
ActivityId: XXXX, Microsoft.Azure.Documents.Common/2.14.0

This is because the CosmosDB firewall has Public network access disabled and the pods in AKS require access via the AKS API Server PIP.

I'm not sure why this is not a problem in Azure Commercial.

To Reproduce Steps to reproduce the behavior:

az cloud set --name "AzureUSGovernment"
az login

The following additional params are required in deploy.paramaters.json

  "AISEARCH_ENDPOINT_SUFFIX": "search.azure.us",
  "AISEARCH_AUDIENCE": "https://search.azure.us",
  "CLOUD_NAME":"AzureUSGovernment",
  "GRAPHRAG_COGNITIVE_SERVICES_ENDPOINT":"https://cognitiveservices.azure.us/.default"
timothymeyers commented 4 months ago

Some notes/workarounds

A proper fix is likely to deploy the AKS cluster in private cluster mode with public fqdn disabled, and establish a private endpoint between the AKS cluster and Cosmos (and the other resources it needs to reach).

timothymeyers commented 3 months ago

Need to test this again after changes from #123 were introduced.