search

Azure / azure-cosmos-dotnet-v3

.NET SDK for Azure Cosmos DB for the core SQL API
MIT License
743 stars 495 forks source link

[BUG] Cosmos client connecting to wrong port when using emulator #2784

Closed goenning closed 3 years ago

goenning commented 3 years ago

Describe the bug I have the cosmos emulator running on a container using port 9999 instead of 8081. I'm using the C# client to connect to it, but it just hangs indefinitly. Looking at the output window, I see that it's trying to connect to port 8081 regardless of what my endpoint is.

Expected behavior It should connect to the correct port.

Actual behavior (include Exception or Stack Trace) It's connecting to the wrong port.

DocDBTrace Information: 0 : DocumentClient with id 1 initialized at endpoint: https://localhost:9999/ with ConnectionMode: Gateway, connection Protocol: Https, and consistency level: null
DocDBTrace Information: 0 : Telemetry Disabled.
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Globalization.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Users\goenning\Desktop\CosmosLocalSDK\bin\Debug\net5.0\Microsoft.Azure.Cosmos.Core.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Numerics.Vectors.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Collections.Immutable.dll'. Symbols loaded.
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Runtime.CompilerServices.Unsafe.dll'. Symbols loaded.
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Security.Cryptography.Encoding.dll'. Symbols loaded.
DocDBTrace Information: 0 : SystemUsageMonitor started
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Threading.Tasks.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.IO.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Dynamic.Runtime.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Runtime.Serialization.Formatters.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Runtime.Numerics.dll'. Symbols loaded.
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Reflection.Emit.ILGeneration.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Reflection.Primitives.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Reflection.Emit.Lightweight.dll'. 
'CosmosLocalSDK.exe' (CoreCLR: clrhost): Loaded 'Anonymously Hosted DynamicMethods Assembly'. 
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.21.0.2:8081/) ReadEndpoints = (https://172.21.0.2:8081/)
DocDBTrace Information: 0 : GlobalEndpointManager: StartLocationBackgroundRefreshWithTimer() refreshing locations
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Sockets.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Http.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Sockets.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Http.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Sockets.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Http.dll
Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in Microsoft.Azure.Cosmos.Client.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
DocDBTrace Warning: 0 : ClientRetryPolicy: Gateway HttpRequestException Endpoint not reachable. Failed Location: https://172.21.0.2:8081/; ResourceAddress: dbs/TEST
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.21.0.2:8081/ unavailable for read
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.21.0.2:8081/) ReadEndpoints = (https://172.21.0.2:8081/)
DocDBTrace Information: 0 : Endpoint https://172.21.0.2:8081/ unavailable for Read added/updated to unavailableEndpoints with timestamp 10/11/2021 09:42:39
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.21.0.2:8081/ unavailable for Write
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.21.0.2:8081/) ReadEndpoints = (https://172.21.0.2:8081/)
DocDBTrace Information: 0 : Endpoint https://172.21.0.2:8081/ unavailable for Write added/updated to unavailableEndpoints with timestamp 10/11/2021 09:42:39
DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.21.0.2:8081/) ReadEndpoints = (https://172.21.0.2:8081/)

To Reproduce

  1. Start a cosmos emulator on linux container ( https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator?tabs=ssl-netstd21 ) using port 9999 instead of 8081
  2. Write a simple C# script that connects to it.
  3. It'll just hang indefinitly, look at output window.

Environment:

image

ealsur commented 3 years ago

The SDK uses the global endpoint, in your case https://localhost:9999/ to connect and request the account information. The global endpoint returns the endpoints the SDK should use. In a real account, you'd use https://youraccount.documents.azure.com as global endpoint and you'd get 1 endpoint per region in the format https://youraccount-region.documents.azure.com that the SDK would use.

Based on the logs, the problem is not the SDK.

DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.21.0.2:8081/) ReadEndpoints = (https://172.21.0.2:8081/)

Those are the endpoints the Emulator is reporting, so the SDK will attempt to use that. Why the emulator is reporting the other endpoints with a different port is something that is outside the scope of this repo.

ealsur commented 3 years ago

@goenning Could you share which command line arguments are you using to start the emulator?

goenning commented 3 years ago

Thanks for the description above, that makes sense. The emulator doesn't know it's on port 9999 because that mapping is done on a higher layer.

I'm using the emulator in docker-compose with this yaml. There must be a way to tell the emulator that we're using a different port then.

version: '2'
services:
  cosmosdb:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
    container_name: cosmosdb
    restart: always
    ports:
      - 9999:8081 
      - 10251:10251 
      - 10252:10252 
      - 10253:10253 
      - 10254:10254
ealsur commented 3 years ago

You should be starting the Emulator with the /Port argument from the documentation: https://docs.microsoft.com/en-us/azure/cosmos-db/emulator-command-line-parameters

image

I don't know how that is done on docker-compose.

goenning commented 3 years ago

Thanks for the info, I reached out to the emulator team and got this reply in case someone is looking for it.

We don't fully document how this can be done today but there are some options that allows you to pass other options to start the emulator.
For instance you can pass in the environment variable "AZURE_COSMOS_EMULATOR_ARGS" via Docker's "-e" option. This could take a list of emulator command line arguments as if you invoke the executable yourself (Command-line and PowerShell reference for Azure Cosmos DB Emulator | Microsoft Docs).

Another option is to override the default container's startup command, (see Docker options) and provide your own script/custom command that will replace it. In Windows Docker container this is a PowerShell script that you have access to. The custom logic will have to take the current command and add to it:
$startEmulatorCmd = "Start-CosmosDbEmulator -AllowNetworkAccess -NoFirewall -NoUI -Key $Key $startEmulatorOptions"

I hope the above steps help.

This is how it looks now:

version: '2'
services:
  cosmosdb:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
    container_name: cosmosdb
    restart: always
    ports:
      - 9999:9999 
      - 10251:10251 
      - 10252:10252 
      - 10253:10253 
      - 10254:10254
    environment:
      AZURE_COSMOS_EMULATOR_ARGS: /Port=9999
ealsur commented 3 years ago

Thanks for sharing the docker steps for future users.