CosmosDB Trigger is not working in Visual Studio development

[AlfredBauer]

I'm developing my functions in Visual Studio and providing testing also from Visual Studio. My function "CosmosDB Trigger" using Local CosmosDB Emulator. When I press to debug I started to see in Output(VS) errors and through 30 sec I see error in console.

Here is my function

[FunctionName("WorkDocumentDBTrigger")] public static void Run([CosmosDBTrigger("AlfredDentDocumentDBWE01", "Clinic", ConnectionStringSetting = "CosmosDBEmulator", CreateLeaseCollectionIfNotExists = true, LeaseCollectionName = "ClinicLease")] IReadOnlyList documents, TraceWriter log) { log.Info($"{DateTimeOffset.Now} [INFO][WorkDocumentDBTrigger] Start"); }

Here is Output Stack trace

An exception is thrown:"System.Net.WebException" в System.dll An exception is thrown: "System.Net.WebException" в System.dll An exception is thrown: "System.Net.Sockets.SocketException" в System.dll DocDBTrace Information: 0 : RefreshLocationAsync() refreshing locations DocDBTrace Information: 0 : Set WriteEndpoint https://127.0.0.1:8081/ ReadEndpoint https://127.0.0.1:8081/ DocDBTrace Information: 0 : DocumentClient with id 2 initialized at endpoint: https://localhost:8081/ with ConnectionMode: Direct, connection Protocol: Tcp, and consistency level: null DocDBTrace Information: 0 : RefreshLocationAsync() refreshing locations DocDBTrace Information: 0 : Set WriteEndpoint https://127.0.0.1:8081/ ReadEndpoint https://127.0.0.1:8081/ An exception is thrown: "System.Security.Authentication.AuthenticationException" in System.dll An exception is thrown:"System.Security.Authentication.AuthenticationException" in System.dll An exception is thrown:"System.Security.Authentication.AuthenticationException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in Microsoft.Azure.Documents.Client.dll An exception is thrown: "System.Security.Authentication.AuthenticationException" in System.dll An exception is thrown: "System.Security.Authentication.AuthenticationException" in System.dll An exception is thrown:"System.Security.Authentication.AuthenticationException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in Microsoft.Azure.Documents.Client.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll DocDBTrace Information: 0 : Вызвано исключение: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll An exception is thrown: "Microsoft.Azure.Documents.GoneException" in mscorlib.dll GetOpenConnection failed: RID: dbs/AlfredDentDocumentDBWE01, ResourceType Database, Op: (operationType: Read, resourceType: Database), Address: rntbd://172.19.32.1:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, Exception: Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server. ActivityId: fbb9e255-983a-4ece-a7d0-bcf08d1ea4f9, Request URI: rntbd://172.19.32.1:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/ ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to authentication results in System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) в System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result) в System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult) в System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End the stack trace from the previous location where the exception occurred--- в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) в Microsoft.Azure.Documents.RntbdConnection.d20.MoveNext() --- End trace of internal exception stack--- **в Microsoft.Azure.Documents.RntbdConnection.d20.MoveNext() --- End the stack trace from the previous location where the exception occurred--- в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) в System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) в Microsoft.Azure.Documents.RntbdConnection.d__0.MoveNext() ---End the stack trace from the previous location where the exception occurred--- в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) в Microsoft.Azure.Documents.RntbdConnectionDispenser.d__0.MoveNext() ---End the stack trace from the previous location where the exception occurred--- в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) в Microsoft.Azure.Documents.ConnectionPool.d__0.MoveNext() --- End the stack trace from the previous location where the exception occurred--- в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) в Microsoft.Azure.Documents.ConnectionPoolManager.d__0.MoveNext() --- End the stack trace from the previous location where the exception occurred---** в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) в Microsoft.Azure.Documents.RntbdTransportClient.d__0.MoveNext()

Here is console error

A ScriptHost error has occurred The listener for function "***" was unable to start. Microsoft.Azure.Documents.Client: Service is currently unavailable. Microsoft.Azure.Documents.Client: The requested resource is no longer available at the server.

CosmosDB Emulator 1.17.43.5 Microsoft.Azure.WebJobs.Extenstions.DocumentDB 1.1.0-beta-4 Microsoft.NET.Sdk.Functions 1.0.6

[paulbatum]

@ealsur Does the change feed work for the cosmos local emulator?

[ealsur]

EDITED: Originally my answer was no, but upon internal check, the Change Feed is available to the Emulator and thus should work in this scenario.

[AlfredBauer]

So it's not working for the emulator? Am I right understand?

[paulbatum]

Yes, the trigger relies on the CosmosDB change feed which apparently does not exist when using the local emulator. So you'll need to use a real cosmos database if you want to dev/test your function.

[ealsur]

@AlfredBauer You can always use the Try Cosmos DB Free Tier.

[ealsur]

@AlfredBauer I stand corrected, after checking internally, the Change Feed should be available for the local emulator and your scenario. What is your CosmosDBEmulator setting value?

[AlfredBauer]

@ealsur Actually I do not know where I can find this "settings". My version is 1.17.43.5 In C:\Users**\AppData\Local\CosmosDBEmulator folder I see Machine.Config and GatewayMachine.Config** files. Are these files represent settings?

[ealsur]

@AlfredBauer the ConnectionStringSetting attribute should point to an Application Setting that contains the connection string of the Cosmos DB instance. Based on this article, it should something like:

AccountEndpoint=https://localhost:8081;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==;

In case you have not changed the key.

[AlfredBauer]

@ealsur Here is my string: "CosmosDBEmulator": "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==;"

I think it's the same. Any other operations with Emulator works fine(I mean another functions(CRUD))

[psfinaki]

So, finally, is this supposed to work?

I have set this exact connection string, have double-checked database and collection names, and still my CosmosDBTrigger spits some similar stuff to what Alfred showed. When set up to a real database, it works.

[ealsur]

@psfinaki This seems to be an issue with the emulator, not the Trigger (otherwise it would not work even with the real database).

Does the Cosmos DB Emulator UI work? ie it let's you create collections and manage documents? Does both the monitored collection and the lease collection exists in the Emulator?

[psfinaki]

@ealsur Oh, thank you for the quick response.

Yes, UI works, collections exist. However, the error text is similar when I deliberately specify some non-existent collection:

The listener for function '...Trigger' was unable to start. Microsoft.Azure.Documents.Client: Service is currently unavailable.
ActivityId: .... Microsoft.Azure.Documents.Client: The requested resource is no longer available at the server.

(repeating multiple times)

[ealsur]

That error indicates that it doesn't seem to be able to connect to the Cosmos DB endpoint and fetch the collections (either the monitored collection or the leases collection).

Is your emulator updated to the latest version?

[psfinaki]

Yep, 1.19.102.5. Anyways, I have reinstalled it just in case, it did not help.

Well, I do believe this is something about endpoints, however looks like Trigger somehow feels the local stuff. For example, I manually deleted 'leases' collection, specified CreateLeaseCollectionIfNotExists = true in CosmosDBTrigger attribute parameters and the collection got recreated.

What other experiments can I conduct?

[remcorakers]

I had the same error, where our Function output console running in Visual Studio debug mode showed this:

DocDBTrace Information: 0 : Set WriteEndpoint https://127.0.0.1:8081/ ReadEndpoint https://127.0.0.1:8081/
DocDBTrace Information: 0 : GetOpenConnection failed: RID: dbs/CCM, ResourceType Database, Op: (operationType: Read, resourceType: Database), Address: rntbd://192.168.10.2:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, Exception: Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: b1baa8cd-d305-444e-8916-bab18f5bf07a, Request URI: rntbd://192.168.10.2:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/ ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

As you can see the CosmosDBTrigger in the Function tries to connect to the Cosmos DB Emulator at rntbd://192.168.10.2:10251. This address might not be directly your local machine, but in my case a virtual network adapter created by VirtualBox, VMWare, HyperV or something alike.

Solution: disable the virtual network adapter, or add 192.168.10.2 localhost (or your equivalent IP address) to your local host file. This solved the connection error of the CosmosDBTrigger and the function connected again to the Cosmos DB Emulator change feed.

[psfinaki]

@remcorakers Thank you for your contribution. I do have a virtual network adapter and the same error, however neither of proposed solutions worked in my case, I keep getting AuthenticationException which seems to be the root of evil here...

[ealsur]

@psfinaki Does the trace you see look similar to @remcorakers ? What is the address being connected to?

[psfinaki]

@ealsur Yes, very similar. Same text, same exception. The address is rntbd://172.30.32.1:10251/...

[ealsur]

@psfinaki is this running in a VM? Did you add 172.30.32.1 localhost as suggested to the hosts file?

[remcorakers]

@psfinaki Have you tried to disable all virtual network adapters and reboot the system (just in case)? Did the exception in the Visual Studio output window during debug execution change?

[psfinaki]

@ealsur No, I am running things locally. I have added the line to the hosts file. @remcorakers Yep, I rebooted the system, disconnected from the network, run the trigger with admin rights. The exception is the same.

The exception actually says something about certificates... Does it make sense to dig in that direction?