Closed minnocenti901 closed 1 month ago
Hello @minnocenti901! What is your full name? I suspect there is an issue with your project setup, Consul configuration, or networking. Let's work together to identify the root cause.
Ocelot is running on
http://localhost:5280
Global configuration:
"GlobalConfiguration": { "RequestIdKey": "ApiGateway", "ServiceDiscoveryProvider": { "Type": "Consul", "Port": "8502", "Host": "192.168.0.204" } }
The BaseUrl
property is missing and is required. As outlined in our Getting Started documentation, the BaseUrl
must be defined in the Configuration section of the JSON file, as shown below:
"GlobalConfiguration": {
"BaseUrl": "http://localhost:5280", // this is crucial!
"RequestIdKey": "ApiGateway",
"ServiceDiscoveryProvider": {
"Type": "Consul",
"Port": "8502",
"Host": "192.168.0.204"
}
}
Please could you provide the complete ocelot.json
file for review?
Expected Behavior
Ocelot should request from Consul the correct service for every route
Steps to Reproduce the Problem
On consul I have 2 services registered: ProjectsService with url
http://localhost:5013
CustomersService with urlhttp://localhost:5004
Ocelot is running onhttp://localhost:5280
š Could you show us complete Consul configuration please, service mappings, nodes etc.?
Regarding networking... I see you use learning solution at local machine in Docker or without, right?
Specifications
- Version: Ocelot 23.3.3, Ocelot.Provider.Consul 23.3.3, Consul 1.12.0
- Platform: .NET8
š Please revert the version to 23.2.2 and conduct the same tests.
Adding BaseUrl didn't solve the issue. This is the complete ocelot.json
{
"Routes": [
{
"DownstreamPathTemplate": "/{all}",
"DownstreamScheme": "http",
"ServiceName": "CustomersService",
"UpstreamPathTemplate": "/customers/{all}",
"UpstreamHttpMethod": [
"GET",
"POST",
"PUT",
"DELETE"
],
"ReRouteIsCaseSensitive": false,
"AuthenticationOptions": {
"AuthenticationProviderKey": "Bearer"
},
"LoadBalancerOptions": {
"Type": "LeastConnection"
}
},
{
"DownstreamPathTemplate": "/{all}",
"DownstreamScheme": "http",
"ServiceName": "ProjectsService",
"UpstreamPathTemplate": "/projects/{all}",
"UpstreamHttpMethod": [
"GET",
"POST",
"PUT",
"DELETE"
],
"ReRouteIsCaseSensitive": false,
"AuthenticationOptions": {
"AuthenticationProviderKey": "Bearer"
},
"LoadBalancerOptions": {
"Type": "LeastConnection"
}
}
],
"GlobalConfiguration": {
"BaseUrl": "http://localhost:5280",
"RequestIdKey": "ApiGateway",
"ServiceDiscoveryProvider": {
"Type": "Consul",
"Port": "8502",
"Host": "192.168.0.204"
}
}
}
I'm running it inside visual studio 2022 professional on windows 11. The issue is also present if running it in a docker container on Ubuntu Server 22.04
I forgot to mention that I have upgraded from Ocelot 22.0.1 on .NET6 to latest version 23.3.3 on .NET8. Before the upgrade everything was working perfectly.
Consul is running on docker compose with this .yml:
consul:
restart: always
image: consul:1.12.0
volumes:
- ./data/consul:/consul/data
command: agent -server -ui -node=server-1 -client=0.0.0.0 -bootstrap-expect=1
ports:
- "8502:8500"
logging:
driver: "json-file"
options:
max-size: "50m"
Reverting to 23.2.2 fixed the issue
Hello @minnocenti901! What is your full name?
Hello! It's Massimiliano Innocenti
Actual Behavior
If concurrent requests, ocelot uses the same servicename for every concurrent request
Then, at startup, I make 3 different requests at the same time:
- GET
http://localhost:5280/projects/api/projects
- GET
http://localhost:5280/projects/api/industrialSectors
- GET
http://localhost:5280/customers/api/customers
logs...
I'm expecting that for the route /customers it uses CustomersService instead of ProjectsService.
It seems we may have encountered a bug following a significant refactoring of the Consul provider by PR https://github.com/ThreeMammals/Ocelot/pull/2067 and commit https://github.com/ThreeMammals/Ocelot/commit/34cb3ebf9768ac8cd8d2c75139da2123e23fdba4
Reverting to 23.2.2 fixed the issue
@minnocenti901 Excellent! Continue using version 23.2.2 until we have resolved or understood the problem.
@ggnaegi, could you please join us and collaborate on this issue? Have you deployed version 23.3.x to Production?
@raman-m Thank you for your time!
@minnocenti901 commented
Hello! It's Massimiliano Innocenti
LoL Massimiliano Innocenti on LinkedIn Which one could be you? š
I prefer not to answer :)
@minnocenti901, are you a C# developer? If so, would you be willing to collaborate on problem-solving: code researching, debugging, design review and probably open a PR?
@minnocenti901, are you a C# developer?
Yes
If so, would you be willing to collaborate on problem-solving: code researching, debugging, design review and probably open a PR?
No, thank you. I don't have time :\ Im sorry
No, thank you. I don't have time :\ Im sorry
How did you detect the issue? Are you using Ocelot for certain projects? Regardless, I appreciate you reporting the bug!
I use it as main api gateway for a microservices architecture. I need to upgrade from NET6 to NET8 due to upcoming end of service and with the update I had to recheck that everything still works. Then comes the issue
Understood. For the time being, please proceed with version 23.2.2 for your solution upgrade.
@minnocenti901 commented Jul 10, 2024, 1:08 PM:
Reverting to 23.2.2 fixed the issue
Good. Did you change anything in the Consul services registrations, or was there nothing except a version increase? It seems the issue was introduced by PR #2067. You are correct; the issue might be related to concurrency. Max, could you please share the JSON from the Consul client? I need the complete list of services registered in Consul.
I just increase version of Ocelot. How do I get the JSON you need?
@raman-m @minnocenti901 I'm on it, cheers
Let me to proceed with this, @ggnaegi. Please concentrate on issue #2116. It appears that the author is not intending to contribute but rather aims to direct our attention to this bug.
@minnocenti901 Max, are you online?
Yes, i've seen you marked as closed. You need some testing?
The hotfix has been released in ver. 23.3.4, so you can download the pack from NuGet. I've completed all the testing on my side šø. Now, I would like to ask if you could perform testing in your environment, preferably manual testing in a real setting. What kind of environment do you have? Production, Staging, or Local?
Local and Staging, cannot run tests on production
Can you handle deployment? What is your role in the project, QA? Reporting back the test results from Staging is sufficient for me.
Yes I can. I have to update to 23.3.4?
Emm... Yes, please, update version, deploy and test.
Updated from 23.2.2 to 23.3.4. I now receive this error for every request I make:
warn: Ocelot.Responder.Middleware.ResponderMiddleware[0]
requestId: 0HN74COOPNRLJ:00000001, previousRequestId: No PreviousRequestId, message: 'Error Code: ConnectionToDownstreamServiceError Message: Error connecting to downstream service, exception: System.Net.Http.HttpRequestException: Resource temporarily unavailable (server-1:80)
---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Ocelot.Requester.TimeoutDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Ocelot.Requester.MessageInvokerHttpRequester.GetResponse(HttpContext httpContext) errors found in ResponderMiddleware. Setting error response for request path:/projects/api/projects, request method: GET'
Configuration is not changed and it is the same as the one is posted
EDIT: server-1 is the node name of consul
Did you read this:
server-1 is the node name of consul
Default service builder uses node name as a host! https://github.com/ThreeMammals/Ocelot/blob/6088515173b70abd52798e544e5ded409680dbdb/src/Ocelot.Provider.Consul/DefaultConsulServiceBuilder.cs#L96-L97 This is current best practice because node may/should contain the same hosts but different ports in load balancing scenarios, when Docker containers are deployed with different ports but same host. In this case it is logical thing to assign real host name to node name.
But I guess if Consul has setup by default, seems node names are abstract. Please apply quick patch from the docs! Or go to Consul admin tool and assign real host names to node names!
It's curious how your environment functioned properly with version 23.2.2, given that the "node name as host name" feature has been available since version 13.5.2.
After the implementation of a "MyConsulServiceBuilder" using only the service.address it works perfectly. Thanks.
It's curious how your environment functioned properly with version 23.2.2, given that the "node name as host name" feature has been available since version 13.5.2.
I don't know :( it worked.
I'm just curious, could you make node name empty in Consul admin? Seems not, right?
I've asked AI and got this answer :D
In Consul administration, it is not possible to have an empty node name; however, you can change the node name and IP address by following a specific procedure which involves removing the node from the cluster, wiping its local data, and then restarting it as a new node with the updated information. Additionally, ACL rules in Consul allow for read-write access to a node with a specific name, and read-only access to any node name using an empty prefix.
After the implementation of a "MyConsulServiceBuilder" using only the service.address it works perfectly. Thanks.
It's curious how your environment functioned properly with version 23.2.2, given that the "node name as host name" feature has been available since version 13.5.2.
I don't know :( it worked.
Great! Congrats! š„³ Can you start deployment and testing please? I am expecting your feedback by the end of the day...
Local and Staging works perfectly. Every concurrent request receives the correct route. Can't test in production, sorry
It appears we've officially resolved the issue. š
Can't test in production, sorry
No worries! Are other team members responsible for Production? It would be beneficial to inform them about the successful tests in Staging so they can initiate a new Prod deployment.
Thank you for your contribution!
Expected Behavior
Ocelot should request from Consul the correct service for every route
Actual Behavior
If concurrent requests, ocelot uses the same servicename for every concurrent request
Steps to Reproduce the Problem
On consul I have 2 services registered: ProjectsService with url
http://localhost:5013
CustomersService with urlhttp://localhost:5004
Ocelot is running on
http://localhost:5280
Global configuration:
Routes configuration:
Then, at startup, I make 3 different requests at the same time:
http://localhost:5280/projects/api/projects
http://localhost:5280/projects/api/industrialSectors
http://localhost:5280/customers/api/customers
This is the log in console with LogLevel Debug
I'm expecting that for the route /customers it uses CustomersService instead of ProjectsService.
Specifications