search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

Eureka 0-downtime Blue/Green deployments issue #4098

Open leonard520 opened 2 years ago

leonard520 commented 2 years ago

Describe the bug We are leveraging eureka for the service discovery function and we have two different deployments for one app as blue-green deployments. At beginning, deployment A is start up as eureka.instance.initialStatus=UP while deployment B as eureka.instance.initialStatus=OUT_OF_SERVICE. After running sometime, we switch them by PUT /eureka/v2/apps/appID/instanceID/status?value={OUT_OF_SERVICE or UP}. It works good usually. However, sometimes, we find eureka server is blocked by network issue to this app, and the lease for the app expires. Then eureka server will ask client to re-register. In this case, the initialized status is not correct for the two deployments. Is there a way that we can override “eureka.instance.initial-status” without restarting these instances? Such as config hot reload? Will it work since app is up and running for a long time.

leonard520 commented 2 years ago

I find that the initial status can be refreshed when EurekaInstanceConfigBean is reloaded. So one possible solution is to use Config Server to store eureka.instance.initialStatus and when want to swap the two deployments,

  1. update the value in Config Server
  2. curl -XPOST http://{eureka-client}/actuator/refresh together with overriding the status to the eureka server-side so that they can both update to the new status.

This is a bit complicated, not sure if any other methods can achieve it.

spencergibb commented 2 years ago

You can use the ServiceRegistry.setStatus() method. https://github.com/spring-cloud/spring-cloud-netflix/blob/main/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/serviceregistry/EurekaServiceRegistry.java#L75

leonard520 commented 2 years ago

@spencergibb Thanks for the response. May I double confirm that the operation ServiceRegistry.setStatus() happens in client app side? If that case, when should the client app set the value? Does it mean the client app should be aware of the time the swap of blue-green happens?

spencergibb commented 2 years ago

ServiceRegistry.setStatus() can happen anywhere. It's just calling the eureka server API. I'd say that something has to be aware of the blue/green swap, but not necessarily a client.

leonard520 commented 2 years ago

@spencergibb I think the problem we found is caused by the obsolete client status. Even the server-side is aware of the blue/green swap, when the client lost connection due to a network issue or so, the client will re-registry with a wrong status.

spencergibb commented 2 years ago

Something has to change the status of that client, there's no other way around it.

spring-cloud-issues commented 2 years ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

leonard520 commented 2 years ago

Do we have a plan to improve this behavior that the client can be notified about the change? I believe this is a common issue for Eureka users.

gzmrst-sr commented 2 years ago

We are running into this problem as a user of a PaaS with a provided Eureka server. The PaaS does blue/green deployments by setting the status as described by @leonard520 above... The application being deployed has no knowledge of how it is deployed or of any change it needs to make in the status.

What should be the intended behaviour in situations like this? Do we have to customise the client side Eureka client code to change the initial status? Are there any plans to fix blue/green deployments with Eureka?

spring-cloud-issues commented 1 year ago

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.

gzmrst-sr commented 1 year ago

We are still struggling with this issue. It happens whenever the app loses connection to Eureka for a time, regardless of the cause.

@spencergibb Could you please answer my question above? How is it supposed to work?

Or is it the case that the initial status was not intended for use like this? (Although I have trouble seeing where else it would be used)

In any case, the problem is that the Eureka client is liable to re-register itself with the initial status every time it loses connection. And there is no mechanism to change the initial status that I am aware of.

gzmrst-sr commented 1 year ago

@spencergibb @OlgaMaciaszek Apologies for bumping an old issue, but it is still a problem for us and we'd greatly appreciate some pointers on how it's supposed to work and/or how we should work around the problem. Right now we have to just accept occasional outages caused by an incorrect Eureka status and restart everything, which seems sub optimal to say the least.

spring-cloud-issues commented 1 year ago

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.

gzmrst-sr commented 1 year ago

@spring-cloud-issues @OlgaMaciaszek Yes please, we would still like to have our questions around this issue answered...

spring-cloud-issues commented 1 year ago

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.

spring-cloud-issues commented 1 year ago

Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.