search

Netflix / eureka

AWS Service registry for resilient mid-tier load balancing and failover.
Apache License 2.0
12.37k stars 3.74k forks source link

JerseyEurekaHttpClient issue with 404 responses in Spring Boot 3.x #1515

Open kvmw opened 1 year ago

kvmw commented 1 year ago

Spring Boot 3.x includes a payload to error responses, by default. exmple:

{
  "timestamp":"<time>",
  "status": 404,
  "error": "Not Found",
  "path": "/eureka/apps/<app-id>/<instance-id>"
}

On the other hand JerseyEurekaHttpClient::sendHeartBeat method tries to read and deserialise the response payload to InstanceInfo , if it is not empty.

Failure in deserialising the above error payload results in 400 (bad request) error and the client does not continue with the registration (which supposed to happened after receiving 404).

A similar issue in Spring Cloud Netflix has been fixed by updating RestTemplateTransportClientFactory and WebClientTransportClientFactory.

I believe a simple solution needs to be implemented for JerseyEurekaHttpClient.

Here is a sample stacktrace :

2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT 2023-09-18T14:12:23.518Z  INFO 7 --- [tbeatExecutor-0] c.n.d.s.t.d.RedirectingEurekaHttpClient  : Request execution error. endpoint=DefaultEndpoint{ serviceUrl='https://SERVER/eureka/}, exception=HTTP 400 Bad Request stacktrace=jakarta.ws.rs.WebApplicationException: HTTP 400 Bad Request
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.provider.DiscoveryJerseyProvider.readFrom(DiscoveryJerseyProvider.java:110)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:233)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:212)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:102)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:298)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.client.InboundJaxrsResponse$1.call(InboundJaxrsResponse.java:93)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:365)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.client.InboundJaxrsResponse.runInScopeIfPossible(InboundJaxrsResponse.java:244)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.glassfish.jersey.client.InboundJaxrsResponse.readEntity(InboundJaxrsResponse.java:90)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.jersey3.AbstractJersey3EurekaHttpClient.sendHeartBeat(AbstractJersey3EurekaHttpClient.java:151)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.MetricsCollectingEurekaHttpClient.execute(MetricsCollectingEurekaHttpClient.java:73)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.executeOnNewServer(RedirectingEurekaHttpClient.java:121)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:80)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:120)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:837)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1401)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at java.base/java.lang.Thread.run(Unknown Source)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Root name ('timestamp') does not match expected ('instance') for type `com.netflix.appinfo.InstanceInfo`
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2] (through reference chain: com.netflix.appinfo.InstanceInfo["timestamp"])
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.DeserializationContext.reportPropertyInputMismatch(DeserializationContext.java:1787)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.DeserializationContext.reportPropertyInputMismatch(DeserializationContext.java:1803)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext._unwrapAndDeserialize(DefaultDeserializationContext.java:348)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:320)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.converters.EurekaJacksonCodec.readValue(EurekaJacksonCodec.java:213)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at org.springframework.cloud.netflix.eureka.server.CloudJacksonJson.decode(CloudJacksonJson.java:84)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT at com.netflix.discovery.provider.DiscoveryJerseyProvider.readFrom(DiscoveryJerseyProvider.java:103)
   2023-09-18T16:12:23.51+0200 [APP/PROC/WEB/0] OUT ... 37 more
ouaibsky commented 5 months ago

Hello

I confirm this is a big issue when using spring-cloud-eureka-server, breaking the replication between 2 server instances (after one instance disappear and try to come back) more information there: https://github.com/spring-cloud/spring-cloud-netflix/issues/4220 It contains an example on how to reproduce.

It's a simple fix I can do but I'm not able to find the source code for eureka-client-jersey3 ? Any clue please ?

pom.xml from eureak-client-jersey3 is referencing this repository. I'm unable to find any repository or source code in github for AbstractJersey3EurekaHttpClient.java

Thx Christophe

ouaibsky commented 5 months ago

Hey Does someone can help me by explaining me where is eureka-client-jersey3 sources ? Then I'll be able to propose PR.

spencergibb commented 5 months ago

https://github.com/Netflix/eureka/tree/2.x

ouaibsky commented 5 months ago

PR is available here: https://github.com/Netflix/eureka/pull/1540

Tested successfully with example provided at: https://github.com/spring-cloud/spring-cloud-netflix/issues/4220#issuecomment-1900788126