search

spring-projects / spring-boot

Spring Boot helps you to create Spring-powered, production-grade applications and services with absolute minimum fuss.
https://spring.io/projects/spring-boot
Apache License 2.0
75.14k stars 40.68k forks source link

Spring Boot 2.7.8 application cannot be reached in Azure #33974

Closed Telamon-m closed 1 year ago

Telamon-m commented 1 year ago

I'm using Spring boot within a Docker image on Azure (App Service). After upgrading from Spring Boot 2.7.7 to 2.7.8 no request reaches the spring container any more, Tomcat responds with 404 to all requests. The strange thing is that everything is working locally within the IDE as well as when running the Docker image locally: The exact same image with the same env variables as in Azure is running locally.

Within the Azure App Service the application starts successfully, however absolutely no request is processed. Even actuators are not reachable. Tomcat responds with a 404 to every request.

After downgrading my app to 2.7.7 everything works again in Azure App Service. (The Spring Boot downgrade was the only change.)

I attached the logfiles where you can see that the application was started successfully, however there is no more log produced, except from scheduled tasks. No http request reaches the spring container.

Spring Boot 2.7.8 issue.log

wilkinsona commented 1 year ago

Thanks for the report. Unfortunately, if no HTTP requests are reaching the container then don't think we'll be able to help as Spring Boot isn't responsible for deciding if requests should be routed to the container. It's Azure that makes this routing decision. It may be that something has changed in 2.7.8 that's affecting this decision, but without knowing the precise details of how Azure behaves, there's nothing we can do here. Please reach out to Microsoft for some support on the Azure side. If they identify why requests aren't being routed to the application and it is caused by Spring Boot, please comment here and we can re-open the issue and take another look.

Telamon-m commented 1 year ago

Hi @wilkinsona , The requests do reach Tomcat. So this cannot be a pure Azure topic. And as downgrading Spring Boot again to 2.7.7 fixes the issue the root cause from my point of view is something that came in with 2.7.8 I can image that similar issues occur in other environments, e. g. k8s etc. Thanks, Michael

bclozel commented 1 year ago

Hello @Telamon-m, This sounds similar to #33921 - is there a Java agent somehow involved? Can you upgrade it or disable it to confirm?

wilkinsona commented 1 year ago

Sorry, I misunderstood what you meant by "No http request reaches the spring container".

If the suggestion from @bclozel does not help, please describe the problem as precisely as possible and provide an example of an HTTP request-response exchange showing the exact details (URI, headers, body, and so on) of both the request and response.

marcomima commented 1 year ago

I'm facing the same issue (also Spring Boot 2.7.7 to Spring Boot 2.7.8, no other changes). I have tried both to update the java-agent (to latest https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.4.8/applicationinsights-agent-3.4.8.jar) and completely removing it from the Dockerfile. With the same result.

I use spring-actuator and as an example this fails: /actuator/info

Headers:
Request URL: .../actuator/info
Request Method: GET
Status Code: 404 
Remote Address: ...
Referrer Policy: strict-origin-when-cross-origin
content-language: en
content-length: 431
content-type: text/html;charset=utf-8
date: Wed, 25 Jan 2023 15:19:44 GMT
request-context: appId=...
x-azure-ref: ...
x-cache: CONFIG_NOCACHE
:authority: ...
:method: GET
:path: /actuator/info
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
cache-control: max-age=0
cookie: JSESSIONID=...
sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

Response:

<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1></body></html>
bclozel commented 1 year ago

@marcomima I guess the request you're sharing is from the point of view of the client? It would be interesting to see the same from the server's perspective. Could you add to your application a simple Servlet filter that logs the HttpServletRequest::getRequestURI? Also, it's not clear why we're seeing Tomcat's default 404 page instead of Spring Boot's error page. Did you configure your application with a specific Servlet mapping?

One more thing, could you try downgrading to Tomcat 9.0.70 with the following in your build.gradle or pom.xml file?

ext['tomcat.version'] = '9.0.70'
<properties>
  <tomcat.version>9.0.70</tomcat.version>
</properties>

Trying these options should help us narrow down where the problem comes from.

marcomima commented 1 year ago

@bclozel Yes, it was from the clients view, I haven't configured any specific Servlet mappings. I get zero logs from the web context, also nothing from the ServletFilter. The application is running as I have a Scheduler (@Scheduled) running every 5 minutes that writes something in the logs.

Downgrading to Tomcat 9.0.70 makes the application work again!

derkoe commented 1 year ago

We ~have~ had the same issue and it is caused by the Application Insights Java agent. We used the agent version 3.2.3.

We have now switched to the current version 3.4.8 which does not have this issue.

bclozel commented 1 year ago

@marcomima the fact that Tomcat 9.0.70 works with Spring Boot 2.7.8 leads me to think that this is indeed a Java agent issue - probably an instrumentation not being compatible with a Tomcat change. If the request is not even reaching a Servlet Filter and there is not specific Servlet mapping setup, this looks really out of Spring Boot's control. Can you have a look at the java launch command, maybe another agent is being applied?

@derkoe thanks for letting us know, this is indeed consistent with the symptoms here.

@Telamon-m could you provide the information I've requested here, but for your case?

marcomima commented 1 year ago

Yes, there was Azure also adding the Java-Agent. After turning that off it works just fine with the latest (3.4.8) java-agent: Had to remove: JAVA_TOOL_OPTIONS=-javaagent:/agents/java/applicationinsights-agent-codeless.jar -e StartupBootstrapper=Microsoft.ApplicationInsights.StartupBootstrapper

wilkinsona commented 1 year ago

Thanks very much, @marcomima.

@Telamon-m Does this work for you too?

spring-projects-issues commented 1 year ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

florentbo commented 1 year ago

@marcomima I am not using a docker image, but I directly deploy to a "java app" and I am encountering the same issue. It works with 2.7.7 but not with 2.7.8. How can we use your fix in an app?

marcomima commented 1 year ago

@florentbo have you checked the java -jar and what parameters are passed in to the app.jar? If there is no java-agent there it looks like another issue altogether

florentbo commented 1 year ago

@marcomima where can I configure this java -jar execution?

stages:

marcomima commented 1 year ago

@florentbo I don't know your environment and how java app work in Azure and how the application is started. Maybe in the logs of the Deployment Center you'll find the start of the application?

wilkinsona commented 1 year ago

There are over 3000 people watching this repository who may be receiving a notification for every comment on this issue. As such, unfortunately, it's not a good place to diagnose Azure's behavior and the details of how it deploys an app.

In the absence of any follow up from @Telamon-m and all the evidence pointing towards this being a Java agent problem that's out of Spring Boot's control, I am going to close this issue. @florentbo you may want to reach out to Microsoft support to learn how they configure the JVM and how you can ensure that either no agent is used or that the agent is up-to-date.

Telamon-m commented 1 year ago

Hi @wilkinsona , I can confirm that the described workaround works for me: Upgrade the azure monitor agent to latest version 3.4.9 fixed the issue. Thank you. For everyone else: We are struggling with Azure, because at least within App Services the App Setting ApplicationInsightsAgent_EXTENSION_VERSION is giving us a hard time, as it automatically adds an old agent version...

olivierboudet commented 1 year ago

Hello, for information for others having this issue, I encountered this with an older version of opentelemetry agent.

bpossolo commented 1 year ago

I experienced the same problem when upgrading my Azure Web App Service from using Spring Boot 2.5.5 to 3.0.4 Request were not reaching the web server and were all returning 404. My app also had Application Insights enabled and the ApplicationInsightsAgent_EXTENSION_VERSION configuration prop was set to ~3. I looked up the latest version available by looking at the Application Insights Java Agent github repo and updated the prop to 3.4.10 Everything starting working again after that. Thanks @Telamon-m 🌟

marcelhoekstra commented 1 year ago

I experienced the same problem when upgrading my Azure Web App Service from using Spring Boot 2.5.5 to 3.0.4 Request were not reaching the web server and were all returning 404. My app also had Application Insights enabled and the ApplicationInsightsAgent_EXTENSION_VERSION configuration prop was set to ~3. I looked up the latest version available by looking at the Application Insights Java Agent github repo and updated the prop to 3.4.10 Everything starting working again after that. Thanks @Telamon-m 🌟

Thank you, you saved my day!

my3sons commented 1 year ago

So this whole Java 17 thing on Azure has been quite an experience. We dealt with the 404 issue for some time by disabling app insights (which is obviously not a good thing), then when the 3.4.x agent got released that finally got fixed, but now we are fighting logging issues that were not there with Java 11, so am curious if anyone else is seeing the same behavior as us. We are using the codeless agent and with Java 11 all that was required was to include the following dependencies in our deployment. The logging framework we are using is the slf4j implementation from lombok.

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-spring-boot-starter</artifactId>
    <version>2.6.4</version>
</dependency>
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-logging-logback</artifactId>
    <version>2.6.4</version>
</dependency>

With that all our app logs ended up as expected in Log Analytics. However, with Java 17 and the 3.4.10 agent, in order for our app logs to make it to Log Analytics, we now need to also include at a minimum the following logback.xml in our deployment as well:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml"/>
    <appender name="aiAppender"
              class="com.microsoft.applicationinsights.logback.ApplicationInsightsAppender">
    </appender>
    <root level="info">
        <appender-ref ref="aiAppender" />
    </root>
</configuration>

This is no big deal, but what we are also observing which is much more impactful is that with the 3.4.10 agent, the "cloud_RoleName" field and some others as well are not get populated anymore in log analytics. We have multiple apps leveraging the same application insights resource and the cloud_RoleName is the one field that allowed us to write queries that would filter just the app logs you were interested in. With Java 11 and the older agents, the cloud_RoleName was automatically populated for us. I am curious if anyone else is observing this and should I create an issue for this?

florentbo commented 1 year ago

So this whole Java 17 thing on Azure has been quite an experience. We dealt with the 404 issue for some time by disabling app insights (which is obviously not a good thing), then when the 3.4.x agent got released that finally got fixed, but now we are fighting logging issues that were not there with Java 11, so am curious if anyone else is seeing the same behavior as us. We are using the codeless agent and with Java 11 all that was required was to include the following dependencies in our deployment. The logging framework we are using is the slf4j implementation from lombok.

<dependency>
  <groupId>com.microsoft.azure</groupId>
  <artifactId>applicationinsights-spring-boot-starter</artifactId>
  <version>2.6.4</version>
</dependency>
<dependency>
  <groupId>com.microsoft.azure</groupId>
  <artifactId>applicationinsights-logging-logback</artifactId>
  <version>2.6.4</version>
</dependency>

With that all our app logs ended up as expected in Log Analytics. However, with Java 17 and the 3.4.10 agent, in order for our app logs to make it to Log Analytics, we now need to also include at a minimum the following logback.xml in our deployment as well:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml"/>
    <appender name="aiAppender"
              class="com.microsoft.applicationinsights.logback.ApplicationInsightsAppender">
    </appender>
    <root level="info">
        <appender-ref ref="aiAppender" />
    </root>
</configuration>

This is no big deal, but what we are also observing which is much more impactful is that with the 3.4.10 agent, the "cloud_RoleName" field and some others as well are not get populated anymore in log analytics. We have multiple apps leveraging the same application insights resource and the cloud_RoleName is the one field that allowed us to write queries that would filter just the app logs you were interested in. With Java 11 and the older agents, the cloud_RoleName was automatically populated for us. I am curious if anyone else is observing this and should I create an issue for this?

I was tired to configure our logs like this and so we pushed the logs on ELK following these 2 repos: https://github.com/lemoncode21/docker-loging-elk/blob/master/docker-compose.yml https://github.com/lemoncode21/springboot-logging-elk/blob/master/src/main/resources/logback-spring.xml And it works very well. We have deployed it on an Azure VM Standard B2s

bpossolo commented 1 year ago

@my3sons looks like Java 18 support was added in agent 3.3.1 so you could try downgrading and see if that fixes your role name problem

https://github.com/microsoft/ApplicationInsights-Java/releases/tag/3.3.1

I would suggest either submitting an issue in their GitHub project or send a pr to fix it yourself. I’ve sent PRs to Microsoft and they’ve merged them.

sbarsbar1 commented 1 year ago

@marcomima I am not using a docker image, but I directly deploy to a "java app" and I am encountering the same issue. It works with 2.7.7 but not with 2.7.8. How can we use your fix in an app?

@florentbo: Just for your information: You can change the Application Insights Java Agent version under the configuration of the App Service. For me it was ~3 and together with Spring Boot 2.7.9 the App Service didn't serve any traffic to my application.

After updating the agent version to 3.4.10 it worked again.

image

florentbo commented 1 year ago

After updating the agent version to 3.4.10 it worked again.

@sbarsbar1, as already mentioned. I update the agent version to 3.4.10 it worked again. No 404. But no more logs :( So, we are using logstash and ELK now and it works very well

antgustech commented 1 year ago

I have a similar issue. When updating tomcat to 9.0.73, requests started to return 404 when using Application Insights auto instrumentation. When I turned it off, requests started working again. Reverting tomcat to older version 9.0.70 or disabling insights completely seems to help so there is some kind of issue using newer tomcat and older insights. If my App Service currently has set "ApplicationInsightsAgent_EXTENSION_VERSION": "~3", which seems to default to insights agent 3.2.11 as of this writing.

Updating this value to 3.4.10 initially seemed to work, but I think that simply made so that insights is not added at all. When inspecting startup logs, i could see that the agent was injected when using ~3 but not when using 3.4.10. I then read in the docs that this app config only controls if its windows or linux?

ApplicationInsightsAgent_EXTENSION_VERSION | Main extension, which controls runtime monitoring. | ~2 in Windows or ~3 in Linux.

It seems to be that ìn order to actual control the version, you will have to download the agent "manually" with a shell script or similar in the container. Or wait until Microsoft updates the default agent version in App Services application insights auto instrumentation. Sadly, there is no way to control the version using app config.

sheeeng commented 1 year ago

I experienced the same problem when upgrading my Azure Web App Service from using Spring Boot 2.5.5 to 3.0.4 Request were not reaching the web server and were all returning 404. My app also had Application Insights enabled and the ApplicationInsightsAgent_EXTENSION_VERSION configuration prop was set to ~3. I looked up the latest version available by looking at the Application Insights Java Agent github repo and updated the prop to 3.4.10 Everything starting working again after that. Thanks @Telamon-m 🌟

Is there a way to specify this ApplicationInsightsAgent_EXTENSION_VERSION property without having to hardcode it to the fixed 3.4.10 version, while supporting Spring Boot 3 / Java 17 environment?

If my App Service currently has set "ApplicationInsightsAgent_EXTENSION_VERSION": "~3", which seems to default to insights agent 3.2.11 as of this writing.

@antgustech, Could you share with us how did you manage to find out this piece of information?

antgustech commented 1 year ago

I experienced the same problem when upgrading my Azure Web App Service from using Spring Boot 2.5.5 to 3.0.4 Request were not reaching the web server and were all returning 404. My app also had Application Insights enabled and the ApplicationInsightsAgent_EXTENSION_VERSION configuration prop was set to ~3. I looked up the latest version available by looking at the Application Insights Java Agent github repo and updated the prop to 3.4.10 Everything starting working again after that. Thanks @Telamon-m 🌟

Is there a way to specify this ApplicationInsightsAgent_EXTENSION_VERSION property without having to hardcode it to the fixed 3.4.10 version, while supporting Spring Boot 3 / Java 17 environment?

If my App Service currently has set "ApplicationInsightsAgent_EXTENSION_VERSION": "~3", which seems to default to insights agent 3.2.11 as of this writing.

@antgustech, Could you share with us how did you manage to find out this piece of information?

Yes, got confirmation from Azure support. There are 3 possible solutions:

1.It is possible to download and upload a newer agent to app service and then injecting it with startup command. 2.Install it "manually" in the docker image. 3.Wait until Azure decides to update the auto injected agent version.

I simply reverted our tomcat change as I don't have time to deal with this right now.

juusechec commented 1 year ago

In my case was a bit different, which could be confused with this problem. Spring boot generates a .war that I could execute with java -jar app.war, but in deployment in GitHub Actions, I changed their name to app.jar to upload as a single file, this caused the 404 error. To solve this I deploy the war into a zip with zip target/package.zip target/app.war and execute with custom start command java -jar /home/site/wwwroot/target/app.war