search

payara / Payara

Payara Server is an open source middleware platform that supports reliable and secure deployments of Java EE (Jakarta EE) and MicroProfile applications in any environment: on premise, in the cloud or hybrid.
http://www.payara.fish
Other
879 stars 301 forks source link

[Regression] New HTTP/2 Warnings #6587

Closed lprimak closed 4 weeks ago

lprimak commented 5 months ago

Brief Summary

New warnings appear below. There were no warnings before. These warnings started appearing with Payara 6.2024.3

[#|2024-03-14T16:09:23.862+0000|WARNING|Payara 6.2024.3|org.glassfish.grizzly.http2.frames.SettingsFrame|_ThreadID=91;_ThreadName=http-thread-pool::http-listener-2(3);_TimeMillis=1710432563862;_LevelValue=900;|
  Setting 2,794 is unknown and will be ignored|#]

[#|2024-03-14T16:09:32.167+0000|WARNING|Payara 6.2024.3|org.glassfish.grizzly.http2.frames.SettingsFrame|_ThreadID=90;_ThreadName=http-thread-pool::http-listener-2(2);_TimeMillis=1710432572167;_LevelValue=900;|
  Setting 6,714 is unknown and will be ignored|#]

Expected Outcome

No warnings

Current Outcome

Warnings

Reproducer

No simple reproducer (at least not yet)

Operating System

Any

JDK Version

Any

Payara Distribution

Payara Server Full Profile

Elifzeynepedman commented 5 months ago

Hi @lprimak,

Could you please provide a simple-to-follow scenario on how to reproduce this on the latest version? A reproducer should ideally follow the SSCCE rules: http://www.sscce.org/. It will greatly help us to find the cause and fix it.

Thank you, Elif

lprimak commented 5 months ago

I’m sorry I don’t have one. I have production monitoring logs and this was a new alert since upgrading

Elifzeynepedman commented 5 months ago

Hi @lprimak,

Could you please provide answers to the following questions, it would help us proceed with the issue;

  1. Can you describe the specific circumstances under which the HTTP/2 warning appears?
  2. Are you encountering the warning while using an HTTP/2 enabled endpoint in your application
  3. Could you specify if you're utilizing a particular development framework that may interact with HTTP/2.

It would be great if you could provide as much information as you could for us to get a better understanding.

Thank you, Elif

lprimak commented 5 months ago
  1. I can't tell when this specifically happens, it's random / sporadic.
  2. Yes, http/2 endpoint is enabled (by default) and that's what I am using.
  3. JSF and PrimeFaces (latest release version)

I am sorry I can't be more helpful but the nature of this problem prevents this unfortunately.

github-actions[bot] commented 4 months ago

Greetings, It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please? We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

lprimak commented 4 months ago

Not abandoned :)

lprimak commented 4 months ago

I found a way to consistently reproduce the issue. Stay tuned.

lprimak commented 4 months ago

Didn't work

Ok, here's how to reproduce. It's not quite trivial but it's simple and self-contained:

I did it in a docker container, but you don't have to. You can run any JDK starting with 17 and higher

Run docker container:

docker run -it --rm azul/zulu-openjdk-alpine:17-latest

Set up docker container:

apk update && apk add git maven curl firefox

Clone and run tests:

git clone https://github.com/flowlogix/flowlogix.git
cd flowlogix
curl https://raw.githubusercontent.com/lprimak/infra/main/scripts/cloud/docker/_builders/agent-maven-settings.xml -o /flowlogix/settings.xml
mvn verify -ntp -gs /flowlogix/settings.xml -Pui-test -Dwebdriver.browser=firefox -Dtest=none -Dsurefire.failIfNoSpecifiedTests=false

See the output in the logs:

less /flowlogix/target/dependency/payara6/glassfish/domains/domain1/logs/server.log

You will see the errors in the log file

lprimak commented 4 months ago

Well, spoke too soon, the reproducer above fails to reproduce the issue. It's still not pinned down when / how this happens but it does.

lprimak commented 4 months ago

I guess the only thing I learned today is that it only happens when Chrome is the client

Elifzeynepedman commented 4 months ago

Hi @lprimak,

Thank you for your efforts in reporting this issue. Unfortunately, as we have not received the necessary reproducer to consistently replicate the problem, we are unable to proceed with further investigation. Therefore, we will have to close this issue. If you are able to provide the details needed in the future, please feel free to reopen the issue or submit a new one.

Thank you, Elif

lprimak commented 4 months ago

Hi, Elif,

Please reopen this. Just because it's not "reliably reproducible" at the moment doesn't mean the issue doesn't exist. I have worked hard to reproduce this issue, and even if unsuccessful, I don't want to see all that work wasted. Thank you

phillipross commented 4 months ago

An advantage to keeping things like this open in the issue tracker can help provide visibility to community users who may run into the issue and have more resources/bandwidth to provide higher quality reproducers that help find the true root cause and and/or fix underlying issues.

Specifically, issues related to http/2 have been notoriously difficult to troubleshoot, fix, detect regressions in, or even make sense of or differentiate between new issues, regressions, or somewhere in between 😬

fturizo commented 3 months ago

@lprimak, @phillipross, we understand your comments. We'd have no problem keeping this issue open until a reliable reproducer is created to properly fix the issue. Still, we have a clear support policy that states that all bug reports must be accompanied by a reliable reproducer to be formally escalated to our Platform engineers, so keep this in mind.

However, in the spirit of conceding that HTTP/2 issues are indeed challenging to troubleshoot, we'll keep this issue open for another two months to give you (or other community users) the opportunity to share a reliable (or semi-reliable) reproducer to help us patch the issue. I'm afraid that if no reproducer is found, we'll proceed to close the issue again.

roberto-cisternino commented 2 months ago

I have similar error along with others but not related to my code.

[2024-06-25T19:00:39.324+0000] [Payara 6.2023.10] [SEVERE] [AS-WEB-CORE-00037] [jakarta.enterprise.web.core] [tid: _ThreadID=36 _ThreadName=http-thread-pool::http-listener-1(4)] [timeMillis: 1719342039324] [levelValue: 1000] [[ An exception or error occurred in the container during the request processing java.net.ProtocolException: Host header not found in request at com.sun.web.security.RealmAdapter.getHostAndPort(RealmAdapter.java:1126) at com.sun.web.security.RealmAdapter.redirect(RealmAdapter.java:1096) at com.sun.web.security.RealmAdapter.hasUserDataPermission(RealmAdapter.java:384) at com.sun.web.security.RealmAdapter.hasUserDataPermission(RealmAdapter.java:326) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:451) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:726) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:372) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) at java.base/java.lang.Thread.run(Thread.java:840) ]]

[2024-06-25T19:13:40.879+0000] [Payara 6.2023.10] [WARNING] [] [org.glassfish.grizzly.http2.frames.SettingsFrame] [tid: _ThreadID=45 _ThreadName=http-thread-pool::http-listener-2(5)] [timeMillis: 1719342820879] [levelValue: 900] [[ Setting 8 is unknown and will be ignored]]

[2024-06-25T19:13:46.985+0000] [Payara 6.2023.10] [WARNING] [] [org.glassfish.grizzly.http2.frames.SettingsFrame] [tid: _ThreadID=41 _ThreadName=http-thread-pool::http-listener-2(1)] [timeMillis: 1719342826985] [levelValue: 900] [[ Setting 8 is unknown and will be ignored]]

[2024-06-25T19:40:34.187+0000] [Payara 6.2023.10] [WARN] [] [org.jboss.weld.Servlet] [tid: _ThreadID=36 _ThreadName=http-thread-pool::http-listener-1(4)] [timeMillis: 1719344434187] [levelValue: 900] [[ WELD-000714: HttpContextLifecycle guard leak detected. The Servlet container is not fully compliant. The value was 1]]

[2024-06-25T19:40:34.193+0000] [Payara 6.2023.10] [WARN] [] [org.jboss.weld.Context] [tid: _ThreadID=36 _ThreadName=http-thread-pool::http-listener-1(4)] [timeMillis: 1719344434193] [levelValue: 900] [[ WELD-000225: Bean store leak detected during org.jboss.weld.module.web.context.http.HttpRequestContextImpl association: org.apache.catalina.connector.RequestFacade@4379c2b5]]

fturizo commented 1 month ago

@roberto-cisternino, would you be able to share a self-contained reproducer that can help us identify how these warnings are triggered?

lprimak commented 1 month ago

I actually got the same errors in Payara 5 so this is a long standing issue not a regression.

roberto-cisternino commented 1 month ago

I am not sure I can isolate the issue easely. My software is J2EE 10 and uses EJB Singletons and REST. I suspect this happen after session expires.

fturizo commented 4 weeks ago

@roberto-cisternino, thanks for the update but sadly we need a self-contained reproducer to fix this issue and with the details we have gotten we have not been able to reproduce it yet. Sadly, as we have given enough time for users to share a viable self-contained reproducer so we can proceed to fix this issue and none has been shared, we'll proceed to close this issue.

If in the future any user can create a consistent reproducer of this issue that you're able to share feel free to reopen it.