search

eclipse-ee4j / glassfish

Eclipse GlassFish
https://eclipse-ee4j.github.io/glassfish/
375 stars 143 forks source link

http client sometimes receives a "400 Bad Request" on running HA LB tests. #18211

Closed glassfishrobot closed 12 years ago

glassfishrobot commented 12 years ago

GF build 17 Setup: Cluster with 10 instances Platform: OEL6 with JRockit Has this passed before: OEL + JRockit combination was last used in GF 3.1 where the LB tests did pass and these "400 Bad Request" response were not seen.

Some of the HA LB tests get a "400 Bad Request" response for some of the requests. Different tests have this issue during different runs. Tests that have the issue in one run have been observed to pass in another run. No pattern has been identified yet to be able to provide a set of steps to reproduce the issue.

This issue was thought to have been related to issue 18060, but is now being filed as a separate issue as issue 18060 has also been observed independent of the "400 Bad Request" response and also based on inputs that the ClassCastException observed in 18060 may not lead to a "400 Bad Request". These issues may still be related though.

Logs are attached from 3 failed tests.

The same tests have passed on the same platform (and other platforms - windows, solaris) and with the same http client. So it is unlikely that there is an error in the http request it self.

Logs have been attached from some failed https tests. The issue has also been observed with http tests. So, no pattern there either.

The "400 Bad Request" response has only been observed with LB test. Now, some of the LB tests add/remove new http(s) listeners that tests from other HA modules do not. We are trying to run all the LB tests minus these add/remove listener scenarios and see if we still have this issue.

If it helps, do let us know of a way to dump the http request received by the web-container ?

Affected Versions

[3.1.2_dev]

glassfishrobot commented 5 years ago
glassfishrobot commented 12 years ago

@glassfishrobot Commented @shingwaichan said: Do we really use "round-robin LB" in the test? Round-robin LB is not supported in 3.1.1. Is it a new requirement? Do those tests pass when sticky LB is used?

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: "round-robin" is the name of the policy that is used. Please see the policy name in the test/sjswsj/loadbalancer.xml. It does function as a sticky LB when sessions are in use.

Yes, sticky LB test has passed on this setup. The issue is intermittent and so its possible that it shows up in a sticky LB test as well.

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: Linking this with 18060 as it might be related.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Does this issue occur on Oracle JDK?

glassfishrobot commented 12 years ago

@glassfishrobot Commented mzh777 said: The results with Oracle JDK 1.6.0_30 shows the issue exits.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: I tried a few runs with SQE setup and can't seem to reproduce the errors.

http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy-jrockit/summary/lb.html http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-nopatch/summary/lb.html http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy0/summary/lb.html http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy1/summary/lb.html http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy2/summary/lb.html http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy3/summary/lb.html http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy4/summary/lb.html

How often do these errors occur (%)? I need a more stable way to reproduce the errors to debug further.

glassfishrobot commented 12 years ago

@glassfishrobot Commented mzh777 said: The runs you made have only 17 tests. The 2 runs by SQE (one for JRockit and the other for Oracle JDK) have 103 tests in each and the error string "400 Bad" can be found in both of them. Please update the gf-ha-qe/functional/lb/loadbalancer.testlist to tip.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Varun suggested the setup with 17 tests. Please let me know where can I find the list of 103 tests so I can update the test list.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: I ran the full LB test list and all 105 tests passed with no "400 Bad Request" errors.

http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-full/summary/lb.html

I'll re-run them again but any pointer to reproduce the error would be helpful.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Varun and Ming,

Can you try running the tests with the attached web-core.jar? When 400 happens, it should log out more info with a heading "#18211".

Thanks, Amy

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: When 400 happens sometimes, the host is null in the MappingData where it should use the default host.

===

host: null context: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/clusterjsp] wrapper: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/clusterjsp].StandardWrapper[jsp] servletName: jsp contextPath: /clusterjsp requestPath: /HaJsp.jsp wrapperPath: /HaJsp.jsp pathInfo: null redirectPath: null

===

host: StandardEngine[glassfish-web].StandardHost[server] context: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/webapps-simple] wrapper: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/webapps-simple].StandardWrapper[HelloWorldExample] servletName: HelloWorldExample contextPath: /webapps-simple requestPath: /servlet/HelloWorldExample wrapperPath: /servlet/HelloWorldExample pathInfo: null redirectPath: null

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: 400 happens when MappingData.host is null. Default host mapping "server" is present. hosts[pos].name equals the defaultHostName "server", however, hosts[pos].object is NULL, hence, mappingData.host is NULL.

com.sun.grizzly.util.http.mapper.Mapper:863.

if (mappingData.host == null) {
    Host[] hosts = this.hosts;
    int pos = findIgnoreCase(hosts, host);
    if ((pos != -1) && (host.equalsIgnoreCase(hosts[pos].name))) {
        mappingData.host = hosts[pos].object;
        hostPos = pos;
        contexts = hosts[pos].contextList.contexts;
        nesting = hosts[pos].contextList.nesting;
    } else {
        if (defaultHostName == null) {
            return;
    }
    pos = findIgnoreCase(hosts, defaultHostName);
    if ((pos != -1) && (defaultHostName.equalsIgnoreCase(hosts[pos].name))) {
        mappingData.host = hosts[pos].object;

// XXX hosts[pos]name equals defaultHostName "server", however, hosts[pos].object is NULL 
        hostPos = pos;
        contexts = hosts[pos].contextList.contexts;
        nesting = hosts[pos].contextList.nesting;
    } else {
        return;
    }
}
glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Looking at the logs, the com.sun.dft.glassfish.lb.RR.http.MultipleFailoverHttpOnly test which fails sometimes follows this scenario.

1. delete-http-listener --target st-cluster http-listener-2 2. deploy --target st-cluster /space/gf-ha/agent-repository/glassfish-samples/webapps-simple.war 3. export-http-lb-config --config gf-ha-qe-lb-config /space/gf-ha/agent-repository//loadbalancer.xml 4. send requests to http://jed-asqe-24.us.oracle.com:80/webapps-simple/session 5. start-cluster st-cluster 6. create-http-listener --target st-cluster http-listener-2 7. undeploy --target st-cluster webapps-simple

The bug report says "We are trying to run all the LB tests minus these add/remove listener scenarios and see if we still have this issue". Has this happened and what is the result?

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: No. We did not get a chance to try a run where there is no change in the listeners.

Are there indications that the issue could be related to addition/removal of listeners ?

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Here is the sequence for some tests that fail intermittently.

The app gets deployed successfully.

Both Grizzly and WebContainer get restarted during load balancer configuration.

The first request is sent to instance105 and the client receives a 200 response from instance105.

[testng] Sending Request 1 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session [testng] Serving Instance for Request 1:instance105

Between Grizzly startup and WebContainer startup, a second request is sent to the same instance105.

at com.sun.enterprise.v3.services.impl.GrizzlyProxy.configureGrizzly(GrizzlyProxy.java:159) at com.sun.enterprise.v3.services.impl.GrizzlyProxy.(GrizzlyProxy.java:121) at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:445) at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct

A second request to instance105 happens before the Mapper is fully configured and Connector is started.

at com.sun.grizzly.util.http.mapper.Mapper.internalMap(Mapper.java:900) at com.sun.grizzly.util.http.mapper.Mapper.map(Mapper.java:821) at com.sun.enterprise.v3.services.impl.ContainerMapper.map(ContainerMapper.java:335) at com.sun.enterprise.v3.services.impl.ContainerMapper.mapUriWithSemicolon(ContainerMapper.java:323) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)

before the mapper is fully set -

at org.apache.catalina.connector.Connector.start(Connector.java:1481) at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529) at org.apache.catalina.startup.Embedded.start(Embedded.java:936) at com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:599)

On all subsequent requests to instances, the request's Mapper contains .object=null because the Mapper wasn't fully configured yet.

[testng] ** Sending Request 2 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=2nd&datavalue=https [testng] Instance failure detected for instance105

[testng] ** Sending Request 3 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=3rd&datavalue=https [testng] Instance failure detected for instance109

[testng] ** Sending Request 4 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=4th&datavalue=https [testng] Instance failure detected for instance107

[testng] ** Sending Request 5 to Webserver jed-asqe-24.us.oracle.com:80> http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=5th&datavalue=https

The issue is caused by concurrent thread, which updates the Mapper at the time, another thread is processing request.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: This issue is related to 18267 since 400 happens when a request comes in between Grizzly and WebContainer startup before the mapper is set. The request shouldn't come in before the mapper is fully set.

"(a) GrizzlyService create a GrizzlyProxy, which creates ContainerMapper and Mapper. (b) WebContainer.postConstruct is invoked and then calls Connector.start and the above mapper is set."

glassfishrobot commented 12 years ago

@glassfishrobot Commented scatari said: As agreed in bug swat, this is not a 3.1.2 stopper, downgrading to P3.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Attach server.log which displays the request comes in between Grizzly and WebContainer start (GrizzlyService.postConstruct & WebContainer.postConstruct) and shows Mapper.mappingData.host=null which causes 400 Bad Request in CoyoteAdapter.doService.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Upon further investigation of the timestamps of the logs, the request that comes in between Grizzly and web container start is actually coming from loadbalancer health-check after the cluster is restarted and shouldn't cause 400 bad request and fail the test in theory.

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: Summary of more findings:

Below are details noted from results from one of the debug runs (http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/summary/lb.html)

Here are details of logs that show a 400 response when the request is NOT received between grizzly and web-container startup:

Some Patterns that were observed from the run:

a) It seems that changes to instance listener's may be triggering the issue. It is possible that the issue begins to appear after deletion and re-creation of the cluster's (i.e of all the instance's) http or https listener. Detail: See http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/failed-tests-list for the list of tests that failed in this run. The complete test list is here: http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/loadbalancer.testlist.cvs. instance108 seemed to be the problem instance in this run. Not all instance's serve requests in all the tests. Now, instance108 served the last request correctly in test "com.sun.dft.glassfish.lb.RR.http.OnlyHttpListeners". This test deletes the https listener and re-creates it. The test that follows deletes the http listener and then re-creates it. b) It seems that once an instance ends up in this state it begins to respond to all requests with a 400 response. Detail: Once the issue showed up in the "com.sun.dft.glassfish.lb.RR.http.QuiesceInstance", every test where instance108 received a request ended up failing with a 400 response from instance108. This can be verified by looking into the logs of the failed tests listed in http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/failed-tests-list

Here are details of Logs that indicate that the 400 response occurring when a request IS received between a grizzly startup and the web-container startup are a response to the LB Health Check ping:

LB health check requests are auto generated by LB and a 400 response from those does not affect the test's pass/fail status. Example test that passed but some instance's show a 400 response: http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/com.sun.dft.glassfish.lb.RR.https_routing_on.ChangeHttpsListener/html/index.html

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: The issue needs further evaluation and hence upgraded it to a P2:

Next steps being worked upon: 1. Amy is trying a patch that would provide more information on how the mapper's host entry ends up being null even when the request is NOT received between grizzly and web-container startup. 2. Some pattern seems to have emerged from the failed tests (see new updates to the bug report). We are checking if there is a specific reproducible case that can be created out of it. 3. We are also trying to run the tests on a different OEL6 setup and see if 18211 is reproduced there.

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: The issue was reproduced with build 21 on a second OEL6 setup.

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: Here is info on a pattern that has been observed:

In both the last run and in the earlier saved run (see comment dated 08/Feb/12 04:08 PM) the StickyRoundRobin test was affected by this issue.

In both the runs, the instance that sent the 400 response had previously served a request correctly in the "OnlyHttpListeners" test.

Here is the test list between the two tests along with the main steps that are executed in those tests.

OnlyHttpListeners: deploys the app, deletes the cluster's (all instances) https listener, sends a number (cluster-size2) requests, re-creates the https listener, undeploys the app OnlyHttpsListeners: deploys the app, deletes the cluster's (all instances) http listener, sends 1 request, re-creates the https listener, undeploys the app QuiesceApplication: deploys the app, sends a request, disables the app, sends a few more sticky requests, undeploys the app QuiesceInstance: deploys the app, sends a request, disables the serving instance, sends a few more sticky requests, undeploys the app StickyRoundRobin: deploys the app, sends a number (cluster-size2) of reuqests, undeploys the app

Now, the issue seems to be exposed by tests that cause requests to be sent to all the instances - like the StickyRoundRobin (failed) and the OnlyHttpListener (passed). The test that failed for us is the StickyRoundRobin. Assuming that until OnlyHttpListener completes the cluster setup has no issues, it could be a step after this test and before the StickyRoundRobin test that caused the problem.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @shingwaichan said: The mapper will be modified when a listener is created.

In http://java.net/jira/browse/GLASSFISH-18060, there is an issue about the mapper. It is caused by difference in HK2 events. So, we may like to debug whether it is the same case here.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: It is confirmed that mapper.host.object is null after https listener is recreated.

In the com.sun.dft.glassfish.lb.RR.http.OnlyHttpListeners test, the scenario follows

asadmin delete-http-listener http-listener-2 asadmin export-http-lb-config asadmin create-http-listener http-listener-2

WebContainer#addConnector

synchronized (mapperUpdateSync) {

            Mapper mapper = null;
            for (Mapper m : habitat.getAllByContract(Mapper.class)) {
if (m.getPort() == port && m instanceof ContextMapper) {
    ContextMapper cm = (ContextMapper) m;
    if (httpListener.getName().equals(cm.getId())) {
        mapper = m;
        break;
    }
}
            }

            if (start) {
connector.start();
            }

The newly created connector's mapper.host.object is confirmed to be null. The correct mapper returned by Habitat.getAllByContract(Mapper.class) should have the correct host.object by the time connector is started. Subsequent tests which send requests to the same instance without any restart of listener/instance sometimes fail with 400 Bad Request since host.object is null.

CoyoteAdapter#doService

connector.requestStartEvent(request.getRequest(),
request.getHost(), request.getContext());
            Container container = connector.getContainer();
            try {
request.lockSession();
if (container.getPipeline().hasNonBasicValves() ||
        container.hasCustomPipeline()) {
    container.getPipeline().invoke(request, response);
} else {
    // Invoke host directly
    Host host = request.getHost();
    if (host == null) {
        response.sendError(HttpServletResponse.SC_BAD_REQUEST);
        response.setDetailMessage(sm.getString(
"coyoteAdapter.noHost",
request.getRequest().getServerName()));
        return;
    }
glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: Assigning to Grizzly team to investigate why Mapper.host.object is null after https listener is recreated.

glassfishrobot commented 12 years ago

@glassfishrobot Commented scatari said: Can we identity the 3.1.2 build where this issue was not seen? This would help us evaluate Grizzly changes quickly.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @amyroh said: The sequence of the mapper events are different at runs. The issue might be related to #18060"). Full log with event debugging info attached.

[#|2012-02-08T18:25:56.733-0800|INFO|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=27;_ThreadName=Thread-2;|DynamicConfigListener-18211 config changed ADD interface com.sun.grizzly.config.dom.NetworkListener GlassFishConfigBean.com.sun.grizzly.config.dom.NetworkListener http-listener-1|#]

[#|2012-02-08T18:25:56.735-0800|INFO|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|WEBCONFIGLISTENER-18211 Web container config changed ADD interface com.sun.grizzly.config.dom.NetworkListener GlassFishConfigBean.com.sun.grizzly.config.dom.NetworkListener http-listener-1|#]

[#|2012-02-08T18:25:56.736-0800|INFO|oracle-glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=34;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [http-listener-1] on host/port [0.0.0.0:28080]|#]

[#|2012-02-08T18:25:56.747-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=27;_ThreadName=Thread-2;|java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1249) at com.sun.grizzly.util.http.mapper.Mapper.addHost(Mapper.java:214) at org.glassfish.internal.grizzly.ContextMapper.addHost(ContextMapper.java:96) at org.glassfish.internal.grizzly.V3Mapper.addHost(V3Mapper.java:103) at com.sun.enterprise.v3.services.impl.ContainerMapper.configureMapper(ContainerMapper.java:143) at com.sun.enterprise.v3.services.impl.GrizzlyProxy.configureGrizzly(GrizzlyProxy.java:159) at com.sun.enterprise.v3.services.impl.GrizzlyProxy.(GrizzlyProxy.java:121) at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:451) at com.sun.enterprise.v3.services.impl.DynamicConfigListener.processNetworkListener(DynamicConfigListener.java:156) at com.sun.enterprise.v3.services.impl.DynamicConfigListener.access$100(DynamicConfigListener.java:79) at com.sun.enterprise.v3.services.impl.DynamicConfigListener$1.changed(DynamicConfigListener.java:105) at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289) at com.sun.enterprise.v3.services.impl.DynamicConfigListener.changed(DynamicConfigListener.java:94) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

|2012-02-08T18:25:56.766-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=27;_ThreadName=Thread-2;|java.lang.Exception: Stack trace

at java.lang.Thread.dumpStack(Thread.java:1249) at com.sun.grizzly.util.http.mapper.Mapper.addContext(Mapper.java:351) at com.sun.grizzly.util.http.mapper.Mapper.addContext(Mapper.java:332) at org.glassfish.internal.grizzly.ContextMapper.addContext(ContextMapper.java:119) at org.glassfish.internal.grizzly.V3Mapper.addContext(V3Mapper.java:135) at com.sun.enterprise.v3.services.impl.ContainerMapper.configureMapper(ContainerMapper.java:144) at com.sun.enterprise.v3.services.impl.GrizzlyProxy.configureGrizzly(GrizzlyProxy.java:159) at com.sun.enterprise.v3.services.impl.GrizzlyProxy.(GrizzlyProxy.java:121) at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:451) at com.sun.enterprise.v3.services.impl.DynamicConfigListener.processNetworkListener(DynamicConfigListener.java:156) at com.sun.enterprise.v3.services.impl.DynamicConfigListener.access$100(DynamicConfigListener.java:79) at com.sun.enterprise.v3.services.impl.DynamicConfigListener$1.changed(DynamicConfigListener.java:105) at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289) at com.sun.enterprise.v3.services.impl.DynamicConfigListener.changed(DynamicConfigListener.java:94) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

[#|2012-02-08T18:25:56.768-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1249) at org.apache.catalina.connector.Connector.start(Connector.java:1483) at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529) at com.sun.enterprise.web.WebContainer.addConnector(WebContainer.java:3263) at com.sun.enterprise.web.reconfig.WebConfigListener$1.changed(WebConfigListener.java:126) at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289) at com.sun.enterprise.web.reconfig.WebConfigListener.changed(WebConfigListener.java:114) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

[#|2012-02-08T18:25:56.785-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1249) at com.sun.grizzly.util.http.mapper.Mapper.addHost(Mapper.java:214) at org.glassfish.internal.grizzly.ContextMapper.addHost(ContextMapper.java:96) at org.glassfish.internal.grizzly.V3Mapper.addHost(V3Mapper.java:103) at org.apache.catalina.connector.MapperListener.registerHost(MapperListener.java:466) at org.apache.catalina.connector.MapperListener.init(MapperListener.java:203) at org.apache.catalina.connector.Connector.start(Connector.java:1537) at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529) at com.sun.enterprise.web.WebContainer.addConnector(WebContainer.java:3263) at com.sun.enterprise.web.reconfig.WebConfigListener$1.changed(WebConfigListener.java:126) at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289) at com.sun.enterprise.web.reconfig.WebConfigListener.changed(WebConfigListener.java:114) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

[#|2012-02-08T18:25:56.787-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1249) at com.sun.grizzly.util.http.mapper.Mapper.addContext(Mapper.java:351) at org.apache.catalina.connector.MapperListener.registerContext(MapperListener.java:550) at org.apache.catalina.connector.MapperListener.init(MapperListener.java:213) at org.apache.catalina.connector.Connector.start(Connector.java:1537) at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529) at com.sun.enterprise.web.WebContainer.addConnector(WebContainer.java:3263) at com.sun.enterprise.web.reconfig.WebConfigListener$1.changed(WebConfigListener.java:126) at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289) at com.sun.enterprise.web.reconfig.WebConfigListener.changed(WebConfigListener.java:114) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401) at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281) at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: Question in comment dated - "09/Feb/12 12:45 AM"

The issue has been observed on OEL6 in all the 3.1.2 runs - first run on OEL6 was with build 16. At that time 18060 was thought to have been causing the 400 responses, but further investigation lead us to file this issue as a separate one.

The issue has not been observed yet on runs done on Solaris 11 and Windows 2008. Further analysis of the cause of the bug is needed to determine if it is platform specific.

glassfishrobot commented 12 years ago

@glassfishrobot Commented varunrupela said: Removed 18267 from the "depends on" link and adding it as a "related to" link.