payara / Payara

Payara Server is an open source middleware platform that supports reliable and secure deployments of Java EE (Jakarta EE) and MicroProfile applications in any environment: on premise, in the cloud or hybrid.
http://www.payara.fish
Other
883 stars 306 forks source link

Bean identifier index inconsistency after Hot re-deployment from NetBeans #300

Closed ratcashdev closed 9 years ago

ratcashdev commented 9 years ago

Editing a CDI bean in netBeans and hitting save redeploys the application in GF. After visiting a JAX-RS resoruce using the CDI bean we get the error below:

<pre>org.jboss.weld.exceptions.IllegalStateException: WELD-000227: Bean identifier index inconsistency detected - the distributed container probably does not work with identical applications
Expected hash: 813144575
Current index: BeanIdentifierIndex [hash=2099704167, indexed=12]:
   0: WELD&#37;AbstractSyntheticBean&#37;JSR196OAuthLogin-1.0-SNAPSHOT&#37;HttpSession
   1: WELD&#37;AbstractSyntheticBean&#37;com.ibm.jbatch.container.cdi.BatchCDIInjectionExtension&#37;HttpSession
   2: WELD&#37;AbstractSyntheticBean&#37;com.sun.faces.flow.FlowCDIExtension&#37;HttpSession
   3: WELD&#37;AbstractSyntheticBean&#37;fish.payara.cdi.jsr107.PayaraHazelcastCDIExtension&#37;HttpSession
   4: WELD&#37;AbstractSyntheticBean&#37;org.glassfish.cdi.transaction.TransactionalExtension&#37;HttpSession
ratcashdev commented 9 years ago

Likely cause of the error is the following code:

BeanManager beanManager = getBeanManager();
        Bean<MyBean> bean = (Bean<MyBean>) beanManager.resolve(beanManager.getBeans(MyBean.class));
        CreationalContext ctx = beanManager.createCreationalContext(bean);
        MyBean mBean= (MyBean) beanManager.getReference(bean, MyBean.class, ctx);

This snippet is used inside a ServerAuthModule. I am trying to release the bean using ctx.release(); as well as bean.destroy(mBean,ctx); but it does not help. If I stop trying to use CDI in the ServerAuthModule alltogether, then WELD is not complaining after hot-redeploy. However, I need to get this working.

smillidge commented 9 years ago

How are you registering/deregistering the JASPIC SAM in your code?

ratcashdev commented 9 years ago

The JASPIC code/jar is copied to the glassfish/lib folder and registered as a MessageSecurity through the admin console. Instructions taken from here: https://www.ibm.com/developerworks/java/library/j-authenticroast/ (see section container configuration).

The actual JASPIC code that is deployed deployed is here: https://github.com/ratcashdev/authenticroast You'll have to compile it yourself. Make sure to copy both the API and the IMPL jars. The app through which I test: https://github.com/ratcashdev/JSR196OAuthLogin (has a dependency on the API)

Try to change the SampleOAuth.java class while it's deployed 2-3 times, and do GET /api/test/secure/oauth with the header: Authorization: BEARER mytoken

BTW: Is there a more straightforward way to register the JASPIC code without messing inside the admin console?

smillidge commented 9 years ago

Take a look here for an example of how to register a SAM programmatically

https://github.com/javaee-samples/javaee7-samples/blob/master/jaspic/custom-principal/src/main/java/org/javaee7/jaspic/customprincipal/sam/SamAutoRegistrationListener.java

ratcashdev commented 9 years ago

Tried with the code from the sample (JaspicUtils). Thank you for pointing me to this solution. Simplifies it a lot. However, it did not help with the exception I was getting in the OP.

I have even tried to register a bean (as a JASPIC ServerAuthModule) that was injected by CDI into my @WebListener but after some edit of SampleOAuth.java and redeploy I got the same result: Bean identifier index inconsistency

smillidge commented 9 years ago

Can you try removing the registration of the SAM module on undeploy?

ratcashdev commented 9 years ago

Unfortunately, even after a successful deregistration of the SAM module and redeploy the exception remains. BTW: all sample codes with programmatic registration completely ignore the request policy (which in this case is 'null'). Is it really so, that this is completely useless data (even if I found a way to set it)?

Edit: I think I got closer: Severe: The web application [/JSR196OAuthLogin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@298e8b71]) and a value of type [name.aikesommer.authenticator.DefaultPrincipalStore] (value [name.aikesommer.authenticator.DefaultPrincipalStore@2dffca54]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Edit2: eliminated all the ThreadLocals in the old code. The exception still remains.

ratcashdev commented 9 years ago

Tried replaced the weld-osgi-bundle jar with a newer version (2.2.10-2.2.12 Final), but it did not help.

ratcashdev commented 9 years ago

Hi Steve, is there anything else I can do to help replicate/solve this issue?

smillidge commented 9 years ago

We just need to try and replicate the issue. We haven't had a chance to replicate with your test application yet. I'll let you know.

smillidge commented 9 years ago

Hi I've tried to reproduce this but can't can you provide me step by step instructions? I have compiled the AuthenticRoast jars and copied them into the GlassFish lib directory. I have compiled your war and opened it in NetBeans I have ran your war from NetBeans so it fires up Payara and deploys the application. I have then used curl curl -G -H "Authorization: BEARER mytoken" http://127.0.0.1:8080/JSR196OAuthLogin/api/test/secure/oauth

To access the application and I get a 401 UnAuthorized error and the server log is; Info: Retrieving main CDI bean. Info: Retrieved: com.ratcash.multilogin.authenticators.SampleOAuth@f4dfa38 Severe: No valid EE environment for injection of name.aikesommer.authenticator.DefaultPrincipalStore

I have then edited SampleOAuth and the application redeploys with server log

Info: RegId = __311_HttpServletserver /JSR196OAuthLogin Info: result = true Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references Info: visiting unvisited references WARN: WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. WARN: WELD-000411: Observer method [BackedAnnotatedMethod] public org.glassfish.jms.injection.JMSCDIExtension.processAnnotatedType(@Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. WARN: WELD-000411: Observer method [BackedAnnotatedMethod] org.glassfish.sse.impl.ServerSentEventCdiExtension.processAnnotatedType(@Observes ProcessAnnotatedType, BeanManager) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. WARN: WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. WARN: WELD-000411: Observer method [BackedAnnotatedMethod] org.glassfish.sse.impl.ServerSentEventCdiExtension.processAnnotatedType(@Observes ProcessAnnotatedType, BeanManager) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. WARN: WELD-000411: Observer method [BackedAnnotatedMethod] public org.glassfish.jms.injection.JMSCDIExtension.processAnnotatedType(@Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. WARN: WELD-001440: Scope type @javax.enterprise.context.SessionScoped() used on injection point [BackedAnnotatedField] @Inject @SessionScoped com.ratcash.multilogin.authenticators.AppAuthModule.store Info: registration returned: __311_HttpServletserver /JSR196OAuthLogin Info: Loading application [JSR196OAuthLogin] at [/JSR196OAuthLogin] Info: JSR196OAuthLogin was successfully deployed in 1,210 milliseconds.

Severe: No valid EE environment for injection of com.ratcash.multilogin.authenticators.AppAuthModule Info: Cr eated Info: Retrieving main CDI bean. Info: Cr eated Severe: No valid EE environment for injection of com.ratcash.multilogin.authenticators.SampleOAuth Info: Retrieved: com.ratcash.multilogin.authenticators.SampleOAuth@75014671 Severe: No valid EE environment for injection of name.aikesommer.authenticator.DefaultPrincipalStore

I have done this multiple times but not received the IllegalStateException I then use curl again and get the same 401 response and the server log is

smillidge commented 9 years ago

BTW Scope type @javax.enterprise.context.SessionScoped() used on injection point [BackedAnnotatedField] @Inject @SessionScoped com.ratcash.multilogin.authenticators.AppAuthModule.store

I don't think you should put a scope annotation on an injection point

ratcashdev commented 9 years ago

@inject: you're right. I misinterpreted some other sources on the internet. The definitive seems to be https://docs.oracle.com/javaee/6/tutorial/doc/gmgil.html

ratcashdev commented 9 years ago

As for the Sample app: commit #d386dc2a2f192f3d519f21f9ca12326c4ab48527 needs the jars in the lib folder of glassfish, however, current master doesn't (does the programmatic-registration) To get the exception, I merely changed the strings inside System.out.println statements to trigger hotdeployment (added or removed a space character): System.out.println("Called. Token=" + access_token);

The third redeploy attempt always caused the exception. I will try it on a virgin installation of payara.

smillidge commented 9 years ago

OK thanks I will try deleting the jars from the lib directory and see what happens

smillidge commented 9 years ago
ratcashdev commented 9 years ago

behaves the same on a virgin installation of Payara 4.1.152.1 with the embedded domain 'domain1'. What's even more disturbing, that after some edits, back and forth, suddenly it works again, regardless of the file's actual content (i.e. whether it actually equals to the original content when deployed for the first time). Windows 7, 64bit, with Netbeans 8.0.2, java 1.8.0_45

smillidge commented 9 years ago

I removed the jars from the lib directory and redeployed via NetBeans many times and can not reproduce the issue on Linux 64bit 1.8.0_45-b14 + NetBeans 8.0.2 + Payara 4.1.153-SNAPSHOT

smillidge commented 9 years ago

I'm afraid also can't reproduce on Payara 4.1.152.1 on Linux

ratcashdev commented 9 years ago

Strange. The only notable difference seems to be the OS, but I strongly doubt it could cause these kind of issues. What else do you suggest to try/check/eliminate?

ratcashdev commented 9 years ago

As a matter of fact, I can reproduce the issue on two different laptops with Windows 7. One DOES have an anti-virus, the other doesn't. Maybe some windows services (background shadow copy, defender, whatnot) are interfering? But in such a case a lot of ppl should have had similar issues.

ratcashdev commented 9 years ago

Got rid of the @SessionScoped annotation on the injection point. The initial deployment is still OK, everything as expected. After a hotdeployment, I am still getting the inconsistency exception, but this time also a second exception:

Info:   JSR196OAuthLogin was successfully deployed in 1�494 milliseconds.
Warning:   Error invoking requestInitialized method on ServletRequestListener org.jboss.weld.servlet.WeldListener
org.jboss.weld.exceptions.IllegalStateException: WELD-000227: Bean identifier index inconsistency detected - the distributed container probably does not work with identical applications
Expected hash: 1876076154
Current index: BeanIdentifierIndex [hash=-275291188, indexed=14]:
   0: WELD%AbstractSyntheticBean%JSR196OAuthLogin-1.0-SNAPSHOT%HttpSession
   1: WELD%AbstractSyntheticBean%WEB-INF/lib/AuthenticRoast-Impl-0.3.4-SNAPSHOT%HttpSession
   2: WELD%AbstractSyntheticBean%com.ibm.jbatch.container.cdi.BatchCDIInjectionExtension%HttpSession
   3: WELD%AbstractSyntheticBean%com.sun.faces.flow.FlowCDIExtension%HttpSession
   4: WELD%AbstractSyntheticBean%fish.payara.cdi.jsr107.PayaraHazelcastCDIExtension%HttpSession
   5: WELD%AbstractSyntheticBean%org.glassfish.cdi.transaction.TransactionScopedContextExtension%HttpSession
   6: WELD%AbstractSyntheticBean%org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider%HttpSession
...

Severe:   No valid EE environment for injection of com.ratcash.multilogin.authenticators.AppAuthModule
Info:   Creat ed
Info:   Retrieving main CDI bean.
Info:   Creat ed
Severe:   No valid EE environment for injection of com.ratcash.multilogin.authenticators.SampleOAuth
Info:   Retrieved: com.ratcash.multilogin.authenticators.SampleOAuth@35ce12ab
Severe:   org.jboss.weld.context.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.SessionScoped
    at org.jboss.weld.manager.BeanManagerImpl.getContext(BeanManagerImpl.java:708)
    at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:90)
    at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:165)

So it seems this has a close relationship with #299 .

ratcashdev commented 9 years ago

Tested it on a plain old vanilla glassfish 4.1 from oracle. That version did not throw the inconsistency exception on the same machine(win 7 64 bit)/java(1.8.0_45) version, whereas payara 4.1.152.x and 4.1.151 did.

ratcashdev commented 9 years ago

Is it possible to somewhere disable the "distributed container"?

Pandrex247 commented 9 years ago

Hi, I've tried to recreate this on a Windows environment, and I'm getting the same results as Steve. Can you lay out some steps to recreate the errors from a clean install to make sure we're not misunderstanding somewhere please?

I did the following on 64 bit Windows:

  • Cloned master branch of https://github.com/ratcashdev/JSR196OAuthLogin
  • Downloaded and extracted Payara 4.1.152.1
  • Downloaded and installed Oracle JDK 1.8.0_45
  • Downloaded and installed NetBeans
  • Added Payara domain domain1 to NetBeans
  • Build and added the 2 JARs from https://github.com/ratcashdev/authenticroast master branch
  • Built JSR196OAuthLogin using NetBeans, and clicked Run to deploy
    • This deployed successfully
  • Ran the GET request using cURL and RESTClient Firefox plugin
    • Same result that Steve gave
  • Made a change to the String and clicked Run again
    • Same result as Steve
  • Repeated previous step a few times, still same results
  • Stopped the server, removed the 2 JARs, and redid deployments
    • Same result as Steve
ratcashdev commented 9 years ago

The procedure seems to me right. I will try to remove EVERYTHING (incl. JAVA and MAVEN) from my windows machine and repeat the steps you outlined. Meanwhile I have realized that maybe the path of the payara/glassfish installation may play a role. My contains spaces, i.e. C:\JAVA Environment\glassfish4\

The maven version is (note the paths):

c:\JAVA Environment\maven\bin>mvn --version
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T13:57:37+02:00)
Maven home: c:\JAVA Environment\maven\bin\..
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_45\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"

JAVA home is visible from the output as well.

ratcashdev commented 9 years ago

Removed JAVA, Netbeans and maven. Reinstalled everything from scratch including a virgin payara distribution. Path still contained spaced. Result: Got the "inconsitency" exception. If Netbeans was using the bundled maven distribution (3.0.5), the result was still the same. On the other PC the paths to glassfish/maven do not contain spaces and the result is still the same (bean inconsistency). I think we can rule out the path as being the difference/culprit.

ratcashdev commented 9 years ago

I just realized, that you're clicking on "RUN" multiple times. This is not what I am doing. I click RUN only the very first time (which in turn asks me which app server to use/start) and I tell Netbeans to remember my selection during the session.

  • Do the REST call
  • edit SampleOauth.java
  • click save
  • !!! Wait till Netbeans performs hotdeploy. I do NOT click RUN any more
  • Do the REST call

If I click RUN after SAVE, the inconsistency exception is not thrown.

Pandrex247 commented 9 years ago

OK I've managed to reproduce your error now, just clicking save does seem to make it appear after three times. We'll look into a bit more for you.

Now that I know what to do, I'll see if I can reproduce it on Linux as well.

Pandrex247 commented 9 years ago

I've managed to reproduce this error on Linux with Payara 4.1.152.1. The error doesn't seem to be present in 4.1.151 though, so I thought I'd check the GlassFish upstream, and the error does get thrown there as well.

With that, it looks like it'll be something we've brought in from one of GlassFish upstream cherry-picks since 4.1.151.

Pandrex247 commented 9 years ago

I just tried recreating it on clean install of 4.1.151 again on a different VM, and couldn't. Would you mind retrying it again on a clean install of Payara 4.1.151 when you have the chance please?

ratcashdev commented 9 years ago

I have just re-tested on 4.1.151 (downloaded from http://bit.ly/1CGCtI9 ) I could not reproduce the bug. So I guess I have to apologize for my former statement.

Pandrex247 commented 9 years ago

No problem, I was just trying to determine if it was an upstream bug or something we'd put in ourselves.

ratcashdev commented 9 years ago

I was checking the issues labelled as "upstream" for Payara and the related commits. I am willing to do the bisection straight on Payara, if I could get access to the nighly builds. Are those reachable somewhere? Could not download them from your jenkins.

ratcashdev commented 9 years ago

OK, tracked it down. Plain Glassfish nightly glassfish-4.1-b13-03_26_2015 is OK, using Weld 2.2.2 (Final). The nighly glassfish-4.1-b13-03_27_2015 is bad, exhibits the problem described here. The changelog of the latter mentions a single thing: "Upgrade to Weld 2.2.10.Final"

ratcashdev commented 9 years ago

Downgrading to Weld 2.2.8 (Final) resolves the issue. 2.2.9 already throws the exception. Upgrading to 2.2.14 does not help. After reading the changelog of 2.2.9, it seems this behavior is not really a bug, but rather a feature: https://issues.jboss.org/browse/WELD-1818 So now the question is,

  1. what causes WELD to think that we're in a cluster?
  2. what causes WELD to discover the incorrect index hash?
smillidge commented 9 years ago

OK thanks for the diagnostic. We can't revert Weld due to fixes we need in the later weld so we'll need to spend some time looking into this in more detail. In particular the new Weld functionality and why it causes problems with hot deployment via NetBeans and not on a full redeploy.

ratcashdev commented 9 years ago

https://issues.jboss.org/browse/WELD-1887 is interesting. It says it can be disabled somehow.

smillidge commented 9 years ago

I think this is related to the fact that I believe NetBeans uses keepState when using "deploy on save" which means it will attempt to retain the session state by serializing and deserializing the session objects. Can you try setting an explicit serialVersionUID on any classes you put into the Session.

Also the property you refer to is org.jboss.weld.serialization.beanIdentifierIndexOptimization can you try setting that as a System property to false on the server NetBeans is deploying to. Although, at this stage, I don't know if setting a system property will be used by Weld.

ratcashdev commented 9 years ago

created a weld.properties file under /src/main/resources with the content org.jboss.weld.serialization.beanIdentifierIndexOptimization = false

This masked the apparent underlying problems and the exception was not thrown. Setting it to true caused the exception to appear again.

The only code that is session scoped is the PrincipalHolder. Giving him a new serialVersionUID for every deployment also helps avoid the exception.

smillidge commented 9 years ago

OK I think this is the weld bug you refer to

ratcashdev commented 9 years ago

WELD-1887 is marked as resolved in 2.2.10 (and I also tried 2.2.14). I guess in our case it must be something else.

smillidge commented 9 years ago

It's resolved in that they have set a property to switch off the bean index optimisation. Not that they have fixed the problem with hot deploy and sessions. Which fixes the problem in your case as well.

ratcashdev commented 9 years ago

I see now. I also found the clarification in the Weld RefGuide. Pasting the link here for completeness. https://docs.jboss.org/weld/reference/latest/en-US/html/configure.html#_bean_identifier_index_optimization

This means this ticket can be closed as resolved.

smillidge commented 9 years ago

We need to explicitly set the property org.jboss.weld.serialization.beanIdentifierIndexOptimization = false in Payara similar to the Weld Servlet.

ratcashdev commented 9 years ago

To my understanding, this new 'feature' of WELD only affects development installations. Is it actually desirable to disable this optimization in production environments? I have not tested this in a clustered setup...

smillidge commented 9 years ago

This also causes problems on redeploys with keep session state.

smillidge commented 9 years ago

Also causes problems in shared session with Payara Micro