search

IQSS / dataverse

Open source research data repository software
http://dataverse.org
Other
878 stars 485 forks source link

Upgrade Glassfish from 4.1 #2628

Closed pdurbin closed 4 years ago

pdurbin commented 8 years ago

Oracle has announced that GlassFish 4.1.1 is now available!

Of particular interest for people running Dataverse is that Glassfish 4.1.1 contains:

Both of these updates are significant because they are newer than the versions recommended in our Installation Guide.

http://guides.dataverse.org/en/4.2/installation/prerequisites.html recommends Weld 2.2.10.SP1 because it has the bug fix for https://issues.jboss.org/browse/WELD-1907 which we investigated in #1516.

http://guides.dataverse.org/en/4.2/installation/shibboleth.html#apply-grizzly-1787-patch recommends downloading http://guides.dataverse.org/en/4.2/_static/installation/files/issues/2180/grizzly-patch/glassfish-grizzly-extra-all.jar which is a custom jar we created to contain the bug fix in https://java.net/jira/browse/GRIZZLY-1787 as part of #2180.

In addition, the blog post says, "Finally, it should also be mentioned that and in addition to various bug fixes, GlassFish 4.1.1 also includes several security related fixes."

For this issue development should (please add or correct):

pdurbin commented 8 years ago

Just a heads up that the new version of Weld caused problems for @phillipross at least. See https://issues.jboss.org/browse/WELD-1941 and below from https://javabot.evanchooly.com/logs/%23glassfish/2015-11-02

_Tenchi_ | that issue describes the problem, but i dont know if that's the same one
         | that i found before                                                     
_Tenchi_ | so, in the example... he has a bean called JSFBean                      
_Tenchi_ | if it was a jsf managed bean... you'd reference it like this -->        
         | #{jsfBean.someProperty}                                                 
_Tenchi_ | in the past... if it was a cdi bean... you'd reference it the same way  
_Tenchi_ | but, apparently, this did not follow the spec                           
_Tenchi_ | this did not follow the cdi spec                                        
_Tenchi_ | so... they fixed weld to follow the cdi spec                            
_Tenchi_ | so now... you have to reference it like this --> #{jSFBean}             
_Tenchi_ | hehe                                                                    
_Tenchi_ | which looks fugly                                                       
_Tenchi_ | but it follows the spec

This may not affect us but I just wanted to raise awareness of the change in behavior.

pdurbin commented 8 years ago

@sekmiller I gave this to you since you mentioned Glassfish in 8768e05 :)

Also, Netbeans 8.1 is out which bundles Glassfish 4.1.1: https://netbeans.org/downloads/

pdurbin commented 8 years ago

From @scolapasta

"Seems like it just has to do with @Named beans that start with multiple caps, e.g JSFBean.

Steve would you mine doing a quick review and see if we have any of these? I think there would only be a few, if any.

The solution, of course, is to just give the @Named annotation the name we use, in those case.

Let me know if you have any questions."

sekmiller commented 8 years ago

When I installed NetBeans 8.1 and took advantage of the automatic project migration from 8.0.2 the glassfish-web.xml file was corrupted and this line was missing

context-root>/</context-root (add back in the opening and closing arrows)

It needs to be returned to the file and the app redeployed or links in the home page won't work properly.

sekmiller commented 8 years ago

Running setup-all.sh after dropping DB and running the install script I get the following error when roles are supposed to be created:

Setting up admin role <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">GlassFish Server Open Source Edition 4.1.1 - Error report

HTTP Status 500 - Internal Server Error

type Exception report

messageInternal Server Error

descriptionThe server encountered an internal error that prevented it from fulfilling this request.

exception 

javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.persistence.jaxb.BeanValidationHelper

root cause 

org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.persistence.jaxb.BeanValidationHelper

root cause 

java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.persistence.jaxb.BeanValidationHelper

note The full stack traces of the exception and its root causes are available in the GlassFish Server Open Source Edition 4.1.1 logs.

GlassFish Server Open Source Edition 4.1.1

sekmiller commented 8 years ago

Web services failing in 4.1.1 is a known issue:

https://java.net/jira/browse/GLASSFISH-21440

pdurbin commented 8 years ago

"That nasty JERSEY-2888 definitely crept into GlassFish 4.1.1" -- https://java.net/jira/browse/GLASSFISH-21440?focusedCommentId=390331&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-390331

[JERSEY-2888] NoClassDefFoundError when doing bean validation - Java.net JIRA

Update: Huh. Jersey 2.21 is what shipped with Glassfish 4.1.1 according to the Oracle annoucement but JERSEY-2888 says the fix version is 2.19. So why is Glassfish 4.1.1 affected by this bug? It has an even newer version of Jersey.

I just confirmed that Glassfish 4.1.1 has Jersey 2.21:

murphy:bin pdurbin$ pwd
/Applications/NetBeans/glassfish-4.1.1/glassfish/bin
murphy:bin pdurbin$ ./asadmin osgi lb | grep -i jersey
  169|Resolved   |    1|jersey-ext-bean-validation (2.21.0)
  170|Resolved   |    1|jersey-ext-cdi1x-servlet (2.21.0)
  171|Installed  |    1|jersey-ext-cdi1x-transaction (2.21.0)
  172|Resolved   |    1|jersey-ext-cdi1x (2.21.0)
  173|Resolved   |    1|jersey-core-client (2.21.0)
  174|Resolved   |    1|jersey-core-common (2.21.0)
  175|Resolved   |    1|jersey-container-grizzly2-http (2.21.0)
  176|Installed  |    1|jersey-container-servlet-core (2.21.0)
  177|Installed  |    1|jersey-container-servlet (2.21.0)
  178|Resolved   |    1|jersey-ext-entity-filtering (2.21.0)
  179|Installed  |    1|jersey-gf-ejb (2.21.0)
  180|Resolved   |    1|jersey-repackaged-guava (2.21.0)
  181|Resolved   |    1|jersey-media-jaxb (2.21.0)
  182|Resolved   |    1|jersey-media-json-jackson (2.21.0)
  183|Resolved   |    1|jersey-media-json-jettison (2.21.0)
  184|Resolved   |    1|jersey-media-json-processing (2.21.0)
  185|Installed  |    1|jersey-media-moxy (2.21.0)
  186|Resolved   |    1|jersey-media-multipart (2.21.0)
  187|Resolved   |    1|jersey-media-sse (2.21.0)
  188|Installed  |    1|Jersey MVC TLD connector implementation module (4.1.1)
  189|Installed  |    1|jersey-ext-mvc-jsp (2.21.0)
  190|Installed  |    1|jersey-ext-mvc (2.21.0)
  191|Resolved   |    1|jersey-core-server (2.21.0)
murphy:bin pdurbin$

I just downloaded payara-4.1.1.154.zip and it seems to have Jersey 2.22:

murphy:bin pdurbin$ ./asadmin osgi lb | grep -i jersey
  170|Resolved   |    1|jersey-ext-bean-validation (2.22.0)
  171|Resolved   |    1|jersey-ext-cdi1x-servlet (2.22.0)
  172|Installed  |    1|jersey-ext-cdi1x-transaction (2.22.0)
  173|Resolved   |    1|jersey-ext-cdi1x (2.22.0)
  174|Resolved   |    1|jersey-core-client (2.22.0)
  175|Resolved   |    1|jersey-core-common (2.22.0)
  176|Resolved   |    1|jersey-container-grizzly2-http (2.22.0)
  177|Installed  |    1|jersey-container-servlet-core (2.22.0)
  178|Installed  |    1|jersey-container-servlet (2.22.0)
  179|Resolved   |    1|jersey-ext-entity-filtering (2.22.0)
  180|Installed  |    1|jersey-gf-ejb (2.22.0)
  181|Resolved   |    1|jersey-repackaged-guava (2.22.0)
  182|Resolved   |    1|jersey-media-jaxb (2.22.0)
  183|Resolved   |    1|jersey-media-json-jackson (2.22.0)
  184|Resolved   |    1|jersey-media-json-jettison (2.22.0)
  185|Resolved   |    1|jersey-media-json-processing (2.22.0)
  186|Installed  |    1|jersey-media-moxy (2.22.0)
  187|Resolved   |    1|jersey-media-multipart (2.22.0)
  188|Resolved   |    1|jersey-media-sse (2.22.0)
  189|Installed  |    1|Jersey MVC TLD connector implementation module (4.1.1.154)
  190|Installed  |    1|jersey-ext-mvc-jsp (2.22.0)
  191|Installed  |    1|jersey-ext-mvc (2.22.0)
  192|Resolved   |    1|jersey-core-server (2.22.0)
pdurbin commented 8 years ago

@sekmiller thanks for opening https://github.com/payara/Payara/issues/529

I left a comment there and linked to the chat in #glassfish on freenode.

pdurbin commented 8 years ago

In 1c0c7be I pushed a branch called 2628-payara in which I switched the Vagrant environment from Glassfish 4.1 to Payara 4.1.1.154 (the latest at http://payara.co/downloads ). However, after vagrant up completes I get a "500 Internal Server Error" when I go to the home page at http://localhost:8888 per the screenshot below:

500_internal_server_error_-_root_dataverse_-_2015-11-19_09 26 22

The Payara server.log shows A system exception occurred during an invocation on EJB PermissionServiceBean, method: public edu.harvard.iq.dataverse.PermissionServiceBean$RequestPermissionQuery edu.harvard.iq.dataverse.PermissionServiceBean.on(edu.harvard.iq.dataverse.DvObject)]] ... javax.ejb.EJBException... at edu.harvard.iq.dataverse.DataversePage.init(DataversePage.java:294)... Caused by: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke protected void edu.harvard.iq.dataverse.DataverseRequestServiceBean.setup() on edu.harvard.iq.dataverse.DataverseRequestServiceBean@53084b6a ... at edu.harvard.iq.dataverse.PermissionServiceBean.on(PermissionServiceBean.java:356) ... Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98) ... 99 more Caused by: java.lang.IllegalStateException: Not inside a request scope. at jersey.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:173)

I should note that it is possible to go to the Login Page at http://localhost:8888/loginpage.xhtml but I'm not able to log in.

Here's the full server.log including the entire setup of Dataverse and the exceptions above when attempting to go to the homepage and a couple attempts to log in with dataverseAdmin/admin: server.log.txt

pdurbin commented 8 years ago

As discussed in a meeting this morning the exceptions I wrote about in https://github.com/IQSS/dataverse/issues/2628#issuecomment-158077013 are a show stopper for us moving to Payara. It is duly noted that @kcondon did get Dataverse working on a previous version of Payara six months ago or so. I think was when we were working on #2180.

I mentioned to @sekmiller that I could attempt to replicate his test to confirm that I'm also seeing the bean validation exception when I run setup-all of Dataverse on Glassfish 4.1.1.

Otherwise, I guess we'll wait for Glassfish 4.1.2 or Glassfish 4.2 or Glassfish 5! I think what we're waiting for is for someone from Oracle to fix the bean validation bug at https://java.net/jira/browse/GLASSFISH-21440 .

pdurbin commented 8 years ago

Glassfish 4.1.1 does seem to break bean validation, as originally reported by @sekmiller at https://github.com/IQSS/dataverse/issues/2628#issuecomment-156489052

I just confirmed this in Vagrant in e992ad7 (a new branch called 2628-glassfish-4.1.1).

Apparently, the first time bean validation is exercised by our setup scripts is when we start adding roles at https://github.com/IQSS/dataverse/blob/e992ad784b4a6f7ec5aa33d4239f1d976ecfcf65/scripts/api/setup-builtin-roles.sh

Here's what you see in Vagrant:

==> standalone: Setting up admin role
==> standalone: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>GlassFish Server Open Source Edition  4.1.1  - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Internal Server Error</h1><hr/><p><b>type</b> Exception report</p><p><b>message</b>Internal Server Error</p><p><b>description</b>The server encountered an internal error that prevented it from fulfilling this request.</p><p><b>exception</b> <pre>javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler</pre></p><p><b>root cause</b> <pre>org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler</pre></p><p><b>root cause</b> <pre>java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler</pre></p><p><b>root cause</b> <pre>java.lang.ClassNotFoundException: org.xml.sax.helpers.DefaultHandler not found by org.eclipse.persistence.moxy [72]</pre></p><p><b>note</b> <u>The full stack traces of the exception and its root causes are available in the GlassFish Server Open Source Edition  4.1.1  logs.</u></p><hr/><h3>GlassFish Server Open Source Edition  4.1.1 </h3></body></html>
==> standalone: 
==> standalone: Setting up file downloader role

In server.log I'm seeing the same BeanValidationHelper exceptions (look around line 1189 and below): glassfish-4.1.1-server.log.txt

Perhaps we are suffering from https://java.net/jira/browse/GLASSFISH-21440

Whatever it is, we're blocked from upgrading to Glassfish 4.1.1 until this issue is resolved. We can't run our setup scripts.

pdurbin commented 8 years ago

@smillidge from Payara asked me to open an issue if I'm seeing a difference between Glassfish 4.1.1 and Payara 4.1.1.154 and I definitely am. I opened https://github.com/payara/Payara/issues/532

To sum up:

This is all quite reproducible in Vagrant.

pdurbin commented 8 years ago

I just chatted with @edburns at https://javabot.evanchooly.com/logs/%23glassfish/2015-12-04 and left a comment at https://java.net/jira/browse/GLASSFISH-21440?focusedCommentId=390816&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-390816 linking back to this issue. They seem well aware of the problem which is something about how moxy/Eclipselink needs to be upgraded to 2.6.1. I think this is the commit (in Payara's mirror of upstream Glassfish commits) where they fixed it: https://github.com/payara/Payara/commit/e054f32b3ec62563abf009f2e60aaa8f39894ae5

pdurbin commented 8 years ago

I just wanted to note that five days ago there was some activity from Payara at https://github.com/payara/Payara/issues/532#issuecomment-172173372 in trying to reproduce the problem.

pdurbin commented 8 years ago

I just wanted to mention that in my pull request at #2895 I recommend that people installing Dataverse avoid using Glassfish 4.1.1. That change can be previewed at http://guides.dataverse.org/en/2884-install-guide-reorg/installation/prerequisites.html#glassfish

kcondon commented 8 years ago

Note noted.

pdurbin commented 8 years ago

There were "many regression bugs in GlassFish 4.1.1" according to @m-reza-rahman at https://java.net/projects/glassfish/lists/users/archive/2016-02/message/0

davidandradeb commented 8 years ago

In this blog exaplained how to fix issue replace an archive. http://ayudasdesarrollo.blogspot.com.co/2016/05/glassfish-441-falla-con-jax-rs-y-json.html

jcjesus commented 7 years ago

After this problem with glasfish 4.1.1, I was downloading glasfish 4.1 version and solve my problem.

pdurbin commented 7 years ago

@jcjesus we did the same. This issue is all about how we tried and failed to upgrade from Glassfish 4.1 to Glassfish 4.1.1. (We also tried to upgrade to 4.1.1.154 per my summary above at https://github.com/IQSS/dataverse/issues/2628#issuecomment-158202002 . See also "Internal defect ID: PAYARA-834" at https://github.com/payara/Payara/issues/532 ) It's sad because on Glassfish 4.1 we make our users patch Weld and Grizzly:

It makes me wonder what the next version of Glassfish will be. 4.1.2? 5.0? Who knows. https://glassfish.java.net/download.html still shows 4.1.1 as the latest and greatest and it's been over a year since it was released.

smillidge commented 7 years ago

@pdurbin we are keen to help. I'm not sure whether the conclusion was the problem was in your code's use of @Context in a non-JAX-RS resource or a bug in Payara. We can work together to work out which if you need to?

pdurbin commented 7 years ago

@smillidge thanks! I spoke with @scolapasta about this today and he says you two chat every year at JavaOne about getting Dataverse running on Payara. ( @kcondon reminded us that he got it working a while back (which I noted at https://github.com/IQSS/dataverse/issues/2628#issuecomment-158129836 ) but that was some time ago.) We've been too busy to think about this but people like @phillipross have asked as recently as yesterday: https://javabot.evanchooly.com/logs/%23glassfish/2016-11-16 😄 . We haven't had time to work with you on a solution. I am genuinely curious if we're hitting a bug in Payara or if we're doing something wrong in our code.

Part of why I brought up Payara to @scolapasta today is that he's interested in an update to Eclipselink for something related to Java 8. I'm sure we're running whatever version of Eclipselink ships with Glassfish 4.1 and for now we may well just add it as a third patch we apply along with Weld and Grizzly. I'd love to get away from all this patching just have our customers install a "stock" version of Glassfish or Payara. Or Wildfly or TomEE, I guess, but I'm not sure how compatible the Dataverse code is with these.

Bottom line is that we're surviving on Glassfish 4.1 for now. 😄

pdurbin commented 7 years ago

Heads up that Glassfish 4.1.2 is out. We could skip 4.1.1, given all the problems above, and attempt upgrading directly from 4.1 to 4.1.2: https://blogs.oracle.com/theaquarium/entry/glassfish_4_1_2_released

pdurbin commented 7 years ago

I just tried upgrading from Glassfish 4.1 to 4.1.2 in https://github.com/IQSS/dataverse/commit/da4af510bcd7ede1427e4378321e6c664b361ba6 but we have the same problem that we had with 4.1.1, I believe. I'm attaching Glassfish's server.log and the log file that the Dataverse installer generates:

Here's the main problem:

[2017-04-05T05:55:40.973-0700] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.api.ApiBlockingFilter] [tid: _ThreadID=31 _ThreadName=http-listener-1(5)] [timeMillis: 1491396940973] [levelValue: 900] [[
  Error processing /api/v1/admin/roles/: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:485)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:334)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at edu.harvard.iq.dataverse.api.ApiBlockingFilter.doFilter(ApiBlockingFilter.java:162)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575)
    at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546)
    at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378)
    at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:34)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:416)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:283)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler
    at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:256)
    at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:238)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:486)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:292)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1139)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:460)
    ... 51 more
Caused by: java.lang.NoClassDefFoundError: org/xml/sax/helpers/DefaultHandler
    at org.eclipse.persistence.jaxb.BeanValidationHelper.<clinit>(BeanValidationHelper.java:53)
    at org.eclipse.persistence.jaxb.JAXBBeanValidator.isConstrainedObject(JAXBBeanValidator.java:257)
    at org.eclipse.persistence.jaxb.JAXBBeanValidator.shouldValidate(JAXBBeanValidator.java:208)
    at org.eclipse.persistence.jaxb.JAXBUnmarshaller.validateAndBuildJAXBElement(JAXBUnmarshaller.java:235)
    at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:339)
    at org.eclipse.persistence.jaxb.rs.MOXyJsonProvider.readFrom(MOXyJsonProvider.java:660)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:256)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:235)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:155)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:74)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:155)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085)
    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:852)
    at org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:270)
    at org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide(EntityParamValueFactoryProvider.java:96)
    at org.glassfish.jersey.server.spi.internal.ParamValueFactoryWithSource.provide(ParamValueFactoryWithSource.java:71)
    at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:93)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:127)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:309)
    ... 60 more
Caused by: java.lang.ClassNotFoundException: org.xml.sax.helpers.DefaultHandler not found by org.eclipse.persistence.moxy [72]
    at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)
    at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
    at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 84 more
]]

That "roles" API endpoint is called like this: curl -H "Content-type:application/json" -d @data/role-admin.json http://localhost:8080/api/admin/roles/ from https://github.com/IQSS/dataverse/blob/2628-glassfish-4.1.2/scripts/api/setup-builtin-roles.sh#L5

The "roles" API endpoint takes a "RoleDTO" object at https://github.com/IQSS/dataverse/blob/2628-glassfish-4.1.2/src/main/java/edu/harvard/iq/dataverse/api/Admin.java#L693

Update: some theories from "_Tenchi_" on this stacktrace: https://javabot.evanchooly.com/logs/%23glassfish/2017-04-15

pdurbin commented 7 years ago

The Glassfish team has been releasing promoted builds of Glassfish 5. https://blogs.oracle.com/theaquarium/entry/glassfish_5_update says

The nightly builds will always be a few days ahead of the latest promoted build but we do encourage the community to use promoted builds. So you can help us by testing the latest promoted GF5 builds. If you find any issue, please raise a ticket against the appropriate GlassFish component here. And if you are confident that the issue is located in a specific component (e.g. Jersey) it might be more efficient to raise the issue directly against that particular component.

GlassFish 5 should be released about six weeks after the Java EE 8 Platform specification (JSR 366) is final. As of today, we are targeting specification completion in July, which would result on a release date in the August/September timeframe. The schedule is subject to change but that is our current direction.

At some point we should try Glassfish 5 since we haven't been able to get Glassfish 4.1.1 or 4.1.2 to work.

Additionally, Glassfish moved from JIRA to GitHub Issues so https://java.net/jira/browse/GLASSFISH-21440 is now https://github.com/javaee/glassfish/issues/21440 .

mjrobbins commented 7 years ago

At some point we should try Glassfish 5

I came across your discussion having gone through the same headache myself with Glassfish 4.1.2 - even the various fixes available didn't work for me. I've just tried running with the latest Glassfish 5 (beta 7) instead, and the problem has disappeared completely. I can't vouch for what other bugs might be lurking in the new beta, but this one at least seems to have been dealt with...

pdurbin commented 7 years ago

@mjrobbins cool, that's promising. Thanks. Community member @donsizemore did try to run Dataverse on Glassfish 5.0.0.b05 and reported some difficultly in #3801. Maybe beta 7 would work better.

donsizemore commented 7 years ago

@mjrobbins just curious which OS you're using? I just tried deploying Dataverse 4.6.1 over CentOS 7 again, once with Glassfish 5.0.0.b07 and once with the latest nightly, and hit the same error as with b05. Everything proceeds smoothly until the creation of the root Dataverse:

[2017-05-30T10:36:54.085-0400] [glassfish 5.0] [INFO] [] [edu.harvard.iq.dataverse.api.Dataverses] [tid: _ThreadID=31 _ThreadName=http-listener-1(3)] [timeMillis: 1496155014085] [levelValue: 800] [[
  Creating root dataverse]]

[2017-05-30T10:36:54.360-0400] [glassfish 5.0] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=134 _ThreadName=__ejb-thread-pool1] [timeMillis: 1496155014360] [levelValue: 900] [[
  A system exception occurred during an invocation on EJB DataverseServiceBean, method: public edu.harvard.iq.dataverse.Dataverse edu.harvard.iq.dataverse.DataverseServiceBean.findRootDataverse()]]

[2017-05-30T10:36:54.360-0400] [glassfish 5.0] [WARNING] [] [javax.enterprise.ejb.container] [tid: _ThreadID=134 _ThreadName=__ejb-thread-pool1] [timeMillis: 1496155014360] [levelValue: 900] [[

javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean
        at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:666)

Caused by: javax.persistence.NoResultException: getSingleResult() did not retrieve any entities.
        at org.eclipse.persistence.internal.jpa.QueryImpl.throwNoResultException(QueryImpl.java:976)

[2017-05-30T10:36:54.371-0400] [glassfish 5.0] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=134 _ThreadName=__ejb-thread-pool1] [timeMillis: 1496155014371] [levelValue: 900] [[
  A system exception occurred during an invocation on EJB SolrIndexServiceBean, method: public edu.harvard.iq.dataverse.search.IndexResponse edu.harvard.iq.dataverse.search.SolrIndexServiceBean.indexPermissionsOnSelfAndChildren(edu.harvard.iq.dataverse.DvObject)]]

I ran through a 3rd install with Glassfish 5.0.0.b07 on Ubuntu 16LTS (Postgres-9.5) and hit the same set of errors at the same point in the installation.

scolapasta commented 7 years ago

Don,

It's been a while since I looked at this code, but I will admit to being confused by the debug line that says "Creating root dataverse". I'm pretty sure it's created by the set up script and not the code.

So if you haven't run the set up script, that could explain the NoResultFound Exception.

(I'm both excited and nervous about switching to Glassfish 5!)

On Tue, May 30, 2017 at 10:47 AM, Don Sizemore notifications@github.com wrote:

@mjrobbins https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mjrobbins&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=AGECPmLNbmpCr7DYHgIZoWJ1G_80pFpAAYHEv-8e91c&s=UNqbiowEpTw7HoDFhnuiYIxJYZVpVXgMLQPQOSnrASI&e= just curious which OS you're using? I just tried deploying Dataverse 4.6.1 over CentOS 7 again, once with Glassfish 5.0.0.b07 and once with the latest nightly, and hit the same error as with b05. Everything proceeds smoothly until the creation of the root Dataverse:

[2017-05-30T10:36:54.085-0400] [glassfish 5.0] [INFO] [] [edu.harvard.iq.dataverse.api.Dataverses] [tid: _ThreadID=31 _ThreadName=http-listener-1(3)] [timeMillis: 1496155014085] [levelValue: 800] [[ Creating root dataverse]]

[2017-05-30T10:36:54.360-0400] [glassfish 5.0] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=134 _ThreadName=__ejb-thread-pool1] [timeMillis: 1496155014360] [levelValue: 900] [[ A system exception occurred during an invocation on EJB DataverseServiceBean, method: public edu.harvard.iq.dataverse.Dataverse edu.harvard.iq.dataverse.DataverseServiceBean.findRootDataverse()]]

[2017-05-30T10:36:54.360-0400] [glassfish 5.0] [WARNING] [] [javax.enterprise.ejb.container] [tid: _ThreadID=134 _ThreadName=__ejb-thread-pool1] [timeMillis: 1496155014360] [levelValue: 900] [[

javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:666)

Caused by: javax.persistence.NoResultException: getSingleResult() did not retrieve any entities. at org.eclipse.persistence.internal.jpa.QueryImpl.throwNoResultException(QueryImpl.java:976)

[2017-05-30T10:36:54.371-0400] [glassfish 5.0] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=134 _ThreadName=__ejb-thread-pool1] [timeMillis: 1496155014371] [levelValue: 900] [[ A system exception occurred during an invocation on EJB SolrIndexServiceBean, method: public edu.harvard.iq.dataverse.search.IndexResponse edu.harvard.iq.dataverse.search.SolrIndexServiceBean.indexPermissionsOnSelfAndChildren(edu.harvard.iq.dataverse.DvObject)]]

Granted, I'm typing on a Monday morning and haven't yet finished my second cup of coffee.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_issues_2628-23issuecomment-2D304901877&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=AGECPmLNbmpCr7DYHgIZoWJ1G_80pFpAAYHEv-8e91c&s=sCi3_-KYhXcGCKpjCICWlIueezNntEx7Pyp3ypMnAh4&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AEEhBMNwwovBGtoNJQxnnUZcReiQVIetks5r-5FCvqgaJpZM4GKub1&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=AGECPmLNbmpCr7DYHgIZoWJ1G_80pFpAAYHEv-8e91c&s=o4jESoyKcJzbSP2omuYsVyo9vGiTIQrx4EyrFNH-iH0&e= .

donsizemore commented 7 years ago

@scolapasta i'm starting from scratch each time, running the "install" perl script included in dvinstall.zip. have any log files you could want but the only thing i've been able to find is the above.

pdurbin commented 7 years ago

@donsizemore I appreciate you taking a look at this. Let's troubleshoot your adventures with Glassfish 5 in #3801 and via chat as time allows. Thanks!

pdurbin commented 7 years ago

This hasn't been a priority for us and this issue is quite long. Let's open a fresh one when we decide to tackle upgrading from Glassfish 4.1 to something newer.

pdurbin commented 7 years ago

"Just a note that I seemingly successfully deployed the Dataverse 4.7 release warfile in Payara 4.1.2.172, on Ubuntu 16 (openjdk 1.8.0_131). Full server log attached in case it's helpful." -- @donsizemore at https://github.com/payara/Payara/issues/532#issuecomment-315408770

pdurbin commented 6 years ago

Over at pull request #4244 I got Dataverse running on Glassfish 5. There are a couple changes I had to make that I don't full understand ( 6f72baa and 1530273 ) but the app basically works, I think! 😄

pdurbin commented 5 years ago

Today at tech hours we discussed upgrading from Glassfish 4.1 and this issue seems to be the most appropriate one to summarize my thoughts in so I'm re-opening it.

First, I think we should should strongly consider an app server that supports both Java EE and MicroProfile. There are two main reasons for this:

releases

There are three app servers that I know of that support both Java EE and MicroProfile:

I don't know enough about TomEE to know if it supports the "full" Java EE profile that I think Dataverse requires.

There are other app servers that aren't open source that I'm not particularly interested in.

Some questions that came up:

I'm probably forgetting a lot of things we talked about and whether you were at the meeting or simply have an opinion on any of the above, I'd be happy to read any feedback! Please just leave a comment. :smile:

smillidge commented 5 years ago

Just a comment here. The Payara team are happy to help if you need assistance migrating to Payara 5. We can also take a look at the war file to see if there's things that can be done to speed up deployment times etc. Because Dataverse is a large open source application with a diverse user base it's potentially a good application for our team to use to drive performance in Payara deployment and understand different usage models e.g. Docker/Kubernetes so we also get useful experience from assisting.

4tikhonov commented 5 years ago

Thanks, @pdurbin and @smillidge! I think this task should get the highest possible priority if we want to scale Dataverse up in Kubernetes.

pdurbin commented 5 years ago

@smillidge thanks!! Yes, Dataverse is a large application. And you're right about the diversity. If you look at the map at https://dataverse.org you'll see that there are now 43 installations of Dataverse around the world. I can't speak for the whole team but I think we'd be very happy to work with you! Oh, and thanks again for giving me a Payara t-shirt at JavaOne a few years ago. 😄

I see @4tikhonov commented after you, within minutes. 😄 He and @poikilotherm are way ahead of the core Dataverse development team in terms of efforts toward running Dataverse on Docker and Kubernetes. From what I understand, they have it working but being stuck on old Glassfish is holding them back somewhat.

Oh, one thing I forgot to comment about here that I mentioned during tech hours is that @donsizemore recently parameterized the URL from which to download the Glassfish zip file in his Ansible playbook, so it's now even easier to swap in a different zip: https://github.com/IQSS/dataverse-ansible/blob/d1dcc4cd83db536de38bdc1c4ae04abb2741eed5/defaults/main.yml#L73

smillidge commented 5 years ago

As Dataverse is a war file in theory it should also run in Payara Micro which may simplify configuration of a Docker image for Kubernetes. Payara Server and Micro 5 both support clustering via Kubernetes service names if that is required. First step I will get one of the Payara team to see if they can get Dataverse deployed on Payara 5 following the work here https://github.com/IQSS/dataverse/issues/4172.

Tech Preview of JDK 11 support will ship imminently as well.

poikilotherm commented 5 years ago

Thanks to @pdurbin for getting the pieces together and reviving the discussion here (I really :heart: that diagram!). Thanks to @smillidge for picking up the baton.

@smillidge please consider taking a look on my notes in my epic #5292 and related sub-stories. @smillidge: a little addendum: you might take a look at this, too

I invested some time in payara/docker-payaraserver-full#61 to make it easier to deploy Dataverse to Payara.

My plan is to rebase my K8s image on upstream Payara image, once Dataverse deploys successfully into usable state (see problems here) and Payara 5.192 is ready (see payara/docker-payaraserver-full#88 and payara/docker-payaraserver-full#89).

smillidge commented 5 years ago

So quick update. After patching for https://github.com/IQSS/dataverse/pull/5894 and patching a flyway upgrade script I could deploy the war and login on Payara 5.192 running on JDK 11. Not sure how to tell it actually works.

pdurbin commented 5 years ago

@smillidge wow! Java 11 too! Please stand by. @donsizemore and I are talking at http://irclog.iq.harvard.edu/dataverse/2019-05-30#i_95415

Thanks!!

pdurbin commented 5 years ago

I could deploy the war and login on Payara 5.192 running on JDK 11

@smillidge last night @poikilotherm and I were debating a bit what you meant. Do you mean you were able to log in to Dataverse using the dataverseAdmin account like in the animated gif below I stole from @rschlaefli over at https://github.com/IQSS/dataverse/issues/5846#issuecomment-497469796 ?

58661220-9ba8b980-8327-11e9-8a39-b1f8554f2dea

I'm going to guess that instead you're talking about being able to log into the Payara admin interface (or API). Is that right?

I see you're already commenting on https://github.com/IQSS/dataverse-ansible/issues/43 where I'm trying to spin up Dataverse running on Payara 5. Thank you!!

smillidge commented 5 years ago

I deployed the war. I loaded up the data through the REST api by running the scripts then could login to the Dataverse admin account as shown in the gif above. I could view my profile etc.

However I didn't bother setting up solr so was getting errors from the solr EJB and seeing errors related to search on the screen.

pdurbin commented 5 years ago

@smillidge wow! You got quite far! That's awesome! 🎉 I just opened https://github.com/IQSS/dataverse-ansible/pull/69 and I'm hacking away. Thanks!

pdurbin commented 5 years ago

@smillidge et al. I just left a long comment on https://github.com/IQSS/dataverse/issues/4172#issuecomment-497707104 with how I'm now easily able to deploy Dataverse to Payara 5 but I'm having some issues. I'd like to move the Payara-specific discussion over there if that's agreeable to everyone. Thank you to all for looking into this! I'd love some help looking into the stack traces and other artifacts I uploaded to that issue.

pdurbin commented 5 years ago

I think we should should strongly consider an app server that supports both Java EE and MicroProfile.

In addition to the other benefits of MicroProfile I listed above @smillidge pointed out in https://github.com/IQSS/dataverse/issues/5794#issuecomment-498005151 that MicroProfile automatically gives you support for Swagger/OpenAPI. I've been under the impression that we'd have to annotate each of our API endpoints but he provide a dump of all of them in that comment. Great stuff.

Also, I'd be remiss if I didn't mention that @smillidge has spent at least part of his weekend troubleshooting Dataverse on Payara. His latest updates are in #4172.

pdurbin commented 5 years ago

I've been making small pull requests with the intention of trying to help us get off of Glassfish 4.1 some day. I have not been doing any exhaustive testing. Rather, I'm just doing simple things like attempting to visit the home page of a fresh installation of Dataverse or trying to log in as the dataverseAdmin user for the first time.

Yesterday this pull request I made was merged: prevent JSF validation error on Payara 5 #5933 (this one was @smillidge 's idea so thanks again!).

Then I created this pull request: avoid "paginationStart must be 0 or greater" #5940

poikilotherm commented 5 years ago

I just created #5943, as this smells like more code nibling worms are creeping in. Fear not, dear Dataversistas, we'll squash 'em all.

pdurbin commented 5 years ago

Yesterday this pull request I made was merged: prevent JSF validation error on Payara 5 #5933 (this one was @smillidge 's idea so thanks again!).

Unfortunately we had to revert pull request #5933 with pull request #5944 because we were unable to create dataverses anymore. 😢