IQSS / dataverse

Open source research data repository software
http://dataverse.org
Other
877 stars 484 forks source link

Publish Dataset - Fails when metadata contains HTML entities w/special characters such as   #3328

Closed rmo-cdsp closed 3 years ago

rmo-cdsp commented 8 years ago

I added datasets in a Dataverse using Datacite for DOI's. Those datasets were added with python scripts, using the dataverse python api to first create a "simple" (only the required elements) dataset, then update its metadatas with a json (made by extracting datas from xlsx documents).

The problem here seems to come when I try to publish the dataset. The publish fails and there is the error message : "Error – This dataset may not be published because the DataCite Service is currently inaccessible. Please try again. Does the issue continue to persist? Please contact Dataverse Support for assistance. "

Logs are here :dataverse_event_published_error_log.txt

We can see a message : [xml] xml error: The entity "nbsp" was referenced, but not declared.] So I tried to use the datacite api to try with "custom" xml files, and there was something: if I had a "&nbsp" element, I receive the same error. When I remove it, everything is fine. I tried by adding an entity element (<!DOCTYPE space[ <!ENTITY nbsp " "> ]>), (it's a &# and 160 between the "") and the nbsp was replaced and everything worked like a charm. Except that it's by using the api. The problem here is that dataverse doesn't handle nbsp thing (coming from my imports maybe).

Here is a json example asked for doing test on a test server using datacite : json.txt

rmo-cdsp commented 8 years ago

By looking more precisely to my datas, some utf-8 "No-Break Space", aka \xa0, aka & nbsp, were present. So I purged them, and the error disappeared. I had a second error after that, an argument, optionnal for Dataverse, but mandatory for Datacite it seems: Response code: 400, [xml] xml error: cvc-minLength-valid: Value '' with length = '0' is not facet-valid with respect to minLength '1' for type '#AnonType_contributorNamecontributorcontributorsresource'.

Here, the contributorName is not part of the contributor field of a Dataset on Dataverse, but it concerns the dataset contact Name, that is optionnal for a dataset, but create an error if you use Datacite, because it's mandatory but not asked anywhere (except in the error message).

After adding a Name to my datasetContact field, I could finally properly Publish my imported Dataset :)

So to resume, there is 1 major problem, and 1 optionnal problem:

pdurbin commented 8 years ago

See also details at http://irclog.iq.harvard.edu/dataverse/2016-09-02#i_40373

djbrooke commented 7 years ago

We've had another report of the "optional" item from this issue regarding “<em>” tags in the metadata. These can be populated through the Dataverse UI, but they do not play well with Datacite.

pdurbin commented 7 years ago

@djbrooke basic includes: a, b, blockquote, br, cite, code, dd, dl, dt, em according to https://github.com/IQSS/dataverse/blob/v4.5.1/src/main/java/edu/harvard/iq/dataverse/util/MarkupChecker.java#L32 . It sounds like for now we should blacklist em.

pdurbin commented 7 years ago

I believe there's another case of this problem reported here today: https://groups.google.com/d/msg/dataverse-community/v6jgyewGqlI/peTMCoW-DAAJ

pdurbin commented 7 years ago

In pull request #4075 for issue #3845 I worked on the problem of how invalid XML was being sent to DataCite. I'm going to put this is QA so we can at least talk about it at standup.

djbrooke commented 7 years ago

We talked about this in standup. The expectation is that code changes that were already made had fixed this, so we should retest.

kcondon commented 7 years ago

So, to test this I need to:

  1. Configure dataverse to use DataCite
  2. Identify which metadata fields are sent to DataCite Title, Author, Description allow HTML
  3. Identify which html tags are unsupported by DataCite DataCite doesn't like any tags (<br>) or entities (&, &nbsp;) so try each of these.
  4. Create a dataset (via API?) which uses unsupported html tags in fields sent to DataCite
    API not needed. Previous code still renders HTML on dataverse but strips tags, characters from metadata sent to DataCite, allowing publication
pdurbin commented 7 years ago

Yeah, HTML content such as &nbsp; as mentioned above.

kcondon commented 7 years ago

Is there a known list or pattern to unsupported tags or would just trying a known case be enough?

kcondon commented 7 years ago

I see, is this issue essentially a duplicate of https://github.com/IQSS/dataverse/issues/3845?

kcondon commented 7 years ago

OK, still fails: Caused by: java.lang.RuntimeException: Response code: 400, [xml] xml error: The entity "nbsp" was referenced, but not declared.

This works: Test DataCite Metadata again & <br>again</br>

This fails: Test DataCite Metadata again & <br>again</br> &nbsp;

kcondon commented 7 years ago

[2017-09-14T17:19:50.302-0400] [glassfish 4.1] [SEVERE] [] [edu.harvard.iq.dataverse.DataCiteRE STfullClient] [tid: _ThreadID=56 _ThreadName=jk-connector(4)] [timeMillis: 1505423990302] [leve lValue: 1000] [[ Response code: 400, [xml] xml error: The entity "nbsp" was referenced, but not declared.]]

[2017-09-14T17:19:50.304-0400] [glassfish 4.1] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.c ontainer] [tid: _ThreadID=56 _ThreadName=jk-connector(4)] [timeMillis: 1505423990304] [levelVal ue: 900] [[ A system exception occurred during an invocation on EJB DOIDataCiteRegisterService, method: p ublic java.lang.String edu.harvard.iq.dataverse.DOIDataCiteRegisterService.createIdentifier(jav a.lang.String,java.util.HashMap,edu.harvard.iq.dataverse.Dataset) throws java.io.IOException]]

[2017-09-14T17:19:50.304-0400] [glassfish 4.1] [WARNING] [] [javax.enterprise.ejb.container] [t id: _ThreadID=56 _ThreadName=jk-connector(4)] [timeMillis: 1505423990304] [levelValue: 900] [[

javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBCont ainerTransactionManager.java:662) at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTrans actionManager.java:507) at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4566) at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2074) at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2044) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocati onHandler.java:220) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObject InvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy348.createIdentifier(Unknown Source) at edu.harvard.iq.dataverse.EJB31_GeneratedDOIDataCiteRegisterServiceIntf__Bean .createIdentifier(Unknown Source) at edu.harvard.iq.dataverse.DOIDataCiteServiceBean.updateIdentifierStatus(DOIDataCiteSe rviceBean.java:233) at edu.harvard.iq.dataverse.DOIDataCiteServiceBean.publicizeIdentifier(DOIDataCiteServi ceBean.java:223) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4758) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy349.publicizeIdentifier(Unknown Source) at edu.harvard.iq.dataverse.EJB31_GeneratedDOIDataCiteServiceBeanIntf__Bean.publicizeIdentifier(Unknown Source) at edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand.publicizeExternalIdentifier(FinalizeDatasetPublicationCommand.java:166) at edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand.execute(FinalizeDatasetPublicationCommand.java:100) at edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand.execute(FinalizeDatasetPublicationCommand.java:42) at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:214) at edu.harvard.iq.dataverse.EjbDataverseEngine$1$1.submit(EjbDataverseEngine.java:385) at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:86) at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:37) at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4758) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy267.submit(Unknown Source) at edu.harvard.iq.dataverse.__EJB31_GeneratedEjbDataverseEngineIntf__Bean.submit(Unknown Source) at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1812) at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1704) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.el.parser.AstValue.invoke(AstValue.java:289) at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304) at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40) at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50) at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105) at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87) at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102) at javax.faces.component.UICommand.broadcast(UICommand.java:315) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282) at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:295) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:415) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(Wor kerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool .java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.ja va:545) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Response code: 400, [xml] xml error: The entity "nbsp" w as referenced, but not declared. at edu.harvard.iq.dataverse.DataCiteRESTfullClient.postMetadata(DataCiteRESTfullClient. java:183) at edu.harvard.iq.dataverse.DOIDataCiteRegisterService.createIdentifier(DOIDataCiteRegi sterService.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 ) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManag er.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager. java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorMana ger.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(Abstrac tEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:5 2) at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 ) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorMan ager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) ... 133 more ]]

[2017-09-14T17:19:50.307-0400] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.DOIDataCiteServiceBean] [tid: _ThreadID=56 _ThreadName=jk-connector(4)] [timeMillis: 1505423990307] [levelValue: 900] [[ modifyMetadata failed]]

pdurbin commented 7 years ago

Bummer. Thanks for checking @kcondon

The reason this got on my radar is that @amberleahey linked to it from the "Dev Efforts by the Dataverse Community" spreadsheet at https://docs.google.com/spreadsheets/d/1pl9U0_CtWQ3oz6ZllvSHeyB0EG1M_vZEC_aZ7hREnhE/edit?usp=sharing and indicated the @kevinworthington put a fix in the production installation for Scholars Portal. She called it "DataCite metadata validation (strip out HTML encoding)". For #3845 we stripped out HTML tags but we didn't look at entities like &nbsp;.

kcondon commented 7 years ago

So if we didn't look at entities, why did we bother testing it?

pdurbin commented 7 years ago

It's good to know where the bugs are. All Dataverse installations using EZID will need to move to DataCite: http://www.cdlib.org/cdlinfo/2017/08/28/ezid-service-update-august-2017/

kcondon commented 7 years ago

I know but if we did not specifically fix entities why would we expect them to work? This does not require an answer, just something to think about going forward.

pdurbin commented 7 years ago

This issue wasn't on my radar when I worked on #3845. In the next retrospective I'll mention that we need a better way to group together related issues so we can fix and QA them in a single pull request.

kcondon commented 7 years ago

I guess my point isn't that you didn't fix it or that it wasn't organized in such a way, it was that we knew it wasn't fixed but we tested it rather than just acknowledging that it was something we have still to do. That does not make sense to me. That's all. We can continue off line in stand up.

pdurbin commented 6 years ago

@kcondon @djbrooke I attempted to talk about this during retrospective yesterday but I don't think I worded it well. I'll try again in the future. Thanks.

pdurbin commented 6 years ago

Unfortunately @philippconzett is suffering from this bug in production as originally reported at https://groups.google.com/d/msg/dataverse-community/kVktk1nzBG8/4wsIsZlBBgAJ

He's running 4.7.1 but the bug is in the "develop" branch as well. In the stack trace below that Philipp sent via https://help.hmdc.harvard.edu/Ticket/Display.html?id=254989 the RuntimeException is being thrown because Dataverse is sending invalid XML to DataCite. A workaround could be to edit the dataset to remove the &nbsp; and trying to re-publish. See #4228 for an example of how a &nbsp; got into a dataset title.

Here's the code in question that should validate the XML before it's sent to DataCite:

    httpPost.setHeader("Content-Type", "application/xml;charset=UTF-8");
    httpPost.setEntity(new StringEntity(metadata, "utf-8"));
    try {
        HttpResponse response = httpClient.execute(httpPost,context);

        String data = EntityUtils.toString(response.getEntity(), encoding);
        if (response.getStatusLine().getStatusCode() != 201) {
            String errMsg = "Response code: " + response.getStatusLine().getStatusCode() + ", " + data;
            logger.log(Level.SEVERE, errMsg);
            throw new RuntimeException(errMsg);

https://github.com/IQSS/dataverse/blob/5efe63b5a36b00c799cbb02c2549ba11201d40df/src/main/java/edu/harvard/iq/dataverse/DataCiteRESTfullClient.java#L183

Here's the stack trace showing line the RuntimeException from 183 above:

[2017-11-02T19:58:24.209+0100] [glassfish 4.1] [SEVERE] [] [edu.harvard.iq.dataverse.DataCiteRESTfullClient] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104209] [levelValue: 1000] [[
  Response code: 400, [xml] xml error: The entity "nbsp" was referenced, but not declared.]]

[2017-11-02T19:58:24.210+0100] [glassfish 4.1] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104210] [levelValue: 900] [[
  A system exception occurred during an invocation on EJB DOIDataCiteRegisterService, method: public java.lang.String edu.harvard.iq.dataverse.DOIDataCiteRegisterService.createIdentifier(java.lang.String,java.util.HashMap,edu.harvard.iq.dataverse.Dataset) throws java.io.IOException]]

[2017-11-02T19:58:24.210+0100] [glassfish 4.1] [WARNING] [] [javax.enterprise.ejb.container] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104210] [levelValue: 900] [[

javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean
    at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:662)
    at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:507)
    at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4566)
    at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2074)
    at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2044)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy327.createIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__DOIDataCiteRegisterService__Intf____Bean__.createIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.DOIDataCiteServiceBean.updateIdentifierStatus(DOIDataCiteServiceBean.java:231)
    at edu.harvard.iq.dataverse.DOIDataCiteServiceBean.publicizeIdentifier(DOIDataCiteServiceBean.java:221)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy303.publicizeIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__DOIDataCiteServiceBean__Intf____Bean__.publicizeIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:293)
    at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:42)
    at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:211)
    at sun.reflect.GeneratedMethodAccessor989.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy330.submit(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__EjbDataverseEngine__Intf____Bean__.submit(Unknown Source)
    at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1685)
    at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1577)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.el.parser.AstValue.invoke(AstValue.java:289)
    at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304)
    at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
    at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
    at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
    at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87)
    at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
    at javax.faces.component.UICommand.broadcast(UICommand.java:315)
    at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790)
    at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282)
    at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
    at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:295)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:415)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Response code: 400, [xml] xml error: The entity "nbsp" was referenced, but not declared.
    at edu.harvard.iq.dataverse.DataCiteRESTfullClient.postMetadata(DataCiteRESTfullClient.java:183)
    at edu.harvard.iq.dataverse.DOIDataCiteRegisterService.createIdentifier(DOIDataCiteRegisterService.java:89)
    at sun.reflect.GeneratedMethodAccessor7485.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    ... 127 more
]]

[2017-11-02T19:58:24.212+0100] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.DOIDataCiteServiceBean] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104212] [levelValue: 900] [[
  modifyMetadata failed]]

[2017-11-02T19:58:24.212+0100] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.DOIDataCiteServiceBean] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104212] [levelValue: 900] [[
  String javax.ejb.EJBTransactionRolledbackException]]

[2017-11-02T19:58:24.212+0100] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.DOIDataCiteServiceBean] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104212] [levelValue: 900] [[
  localized message null]]

[2017-11-02T19:58:24.213+0100] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.DOIDataCiteServiceBean] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104213] [levelValue: 900] [[
  cause
javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean
    at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:662)
    at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:507)
    at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4566)
    at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2074)
    at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2044)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy327.createIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__DOIDataCiteRegisterService__Intf____Bean__.createIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.DOIDataCiteServiceBean.updateIdentifierStatus(DOIDataCiteServiceBean.java:231)
    at edu.harvard.iq.dataverse.DOIDataCiteServiceBean.publicizeIdentifier(DOIDataCiteServiceBean.java:221)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy303.publicizeIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__DOIDataCiteServiceBean__Intf____Bean__.publicizeIdentifier(Unknown Source)
    at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:293)
    at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:42)
    at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:211)
    at sun.reflect.GeneratedMethodAccessor989.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy330.submit(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__EjbDataverseEngine__Intf____Bean__.submit(Unknown Source)
    at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1685)
    at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1577)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.el.parser.AstValue.invoke(AstValue.java:289)
    at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304)
    at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
    at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
    at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
    at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87)
    at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
    at javax.faces.component.UICommand.broadcast(UICommand.java:315)
    at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790)
    at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282)
    at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
    at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:295)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:415)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Response code: 400, [xml] xml error: The entity "nbsp" was referenced, but not declared.
    at edu.harvard.iq.dataverse.DataCiteRESTfullClient.postMetadata(DataCiteRESTfullClient.java:183)
    at edu.harvard.iq.dataverse.DOIDataCiteRegisterService.createIdentifier(DOIDataCiteRegisterService.java:89)
    at sun.reflect.GeneratedMethodAccessor7485.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    ... 127 more
]]

[2017-11-02T19:58:24.213+0100] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.DOIDataCiteServiceBean] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104213] [levelValue: 900] [[
  message null]]

[2017-11-02T19:58:24.214+0100] [glassfish 4.1] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104214] [levelValue: 900] [[
  A system exception occurred during an invocation on EJB PermissionServiceBean, method: public java.util.Set edu.harvard.iq.dataverse.PermissionServiceBean.permissionsFor(edu.harvard.iq.dataverse.engine.command.DataverseRequest,edu.harvard.iq.dataverse.DvObject)]]

[2017-11-02T19:58:24.214+0100] [glassfish 4.1] [WARNING] [] [javax.enterprise.ejb.container] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104214] [levelValue: 900] [[

javax.ejb.TransactionRolledbackLocalException: Client's transaction aborted
    at com.sun.ejb.containers.EJBContainerTransactionManager.useClientTx(EJBContainerTransactionManager.java:357)
    at com.sun.ejb.containers.EJBContainerTransactionManager.preInvokeTx(EJBContainerTransactionManager.java:251)
    at com.sun.ejb.containers.BaseContainer.preInvokeTx(BaseContainer.java:4524)
    at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1986)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy287.permissionsFor(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__PermissionServiceBean__Intf____Bean__.permissionsFor(Unknown Source)
    at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:190)
    at edu.harvard.iq.dataverse.EjbDataverseEngine$1$1.submit(EjbDataverseEngine.java:382)
    at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:297)
    at edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand.execute(PublishDatasetCommand.java:42)
    at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:211)
    at sun.reflect.GeneratedMethodAccessor989.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
    at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
    at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy330.submit(Unknown Source)
    at edu.harvard.iq.dataverse.__EJB31_Generated__EjbDataverseEngine__Intf____Bean__.submit(Unknown Source)
    at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1685)
    at edu.harvard.iq.dataverse.DatasetPage.releaseDataset(DatasetPage.java:1577)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.el.parser.AstValue.invoke(AstValue.java:289)
    at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304)
    at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
    at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
    at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
    at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87)
    at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
    at javax.faces.component.UICommand.broadcast(UICommand.java:315)
    at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790)
    at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282)
    at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
    at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:295)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:415)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
    at java.lang.Thread.run(Thread.java:748)
]]

[2017-11-02T19:58:24.241+0100] [glassfish 4.1] [SEVERE] [] [edu.harvard.iq.dataverse.DatasetPage] [tid: _ThreadID=50 _ThreadName=jk-connector(5)] [timeMillis: 1509649104241] [levelValue: 1000] [[
  Command edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand@2e107f1d failed: null]]
pdurbin commented 6 years ago

See the note by @philippconzett at https://groups.google.com/d/msg/dataverse-community/kVktk1nzBG8/bhFkuO9cBwAJ for his workaround:

Finally, I figured out what caused the error message: The author had used [ and ] in the description field, rendering the following html code:

[article abstract] Phonological variation ...

I simply used the source code of the dataset page to search for "nbsp" :)

pdurbin commented 6 years ago

I just re-tested this bug on the phoenix server as of 8cfca31 now that pull request #4807 has been merged for issue #4819 and I'm happy to report that while the end user doesn't get a specific error. Instead of some weird error about "FinalizeDatasetPublicationCommand" and "null", the user gets a more appropriate error saying "This dataset may not be published due to an error when contacting the DataCite Service" like this:

screen shot 2018-07-26 at 7 39 15 pm

Somewhat oddly, the error in Glassfish's server.log changed. I was expecting to see something like Response code: 400, [xml] xml error: The entity "nbsp" was referenced, but not declared as I had seen before (and as shown above) but this time it said Response code: 400, xmlns namespace not found. Weird.

djbrooke commented 5 years ago

Hi @mfenner! We discussed this in our backlog grooming today and we're looking at sanitizing on Dataset create/edit or just on output to Datacite. Any opinions on which approach we should take?

mheppler commented 5 years ago

@pdurbin I saw that you retested this last July. I gave it a whirl locally after #4077 was merged and got a successful publish with ampersand &#38; &amp; & nonbreaking space &#160; &nbsp; for the metadata.

Screen Shot 2019-08-12 at 1 46 04 PM

pdurbin commented 5 years ago

@mheppler cool but are you using the "FAKE" DOI provider locally? We need to test with real DataCite.

mheppler commented 5 years ago

Ah, yes, I am set up locally with the FAKE DOI. Apologies for missing that.

4tikhonov commented 4 years ago

We would like to have a line break in the Description field but it seems it's removed on the landing page. However, \n is available in the metadata editor:

Screen Shot 2019-12-13 at 3 33 41 PM

Screen Shot 2019-12-13 at 3 33 57 PM

The final result should look like on the screenshot, with line breaks:

screen @pdurbin @djbrooke

mfenner commented 4 years ago

FWIW, DataCite supports only a few HTML tags in the DOI description field, but that includes </br>. The complete list of tags is strong em b i code pre sub sup br.

pdurbin commented 4 years ago

@mfenner thanks!

@4tikhonov you'll need to add the </br> tags manually as part of your migration process. The description field supports HTML and HTML ignores whitespace, generally.

4tikhonov commented 4 years ago

@pdurbin, it makes no sense to add
tags to metadata, it can bring unexpected behavior to the machine-to-machine interaction. At the moment we've modified Dataverse application on frontend but would like to get more sustainable solution.

pdurbin commented 4 years ago

@4tikhonov ok, what would be a more sustainable solution? Markdown support?

4tikhonov commented 4 years ago

@pdurbin, probably it would be better to replace '\n' with <br /> in metadata when rendering on the dataset landing page.

mfenner commented 4 years ago

The subset that we picked (strong em b i code pre sub sup br) is what is typically used in forums and other places that allow "unsafe" input. We are of course maybe too careful, as we also don't support for example links or lists. But I think this is what you typically see with indexing services.

Something that we "unofficially" support is Latex, which for example the folks at CERN make heavy use of and that we support via Mathjax (but that is tricky).

scolapasta commented 4 years ago

@4tikhonov one challenge with that is for those that have used <br/> you will get additional line breaks that may not have been intended. For machine to machine interaction, I think we may strip out the html, or if we don't maybe we should. (we do strip out, for example, when we store the values in solr)

djbrooke commented 3 years ago

Hey @qqmyers - you mentioned you may have a fix for this issue that you deployed on QDR.

qqmyers commented 3 years ago

@djbrooke, @jmjamison - not sure if anything is needed. I just repeated putting odd chars in the description as mheppler did in Aug 2019 (above) and it publishes OK: 'https://demo.dataverse.org/dataset.xhtml?persistentId=doi%3A10.70122%2FFK2%2F0LD1LQ&version=1.1

Does that mean this issue is fixed? Or is there a better test case that demonstrates the problem? The actual content I entered is 'this & that' and then ampersand &#38; &amp; & nonbreaking space &#160; &nbsp; which is still visible when I edit the metadata.

mheppler commented 3 years ago

Ah, yes, I am set up locally with the FAKE DOI. Apologies for missing that.

Is demo configured with a fake DOI provider? That was what I missed if you are referencing my test on my local dev environment above.

Just putting these here...

qqmyers commented 3 years ago

I'm assuming the 10.70122 is a real DOI of some sort (on the test DataCite server?), but I don't know.

mheppler commented 3 years ago

@qqmyers I believe that the rest of the DOI "10.70122/FK2/0LD1LQ" with the "FK2" shoulder is the "fake" provider part.

qqmyers commented 3 years ago

The FakePIDProvider just returns "fakeIdentifier" (see https://github.com/IQSS/dataverse/blob/54a47cf0c23127caacd01a979e7f040484d57e88/src/main/java/edu/harvard/iq/dataverse/pidproviders/FakePidProviderServiceBean.java#L46-L47) so I'm guessing demo uses the test DataCite server - real-looking DOIs but they aren't resolvable.

mheppler commented 3 years ago

Not only are they not resolvable (DataCite 404 when clicked), but the Developer Guide implies the registration is disabled, which I believe is why these special characters only fail on production installations with "real" DOI providers.

This will disable DOI registration by using a fake (in-code) DOI provider.

qqmyers commented 3 years ago

Can someone verify what demo.dataverse.org is running? While it's true that the FAKE provider would generate 'realistic looking' DOIs and not be a good test, running with the DataCite provider would be a good test and looks similar. (demo.dataverse.org is using the authority 10.70122 which looks more like a DataCite test authority than the obsolete 10.5072 authority I've seen used for the FAKE provider).

Also - this is working at QDR on a dev server using the DataCite provider and their test system so either demo is also using DataCite, this is a valid test, the basic problem is fixed, or demo is on FAKE, not a valid test, and QDR must have some code change that needs to be merged since the problem is fixed there.

kcondon commented 3 years ago

@qqmyers Yes, it is a test domain: https://mds.test.datacite.org

qqmyers commented 3 years ago

@kcondon - thanks. So - I think that means this issue is solved in 5.3. @jmjamison - can you try your specific case on https://demo.dataverse.org and confirm?

Are there any other test cases in the notes above that need to be tested? (I tried & in the description and mheppler's string with more variants, but there could be some other test that is not yet handled.)

Also - adding a note from the QDR dev server - I can see the & at DataCite in the Fabrica interface, so it is definitely being transmitted correctly (and not just getting cleaned out of the metadata before sending, etc.)

qqmyers commented 3 years ago

I can't explain why things are working on demo and not elsewhere (see https://groups.google.com/d/msgid/dataverse-community/cf4032af-e12b-4dfc-ba6d-7804b35e4f40n%40googlegroups.com). That said, I do see that the datacite.xml produced on demo is not correctly escaped, i.e. in my browser I see an error when trying to view https://demo.dataverse.org/api/datasets/export?exporter=Datacite&persistentId=doi%3A10.70122/FK2/0LD1LQ : image

QDR has a change that does the escaping which may also resolve the problem with publishing (at least at QDR, my browser shows the formatted xml when I look at a dataset with '&' > < chars in the description and can publish and retrieve the XML from the test DataCite server (via Fabrica) to see that the description is there with escaping, eg. <description descriptionType="Abstract">** asterisks indicates p &lt; 0.05 * asterisk indicates 0.10 &gt; p ≤ 0.05 no asterisk indicates p ≥ 0.10</description> for the example in the email thread.

PR to follow.

qqmyers commented 3 years ago

More investigation: It looks like DataCite is silently removing unescaped chars rather than failing during publish. @jmjamison - guessing that you use EZID (which may not be doing that and just failing instead)?

For DataCite, the PR above, which escapes the description, results in DataCite (i.e. as seen in Fabrica) having the escaped chars and the Dataverse DataCite.xml export being valid xml.

For EZID, we'll need a similar change in AbstractGlobalIdServiceBean.getMetadataFromDvObject() - shouldn't hurt anything regardless but would be to confirm that the publication failure is with that service first.

Another fun note for DataCite: If you put an & in the title (now or after we add escaping to the description), DataCite decides to remove the unescaped and escaped versions of the characters from the whole xml it is sent, so even the properly escaped & chars in a description disappear if DataCite sees an unescaped & in the title. So - we may want to do more work to make sure all fields, even ones that don't normally get special chars in them to avoid odd effects if/when someone does enter them in some other field.

qqmyers commented 3 years ago

Confirmed with @jmjamison that they use EZID, so the mysteries are solved. The PR doesn't yet have the same fix in the EZID code, but should be straight-forward to add. Not a huge priority as removing the offending chars is a work-around.