vivo-project / VIVO

VIVO is an extensible semantic web application for research discovery and showcasing scholarly work
http://vivoweb.org
BSD 3-Clause "New" or "Revised" License
208 stars 131 forks source link

VIVO-1615: Linked Data Fragment server doesn't work with TDB #3205

Closed chenejac closed 3 years ago

chenejac commented 6 years ago

Benjamin Gross (Migrated from VIVO-1615) said:

The Linked Data Fragment/TPF server doesn't appear to work when running VIVO on a TDB database. It works with SDB.

The TPF server should show results if you go to http://localhost:8080/tpf/core (e.g. https://connect.unavco.org/tpf/core)

When running TDB, I get the following error:

HTTP Status 500 – Internal Server Error
Type Exception Report

Message freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

javax.servlet.ServletException: freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:
==> (title!header)  [in template "base.ftl.html" at line 7, column 44]

----
Tip: If the failing expression is known to be legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing</#if>. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??
----

----
FTL stack trace ("~" means nesting-related):
    - Failed at: ${(title!header)?ensure_starts_with("...  [in template "base.ftl.html" in macro "display_page" at line 7, column 41]
    - Reached through: @display_page  [in template "error.ftl.html" at line 11, column 1]
----
    org.vivoweb.linkeddatafragments.servlet.VitroLinkedDataFragmentServlet.doGet(VitroLinkedDataFragmentServlet.java:204)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    edu.cornell.mannlib.vitro.webapp.controller.VitroHttpServlet.service(VitroHttpServlet.java:71)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1232)
    edu.cornell.mannlib.vitro.webapp.filters.PageRoutingFilter.doNonDisplayPage(PageRoutingFilter.java:111)
    edu.cornell.mannlib.vitro.webapp.filters.PageRoutingFilter.doFilter(PageRoutingFilter.java:102)
    edu.cornell.mannlib.vitro.webapp.filters.ClickjackFilter.doFilter(ClickjackFilter.java:34)
    edu.cornell.mannlib.vitro.webapp.filters.RequestModelsPrep.doFilter(RequestModelsPrep.java:59)
    edu.cornell.mannlib.vitro.webapp.filters.URLRewriterFilter.doFilter(URLRewriterFilter.java:40)
    edu.cornell.mannlib.vitro.webapp.filters.JSessionStripFilter.doFilter(JSessionStripFilter.java:63)
    edu.cornell.mannlib.vitro.webapp.filters.CachingResponseFilter.produceBasicResponse(CachingResponseFilter.java:321)
    edu.cornell.mannlib.vitro.webapp.filters.CachingResponseFilter.doFilter(CachingResponseFilter.java:120)
    edu.cornell.mannlib.vitro.webapp.i18n.selection.LocaleSelectionFilter.doFilter(LocaleSelectionFilter.java:64)
    edu.cornell.mannlib.vitro.webapp.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:36)
    edu.cornell.mannlib.vitro.webapp.filters.SessionTimeoutLimitingFilter.doFilter(SessionTimeoutLimitingFilter.java:41)
    edu.cornell.mannlib.vitro.webapp.filters.StartupStatusDisplayFilter.doFilter(StartupStatusDisplayFilter.java:66)
Root Cause

freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:
==> (title!header)  [in template "base.ftl.html" at line 7, column 44]

----
Tip: If the failing expression is known to be legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing</#if>. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??
----

----
FTL stack trace ("~" means nesting-related):
    - Failed at: ${(title!header)?ensure_starts_with("...  [in template "base.ftl.html" in macro "display_page" at line 7, column 41]
    - Reached through: @display_page  [in template "error.ftl.html" at line 11, column 1]
----
    freemarker.core.InvalidReferenceException.getInstance(InvalidReferenceException.java:131)
    freemarker.core.EvalUtil.coerceModelToString(EvalUtil.java:355)
    freemarker.core.Expression.evalAndCoerceToString(Expression.java:82)
    freemarker.core.BuiltInForString._eval(BuiltInForString.java:26)
    freemarker.core.Expression.eval(Expression.java:78)
    freemarker.core.MethodCall._eval(MethodCall.java:55)
    freemarker.core.Expression.eval(Expression.java:78)
    freemarker.core.Expression.evalAndCoerceToString(Expression.java:82)
    freemarker.core.BuiltInForString._eval(BuiltInForString.java:26)
    freemarker.core.Expression.eval(Expression.java:78)
    freemarker.core.MethodCall._eval(MethodCall.java:55)
    freemarker.core.Expression.eval(Expression.java:78)
    freemarker.core.Expression.evalAndCoerceToString(Expression.java:82)
    freemarker.core.DollarVariable.accept(DollarVariable.java:41)
    freemarker.core.Environment.visit(Environment.java:324)
    freemarker.core.MixedContent.accept(MixedContent.java:54)
    freemarker.core.Environment.visit(Environment.java:324)
    freemarker.core.Macro$Context.runMacro(Macro.java:184)
    freemarker.core.Environment.invoke(Environment.java:701)
    freemarker.core.UnifiedCall.accept(UnifiedCall.java:84)
    freemarker.core.Environment.visit(Environment.java:324)
    freemarker.core.MixedContent.accept(MixedContent.java:54)
    freemarker.core.Environment.visit(Environment.java:324)
    freemarker.core.Environment.process(Environment.java:302)
    freemarker.template.Template.process(Template.java:325)
    org.vivoweb.linkeddatafragments.views.HtmlTriplePatternFragmentWriterImpl.writeError(HtmlTriplePatternFragmentWriterImpl.java:219)
    org.vivoweb.linkeddatafragments.servlet.VitroLinkedDataFragmentServlet.doGet(VitroLinkedDataFragmentServlet.java:200)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    edu.cornell.mannlib.vitro.webapp.controller.VitroHttpServlet.service(VitroHttpServlet.java:71)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1232)
    edu.cornell.mannlib.vitro.webapp.filters.PageRoutingFilter.doNonDisplayPage(PageRoutingFilter.java:111)
    edu.cornell.mannlib.vitro.webapp.filters.PageRoutingFilter.doFilter(PageRoutingFilter.java:102)
    edu.cornell.mannlib.vitro.webapp.filters.ClickjackFilter.doFilter(ClickjackFilter.java:34)
    edu.cornell.mannlib.vitro.webapp.filters.RequestModelsPrep.doFilter(RequestModelsPrep.java:59)
    edu.cornell.mannlib.vitro.webapp.filters.URLRewriterFilter.doFilter(URLRewriterFilter.java:40)
    edu.cornell.mannlib.vitro.webapp.filters.JSessionStripFilter.doFilter(JSessionStripFilter.java:63)
    edu.cornell.mannlib.vitro.webapp.filters.CachingResponseFilter.produceBasicResponse(CachingResponseFilter.java:321)
    edu.cornell.mannlib.vitro.webapp.filters.CachingResponseFilter.doFilter(CachingResponseFilter.java:120)
    edu.cornell.mannlib.vitro.webapp.i18n.selection.LocaleSelectionFilter.doFilter(LocaleSelectionFilter.java:64)
    edu.cornell.mannlib.vitro.webapp.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:36)
    edu.cornell.mannlib.vitro.webapp.filters.SessionTimeoutLimitingFilter.doFilter(SessionTimeoutLimitingFilter.java:41)
    edu.cornell.mannlib.vitro.webapp.filters.StartupStatusDisplayFilter.doFilter(StartupStatusDisplayFilter.java:66)
Note The full stack trace of the root cause is available in the server logs.

Apache Tomcat/8.5.34

I think the Freemarker error being thrown is obscuring the actual problem.

chenejac commented 6 years ago

Graham Triggs said:

The Freemarker error appears to be a consequence of it going through the error handling routines - which itself is a result of the VitroLinkedDataFragmentServlet throwing an exception.

The exception from the servlet code should be in the vivo log.

 

Note that the standard implementation of LDF is simply executing construct queries. For Vitro, this has been wrapped in the RDFService, and so we run construct queries against the triple store.

The only difference with SDB is that if you are doing an empty query (no subject, predicate or object), and the code knows the database it is running against, it uses a custom implementation to query SQL database directly to retrieve the required rows, because that is much faster than the way construct queries get translated by SDB.

Otherwise, SDB and TDB should be doing exactly the same thing in querying the store with constructs.

chenejac commented 6 years ago

Benjamin Gross said:

Thanks for the comments. I am not seeing any errors other than the Freemarker error in vivo.all.log. I tried setting the log level to DEBUG for all the RDFService loggers listed but no luck. I don't see any errors from VitroLinkedDataFragmentServlet in either vivo.all.log or catalina.out.

chenejac commented 5 years ago

Brian Lowe said:

I see the exact same behavior as Benjamin.  Switching from tdbContentTripleSource to sdbContentTripleSource eliminates the error and displays the Linked Data Fragments server.

chenejac commented 5 years ago

Brian Lowe said:

OK, so it appears that the underlying problem is that there's a method in RDFServiceJena called getTriples() that is not overridden in RDFServiceTDB to add transaction handling.  The surface problem is that VitroLinkedDataFragmentServlet's call to writeError() triggers the template exception, and VitroLinkedDataFragmentServlet doesn't log the actual exception (in fact, doesn't even use the log at all).

I added a log and the statement log.error(e, e) to line 204 and got this:

 

2018-12-14 17:40:50,039 ERROR [VitroLinkedDataFragmentServlet] org.apache.jena.tdb.transaction.TDBTransactionException: Not in a transaction org.apache.jena.tdb.transaction.TDBTransactionException: Not in a transaction at org.apache.jena.tdb.transaction.DatasetGraphTransaction.get(DatasetGraphTransaction.java:117) at org.apache.jena.tdb.transaction.DatasetGraphTransaction.getDatasetGraphToQuery(DatasetGraphTransaction.java:80) at org.apache.jena.tdb.solver.QueryEngineTDB$QueryEngineFactoryTDB.dsgToQuery(QueryEngineTDB.java:157) at org.apache.jena.tdb.solver.QueryEngineTDB$QueryEngineFactoryTDB.create(QueryEngineTDB.java:168) at org.apache.jena.sparql.engine.QueryExecutionBase.getPlan(QueryExecutionBase.java:524) at org.apache.jena.sparql.engine.QueryExecutionBase.startQueryIterator(QueryExecutionBase.java:472) at org.apache.jena.sparql.engine.QueryExecutionBase.execConstructTriples(QueryExecutionBase.java:213) at org.apache.jena.sparql.engine.QueryExecutionBase.execConstruct(QueryExecutionBase.java:186) at edu.cornell.mannlib.vitro.webapp.rdfservice.impl.jena.RDFServiceJena.getTriples(RDFServiceJena.java:685) at edu.cornell.mannlib.vitro.webapp.rdfservice.impl.RDFServiceFactorySingle$UnclosableRDFService.getTriples(RDFServiceFactorySingle.java:203) at edu.cornell.mannlib.vitro.webapp.rdfservice.impl.logging.LoggingRDFService.getTriples(LoggingRDFService.java:193) at org.vivoweb.linkeddatafragments.datasource.rdfservice.RDFServiceBasedRequestProcessorForTPFs$Worker.createFragment(RDFServiceBasedRequestProcessorForTPFs.java:105) at org.linkeddatafragments.datasource.AbstractRequestProcessorForTriplePatterns$Worker.createRequestedFragment(AbstractRequestProcessorForTriplePatterns.java:102) at org.linkeddatafragments.datasource.AbstractRequestProcessor.createRequestedFragment(AbstractRequestProcessor.java:29) at org.vivoweb.linkeddatafragments.servlet.VitroLinkedDataFragmentServlet.doGet(VitroLinkedDataFragmentServlet.java:191) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at edu.cornell.mannlib.vitro.webapp.controller.VitroHttpServlet.service(VitroHttpServlet.java:71) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1232) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.PageRoutingFilter.doNonDisplayPage(PageRoutingFilter.java:111) at edu.cornell.mannlib.vitro.webapp.filters.PageRoutingFilter.doFilter(PageRoutingFilter.java:102) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.ClickjackFilter.doFilter(ClickjackFilter.java:34) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.RequestModelsPrep.doFilter(RequestModelsPrep.java:59) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.URLRewriterFilter.doFilter(URLRewriterFilter.java:40) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.JSessionStripFilter.doFilter(JSessionStripFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.CachingResponseFilter.produceBasicResponse(CachingResponseFilter.java:321) at edu.cornell.mannlib.vitro.webapp.filters.CachingResponseFilter.doFilter(CachingResponseFilter.java:120) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.i18n.selection.LocaleSelectionFilter.doFilter(LocaleSelectionFilter.java:64) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:36) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.SessionTimeoutLimitingFilter.doFilter(SessionTimeoutLimitingFilter.java:41) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at edu.cornell.mannlib.vitro.webapp.filters.StartupStatusDisplayFilter.doFilter(StartupStatusDisplayFilter.java:66) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

chenejac commented 3 years ago

Brian Lowe said:

Flagging as blocker so that we can decide whether it should be included in 1.12 release.

chenejac commented 3 years ago

Brian Lowe said:

Pull request: https://github.com/vivo-project/Vitro/pull/222

chenejac commented 3 years ago

Benjamin Gross said:

Per our dev call, add a note to example.runtime.properties addressing the visibility concerns of TPF. This is detailed in the wiki already: https://wiki.lyrasis.org/display/VIVODOC111x/Triple+Pattern+Fragments.

chenejac commented 3 years ago

Brian Lowe said:

Additional VIVO pull request for example.runtime.properties: https://github.com/vivo-project/VIVO/pull/236