Unable to handle Neo4j connections with the PDI EE repository - Githubissues

knowbi / knowbi-pentaho-pdi-neo4j-output

Pentaho Data Integration output step for Neo4J

Apache License 2.0

41 stars 19 forks source link

Unable to handle Neo4j connections with the PDI EE repository #138

Closed jfmonteil closed 5 years ago

jfmonteil commented 5 years ago

While using Pentaho 8.3 with the database repository, i have the following error

Namespace Neo4j does not exist in the repository

I had the following comment form Hitachi developpers

What I see happening first is that the plugin is making a request for Neo4j connections. If none have been created yet, then the namespace does not yet exist in the metastore. When making a query to the metastore for a namespace which does not yet exist, working locally, this situation is handled a little more gracefully and does not result in an uncaught exception. When connected to a repository, a MetaStoreException is thrown. Currently, this exception is not being caught in the neo4j kettle plugin. I downloaded the source code for both the neo4j-core and neo4j-output projects. With a little tweaking, including catching the MetaStoreException, as well as some additional null checks, I am able to get the neo4j plugin past that exception. I am able to create new Neo4j connections, as well as save the step and transformation to the repository. I can see the Neo4j namespace and connections in the repository metastore. I did not go as far as running the transformation, but that is enough to tell me that the issues with the Neo4j plugin and the metastore can be addressed with some of the measures I discussed above.

Many thanks for your help, those steps are much appreciated !!

JFM

mattcasters commented 5 years ago

Can you hit the "Details" button and paste the stack trace over here? Thanks in advance!

mattcasters commented 5 years ago

In principle, none of the other MetaStore implementations throw exceptions if you ask for elements in a namespace that doesn't exist. You simply get an empty list back. Fortunately we already have a fork of the MetaStoreFactory in the Neo4j Core plugin project (because of bugs in the Pentaho version, a recurring theme) so we can fix it there.

mattcasters commented 5 years ago

Possible fix in https://github.com/mattcasters/kettle-neo4j-core/issues/4

jfmonteil commented 5 years ago

Hello, tested 4.1.1 No problem when opening the step while connected to the EE repository. However the problem happens once I want to save a new connection :

org.pentaho.metastore.api.exceptions.MetaStoreException: Namespace 'Neo4j doesn't exist in the repository at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.validateNamespace(PurRepositoryMetaStore.java:479) at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.getElementTypes(PurRepositoryMetaStore.java:223) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.pentaho.di.ui.repo.timeout.MetaStoreSessionTimeoutHandler.invoke(MetaStoreSessionTimeoutHandler.java:45) at com.sun.proxy.$Proxy194.getElementTypes(Unknown Source) at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypes(DelegatingMetaStore.java:205) at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypeByName(DelegatingMetaStore.java:236) at org.neo4j.kettle.core.metastore.MetaStoreFactory.loadElement(MetaStoreFactory.java:78) at bi.know.kettle.neo4j.shared.NeoConnectionUtils.newConnection(NeoConnectionUtils.java:39) at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.newConnection(Neo4JOutputDialog.java:1069) at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.access$000(Neo4JOutputDialog.java:57) at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog$1.widgetSelected(Neo4JOutputDialog.java:176) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.open(Neo4JOutputDialog.java:718) at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120) at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8814) at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3287) at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1384) at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7949) at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9331) at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)


> org.pentaho.metastore.api.exceptions.MetaStoreException: Namespace 'Neo4j doesn't exist in the repository
>   at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.validateNamespace(PurRepositoryMetaStore.java:479)
>   at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.getElementTypes(PurRepositoryMetaStore.java:223)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.pentaho.di.ui.repo.timeout.MetaStoreSessionTimeoutHandler.invoke(MetaStoreSessionTimeoutHandler.java:45)
>   at com.sun.proxy.$Proxy194.getElementTypes(Unknown Source)
>   at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypes(DelegatingMetaStore.java:205)
>   at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypeByName(DelegatingMetaStore.java:236)
>   at org.neo4j.kettle.core.metastore.MetaStoreFactory.loadElement(MetaStoreFactory.java:78)
>   at bi.know.kettle.neo4j.shared.NeoConnectionUtils.newConnection(NeoConnectionUtils.java:39)
>   at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.newConnection(Neo4JOutputDialog.java:1069)
>   at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.access$000(Neo4JOutputDialog.java:57)
>   at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog$1.widgetSelected(Neo4JOutputDialog.java:176)
>   at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
>   at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
>   at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
>   at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
>   at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
>   at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
>   at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.open(Neo4JOutputDialog.java:718)
>   at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
>   at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8814)
>   at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3287)
>   at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
>   at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
>   at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
>   at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
>   at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
>   at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
>   at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
>   at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1384)
>   at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7949)
>   at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9331)
>   at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:710)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
>

mattcasters commented 5 years ago

OK, we get a bit further. Will harden more code and push out 4.1.2 then.

mattcasters commented 5 years ago

Try the new neo4j core library: https://github.com/mattcasters/kettle-neo4j-core/releases/tag/1.0.3 Replace version 1.0.2 in data-integration/plugins/Neo4JOutput/lib/ If that works I'll roll out a new version of the plugin so that the new core library is included.

jfmonteil commented 5 years ago

GREAT Job ! It works fine now.

mattcasters commented 5 years ago

Thanks for the feedback JFM!