Can't connect to Gremlin Neptune DB

[ericmillsio]

Hey! Awesome that you brought this open source. I'm just getting started, docker installation was a breeze.

I'm trying to connect my AWS Neptune DB as a datasource. I'm running a local docker instance while VPN'd in from my computer, which connects me to my AWS VPC. I shouldn't need any database/user/password/ssh or anything. Right?

Type: Gremlin Gremlin Implementation: Neptune Server: my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com Port: 8182

I found this article you guys have, but many variations of this did not work.

I've been using graphexp and connecting works fine for that.

Any suggestions? Thanks for the help! Super excited to start using this.

[robfrank]

hi @crearc , you should be able to "ping" your neptune clusterfrom the command line while in VPN:

ping my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com

if the ping command doesn't work, try with the cluster IP:

ping <neptune IP>

if the IP works, configure the datasource with the ip instead of the address.

BTW, you can watch the logs of the container and report the error message :

docker logs -f   docker_arcadeanalytics-app_1

[ericmillsio]

As far as the logs go, I'm getting:

c.a.p.gremlin.GremlinDataProvider        : testing connection to :: 'my-db.xxxxxxxxxxxx.us-east-1.neptune.amazonaws.com'
o.z.p.spring.web.advice.AdviceTrait      : Internal Server Error
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at com.arcadeanalytics.provider.gremlin.GremlinDataProvider.testConnection(GremlinDataProvider.kt:298)
at com.arcadeanalytics.web.rest.DataSourceResource.testDataSource(DataSourceResource.java:123)

ping is not effective for Neptune instances. By default AWS does not allow ping access. I've attempted to add a 'Custom ICMP Rule' to allow for ping which usually works for EC2 instances, but Neptune totally ignores this rule and it does not apply (similar to other database services on AWS).

For RDS, but same rules apply:

"RDS Instances are not configured to accept and respond to an ICMP packet for pings. The only way you can establish connectivity to your RDS instance is through a standard SQL client application." This means, that adding ICMP rule into particular RDS security group, doesn't make your RDS instance reachable over ICMP.

However, nmap works fine (cluster IP works too): nmap -Pn -p 8182 my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com.

Starting Nmap 7.70 ( https://nmap.org ) at 2019-02-19 10:56 CST
Nmap scan report for my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com (XXX.XX.XX.XX)
Host is up (0.041s latency).
rDNS record for XXX.XX.XX.XX: ip-XXX-XX-XX-XX.ec2.internal

PORT     STATE SERVICE
8182/tcp open  vmware-fdm

Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds

Note the -PN is required because I was getting Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn.

Curling the status also works fine (cluster IP works too):

curl -G http://my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com:8182/status
{"status":"healthy"}

It's most definitely available. As I said I've been able to connect graphexp fine and was able to interact with the DB.

[ericmillsio]

Full log output:

arcadeanalytics-app_1            | 2019-02-19 16:26:17.683  INFO 1 --- [ XNIO-2 task-30] c.a.p.gremlin.GremlinDataProvider        : testing connection to :: 'my-db.xxxxxxxxxx.us-east-1.neptune.amazonaws.com'
arcadeanalytics-app_1            | 2019-02-19 16:26:18.146 ERROR 1 --- [ XNIO-2 task-30] o.z.p.spring.web.advice.AdviceTrait      : Internal Server Error
arcadeanalytics-app_1            |
arcadeanalytics-app_1            | java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
arcadeanalytics-app_1            |  at com.arcadeanalytics.provider.gremlin.GremlinDataProvider.testConnection(GremlinDataProvider.kt:298)
arcadeanalytics-app_1            |  at com.arcadeanalytics.web.rest.DataSourceResource.testDataSource(DataSourceResource.java:123)
arcadeanalytics-app_1            |  at com.arcadeanalytics.web.rest.DataSourceResource$$FastClassBySpringCGLIB$$89a9850e.invoke(<generated>)
arcadeanalytics-app_1            |  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
arcadeanalytics-app_1            |  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
arcadeanalytics-app_1            |  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
arcadeanalytics-app_1            |  at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:48)
arcadeanalytics-app_1            |  at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:34)
arcadeanalytics-app_1            |  at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
arcadeanalytics-app_1            |  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
arcadeanalytics-app_1            |  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
arcadeanalytics-app_1            |  at com.arcadeanalytics.web.rest.DataSourceResource$$EnhancerBySpringCGLIB$$6affad1b.testDataSource(<generated>)
arcadeanalytics-app_1            |  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
arcadeanalytics-app_1            |  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
arcadeanalytics-app_1            |  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
arcadeanalytics-app_1            |  at java.lang.reflect.Method.invoke(Method.java:498)
arcadeanalytics-app_1            |  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
arcadeanalytics-app_1            |  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
arcadeanalytics-app_1            |  at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
arcadeanalytics-app_1            |  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
arcadeanalytics-app_1            |  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
arcadeanalytics-app_1            |  at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:111)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
arcadeanalytics-app_1            |  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
arcadeanalytics-app_1            |  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at com.arcadeanalytics.security.jwt.JWTFilter.doFilter(JWTFilter.java:36)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
arcadeanalytics-app_1            |  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
arcadeanalytics-app_1            |  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
arcadeanalytics-app_1            |  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
arcadeanalytics-app_1            |  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
arcadeanalytics-app_1            |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
arcadeanalytics-app_1            |  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
arcadeanalytics-app_1            |  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
arcadeanalytics-app_1            |  at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
arcadeanalytics-app_1            |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
arcadeanalytics-app_1            |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
arcadeanalytics-app_1            |  at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
arcadeanalytics-app_1            |  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
arcadeanalytics-app_1            |  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)
arcadeanalytics-app_1            |  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
arcadeanalytics-app_1            |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
arcadeanalytics-app_1            |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
arcadeanalytics-app_1            |  at java.lang.Thread.run(Thread.java:748)
arcadeanalytics-app_1            | Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client.submit(Client.java:225)
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client.submit(Client.java:209)
arcadeanalytics-app_1            |  at com.arcadeanalytics.provider.gremlin.GremlinDataProvider.testConnection(GremlinDataProvider.kt:288)
arcadeanalytics-app_1            |  ... 123 common frames omitted
arcadeanalytics-app_1            | Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:321)
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:253)
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client.submit(Client.java:223)
arcadeanalytics-app_1            |  ... 125 common frames omitted
arcadeanalytics-app_1            | Caused by: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient.chooseConnection(Client.java:510)
arcadeanalytics-app_1            |  at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:316)
arcadeanalytics-app_1            |  ... 127 common frames omitted
arcadeanalytics-app_1            |

[robfrank]

We tested only using an EC2 in the same VPC instance as SSH gateway: https://docs.aws.amazon.com/neptune/latest/userguide/security-vpc.html

Which kind of configuration do you provide to graphexp ? I mean, accesKey and so on?

[ericmillsio]

Ahh okay. I'm running locally while connected to my VPN (that gives access to the same VPC where Neptune lives). I just figured out that nmap -Pn -p 8182 my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com does not work while docker exec -it docker_arcadeanalytics-app_1 sh into the container. It works fine on my Mac host machine. I'm going to dig further into this, might just be a docker config <=> VPN issue.

Below is the answer to your questions anyways:

I don't need to provide any authentication at the moment with graphexp. As long as I'm within the VPC I can access it without any access keys. You'll notice the requests in graphexp are pretty simple. I did not enable 'IAM DB authentication' yet although I may in the future.

The only thing used when running the gremlin console is the Amazon Root CA Certificate. (See Root CA Certificates) But I don't think this is used with graphexp

[robfrank]

Oh yes!!! Arcade is running locally but inside a container in another subnet!

I will be back with a solution tomorrow.

[ericmillsio]

More playing around, let me know if this should be a different issue:

So I decided to role this into an actually EC2 instance because I was going to need to eventually anyway. Was able to get the server up and running, and it seems being exec sh'd into the container that Neptune is reachable now. I filled out the info, but got a new error. Seems like 'gateway' and 'sshUser' are coming in as 'null' instead of actual null which is causing an error to be thrown. Note that I left 'Use SSH Tunnel' unchecked.

DataSourceUtil.java:26

Optional.ofNullable(ds.getSshUser()).orElse("")

Seems like it's just the transfer from frontend ends up quoting null inputs in the API. I feel like it might work on this cloud instance if I get passed this...

Docker container log:

2019-02-20 17:11:17.761  INFO 1 --- [  XNIO-2 task-2] c.a.web.rest.DataSourceResource          : REST request to test DataSource : DataSourceDTO{id=null, name='Graph', description='null', type='GREMLIN_NEPTUNE', indexing='null', server='my-db.xxxxxxxx.neptune.amazonaws.com', port=8182, database='na', username='', password='', remote='false', gateway='null', sshPort=null, sshUser='null'}
2019-02-20 17:11:17.762  INFO 1 --- [  XNIO-2 task-2] c.a.web.rest.DataSourceResource          : DataSource : DataSource{id=null, name='Identity Graph', description='null', type='GREMLIN_NEPTUNE', indexing='null', server='my-db.xxxxxxxx.neptune.amazonaws.com', port=8182, database='na', username='', password='', remote='false', gateway='null', sshPort=null, sshUser='null'}
2019-02-20 17:11:17.764 ERROR 1 --- [  XNIO-2 task-2] o.z.p.spring.web.advice.AdviceTrait      : Internal Server Error
2019-02-20T16:57:46.176551482Z        java.lang.IllegalArgumentException: Parameter specified as non-null is null: method com.arcadeanalytics.provider.DataSourceInfo.\u003cinit\u003e, parameter description
2019-02-20T16:57:46.176555732Z     at com.arcadeanalytics.provider.DataSourceInfo.\u003cinit\u003e(DataSourceInfo.kt)
2019-02-20T16:57:46.176559675Z     at com.arcadeanalytics.service.util.DataSourceUtil.toDataSourceInfo(DataSourceUtil.java:26)
2019-02-20T16:57:46.176563594Z     at com.arcadeanalytics.web.rest.DataSourceResource.testDataSource(DataSourceResource.java:120)
2019-02-20T16:57:46.176567429Z     at com.arcadeanalytics.web.rest.DataSourceResource$$FastClassBySpringCGLIB$$89a9850e.invoke(\u003cgenerated\u003e)
2019-02-20T16:57:46.176571408Z     at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
2019-02-20T16:57:46.176575124Z     at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
2019-02-20T16:57:46.176578969Z     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
2019-02-20T16:57:46.176582757Z     at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:48)
2019-02-20T16:57:46.176586639Z     at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:34)
2019-02-20T16:57:46.176590369Z     at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
2019-02-20T16:57:46.176639709Z     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
2019-02-20T16:57:46.176645974Z     at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
2019-02-20T16:57:46.176650058Z     at com.arcadeanalytics.web.rest.DataSourceResource$$EnhancerBySpringCGLIB$$7adc8e07.testDataSource(\u003cgenerated\u003e)
2019-02-20T16:57:46.176654134Z     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-02-20T16:57:46.176657972Z     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-02-20T16:57:46.176661803Z     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-02-20T16:57:46.176665628Z     at java.lang.reflect.Method.invoke(Method.java:498)
2019-02-20T16:57:46.176670223Z     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
2019-02-20T16:57:46.176680918Z     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
2019-02-20T16:57:46.176685148Z     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
2019-02-20T16:57:46.176689031Z     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
2019-02-20T16:57:46.176692957Z     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
2019-02-20T16:57:46.176696867Z     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
2019-02-20T16:57:46.176700727Z     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
2019-02-20T16:57:46.176704572Z     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
2019-02-20T16:57:46.176708366Z     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
2019-02-20T16:57:46.176712084Z     at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
2019-02-20T16:57:46.176715794Z     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
2019-02-20T16:57:46.17671959Z      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
2019-02-20T16:57:46.176723396Z     at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2019-02-20T16:57:46.176727094Z     at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
2019-02-20T16:57:46.176730868Z     at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
2019-02-20T16:57:46.176734623Z     at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:111)
2019-02-20T16:57:46.176738351Z     at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2019-02-20T16:57:46.176742043Z     at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2019-02-20T16:57:46.176745783Z     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
2019-02-20T16:57:46.176749553Z     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
2019-02-20T16:57:46.180774911Z     at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2019-02-20T16:57:46.180805507Z     at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2019-02-20T16:57:46.180811481Z     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
2019-02-20T16:57:46.180815529Z     at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2019-02-20T16:57:46.180819301Z     at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2019-02-20T16:57:46.180831363Z     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
2019-02-20T16:57:46.180835499Z     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
2019-02-20T16:57:46.180839346Z     at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
2019-02-20T16:57:46.18084316Z      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
2019-02-20T16:57:46.18084713Z      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
2019-02-20T16:57:46.180850924Z     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
2019-02-20T16:57:46.180854828Z     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
2019-02-20T16:57:46.180858642Z     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180862411Z     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
2019-02-20T16:57:46.180866219Z     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180870104Z     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
2019-02-20T16:57:46.180873876Z     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180877806Z     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
2019-02-20T16:57:46.18088163Z      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180885453Z     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
2019-02-20T16:57:46.180890449Z     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180894443Z     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
2019-02-20T16:57:46.180898243Z     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180902002Z     at com.arcadeanalytics.security.jwt.JWTFilter.doFilter(JWTFilter.java:36)
2019-02-20T16:57:46.180905703Z     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
2019-02-20T16:57:46.180909573Z     at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)

[Auricopter]

Having similar problems. Here's my config: And here are several errors I get when trying to write gremlin queries and using the full-text-search. Also the "test connection" button is greyed out. So far I just started the source using docker-compose -f src/main/docker/app.yml up

[robfrank]

try this: setup "na" (or any other not null value) as value for user, password and database. Let me know.

We are working to improve this and among over to improve our docs

[Auricopter]

try this: setup "na" (or any other not null value) as value for user, password and database. Let me know.

We are working to improve this and among over to improve our docs

Now I can hit "Test Connection" but the connection is still dead. This config worked 2 months ago, when I used the Arcade Playground on your server as a SaaS.

I suspect the problem is, that somehow the docker container isn't allowed to communicate through the VPC of the EC2/ECS Cluster.

I just remembered that you need to place Arcade's public key into the authorized_keys folder of the EC2

[Auricopter]

Yes, now it works! You need to add the rsa-ssh public key from Arcade Analytics into your authorized_keys file in your EC2 instance. E.g. connect to your EC2 instance via WinSCP. Navigate to home/ec2-user/.ssh (hidden folder). Open authorized_keys. Add the public key.

[ericmillsio]

Where do you get the public key? This page's download link is broken for me: https://arcadeanalytics.com/usermanual/#how-to-integrate-with-your-database

[ericmillsio]

Another day and deeper I go. I don't think I need the public key or ssh for my situation. I've gotten a success on testing the connection (I was mistakenly testing with the neptune instance instead of cluster url)! When I try and index the Neptune DB I get this error. It seems like it is seeing my dataset now for sure because Account and Domain are labels in my DB. (This case is within the context of my EC2 instance.)

Getting closer though!

2019-02-21 18:18:57.009  INFO 1 --- [tics-Executor-2] c.a.p.gremlin.GremlinGraphProvider       : start indexing of data-source 1351 - total nodes:: 9346 
2019-02-21 18:18:59.368  INFO 1 --- [tics-Executor-2] c.a.index.ElasticBulkGraphIndexer        : sending bulk to index 1351 of size 85 -  total indexed:: 85 
2019-02-21 18:18:59.442  INFO 1 --- [tics-Executor-2] c.a.index.ElasticBulkGraphIndexer        : flushing index 1351 - total docs indexed::  85
2019-02-21 18:18:59.443  INFO 1 --- [tics-Executor-2] c.a.index.ElasticBulkGraphIndexer        : indexed by class:: {Account=50, Domain=35}
error while indexing datasource DataSource{id=1351, name='Graph', description='graph', type='GREMLIN_NEPTUNE', indexing='INDEXING', server='my-db.xxxxxxxxxxx.us-east-1.neptune.amazonaws.com', port=8182, database='na', username='na', password='na', remote='false', gateway='', sshPort=null, sshUser=''}
java.lang.IllegalStateException: Multiple properties exist for the provided key, use Vertex.properties(aliases)
        at org.apache.tinkerpop.gremlin.structure.Vertex$Exceptions.multiplePropertiesExistForProvidedKey(Vertex.java:179)
        at org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex.property(DetachedVertex.java:109)
        at org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex.property(DetachedVertex.java:50)
        at org.apache.tinkerpop.gremlin.structure.Element.value(Element.java:94)
        at com.arcadeanalytics.provider.gremlin.GremlinGraphProvider.lambda$provideNodes$0(GremlinGraphProvider.java:93)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
        at com.arcadeanalytics.provider.gremlin.GremlinGraphProvider.provideNodes(GremlinGraphProvider.java:93)
        at com.arcadeanalytics.provider.gremlin.GremlinGraphProvider.provideTo(GremlinGraphProvider.java:62)
        at com.arcadeanalytics.service.ElasticGraphIndexerService.index(ElasticGraphIndexerService.java:118)
        at com.arcadeanalytics.service.ElasticGraphIndexerService$$FastClassBySpringCGLIB$$7483a18d.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at io.github.jhipster.async.ExceptionHandlingAsyncTaskExecutor.lambda$createWrappedRunnable$1(ExceptionHandlingAsyncTaskExecutor.java:68)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[robfrank]

Thank you very much for all your feedbacks.

As you can image, after opening up the project we received a lot of interest and we are trying to manage all.

BTW, we are working on a new release that will enable to provide your own keys to the application for ssh connection.

About the stack-trace, I investigated and discovered that we don't handle multivalued properties. We are on it as well.

The plans are to release a new version with these fixes and other on Monday.

[ericmillsio]

Awesome! There's no rush. I totally understand how a new public release like this can go. Just trying to provide as much detail as I can to be helpful for you and future users.

I totally understand, don't stress! Take your time and let me know how I can help. :)

[robfrank]

hi, the new images, even if not yet tagged with a stable version, are available.

They use the new version of our connectors: https://github.com/ArcadeAnalytics/arcade-connectors/releases/tag/v1.0.2

This version fixes the problem with multivalued properties. Let me know if it works

[ericmillsio]

On startup now of the app, I'm hitting this error where it can't find /content/media. Made a folder at that location, but didn't help.

2019-03-05 23:15:32.479  INFO 1 --- [           main] com.arcadeanalytics.util.MediaLoader     : start loading images from:: /content/media/ 
2019-03-05 23:15:32.482  WARN 1 --- [           main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mediaLoader' defined in class path resource [com/arcadeanalytics/util/MediaLoaderConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.arcadeanalytics.util.MediaLoader]: Factory method 'mediaLoader' threw exception; nested exception is java.io.FileNotFoundException: ServletContext resource [/content/media/] cannot be resolved to URL because it does not exist
2019-03-05 23:15:32.615 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mediaLoader' defined in class path resource [com/arcadeanalytics/util/MediaLoaderConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.arcadeanalytics.util.MediaLoader]: Factory method 'mediaLoader' threw exception; nested exception is java.io.FileNotFoundException: ServletContext resource [/content/media/] cannot be resolved to URL because it does not exist
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1177)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1071)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
        at com.arcadeanalytics.ArcadeanalyticsApp.main(ArcadeanalyticsApp.java:46)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:59)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.arcadeanalytics.util.MediaLoader]: Factory method 'mediaLoader' threw exception; nested exception is java.io.FileNotFoundException: ServletContext resource [/content/media/] cannot be resolved to URL because it does not exist
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        ... 24 common frames omitted
Caused by: java.io.FileNotFoundException: ServletContext resource [/content/media/] cannot be resolved to URL because it does not exist
        at org.springframework.web.context.support.ServletContextResource.getURL(ServletContextResource.java:156)
        at org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(PathMatchingResourcePatternResolver.java:492)
        at org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(PathMatchingResourcePatternResolver.java:293)
        at org.springframework.context.support.AbstractApplicationContext.getResources(AbstractApplicationContext.java:1295)
        at org.springframework.context.support.GenericApplicationContext.getResources(GenericApplicationContext.java:228)
        at com.arcadeanalytics.util.MediaLoader.loadImages(MediaLoader.kt:22)
        at com.arcadeanalytics.util.MediaLoaderConfiguration.mediaLoader(MediaLoaderConfiguration.kt:25)
        at com.arcadeanalytics.util.MediaLoaderConfiguration$$EnhancerBySpringCGLIB$$6c9a9b65.CGLIB$mediaLoader$0(<generated>)
        at com.arcadeanalytics.util.MediaLoaderConfiguration$$EnhancerBySpringCGLIB$$6c9a9b65$$FastClassBySpringCGLIB$$640334b0.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
        at com.arcadeanalytics.util.MediaLoaderConfiguration$$EnhancerBySpringCGLIB$$6c9a9b65.mediaLoader(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        ... 25 common frames omitted

[robfrank]

in which way do you run the application? Are you using mvn ? (we moved to maven last week) Or the container?

[msg-q]

Hi I am also trying to get Arcade Analytics to run internally against Neptune. The application seems to install fine using Ubuntu 18.04 clean image with Java 1.8 using Maven. I am presented with:

---

    Application 'arcadeanalytics' is running! Access URLs:
    Local:          http://localhost:8080
    External:       http://172.18.11.192:8080
    Profile(s):     [swagger, dev]

---

Which looks correct. However when I go to the URL I get a 404 on http://172.18.11.192:8080/ckeditor/ckeditor.js.

The steps were pretty straight forward:

1. Installed java
2. Installed maven
3. Installed git
4. Cloned your github repo
5. Ran Maven

Am I missing anything? Thanks!

[robfrank]

if you run from the sources you need two shell:

• first shell: ./mvnw
  • second shell : 'yarn start'

so you need yarn/node installed too

[msg-q]

That got it running but there are a bunch of errors with the GUI connecting to localhost. Would it be possible for you guys build an AWS EC2 Image and just add it to the AWS Marketplace? It would be much easier for people to adopt.

[robfrank]

@msg-q we provide the docker image, is it enough?

[msg-q]

RobFrank, appreciate the work you are putting in here.

My opinion is it depends on who you want to adopt your software. If you want developers running it locally on their laptops to adopt, then the docker containers are fine. If you want businesses who are running in AWS, adding it to the marketplace as an AMI and having an image that is tested and easy to spin up would be better.

Just my opinion and I hope it helps!

[robfrank]

Is there a way to automate AMI creation as we do with Docker images? Any help in the form of script or tutorial link is appreciated.

[farari7]

Hello everyone,

I cannot find the public key from arcade to add it to my authorized keys in EC2, can someone help me?

[ericmillsio]

Automatic AMI and Docker Image creation is very easy to do with Packer.

[robfrank]

@farari7 please read the readme, I added some doc on how use your own ssh keys to connect to your own neptune.

[farari7]

Ok I did all the steps (create the keys -with name id_rsa-, place it on .ssh directory and add the public key to my EC2 instance and I get the following error: java.nio.file.NoSuchFileException: /.arcade/.ssh/id_rsa .

It looks like I am getting in but then it cannot read my key and it drop out.

[robfrank]

If the application emits the error:

java.nio.file.NoSuchFileException: /.arcade/.ssh/id_rsa

you placed the ssh keys in the wrong place. The path printed out isn't equal to the one provided in the compose file. Did you modify it?

This snippet is from the app.yml descriptor

      - JAVA_OPTS=-DSSH_PRIV_KEY=/arcade/.ssh/id_rsa -DSSH_PUB_KEY=/arcade/.ssh/id_rsa.pub
    ports:
      - 8080:8080
    volumes:
      - ~/.arcade/:/arcade/

from the readme: Create a directory in ~/.arcade/ named .ssh and put the private and public keys inside. The public key should be copied on the server used as ssh gateway too.

So, after the first run using the provided compose file, you should have in your home dir in the host a hidden directory called .arcade

I need more details to help you:

• configuration files if you have modified them
• commands used to run the application
• operative system
• any other useful detail

[farari7]

Ok this is what I am doing:

• I am launching Arcade with docker with: docker-compose -f src/main/docker/app.yml up. -My operating system is Mac OS High Sierra.
• This is how I am trying to connect to my Neptune database (for the Gateway i am using the public IP of my EC2 instance and for the server the endpoint of my Neptune VPC):

I didn't move any file but I am going to erase it and download it again. Literally what I did was to create the keys, place them on the .ssh file (inside .arcade) and share the public key to my EC2 instance.

[psoares-resilient]

Hello we are also keen to try this,

We are having the same problem with timeouts when trying to connect to neptune. Key is in the right place, added to the EC2. everything works normally when I gremlin from my machine to Neptune with a tunnel open.

I also logged in the container and ran: ssh -4 -i id_rsa -L 8182:neptune-cluster.custom-private-domain:8182 ec2-user@[IPEC2]

The result is a tunnel on localhost (container perspective).

lsof -i -P -n | grep LISTEN outputs:

java      7 root  123u  IPv4 561445      0t0  TCP *:8080 (LISTEN)
ssh     107 root    4u  IPv4 560663      0t0  TCP 127.0.0.1:8182 (LISTEN)

And the curl: curl -k https://localhost:8182/status works just fine (again container context).

{"status":"healthy","startTime":"Sun Apr 12 01:28:44 UTC 2020","dbEngineVersion":"1.0.2.2.R2","role":"writer","gremlin":{"version":"tinkerpop-3.4.3"},"sparql":{"version":"sparql-1.1"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"}}

So my understanding is that the container can talk to neptune via the tunnel. Simple. however, when trying to update the values on the ui for the data source, the following exception happen:

I wonder if it has anything with the connection trying to be established via ws:// instead of wss://. All the other permutations of ssh tunnels, and attempt to switch url returns the same exception.

Any ideas?

p.s.: I had to install ping, lsof and ssh into the container to get to that point

[arcade-player]

as I can see from the picture, the "remote" flag is set to false, so arcade will not setup the ssh tunnel

https://arcadeanalytics.com/amazon-neptune-hands-on/

the article refers to OUR public/private keys, you shoul ues yours.

take a look to the parameters showed here:

[psoares-resilient]

Sorry

as I can see from the picture, the "remote" flag is set to false, so arcade will not setup the ssh tunnel

https://arcadeanalytics.com/amazon-neptune-hands-on/

the article refers to OUR public/private keys, you shoul ues yours.

take a look to the parameters showed here:

Sorry if I wasn't clear. I did have the setup using my credentials and keys and tunnel to EC2 via the app. What i did on the container with the tunnel was to prove that even though the container itself can open a ssh tunnel and use localhost:8182 for the query and reach my Neptune, the actual application running on the same container cannot.

I have tried the setup without any local tunnels and it also gets a timeout. I disabled the tunnels and tried to use the host machine tunnel I had setup. Also failed. Finally I opened the tunnel on the container itself to check it could talk to neptune. It could, but again, not the app.

Does that help to understand or maybe more logs/screenshots to help clarify?

[arcade-player]

Just to check., are you passing your keys to the container in the volume:

https://github.com/ArcadeData/arcadeanalytics#configure-ssh

[psoares-resilient]

So,

I realized I was using the arcadeanalytics-recipes project instead of this one as recommended on the readme. I ran: docker-compose -f recipes/arcade-standalone.yml up which uses image: arcadeanalytics/arcadeanalytics:1.0.5 for the image and the volume mounted there is in a different path:

I added the keys to the one there. (Also, when inside the container I ls -lar /arcade/.ssh and the keys are there. (Ignore the other docker running)

I have include the PUBKEY as well which I derived from the private key and added that to the ec2. I know the key works, because I can ssh from the container into the ec2.

Lastly, I update app that I can reach in localhost:8080, login, add datasource and use the following (edit, missing image):

By the params in the log, you can see the ui was populated correctly... I ruled out typos and wrong keys by using the same values in the container itself and ssh-ing into the ec2. All works fine. The app does not connect though.

Just now I tried both building the code and running the yarn start same result. When trying to run the arcadeanalytics/src/main/docker/app.yml (and adding the keys to the right place ~/.arcade/.ssh. However it is using the image image: arcadeanalytics/arcadeanalytics:1.0.1 and docker fails to spin up the postgres which in turn fails the actual app:

which goes into:

Thanks for the help :D

[arcade-player]

Sorry, my fault. Use the repices standalone, it's more updated. I will move the missing documentation about ssh configuration to it.

Jus put the keys in ~/.arcade-single/.ssh on your host.

So, to recap

• use standalone from recipes
• put the keys in ~/.arcade-single/.ssh on you "host"
• start the compose
• rememebr to copy the public key on the gateway machine!
• configure the connection to netpune

[psoares-resilient]

Only the last paragraph is about the Dockerfile with postgres failing. The whole first section is basically all the points covered that result in the Timeout exception.

• use standalone from recipes ✅
• put the keys in ~/.arcade-single/.ssh on you "host" ✅
• start the compose ✅
• rememebr to copy the public key on the gateway machine! ✅
• configure the connection to netpune ✅
• click test connection ❌

Result:

[psoares-resilient]

Follow up on this after coming across this: https://github.com/aws-samples/amazon-neptune-samples/issues/13

I updated GremlinUtil.kt on the Connectors project to force wss on NEPTUNE:

...
.enableSsl(dataSource.type == "GREMLIN_COSMOSDB" || dataSource.type == "GREMLIN_NEPTUNE")
...

referenced the SNAPSHOT on the the main project and finally built it.

After doing it, I still could not connect via the SSH Tunnel option on the ui, I take the path for the .ssh might be different when running from there... So, I removed the ssh tunnel option and opened a tunnel on the host to the ec2 as I was doing before:

ssh -4 -i id_rsa -L 8182:neptune-cluster.custom-private-domain:8182 ec2-user@[IPEC2]

Then I referenced the localhost as the server and I got a testing pass. It actually connected to neptune. I still don't understand how to use the software. (I tried querying a node and on the BE it says it found, but it can't show anything because it doesn't know something called a class... Probably a different issue... But test connection passes!!).

So:

1 - The tunnel used by the app in docker is somehow broken (for me) (I also tried playing with ssh permission and locking/unlocking the keys to 400/755) 2 - Even when you don't need the tunnel in the app (because it was set it upfront) it should still use wss to connect otherwise it will timeout 3 - It may depend on the version of neptune being used, it may not...

{"status":"healthy","startTime":"Sun Apr 12 01:28:44 UTC 2020","dbEngineVersion":"1.0.2.2.R2","role":"writer","gremlin":{"version":"tinkerpop-3.4.3"},"sparql":{"version":"sparql-1.1"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"}}

I hope this helps and this can be reproduced on your side as well so we can get a way to connect it and see the data.

Thanks again.

[thiagodiogo]

Any update or plan to implement this wss change to a release?

Arcade can't be used with AWS Neptune.

Thanks in advance

[arcade-player]

We can fix and release it, it should take some time to us.

In the meanwhile, can you deploy arcade inside AWS on a EC2 instance and try to connect to your neptune instance (no ssh tunnel needed) to check if it works?

[thiagodiogo]

I can try to do that however all the connections must use encryption: Clients must support Transport Layer Security (TLS) 1.0 or later. We recommend TLS 1.2 or later

Source: https://docs.aws.amazon.com/neptune/latest/userguide/data-protection.html

Part of my challenge here are Compliance requirements (encryption in-transit) and I really appreciate your time and effort to fix this.

Thanks!

[arcade-player]

we have released a new version of connectors (1.0.11) that enable ssl for neptune and a new version on analytics (1.0.6) that uses the new connector version. Please, test it!

[dominusmi]

I've followed the instructions and am currently using the app-single.yml from master, however it's still not working. I manage to connect to the EC2 via ssh as the following logs state:

arcadeanalytics-single_1  | 2021-07-14 16:54:57.678  INFO 1 --- [ XNIO-2 task-38] c.a.provider.SshTunnelTemplate           : Authentication succeeded (publickey).
arcadeanalytics-single_1  | 2021-07-14 16:54:57.679  INFO 1 --- [ XNIO-2 task-38] c.a.p.gremlin.GremlinDataProvider        : testing connection to :: 'localhost' 
arcadeanalytics-single_1  | 2021-07-14 16:55:12.692 ERROR 1 --- [ XNIO-2 task-38] o.z.p.spring.web.advice.AdviceTrait      : Internal Server Error

This all happens when I test the connection configuration. However, the second line you can see it tries to connect to localhost, and times out on it? No idea where that's coming from, my connection configurations are pointing toward the neptune.cluster-*.*.neptune.amazonaws.com as expected, and the connection to that endpoint from inside the ec2 instance works. Any idea what's going on and how to fix this?

Update: I've further discovered that when you try to index the database (even though the test connection doesn't work) this error message pops up:

arcadeanalytics-single_1  | 2021-07-14 18:52:23.903  WARN 7 --- [ XNIO-2 task-46] o.apache.tinkerpop.gremlin.driver.Host   : Marking Host{address=localhost/127.0.0.1:39573, hostUri=ws://localhost:39573/gremlin} as unavailable. Trying to reconnect.
arcadeanalytics-single_1  | 2021-07-14 18:52:25.807  WARN 7 --- [tics-Executor-2] o.apache.tinkerpop.gremlin.driver.Host   : Marking Host{address=localhost/127.0.0.1:43869, hostUri=ws://localhost:43869/gremlin} as unavailable. Trying to reconnect.

Again with a reference to localhost, and some random ports (at least to the best of my knowledge). There's also a websocket connection attempt instead of using the https endpoint.