Plugins not loading

[znorris]

I mapped a volume to /var/lib/neo4j/plugins however my jars are not loaded on startup. I've tested using the latest framework and timetree jars.

[benbc]

@znorris Could you tell us which version of Neo4j you are using and show us your Docker invocation, please?

[znorris]

@benbc Thank you for getting back to me so quickly.

Host

Docker version 1.11.1, build 5604cbe docker run -d --publish=7474:7474 --publish=7687:7687 --volume=$HOME/neo4j/data:/data --volume=$HOME/neo4j/plugins:/var/lib/neo4j/plugins neo4j:3.0

Container

root@8fff8ab667a3:/var/lib/neo4j/bin# curl -v localhost:7474/db/data/ -u neo4j:mypassword    
* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 7474 (#0)
* Server auth using Basic with user 'neo4j'
> GET /db/data/ HTTP/1.1
> Authorization: Basic bmVvNGo6YmJyb3lnYiFA
> User-Agent: curl/7.38.0
> Host: localhost:7474
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 08 Jun 2016 16:15:23 GMT
< Content-Type: application/json; charset=UTF-8
< Access-Control-Allow-Origin: *
< Content-Length: 795
* Server Jetty(9.2.9.v20150224) is not blacklisted
< Server: Jetty(9.2.9.v20150224)
< 
{
  "extensions" : { },
  "node" : "http://localhost:7474/db/data/node",
  "relationship" : "http://localhost:7474/db/data/relationship",
  "node_index" : "http://localhost:7474/db/data/index/node",
  "relationship_index" : "http://localhost:7474/db/data/index/relationship",
  "extensions_info" : "http://localhost:7474/db/data/ext",
  "relationship_types" : "http://localhost:7474/db/data/relationship/types",
  "batch" : "http://localhost:7474/db/data/batch",
  "cypher" : "http://localhost:7474/db/data/cypher",
  "indexes" : "http://localhost:7474/db/data/schema/index",
  "constraints" : "http://localhost:7474/db/data/schema/constraint",
  "transaction" : "http://localhost:7474/db/data/transaction",
  "node_labels" : "http://localhost:7474/db/data/labels",
  "neo4j_version" : "3.0.1"
* Connection #0 to host localhost left intact

root@8fff8ab667a3:/var/lib/neo4j/plugins# ls -la
total 13104
drwxr-xr-x  2 1000    1000     4096 Jun  8 04:57 .
drwxr-xr-x 13  111 nogroup     4096 Jun  8 05:26 ..
-rw-rw-r--  1 1000    1000 12709287 May 10 11:51 graphaware-server-community-all-3.0.1.38.jar
-rw-rw-r--  1 1000    1000   699498 May 10 11:51 graphaware-timetree-3.0.1.38.24.jar

Log snippet:

2016-06-08 05:26:03.711+0000 INFO  [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics END ---
2016-06-08 05:26:03.919+0000 INFO  [o.n.b.v.r.Sessions] Bolt Server extension loaded.
2016-06-08 05:26:03.919+0000 INFO  [o.n.b.v.r.Sessions] Bolt enabled on 0.0.0.0:7687.
2016-06-08 05:26:04.483+0000 INFO  [o.n.k.i.DatabaseHealth] Database health set to OK
2016-06-08 05:26:05.225+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.apache.commons.logging.impl.AvalonLogger` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/apache/avalon/framework/logger/Logger
2016-06-08 05:26:05.227+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.apache.commons.logging.impl.Log4JLogger` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/apache/log4j/Category
2016-06-08 05:26:05.231+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.apache.commons.logging.impl.LogKitLogger` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/apache/log/Logger
2016-06-08 05:26:05.468+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.springframework.core.type.filter.AspectJTypeFilter` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/aspectj/weaver/World
2016-06-08 05:26:05.573+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.springframework.cglib.transform.AbstractProcessTask` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/apache/tools/ant/Task
2016-06-08 05:26:05.573+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.springframework.cglib.transform.AbstractTransformTask` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/apache/tools/ant/Task
2016-06-08 05:26:05.640+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `org.springframework.http.client.OkHttpClientHttpRequest` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: com/squareup/okhttp/MediaType
... more of the same ...
2016-06-08 05:26:06.803+0000 WARN  [o.n.k.i.p.Procedures] Failed to load `com.graphaware.runtime.schedule.RotatingTaskScheduler` from plugin jar `/var/lib/neo4j/plugins/graphaware-server-community-all-3.0.1.38.jar`: org/neo4j/management/HighAvailability
2016-06-08 05:26:07.996+0000 INFO  [o.n.k.AvailabilityGuard] Fulfilling of requirement makes database available: Database available
2016-06-08 05:26:07.997+0000 INFO  [o.n.k.i.f.CommunityFacadeFactory] Database is now ready
2016-06-08 05:26:07.997+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:Versions START ---

[spacecowboy]

@znorris the logs state that it can't load org.apache.xxx, and org.springframework.xxx things which your plugins depend on. Are these packages really located inside your jars? Or are you actually missing jars for those dependencies?

[spacecowboy]

Also btw (this is not your problem) but you should mount plugins into /plugins and not /var/lib/neo4j/plugins

[znorris]

@spacecowboy I was following the instructions on the timetree page that indicated the timetree plugin's only requirement is the framework jar. I can't really speak intelligently on the dependencies as I'm trying neo4j with plugins for the first time. I don't see any dependencies on the part of the framework.

Thanks for letting me know about /plugins. I had assumed it wasn't setup that way because /var/lib/neo4j/data is a symlink to /data but /var/lib/neo4j/data/plugin is not a symlink to /plugins. It certainly attempts to load plugins from /plugins though! Also, I don't believe there is any mention of using /plugins in the docs, but maybe I missed it.

[spacecowboy]

It's easy to miss, it's listed under User-defined procedures at http://neo4j.com/developer/docker-3.x/

I am not able to reproduce your problem. But I think you are missing a configuration volume, which is the only way to enable unmanaged extensions in the docker volume.

This is mentioned in the graphaware docs here: https://github.com/graphaware/neo4j-framework/

When using Neo4j in the standalone server mode, deploying the GraphAware Framework (and any code using it) is a matter of :

• downloading the appropriate .jar files
• copying them into the plugins directory in your Neo4j installation
• adding dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware to neo4j.conf
• restarting the server

Working example

Here are my files:

.
├── conf
│   └── neo4j.conf
└── plugins
    ├── graphaware-server-community-all-3.0.1.38.jar
    └── graphaware-timetree-3.0.1.38.24.jar

The contents of the conf file is (notice that very last line):

#*****************************************************************
# Neo4j configuration
#*****************************************************************

# The name of the database to mount
#dbms.active_database=graph.db

# Paths of directories in the installation.
#dbms.directories.data=data
#dbms.directories.plugins=plugins
#dbms.directories.certificates=certificates

# This setting constrains all `LOAD CSV` import files to be under the `import` directory. Remove or uncomment it to
# allow files to be loaded from anywhere in filesystem; this introduces possible security problems. See the `LOAD CSV`
# section of the manual for details.
dbms.directories.import=import

# Whether requests to Neo4j are authenticated.
# To disable authentication, uncomment this line
dbms.security.auth_enabled=false

# Enable this to be able to upgrade a store from an older version.
#dbms.allow_format_migration=true

# The amount of memory to use for mapping the store files, in bytes (or
# kilobytes with the 'k' suffix, megabytes with 'm' and gigabytes with 'g').
# If Neo4j is running on a dedicated server, then it is generally recommended
# to leave about 2-4 gigabytes for the operating system, give the JVM enough
# heap to hold all your transaction state and query context, and then leave the
# rest for the page cache.
# The default page cache memory assumes the machine is dedicated to running
# Neo4j, and is heuristically set to 50% of RAM minus the max Java heap size.
dbms.memory.pagecache.size=512M

#*****************************************************************
# Network connector configuration
#*****************************************************************

# Bolt connector
dbms.connector.bolt.type=BOLT
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=OPTIONAL
# To have Bolt accept non-local connections, uncomment this line
dbms.connector.bolt.address=0.0.0.0:7687

# HTTP Connector
dbms.connector.http.type=HTTP
dbms.connector.http.enabled=true
#dbms.connector.http.encryption=NONE
# To have HTTP accept non-local connections, uncomment this line
dbms.connector.http.address=0.0.0.0:7474

# HTTPS Connector
dbms.connector.https.type=HTTP
dbms.connector.https.enabled=true
dbms.connector.https.encryption=TLS
dbms.connector.https.address=0.0.0.0:7473

# Number of Neo4j worker threads.
#dbms.threads.worker_count=

#*****************************************************************
# Logging configuration
#*****************************************************************

# To enable HTTP logging, uncomment this line
#dbms.logs.http.enabled=true

# To enable GC Logging, uncomment this line
#dbms.logs.gc.enabled=true

# GC Logging Options
# see http://docs.oracle.com/cd/E19957-01/819-0084-10/pt_tuningjava.html#wp57013 for more information.
#dbms.logs.gc.options=-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution

# Number of GC logs to keep.
#dbms.logs.gc.rotation.keep_number=5

# Size of each GC log that is kept.
#dbms.logs.gc.rotation.size=20m

# Size threshold for rotation of the debug log. If set to zero then no rotation will occur. Accepts a binary suffix "k",
# "m" or "g".
#dbms.logs.debug.rotation.size=20m

# Maximum number of history files for the internal log.
#dbms.logs.debug.rotation.keep_number=7

#*****************************************************************
# Miscellaneous configuration
#*****************************************************************

# Enable this to specify a parser other than the default one.
#cypher.default_language_version=3.0

# Determines if Cypher will allow using file URLs when loading data using
# `LOAD CSV`. Setting this value to `false` will cause Neo4j to fail `LOAD CSV`
# clauses that load data from the file system.
#dbms.security.allow_csv_import_from_file_urls=true

# Retention policy for transaction logs needed to perform recovery and backups.
dbms.tx_log.rotation.retention_policy=100M size

# Enable a remote shell server which Neo4j Shell clients can log in to.
#dbms.shell.enabled=true
# The network interface IP the shell will listen on (use 0.0.0.0 for all interfaces).
#dbms.shell.host=127.0.0.1
# The port the shell will listen on, default is 1337.
#dbms.shell.port=1337

# Only allow read operations from this Neo4j instance. This mode still requires
# write access to the directory for lock purposes.
#dbms.read_only=false

# Comma separated list of JAX-RS packages containing JAX-RS resources, one
# package name for each mountpoint. The listed package names will be loaded
# under the mountpoints specified. Uncomment this line to mount the
# org.neo4j.examples.server.unmanaged.HelloWorldResource.java from
# neo4j-server-examples under /examples/unmanaged, resulting in a final URL of
# http://localhost:7474/examples/unmanaged/helloworld/{nodeId}
#dbms.unmanaged_extension_classes=org.neo4j.examples.server.unmanaged=/examples/unmanaged
dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware

Result

Now, the following docker invocation works:

docker run --rm --name neo4jtest -p 7474:7474 -v $(pwd)/plugins:/plugins -v $(pwd)/conf:/conf neo4j:3.0

Which outputs:

Starting Neo4j.
2016-06-14 09:09:16.780+0000 INFO  No SSL certificate found, generating a self-signed certificate..
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2016-06-14 09:09:17.461+0000 INFO  Starting...
2016-06-14 09:09:18.032+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2016-06-14 09:09:23.768+0000 INFO  Started.
2016-06-14 09:09:23.914+0000 INFO  Mounted unmanaged extension [com.graphaware.server] at [/graphaware]
2016-06-14 09:09:26.364+0000 INFO  Remote interface available at http://0.0.0.0:7474/

It works, as can be seen by the error message in this call:

$ curl -v localhost:7474/graphaware/timetree/single/0
*   Trying ::1...
* Connected to localhost (::1) port 7474 (#0)
> GET /graphaware/timetree/single/0 HTTP/1.1
> Host: localhost:7474
> User-Agent: curl/7.48.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Tue, 14 Jun 2016 09:11:25 GMT
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Server: Jetty(9.2.9.v20150224)
< 
* Connection #0 to host localhost left intact
{"message":"There is no time instant for time 0"}

However, notice that the curl call you made still does not list the extension:

$ curl -v localhost:7474/db/data/
*   Trying ::1...
* Connected to localhost (::1) port 7474 (#0)
> GET /db/data/ HTTP/1.1
> Host: localhost:7474
> User-Agent: curl/7.48.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 14 Jun 2016 09:12:09 GMT
< Content-Type: application/json; charset=UTF-8
< Access-Control-Allow-Origin: *
< Content-Length: 795
< Server: Jetty(9.2.9.v20150224)
< 
{
  "extensions" : { },
  "node" : "http://localhost:7474/db/data/node",
  "relationship" : "http://localhost:7474/db/data/relationship",
  "node_index" : "http://localhost:7474/db/data/index/node",
  "relationship_index" : "http://localhost:7474/db/data/index/relationship",
  "extensions_info" : "http://localhost:7474/db/data/ext",
  "relationship_types" : "http://localhost:7474/db/data/relationship/types",
  "batch" : "http://localhost:7474/db/data/batch",
  "cypher" : "http://localhost:7474/db/data/cypher",
  "indexes" : "http://localhost:7474/db/data/schema/index",
  "constraints" : "http://localhost:7474/db/data/schema/constraint",
  "transaction" : "http://localhost:7474/db/data/transaction",
  "node_labels" : "http://localhost:7474/db/data/labels",
  "neo4j_version" : "3.0.2"
* Connection #0 to host localhost left intact
}

[znorris]

Thank you for breaking it down @spacecowboy. I was missing the neo4j.conf line: dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware

This step isn't listed on the plugin installation steps for TimeTree or Neo4j Spatial. TimeTree requires the line you have given me or I don't see it loaded on start. On the other hand, Spatial appears to load as long as it's in the plugins folder. As I play around with different plugins is there a more intelligent way for me to discern the proper way to load plugins?

[spacecowboy]

I agree that the docs seem a bit sparse on this. I had trouble finding the relevant sections in our new manual (this will need correcting), but the sections in the 2.3 manual were fairly informative (see links below).

Neo4jSpatial is a ServerPlugin, whereas TimeTree is an Unmanaged Extension. ServerPlugins are loaded automatically, and made available under /ext, which can be seen by dropping the Neo4jSpatial jar into the plugins directory (and not changing any configuration):

$ curl -v localhost:7474/db/data/
*   Trying ::1...
* Connected to localhost (::1) port 7474 (#0)
> GET /db/data/ HTTP/1.1
> Host: localhost:7474
> User-Agent: curl/7.48.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 15 Jun 2016 12:19:13 GMT
< Content-Type: application/json; charset=UTF-8
< Access-Control-Allow-Origin: *
< Content-Length: 2140
< Server: Jetty(9.2.9.v20150224)
< 
{
  "extensions" : {
    "SpatialPlugin" : {
      "addSimplePointLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addSimplePointLayer",
      "addNodesToLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addNodesToLayer",
      "findClosestGeometries" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/findClosestGeometries",
      "addGeometryWKTToLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addGeometryWKTToLayer",
      "findGeometriesWithinDistance" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/findGeometriesWithinDistance",
      "addEditableLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addEditableLayer",
      "addNodeToLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addNodeToLayer",
      "addCQLDynamicLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addCQLDynamicLayer",
      "getLayer" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/getLayer",
      "findGeometriesInBBox" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/findGeometriesInBBox",
      "updateGeometryFromWKT" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/updateGeometryFromWKT",
      "findGeometriesIntersectingBBox" : "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/findGeometriesIntersectingBBox"
    }
  },
  "node" : "http://localhost:7474/db/data/node",
  "relationship" : "http://localhost:7474/db/data/relationship",
  "node_index" : "http://localhost:7474/db/data/index/node",
  "relationship_index" : "http://localhost:7474/db/data/index/relationship",
  "extensions_info" : "http://localhost:7474/db/data/ext",
  "relationship_types" : "http://localhost:7474/db/data/relationship/types",
  "batch" : "http://localhost:7474/db/data/batch",
  "cypher" : "http://localhost:7474/db/data/cypher",
  "indexes" : "http://localhost:7474/db/data/schema/index",
  "constraints" : "http://localhost:7474/db/data/schema/constraint",
  "transaction" : "http://localhost:7474/db/data/transaction",
  "node_labels" : "http://localhost:7474/db/data/labels",
  "neo4j_version" : "3.0.2"
* Connection #0 to host localhost left intact
}

Unmanaged extensions on the other hand require manual mounting via the config file, or neo4j does not know to look for them, as mentioned in the old manual:

Having built your code, the resulting jar file (and any custom dependencies) should be placed in the $NEO4J_SERVER_HOME/plugins directory. We also need to tell Neo4j where to look for the extension by adding some configuration in the conf/neo4j-server.properties file:

Comma separated list of JAXRS packages containing JAXRS Resource, one package name for each mountpoint.

org.neo4j.server.thirdparty_jaxrs_classes=org.neo4j.examples.server.unmanaged=/examples/unmanaged

My only recommendation is to verify if the plugins you are using are ServerPlugins or Unmanaged Extensions. Ideally, the plugins' respective readme files should mention this.

Closing now as it's not a docker related issue. I opened an issue against TimeTree to mention the required configuration in their readme.