equinor / helm-charts

Repository of helm-charts published on hub.helm.sh
MIT License
1 stars 14 forks source link

Standalone pod erroring out with lock issue in the very first run with a clean volume #8

Closed chprasad closed 3 years ago

chprasad commented 3 years ago

Very first helm install of the neo4j-community chart with a clean volume is resulting in: Unable to obtain lock on file: /data/databases/store_lock issue.

Directories in use:
home: /var/lib/neo4j
config: /var/lib/neo4j/conf
logs: /logs
plugins: /plugins
import: /var/lib/neo4j/import
data: /var/lib/neo4j/data
certificates: /var/lib/neo4j/certificates
run: /var/lib/neo4j/run
Starting Neo4j.
2021-06-18 14:22:18.148+0000 INFO Starting...
2021-06-18 14:22:21.255+0000 INFO ======== Neo4j 4.2.4 ========
2021-06-18 14:22:32.893+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address.
java.lang.RuntimeException: Error starting Neo4j database server at /var/lib/neo4j/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:207) ~[neo4j-4.2.4.jar:4.2.4]
...
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.locker.LockerLifecycleAdapter@17ca8b92' was successfully initialized, but failed to start. Please see the attached cause exception "No locks available".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463) ~[neo4j-common-4.2.4.jar:4.2.4]
... 5 more
Caused by: org.neo4j.kernel.internal.locker.FileLockException: Unable to obtain lock on file: /var/lib/neo4j/data/databases/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
at org.neo4j.kernel.internal.locker.Locker.storeLockException(Locker.java:175) ~[neo4j-kernel-4.2.4.jar:4.2.4]

Here are the versions in my environment:

Kubernetes v1.19 Helm v3.6.0 kubeadm 1.19.7 Nodes running Ubuntu 20.04.1 LTS

I suspect neo4j is restarting for some reason and as reported in https://github.com/neo4j/docker-neo4j/issues/289, neo4j is not clearing the lock from previous start and failing.

Have you experienced a similar issue anytime? Any ideas why an auto-restart may be happening? I'm seeing the same issue with the helm chart from neo4j-contrib.

chprasad commented 3 years ago

https://github.com/neo4j/neo4j/issues/10625 provided the answer. I was using a NFS mount. Setting the nolock option on the mount fixed the issue.