Neo4j recovery

[noboruma]

After a crash (typically Java Heap OOM) or when the disk filesystem is saturated, neo4j DB is in inconsistent state. There is little we can achieve in that state as the DB is not accessible.

What was tried:

• Delete transactions folder from neo4j data volume: failed
• Use neo4j-admin check-consistency --database=neo4j: failed, asking to graceful shutdown
• start, stop, check, start (to gracefully shutdown once): failed, still asking for a graceful shutdown
• Take a snapshot for neo4j_data volume via docker & tar and uncompress: works

Given that the tooling around backup is only available in neo4j enterprise edition, we need to come up with our own strategy. We need to add a snapshot mechanism on neo4j so we can go back in stable state if something wrong happens.

[noboruma]

Added backup mechanism to neo4j: https://github.com/deepfence/ThreatMapper/commit/8d4b87da250fc5da5592cc7420f409eceaa64f87 Next step: See how we can use remote volumes in k8s

[noboruma]

For record, we can keep PV volumes by following: https://acv.engineering/posts/migrating-a-kubernetes-pvc/ For docker, we have utils to tar/untar the volume and do migration manually No more actions for this task, resolving.