DB cannot be started - Githubissues

den1k commented 9 months ago

Restarted the JVM and suddenly can't create a conn

(def d (d/create-conn (db-dir)
                      schema
                      {:auto-entity-time? true
                       :validate-data?    env/dev?}))
Execution error (NullPointerException) at datalevin.storage.Store/init_max_eid (storage.cljc:329).
Cannot read field "e"

huahaiy commented 9 months ago

The DB file looks corrupted. We could put in a JVM shutdown hook to close DB, but it won't help if the machine is shutdown or the process is killed unexpectedly. So, 0.9.0 will default to the safe write mode to prevent this from happening. We will also add a JVM shutdown hook in the next release to reduce the chance.

andersmurphy commented 7 months ago

In the meantime I believe you can turn on safe write mode for your application by passing :flags as [:nordahead :writemap :notls] (note: this will mean sync flushes to disk, so no concurrent inserts/updates):

(defonce !conn
     (d/get-conn "db" schema
       {:kv-opts {:flags [:nordahead :writemap :notls]}}))

The default flags are:

https://github.com/juji-io/datalevin/blob/f7b10d4bb6ba1c6522e2c4aed36a32c102016604/src/datalevin/constants.cljc#L45

The best documentation I have found on these flags is here:

https://github.com/lmdbjava/lmdbjava/blob/9b65ffdf142278fd7da4fc18db8e86c8174bb4eb/src/main/java/org/lmdbjava/EnvFlags.java#L69C1-L69C1

I believe it's :mapasync that can cause database corruption on crash.

  /**
   * Use asynchronous msync when {@link #MDB_WRITEMAP} is used.
   *
   * <p>
   * When using {@link #MDB_WRITEMAP}, use asynchronous flushes to disk.
   * As with {@link #MDB_NOSYNC}, a system crash can then corrupt the database
   * or lose the last transactions. Calling {@link Env#sync(boolean)} ensures
   * on-disk database integrity until next commit.
   */
  MDB_MAPASYNC(0x10_0000),

Hope that helps.

juji-io / datalevin

DB cannot be started #228