Level / classic-level

An abstract-level database backed by LevelDB.
MIT License
58 stars 11 forks source link

How to make backups? #50

Closed siddjain closed 1 year ago

siddjain commented 1 year ago

Sorry if its a noob question but how do I make backups of the database?

vweevers commented 1 year ago

You have several options.

The first is the easiest and fastest: copy the whole directory on disk (close the database before you do). You may want to run await ClassicLevel.repair(location) or await db.compactRange(..) to trigger a compaction, either on the source database (beforehand) or the backup database (afterwards).

Second option, if you need to do it live:

const { ClassicLevel } = require('classic-level')
const { EntryStream } = require('level-read-stream')
const { pipeline } = require('readable-stream')
const WriteStream = require('level-ws')

const db1 = new ClassicLevel('./db', { keyEncoding: 'buffer', valueEncoding: 'buffer' })
const db2 = new ClassicLevel('./backup', { keyEncoding: 'buffer', valueEncoding: 'buffer' })

// Note, this will not capture data that's written afterwards
const src = new EntryStream(db1)
const dst = new WriteStream(db2)

await pipeline(src, dst)
console.log('done')

Third option, using iterators instead of streams:

const iterator = db1.iterator()

try {
  while (true) {
    // Read 100 entries at a time
    const entries = await iterator.nextv(100)

    if (entries.length === 0) {
      console.log('done')
      break
    }

    // Convert entries to batch operations
    const operations = entries.map(([key, value]) => ({ type: 'put', key, value }))

    // Note, you may want to parallelize reading & writing
    await db2.batch(operations)
  }
} finally {
  await iterator.close()
}