search

attic-labs / noms

The versioned, forkable, syncable database
Apache License 2.0
7.45k stars 267 forks source link

Does noms has any interface I can use to replace NBS with levelDB? #3839

Closed poppindouble closed 5 years ago

poppindouble commented 5 years ago

Hi: I am really new to noms and also I am really new to go lang as well, but I really like the idea of noms :)

I just dive into the code for few days, I have several questions:

  1. The documentation mentions that now NBS is the default config for local storage, is there any interface I can switch NBS with LevelDB or maybe other KV storage engine like rocksDB, redis?

  2. I know LevelDB is using the idea of LSM which might not be a good fit for noms scenario, the documentation also mentions that noms is using Prolly Trees to deal with the Blobs, sets, lists, and maps. Is this also the reason behind NBS is better than levelDB? I might be a bit confusing with the relation between NBS and Prolly Trees.

Sorry if I ask some stupid questions, but this project really attracts me, I wanna get familiar with the code as soon as possible. :)

Thanks

aboodman commented 5 years ago

On Mon, Mar 4, 2019 at 12:45 PM Shuangshuang Zhao notifications@github.com wrote:

Hi: I am really new to noms and also I am really new to go lang as well, but I really like the idea of noms :)

I just dive into the code for few days, I have several questions:

1.

The documentation mentions that now NBS is the default config for local storage, is there any interface I can switch NBS with LevelDB or maybe other KV storage engine like rocksDB, redis?

Yeah. That interface is ChunkStore https://github.com/attic-labs/noms/blob/master/go/chunks/chunk_store.go. You can see where the current implementations are hooked in here https://github.com/attic-labs/noms/blob/master/go/spec/spec.go#L186.

We used to have a LevelDB implementation, but it was removed https://github.com/attic-labs/noms/pull/3193 because it had no advantages over the NBS one. So my question would be: why do you think you want an ldb impl?

  1. I know LevelDB is using the idea of LSM which might not be a good fit for noms scenario, the documentation also mentions that noms is using Prolly Trees to deal with the Blobs, sets, lists, and maps. Is this also the reason behind NBS is better than levelDB? I might be a bit confusing with the relation between NBS and Prolly Trees.

The storage layer (where NBS/LDB/etc sit) knows nothing about prolly trees.

LDB works fine for Noms, and we used it for a couple years. The reason we replaced it was because LevelDB provides features (and incurs cost for) features that we don't use.

Namely: LDB (and all other key/value stores) support mutation. The key/value pairs are mutable. Noms does not require this feature. In Noms, a key can only ever have a single value. Eliminating mutation allows the design to be much simpler, and faster as a result.

1.

Sorry if I ask some stupid questions, but this project really attracts me, I wanna get familiar with the code as soon as possible. :)

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/attic-labs/noms/issues/3839, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE6BJbcHXTdvkoFVWQX-4T41-IWPWzLks5vTYX7gaJpZM4bdLF3 .

poppindouble commented 5 years ago

On Mon, Mar 4, 2019 at 12:45 PM Shuangshuang Zhao @.***> wrote: Hi: I am really new to noms and also I am really new to go lang as well, but I really like the idea of noms :) I just dive into the code for few days, I have several questions: 1. The documentation mentions that now NBS is the default config for local storage, is there any interface I can switch NBS with LevelDB or maybe other KV storage engine like rocksDB, redis? Yeah. That interface is ChunkStore https://github.com/attic-labs/noms/blob/master/go/chunks/chunk_store.go. You can see where the current implementations are hooked in here https://github.com/attic-labs/noms/blob/master/go/spec/spec.go#L186. We used to have a LevelDB implementation, but it was removed <#3193> because it had no advantages over the NBS one. So my question would be: why do you think you want an ldb impl?

I am just curious haha, I know ldb is not necessary, and there must be some disadvantage about levelDB in this scenario, but in the early stage of db project, it might be a bit costly to directly implement your own storage engine, so I was thinking you guys must used some mature storage engine, and I wanna have a look about how you design the interface. Thanks!

  1. I know LevelDB is using the idea of LSM which might not be a good fit for noms scenario, the documentation also mentions that noms is using Prolly Trees to deal with the Blobs, sets, lists, and maps. Is this also the reason behind NBS is better than levelDB? I might be a bit confusing with the relation between NBS and Prolly Trees. The storage layer (where NBS/LDB/etc sit) knows nothing about prolly trees. LDB works fine for Noms, and we used it for a couple years. The reason we replaced it was because LevelDB provides features (and incurs cost for) features that we don't use. Namely: LDB (and all other key/value stores) support mutation. The key/value pairs are mutable. Noms does not require this feature. In Noms, a key can only ever have a single value. Eliminating mutation allows the design to be much simpler, and faster as a result.