search

dat-ecosystem-archive / book

Documentation on how to implement the Dat Protocol [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
https://datprotocol.github.io/book/
Apache License 2.0
68 stars 9 forks source link

Need more basic explanations and examples for bitfield #1

Open ZhouHansen opened 6 years ago

ZhouHansen commented 6 years ago

These are some 🔦questions:

I red up bitfield, sparse-bitfield though. I've still struggled to figure out the bitfield module for few days. The source of indexed-field and hypercore/bitfield are both hard to read. And sleep.md/bitfiled is similar to this document. So I decide to ask questions now.

  1. Is this module for sparse file, or something similar to it？
  2. In Types of Bitfields "Data Bitfield: Indicates which data you have, and which data you don't. " What does 'the data you have' means? Does this means data entries I've already downloaded, then when update coming, I don't need to download them again? I don't know. Hypercore is a append-only log. So if there are 3 entries in metadata.data, the Data Bitfield will be 111000000?

I want to figure out this one first.

yoshuawuyts commented 6 years ago

What does 'the data you have' means? Does this means data entries I've already downloaded, then when update coming, I don't need to download them again?

Yeah, that covers the gist of it! -- because Hypercore is append-only, data can be tracked by just flipping a bit, and once it's flipped we don't need to download that piece of data ever again. sparse-bitfield is mostly an efficient way of storing that data on disk.

So if there are 3 entries in metadata.data, the Data Bitfield will be 111000000?

The data is stored in a tree format, so it would be represented differently on disk. But it seems you've got the right intuition about it!

I'll be spending some time after this month to write more docs. Questions like these are super helpful to know where we should focus more on writing docs. Please keep the questions coming! :tada:

ZhouHansen commented 6 years ago

From bitfield.md

Example

bits:  00101
index: 01234

The set above contains 2 and 4.

Hypercore is append-only, It seems when the previous bit is 0 then the next rest of bits should all be 0, so why this happens: the 0 and 1 is 0, but 2 is 1.

yoshuawuyts commented 6 years ago

Oh yeah, so while Hypercore is append-only, it also supports sparse replication. So:

The bitfield represents the data that you have stored on your machine. Because we can do sparse copies (e.g. only copy a few entries in hypercore, or just one file in Hyperdrive) it must be able to represent that in its data structure, which means interior mutability.

Does that make sense?

ZhouHansen commented 6 years ago

Yeah, that's what I want to make sure. Thanks

ZhouHansen commented 6 years ago

I'm learning hypercore, I think SLEEP files prefixed by content are in charge of hypercore, files prefixed by metadata are in charge of hyperdrive. Right?

yoshuawuyts commented 5 years ago

I'm learning hypercore, I think SLEEP files prefixed by content are in charge of hypercore, files prefixed by metadata are in charge of hyperdrive. Right?

Not quite; they're both hypercore instances. One keeps track of the content, the other keeps track of the metadata. In newer version of hyperdrive, the metadata core is a hypercore that's formatted as a hyperdb.