lay out general structure

[yoshuawuyts]

Added a general structure to the book, and laid some base files that can be filled in later. This is in line with the general plan that was sketched out here: https://paper.dropbox.com/doc/2018-10-24-Dat-Docs--AQdGf4QRETwxAsBb2bgBTlZ9AQ-1J9bx3D9XTgiArGIRMiuC (posted on IRC).

I don't want to commit to this structure too much quite yet, as it's still pretty early and I'm trying to get a feel for what works well.

That said, I'm planning to draft the following sections:

• Introduction: explain what this guide is, who it is for, and which sections we have. Also where to contribute, and how to contribute.
• Key Concepts: explain the core concepts that are part of Dat. This is mostly the why and what of Hypercore so people get a feeling for what is what.
• Implementation Guide: this is the reference level implementation of the guide. This is where we explain which modules need to be built, and which functionality/methods they should have. The intent here is so when someone wants to implement Hypercore in the language of their choice, they'll have a clear implementation path ahead of them, and a sense of what each module does. A stretch goal here is to also include reference test cases.

I hope this all makes sense! cc/ @vtduncan keen to hear your thoughts on this general direction.

[ghost]

Hey @yoshuawuyts, this is looking good! Here are some thoughts:

I think the Implementation Guide is going to be the most important part for readers and would be more readable if key concepts were introduced along the way instead of their own section. This helps it adhere to the book format where you can read it from start to end, rather than a wiki where the reader has to piece together what fits where until finally it all comes together in the final step.

An approach I want to try with the Implementation Guide is to get to the juicy part as quickly as possible then stay there. For Dat I think the juicy parts are:

• Given a Dat URL, how do I get the files in it?
• Given a bunch of files I want to share, how can I make a Dat URL that others can get?

(My current understanding is that Dat can do way more than just files, but I think this would be a strong enough kickstart for implementers that more exotic ways of using the protocol would be within reach.)

So the idea is rather than telling implementers what modules they should write, instead show them how to do something interesting with the protocol. Then the implementer can decide how to structure the modules (which implementers tend to be good at themselves).

Anyway I think it’s totally OK to change the structure as we figure out more concrete details and this gives us a great starting point. Thank you for writing it!

[yoshuawuyts]

@vtduncan yay, thanks for your reply!

Hmm, a big constraint here for me is that I haven't re-implemented all of the protocol in Rust yet. I've only really touched the core protocol (hypercore), and not even gotten to the networking part of it. While that might not sound as if it's much, there's quite some gotchas in the process.

I think my angle here is that I'm thinking of making the guide that I wish I had when implementing Hypercore; removing the need to reverse-engineer prior implementations. But perhaps this is an artifact of how I approached implementing the protocol, and there's better ways of approaching this.

This seems like a different angle than which you had in mind, but I don't think they're necessarily at odds with each other! I agree it might be hard to consolidate all details up front, so instead maybe there's a few things we can agree on that need explaining, and then shuffle things around as things are written.

I think the following points would be super helpful to start explaining (and are also within my comfort zone, so I can start writing about these straight away):

1. We need a good explanation of how flat-trees work
2. We need a good explanation of the relationship between content, signatures and bitfield structures.
3. We need a thorough explanation of the 3 different bitfields, and how they are packed together into a single .bitfield file on disk.

I'll start on the flat-tree one I think first, and then work my way forward.

---

Apologies for the somewhat lengthy reply; I hope it sheds a bit more light on my perspectives here. Thanks heaps for taking your time to read through it!

[ghost]

Thanks for explaining this @yoshuawuyts! I understand your angle a lot more now, and I think it's a good one.

Really looking forward to reading your explanations of flat-trees and the three different bitfields. I'm definitely for shuffling things around as they are written, and seeing as you have just implemented a big part of the protocol it’s a great time to get started writing about it. Please never let me get between you an writing, we’ll be able to figure out the structure later!

Good stuff!

[yoshuawuyts]

Awesome; merging this then, and starting work on the next chapter!