Merge with component repos

[khernyo]

This issue is a proposal for merging the hypercore repo with its dependency libraries from the datrs project (and keeping them as separate crates if necessary). The merged dependency libraries would be the following:

• flat-tree
• memory-pager
• merkle-tree-stream
• random-access-disk
• random-access-memory
• random-access-storage
• sleep-parser
• sparse-bitfield
• tree-index

There are a couple of benefits in doing this and not a lot of downsides that I know of. In short, it would facilitate faster development with fewer breakages and easier understanding of the whole project.

On the plus side:

• it would be trivial to check whether a change breaks any of the dependent libs. Currently, you need to find which are these libraries, change their Cargo.toml to use your locally modified library and run the tests. With this proposal, it would be a matter of running cargo test in the repo root.
• rewrites touching several libs could be done in one single commit (e.g., moving a function from one library to another)
• dependency versions would be consistent for all included libraries/crates (one Crate.lock file in the repo root)

As for the disadvantages, these are what I could think of:

• github issues of dependency libraries would be lost. These can be migrated, but the issue references in commits would point to the wrong issue.
• every push will start a Travis build for the whole project (hypercore and its dependencies). Note that this can also be regarded as a benefit because the Travis runtime is dominated by rustup, the compilation of clippy, rustfmt and other dependencies, and these would be done only once, not for each library.

All in all, I think it's a net positive.

Why do it in the hypercore repo? It seems to me that this is the first layer of the Dat Project which should be usable on its own. The dependencies of this project are implementation details. Some of those could be usable on their own, so it might be worthwhile to keep them as separate crates. And I guess the least controversial move is to keep them as separate crates.

To keep the dependencies as separate crates, the workspace feature of cargo can be used: https://github.com/rust-lang/rfcs/blob/master/text/1525-cargo-workspace.md. Here is a project using this feature: https://github.com/gfx-rs/gfx

What do you think?

If you think it might be worthwhile, I can do the merge in a test repo where anyone can try how it would work before committing to this repo overhaul.

[yoshuawuyts]

Hey, I appreciate you opening this issue - but I much prefer working with individual GitHub repositories. This allows scoping issues without needing to manually label them all.

We could of course go down the path of automation for labelling, but with that we'd have to maintain infrastructure & probably code up a bunch of tools.

All of this will end up taking time, which is something that is currently the constraint (only have 12 more weeks left on my grant). If possible I'd like to focus on getting Hypercore working, and I think the current setup is good enough to achieve that.

That said, I do think workspaces are useful. I'm thinking of bringing the WASM version of Hypercore + the C version of Hypercore into the same repository as Hypercore using workspaces. That should hopefully allow us to keep concerns scoped, and ensure no breaking changes occur between the (tightly) related modules.

Thanks again for bringing this up, I hope my explanation here makes enough sense!

[khernyo]

Sure, thanks for taking the time to consider it!