Consensus reorganization of packages and migration

[jasagredo]

Since the beginning, the consensus layer of Cardano has co-existed with the network layer. However, as we progress towards a more Open-Source structure, it happens to be a good moment to part sides with the Network layer.

Also, Consensus grew organically into many packages that although they split nicely the functionality in “separate parts”, they become quite difficult to keep in sync. We made the decision to distribute those packages in bundles but that is just an artificial grouping (not completely artificial though) that Cabal doesn’t understand. Therefore problems started appearing as bundles were broken and supposedly impossible situations took place.

So now we took the decision to make a big reorganization and at the same time migrate the code to a new repository.

Structure

The package structure will now be as follows:

• ouroboros-consensus: Contains the core definitions of the Consensus Layer.

  • Concepts defined here
    • LedgerState definitions
    • Protocol typeclasses, BFT and PBFT
    • the Storage Layer
    • the Mempool implementation
    • the Hard Fork Combinator
    • MiniProtocol definitions
    • ChainSelection
  • Sublibraries and test-suites
    • consensus-testlib: what previously was the library of ouroboros-consensus-test
    • mock-block: definition of a simple mock Ledger for testing
    • tutorials: toy implementations of Block definitions that implement the necessary typeclasses defined by the Consensus library
    • consensus-test: what previously was the test-consensus test-suite. Tests the Consensus core library
    • storage-test: what previously was the test-storage test-suite. Tests our ChainDB implementation
    • mempool-bench: a benchmark of the time it takes to add transactions to a mempool

• ouroboros-consensus-protocol: defines Praos and TPraos protocols which implement the Protocol instances from Consensus. It has a small test sublibrary.

• ouroboros-consensus-cardano: defines Blocks for Byron, Shelley and Cardano. In particular this has absorbed all the previous Cardano packages (ouroboros-consensus-(byron|shelley|cardano)) as well as their test packages, test-suites, ouroboros-consensus-cardano-tools.

• ouroboros-consensus-diffusion: defines the NodeKernel and the booting up sequence for the node, as well as configuration options. It also boots the Network miniprotocols so it is the glue between Consensus and Network. As ThreadNet tests require things from this library, they now live here in several test-suites.

The graph of dependencies for the implementation packages is as follows:

• Cardano -> Protocol
• Cardano -> Consensus
• Protocol -> Consensus
• Diffusion -> Consensus

This reorganization will impact downstream clients and there will be some time of adaptation.

Steps of the migration

1. Merge the whole history of the old repo into this one.
2. Delete the unneeded packages.
3. Do the reorganization
4. Set up side services, I will list a few to not forget them
   1. CI
   2. Nix
   3. PR checks
   4. Code owners
   5. Branch protection
   6. Docs website
   7. Haddock website
   8. Readme
5. Migrate the issues (discussion needed)
6. Migrate PRs (discussion needed)
7. Remove the packages from the old repository
8. Update downstream clients

[nfrisby]

The following are my notes from the discussion we had about migrating the issues. We also thought to ping MPJ and @abailly-iohk for their thoughts on this issue.

• You can click "Transfer issue" to move an issue from one repo to another. The gh exe also lets you do so programmatically: https://fig.io/manual/gh/issue/transfer

• Using that Transfer feature results in several nice behaviors:

  • redirects, eg https://github.com/input-output-hk/ouroboros-network/issues/2608 redirects to https://github.com/input-output-hk/io-sim/issues/29
  • all references in GitHub metadata (eg Issue/PR descriptions, Issue/PR comments, etc) in both repositories are seamlessly updated. (I just now did an experiment with two one-off personal repositories.)
  • (my anticipated mental model is that eg ouroboros-network#2608 is now internally stored as a pointer to io-sim#29, so eg GH can show the most up-to-date thing at render-time in all of its pages/apps/etc. But the raw commit messages don't change, of course)

The only downside is that occurrences of old issue numbers outside of GitHub metadata cannot be automatically updated.

• A full link "https://github.com/input-output-hk/ouroboros-network/issues/2608" or even "input-output-hk/ouroboros-network#2608" would be ultimately fine and unambiguous
• But just "#2608" will be confusing: either input-output-hk/ouroboros-consensus#2608 doesn't exist, or some day it will but it'll be some entirely unrelated issue (!)
• We could do a sed on eg our code, for example, but there's probably plenty (not the majority, but still plenty) of "#XXX" strings without the ouroboros-network prefix in the wild old: Slack messages, personal text files, Google Docs, Jira? :grimacing:, etc

The "perfect" scenario would be:

• Find the list of issues we want to transfer
• For each such issue in ascending order, create enough dummy issues in the new repository so that the transferred issue will have the same issue number as the existing issue (https://fig.io/manual/gh/issue/create)
• Then transfer the issue
• (Optional: add a new comment on the issue, stating what its old issue number in ouroboros-network was. https://fig.io/manual/gh/issue/comment)
• Repeat
• Then delete all of the dummy issues (https://fig.io/manual/gh/issue/delete) (FYI even if you delete the youngest issue, the next one you create will have a universally unique (ie incremented) issue number.)

The only remaining risk of ambiguity would then be among issues created after the migration: Networking will eventually create a Issue numbered 5000 and so will Consensus too. But that is inevitable (unless we artificially inflate our counts to start at 100000 for new issues etc :grimacing:)

I'd say the biggest risk of that "perfect" plan is if our list of issues to transfer was incomplete. If we missed one during the original attempt, then when we later Transfer it, there's now way for it to end up having the same number in the new repository (because we already "consumed" that number with a "dummy" issue).

[nfrisby]

I think migrating PRs is less of a concern; there aren't as many of them.

We probably do have commit messages that refer to Issue and/or PR "#30" for example. But there's nothing we can do about that.... except create a dummy Issues (from my above comment) that link to the corresponding ouroboros-network Issue or PR instead of just being deleted.

[jasagredo]

These are the two PRs that implement this migration:

• https://github.com/input-output-hk/ouroboros-network/pull/4522

• 34

[jasagredo]

I'm closing this as complete. We can always refer back to it.