Split Dredd into multiple projects

[honzajavorek]

For Dredd it would make sense to split into multiple projects:

• dredd - command line tool
• dredd-core - the core functionality of Dredd, without the command-line interface
• dredd-hooks - support for hooks, communication with hook handlers
• dredd-transactions - parses API description and outputs a list of HTTP transactions to execute

This would make easier to use Dredd also in other environments than CLI (imagine all kinds of integrations, such as the one @XVincentX did for Visual Studio Code).

[honzajavorek]

When this happens, investigate whether lerna wouldn't help with managing. (Thanks @XVincentX for the tip.)

[w-vi]

When you say multiple projects, do you mean multiple repos?

[honzajavorek]

That is an implementation detail to be decided. I mean multiple packages in the first place.

[honzajavorek]

I'm going to make this a ticket tracking our "monorepo" efforts, which are currently under-documented and scattered all over our repositories.

I think the initial ideas were formulated around this conversation https://github.com/apiaryio/dredd/issues/600#issuecomment-324887713, then it was further discussed in Apiary internally (Slack link to the initial message, for employees) and also externally with the original Dredd author @netmilk. This issue was created to make the intention public, but I'm not sure it sums up all the reasoning sufficiently. Issues https://github.com/apiaryio/gavel.js/issues/126 and https://github.com/apiaryio/dredd-transactions/issues/252 were created, but unlinked from any other mentions of the decisions behind. There are additional mentions of the "future monorepo" in https://github.com/apiaryio/dredd-transactions/issues/105#issuecomment-346317514 or in https://github.com/apiaryio/dredd/issues/1186. The v7 of Dredd Transactions is one of the first steps in the effort, splitting parsing and compilation into two distinct modules (packages, if you want).

Reasons for the monorepo and additional notes

• The split between dredd and dredd-transactions projects makes the development more difficult. To have those two under one roof removes overhead for both the maintainers team and contributors.
• The dredd-transactions project itself didn't start with clear interfaces. When it was refactored to have clear boundaries, it actually ended up to be two things - API description parser, and transactions compiler. This makes it at least three libraries in the monorepo scenario: dredd, dredd-parse, dredd-compile
• The dredd project itself is getting harder to maintain over time, as it consists of a series of monolithic classes entangled together. The intention of the maintainers was to find clear boundaries between various areas of code, to document the boundaries, and to give names to the areas: dredd (CLI), dredd-core, dredd-hooks, ... While this can definitely happen only on a level of separate folders with well-specified interfaces, having a monorepo in place could accelerate the impacts and force both maintainers and contributors to keep the areas separated and the interfaces properly documented, with the benefit of having these as packages somebody can use and build on top of.
• People find their way to Gavel almost exclusively through Dredd. While Gavel was designed as a standalone tool, it is effectively being used "just" as a core of Dredd's validation for most people outside of Apiary. Bugs in Gavel are almost always being filed as bugs for Dredd. It is annoying overhead for both users and maintainers to have several levels of issue trackers. API Elements monorepo proved that one umbrella issue tracker can work better in such cases. Contributors to Dredd also won't need to fork multiple projects, search multiple projects, npm link multiple projects, etc.
• Gavel behavior is fully inherited by Dredd, so to describe in docs how Dredd behaves one needs to describe how Gavel behaves. Doing it at two different places doesn't seem efficient given the size of the team taking responsibility of both Dredd and Gavel development.
• While dropping the Gavel name and rebranding it to dredd-validate or something like that might be way too radical, I think we agree that having Gavel still as a gavel package, but managed within the monorepo doesn't hurt anyone's feelings and at the same time it makes everyone's life easier. Dredd monorepo was always considered with the intent that all the individual packages are reusable LEGO pieces published to npm, so having Gavel there fits the overall design.
• With the flexibility CircleCI currently offers there should be no significant blocker in having an elaborate (but efficient and parallel) CI build testing of all the included projects, and a separate set of integration tests (perhaps based on Cucumber).
• While Dredd offers docs infrastructure, its smaller friends do not. Having interfaces documented in random READMEs is not nice, while having (and maintaining) full docs infrastructure for each small repository is not efficient. Docs should be ideally just at one place, for maintainers to work on, and for readers to search for stuff.
• Dependencies and other stuff can be tracked under one roof as well. There's now multiple levels of release train, and there are projects which use other projects in their test suites (e.g. gavel-spec is used in gavel, dredd-example is used in dredd, etc.) Again, dropping overhead, making synchronization simpler.
• Granularity of Oracle's open-sourcing policies generally ends on a repository level. Adding code to a (monorepo) repository which already exists, has a license, etc., is something which doesn't trigger any notable internal processes. Open-sourcing a brand new repository does.
• I personally wouldn't include the gavel2html project into the monorepo until it is used within Dredd. So far it renders diffs for Apiary. If we ever use it for e.g. rendering Dredd's console output or Dredd's HTML reporter, I don't see why it should be in the monorepo.

Plan

• [x] split dredd-transactions to two subprojects
• [x] make Dredd's tests passing on CircleCI and migrate to CircleCI for flexibility in CI setup
• [x] turn Dredd into a monorepo, perhaps with yarn etc. (get inspired by the work done on API Elements, ask around in @apiaryio/adt for help)
• [ ] add dredd-transactions as two separate packages
• [ ] have a concept of tests specific to a certain package and ran in parallel in CI, and monorepo-wide integration/e2e tests (current integration tests of Dredd or Dredd Transactions)
• [ ] add Gavel (gavel and gavel-spec)
• [ ] unify docs - put all docs into Dredd's docs, either as user-facing docs or as internals
• [ ] migrate all issues to the Dredd repository
• [ ] consider other candidate packages (e.g. dredd-docker, dredd-hooks-template) - note: do NOT include dredd-example, which should stay as a separate repo mainly for discoverability and to isolate the showcase of CI setups from the rest of Dredd
• [ ] split Dredd itself into separate packages, such as dredd, dredd-core, dredd-hooks