[Proposal] Development lifecycle

[szabo137]

With this issue, I would like to summarize the current development practices in the QEDjl-project and propose some enhancements regarding testing and release procedures.

Preliminaries

Git branching rules

• every package has two long-lived branches: 'main' and 'dev'
• 'main' contains only releases with a version tag
• 'dev' contains the current head of the development
• feature branches come from and are merged to dev (usually from a fork)
• releasing brings a collection of commits from dev onto main (see release workflow below)

Dependency rules

• main branches (i.e. released versions) only depend on other released versions from the general registry
• dev branches also depend only on released versions (not on other dev branches anymore)
• feature branches (and fixes, see below) may depend on dev branches during development, but must be updated using released versions before merging to dev

Testing workflows

Unit tests

• all PRs (for feature branches, fixes, releases, and hotfixes) must run the local unit tests with the versions of the dependencies given by the compat entries.
• Warnings should be thrown if feature branches test against non-released branches, e.g., devs.

Integration tests

• In a PR of a feature branch of a package, the unit tests for all dependent packages should be triggered with the changes made in the PR.
• Status before: To illustrate the workflow, let's assume two packages A and B, both registered as Av0.1 and Bv0.1. In our case, B depends on A.
• Using this setup, a new feature in A waits to be added. The following workflow must be applied:
  1. Checkout a branch featA from dev (usually on a fork), implement the feature and open PR:featA targeting A#dev.
  2. This triggers the unit tests of B against A#featA. If these unit tests pass, nothing needs to be done. PR:featA can be merged. If not, we have introduced a breaking change and need to proceed.
  3. Check out a branch fixB from B#dev adopting the changes introduced in featA, which will depend on A#featA. Open PR:fixB targeting B#dev.
  4. Inform PR:featA about the fix given on fixB and run the integration test (aka the unit test in B) again, but against the combination A#featA and B#fixB.
  5. Because we introduce a breaking change in A, the version of A in the Project.toml should be bumped to Av0.2-DEV, where -DEV indicates, that this version is not been released yet, but other PRs on A can refer to/be built on.
  6. Since the unit tests on PR:fixB are passed, the integration test in PR:featA should also pass. Therefore PR:featA can be merged. This breaks the integration tests again (dev must depend on released versions), which is not a problem, see release workflow.
• Status after: A#dev has a new head including the feature from featA. PR:fixB is still pending and can not be merged because its unit test failed (remember, feature branches must depend on released versions to be merged). However, this is also not a problem, because A will be eventually released as Av0.2, bringing the changes necessary for PR:fixB to a release, which means in fixB the compat entry for A can be bumped and PR:fixB can be merged.
• Extra: If there is another package C depending on A another fixC branch needs to be checked out on C following the same workflow. If C also depends on B, C#fixC needs to address this too, and PR:fixB needs to be informed about the fix given in C#fixC as well.

Release workflow

• The phrase release means following the workflow below bringing a certain commit from dev onto main, registering the new commit on main in General, tagging the new commit on main with a version number, and building a new GitHub release.
• We use the setting introduced above to illustrate the integration test pipeline
• Status before: The head of A#dev works together with B#fixB currently pending in PR:fixB. However, PR:fixB can not be merged, because its unit tests fail by still using Av0.1 (the current top release), or by using the compat entry on Av0.2 which is not released yet.
• We now want to release A to Av0.2 and must follow the release workflow:
  1. Tag all PRs that went into A#dev and are supposed to be part of the release and add them to a common milestone.
  2. create a release branch release-Av0.2
  3. remove -DEV additions in the Project.toml to the version of A
  4. update CHANGELOG.md
  5. open PR:release-Av0.2 targeting A#main. One review is needed to check the correctness of 1-4 of the release workflow.
  6. after merge: open another PR:release-Av0.2 targeting A#dev. Not much review needed here.
  7. register new head of A#main
  8. tag the new head of A#main with v0.2
  9. build a GitHub release
• Status after:
  • PR:fixB can now bump the compat entry to Av0.2 allowing it to be merged.
  • if a user does Pkg.add.(["A","B"]), the top compatible versions of A and B will be resolved, i.e. Av0.1 and Bv0.1, even if Av0.2 is already in the registry.
  • However, if Bv0.2 (including the new compat to Av0.2) is released eventually, the Pkg.add.(["A","B"]) will add Av0.2 and Bv0.2.

Documentation

• Docs should always be handled like unit tests, even if they don't contain jldoctests
• Docs must be built against released versions of their dependencies.
• This means, PRs can only be merged to dev, if all upstream packages used are released.

Final remarks and additional requirements

• We need good integration test tooling, which includes testing and reporting of fixed down the whole dependency graph. This also includes informing a PR about multiple fixes in other packages at the same time (see the integration test workflow above). I guess we already have this thanks to @SimeonEhrig :pray:.
• Since the dev branches are not supposed to work together anymore, one could drop the trigger of unit and integration tests for the merge commit. The same applies to merging commits of release branches and hotfixes.
• Regarding the dev branches, there is no need for them anymore, because they are just storing the current head of development. Therefore, one could discuss a move from the git-branching model with two long-lived branches to a model with just one long-lived branch main. This would simplify the release workflow by having one less PR to merge but introducing a freeze for merges during the release procedure.
• The proposal above will produce more PRs pending and waiting for packages to be released. Therefore, we need a quick and frictionless release procedure. One idea would be to drop the CHANGELOG.md file and just go with the GitHub release page, which can be updated automatically with Julia's TagBot. This and having just one long-lived branch would reduce the release procedure to just one PR:release-Av0.2 where the -DEV is removed. Even the registration call could be done in this PR, which automates away all other steps of the release workflow, I guess.
• We can lose the testing requirements for PRs coming from release branches, maybe triggering only the unit tests but not the integration test.
• Open question to be verified: if fixB bumps the compat entry of A to Av0.2 and A bumps its own version to Av0.2-DEV, do the unit tests of fixB pass?

[szabo137]

I am not entirely sure about the Extra point for the integration tests and the documentation.