With this issue, I would like to summarize the current development practices in the QEDjl-project and propose some enhancements regarding testing and release procedures.
Preliminaries
Git branching rules
every package has two long-lived branches: 'main' and 'dev'
'main' contains only releases with a version tag
'dev' contains the current head of the development
feature branches come from and are merged to dev (usually from a fork)
releasing brings a collection of commits from dev onto main (see release workflow below)
Dependency rules
main branches (i.e. released versions) only depend on other released versions from the general registry
dev branches also depend only on released versions (not on other dev branches anymore)
feature branches (and fixes, see below) may depend on dev branches during development, but must be updated using released versions before merging to dev
Testing workflows
Unit tests
all PRs (for feature branches, fixes, releases, and hotfixes) must run the local unit tests with the versions of the dependencies given by the compat entries.
Warnings should be thrown if feature branches test against non-released branches, e.g., devs.
Integration tests
In a PR of a feature branch of a package, the unit tests for all dependent packages should be triggered with the changes made in the PR.
Status before: To illustrate the workflow, let's assume two packages A and B, both registered as Av0.1 and Bv0.1. In our case, B depends on A.
Using this setup, a new feature in A waits to be added. The following workflow must be applied:
Checkout a branch featA from dev (usually on a fork), implement the feature and open PR:featA targeting A#dev.
This triggers the unit tests of B against A#featA. If these unit tests pass, nothing needs to be done. PR:featA can be merged. If not, we have introduced a breaking change and need to proceed.
Check out a branch fixB from B#dev adopting the changes introduced in featA, which will depend on A#featA. Open PR:fixB targeting B#dev.
Inform PR:featA about the fix given on fixB and run the integration test (aka the unit test in B) again, but against the combination A#featA and B#fixB.
Because we introduce a breaking change in A, the version of A in the Project.toml should be bumped to Av0.2-DEV, where -DEV indicates, that this version is not been released yet, but other PRs on A can refer to/be built on.
Since the unit tests on PR:fixB are passed, the integration test in PR:featA should also pass. Therefore PR:featA can be merged. This breaks the integration tests again (dev must depend on released versions), which is not a problem, see release workflow.
Status after:A#dev has a new head including the feature from featA. PR:fixB is still pending and can not be merged because its unit test failed (remember, feature branches must depend on released versions to be merged). However, this is also not a problem, because A will be eventually released as Av0.2, bringing the changes necessary for PR:fixB to a release, which means in fixB the compat entry for A can be bumped and PR:fixB can be merged.
Extra: If there is another package C depending on A another fixC branch needs to be checked out on C following the same workflow. If C also depends on B, C#fixC needs to address this too, and PR:fixB needs to be informed about the fix given in C#fixC as well.
Release workflow
The phrase release means following the workflow below bringing a certain commit from dev onto main, registering the new commit on main in General, tagging the new commit on main with a version number, and building a new GitHub release.
We use the setting introduced above to illustrate the integration test pipeline
Status before: The head of A#dev works together with B#fixB currently pending in PR:fixB. However, PR:fixB can not be merged, because its unit tests fail by still using Av0.1 (the current top release), or by using the compat entry on Av0.2 which is not released yet.
We now want to release A to Av0.2 and must follow the release workflow:
Tag all PRs that went into A#dev and are supposed to be part of the release and add them to a common milestone.
create a release branch release-Av0.2
remove -DEV additions in the Project.toml to the version of A
update CHANGELOG.md
open PR:release-Av0.2 targeting A#main. One review is needed to check the correctness of 1-4 of the release workflow.
after merge: open another PR:release-Av0.2 targeting A#dev. Not much review needed here.
register new head of A#main
tag the new head of A#main with v0.2
build a GitHub release
Status after:
PR:fixB can now bump the compat entry to Av0.2 allowing it to be merged.
if a user does Pkg.add.(["A","B"]), the top compatible versions of A and B will be resolved, i.e. Av0.1 and Bv0.1, even if Av0.2 is already in the registry.
However, if Bv0.2 (including the new compat to Av0.2) is released eventually, the Pkg.add.(["A","B"]) will add Av0.2 and Bv0.2.
Documentation
Docs should always be handled like unit tests, even if they don't contain jldoctests
Docs must be built against released versions of their dependencies.
This means, PRs can only be merged to dev, if all upstream packages used are released.
Final remarks and additional requirements
We need good integration test tooling, which includes testing and reporting of fixed down the whole dependency graph. This also includes informing a PR about multiple fixes in other packages at the same time (see the integration test workflow above). I guess we already have this thanks to @SimeonEhrig :pray:.
Since the dev branches are not supposed to work together anymore, one could drop the trigger of unit and integration tests for the merge commit. The same applies to merging commits of release branches and hotfixes.
Regarding the dev branches, there is no need for them anymore, because they are just storing the current head of development. Therefore, one could discuss a move from the git-branching model with two long-lived branches to a model with just one long-lived branch main. This would simplify the release workflow by having one less PR to merge but introducing a freeze for merges during the release procedure.
The proposal above will produce more PRs pending and waiting for packages to be released. Therefore, we need a quick and frictionless release procedure. One idea would be to drop the CHANGELOG.md file and just go with the GitHub release page, which can be updated automatically with Julia's TagBot. This and having just one long-lived branch would reduce the release procedure to just one PR:release-Av0.2 where the -DEV is removed. Even the registration call could be done in this PR, which automates away all other steps of the release workflow, I guess.
We can lose the testing requirements for PRs coming from release branches, maybe triggering only the unit tests but not the integration test.
Open question to be verified: if fixB bumps the compat entry of A to Av0.2 and A bumps its own version to Av0.2-DEV, do the unit tests of fixB pass?
With this issue, I would like to summarize the current development practices in the
QEDjl-project
and propose some enhancements regarding testing and release procedures.Preliminaries
Git branching rules
dev
(usually from a fork)dev
ontomain
(see release workflow below)Dependency rules
main
branches (i.e. released versions) only depend on other released versions from the general registrydev
branches also depend only on released versions (not on otherdev
branches anymore)dev
branches during development, but must be updated using released versions before merging todev
Testing workflows
Unit tests
compat
entries.dev
s.Integration tests
A
andB
, both registered asAv0.1
andBv0.1
. In our case,B
depends onA
.A
waits to be added. The following workflow must be applied:featA
fromdev
(usually on a fork), implement the feature and openPR:featA
targetingA#dev
.B
againstA#featA
. If these unit tests pass, nothing needs to be done.PR:featA
can be merged. If not, we have introduced a breaking change and need to proceed.fixB
fromB#dev
adopting the changes introduced infeatA
, which will depend onA#featA
. OpenPR:fixB
targetingB#dev
.PR:featA
about the fix given onfixB
and run the integration test (aka the unit test inB
) again, but against the combinationA#featA
andB#fixB
.A
, the version ofA
in theProject.toml
should be bumped toAv0.2-DEV
, where-DEV
indicates, that this version is not been released yet, but other PRs onA
can refer to/be built on.PR:fixB
are passed, the integration test inPR:featA
should also pass. ThereforePR:featA
can be merged. This breaks the integration tests again (dev
must depend on released versions), which is not a problem, see release workflow.A#dev
has a new head including the feature fromfeatA
.PR:fixB
is still pending and can not be merged because its unit test failed (remember, feature branches must depend on released versions to be merged). However, this is also not a problem, becauseA
will be eventually released asAv0.2
, bringing the changes necessary forPR:fixB
to a release, which means infixB
thecompat
entry forA
can be bumped andPR:fixB
can be merged.C
depending onA
anotherfixC
branch needs to be checked out onC
following the same workflow. IfC
also depends onB
,C#fixC
needs to address this too, andPR:fixB
needs to be informed about the fix given inC#fixC
as well.Release workflow
dev
ontomain
, registering the new commit onmain
inGeneral
, tagging the new commit onmain
with a version number, and building a new GitHub release.A#dev
works together withB#fixB
currently pending inPR:fixB
. However,PR:fixB
can not be merged, because its unit tests fail by still usingAv0.1
(the current top release), or by using thecompat
entry onAv0.2
which is not released yet.A
toAv0.2
and must follow the release workflow:A#dev
and are supposed to be part of the release and add them to a common milestone.release-Av0.2
-DEV
additions in theProject.toml
to the version ofA
CHANGELOG.md
PR:release-Av0.2
targetingA#main
. One review is needed to check the correctness of 1-4 of the release workflow.PR:release-Av0.2
targetingA#dev
. Not much review needed here.A#main
A#main
withv0.2
PR:fixB
can now bump thecompat
entry toAv0.2
allowing it to be merged.Pkg.add.(["A","B"])
, the top compatible versions ofA
andB
will be resolved, i.e.Av0.1
andBv0.1
, even ifAv0.2
is already in the registry.Bv0.2
(including the newcompat
toAv0.2
) is released eventually, thePkg.add.(["A","B"])
will addAv0.2
andBv0.2
.Documentation
jldoctest
sdev
, if all upstream packages used are released.Final remarks and additional requirements
dev
branches are not supposed to work together anymore, one could drop the trigger of unit and integration tests for the merge commit. The same applies to merging commits of release branches and hotfixes.dev
branches, there is no need for them anymore, because they are just storing the current head of development. Therefore, one could discuss a move from the git-branching model with two long-lived branches to a model with just one long-lived branchmain
. This would simplify the release workflow by having one less PR to merge but introducing a freeze for merges during the release procedure.CHANGELOG.md
file and just go with the GitHub release page, which can be updated automatically with Julia's TagBot. This and having just one long-lived branch would reduce the release procedure to just onePR:release-Av0.2
where the-DEV
is removed. Even the registration call could be done in this PR, which automates away all other steps of the release workflow, I guess.fixB
bumps thecompat
entry ofA
toAv0.2
andA
bumps its own version toAv0.2-DEV
, do the unit tests offixB
pass?