Branching strategy for big updates

[benoit74]

We are regularly doing very significant updates of our scrapers (but probably also other stuff).

These updates are breaking stuff, i.e. during some weeks (or more), the new system is way less reliable / feature-full than the old system.

For instance, warc2zim2 is only beginning to be equivalent to warc2zim 1.x since few weeks/days.

The strategy so far (for warc2zim, zimit, but also for kolibri UI revamp) has been to create a new branch (warc2zim, zimit2, new_ui) which becomes the current target of all issues solved during the big update. And only once all is over, we merge the branch to main. Meaning that all developers working on an issue have to create a branch on top of this new branch, and will do a PR to merge their branch to the new branch (and not to main branch).

This strategy was said to be necessary to not "break" the main branch during many weeks and have regression on this branch in terms of functionalities.

It however proved to have some drawbacks (and I even heard @kelson42 saying it probably wasn't a good idea if I understood correctly):

• new contributors tend to contribute to the main branch, they do not immediately realize there is a
• we need to temporarily adapt the CI to publish a new Docker tag to test this version on the Zimfarm (we cannot use the usual dev tag)
• issues linked to a PR are not automatically marked as closed when the PR is merged
• when creating a PR, Github automatically proposes to merge to main branch and we tend to realise a bit late it is not the right target branch

The alternative would be to merge stuff "ready" (i.e. considered to be the future) to main as soon as they are ready, even if they create regressions.

Do we want to continue to work with approach chosen so far despite its drawbacks, or is it time for a lessons-learned and a change of direction (or at least an experimentation)? We need this for Youtube revamp with @dan-niles

My personal feeling on this is that it is preferable to merge to main everything which is ready. Should we really need to hot-fix a production image, we will always be able to create a hot-fix branch from the current production tag. And I never saw a situation where we had to abandon a development and regretted to have everything already merged to main.

[benoit74]

FYI @jaifroid @veloman-yunkan @audiodude, your feedback is welcomed on this even if you are probably way less concerned, especially since current use-cases are more linked to Python scrapers

[mgautierfr]

I would say that git tags are here for that. We can tag a stable version/commit and use that in prod. Developers would be free to push on main.

If tags are too restrictives, we can also have a prod branch which is always after (or equal) to main. We can simply update prod branch once main has proved its quality.

[kelson42]

Lack of quality of releases is directly linked to lack of QA measures. I don't think this is much linked to CD pressure or git branching strategy. I will see if I can elaborate in a second comment.

The situation around warc2zim2 dev is pretty exceptional as the revamp work is hard to split in smaller tasks and keep everything working and coherent on main during the whole life of milestone (so until release). That said, this won't be the last time.

Still such exceptional situations should confirm the general rule. That said, I would prefer (more pros) to break main if no other choice and situation is managed... than moving rhings in long living "develop" branches.

[benoit74]

The situation around warc2zim2 dev is pretty exceptional

I don't agree, this is the case for kolibri and will be the case for youtube. For exactly the same reasons you gave.

[veloman-yunkan]

IMHO, there is no single best approach.

Depending on the amount (and character) of changes envisioned for a piece of functionality they can be carried out in the form of a:

• single commit
• PR (a collection of several commits)
• development branch (the changes on this branch should still be grouped into PRs)
• forked repository
• new project/repository

I guess that problems arise when the actual volume of changes starts to exceed the initial estimate. In theory nothing prevents you from changing the approach - for example, if your initial assessment was that implementing the change on a branch would be appropriate, but then in the middle of the work you figure out that it will take so much longer, you can fork the repo and continue the development on its main branch (once the work is complete the code can be merged back into the parent repo).

[kelson42]

The situation around warc2zim2 dev is pretty exceptional

I don't agree, this is the case for kolibri and will be the case for youtube. For exactly the same reasons you gave.

Please keep it exceptional and ask me anytime you foresee to brake the rule of CD.