Changelog feature

[PierreLoupG]

Context

I work for a client who produces several Yocto builds. They use repo (git-repo) to manage layer versions and generate a changelog from the manifest files. Switching between build is done with bash scripts. As part of a system redesign I proposed the use of yocto-cooker for a better build management.

However, the client like the changelog generation of repo (repo diffmanifests old-manifest.xml new-manifest.xml). Since the repo manifest and the cooker menu file both list the sources of the layers it would be redundant to continue to use repo only for the changelog.

Before starting development, I would like to know if a contribution to add a changelog feature to cooker would be welcome or off topic. Otherwise, I would develop it as an external solution.

Feature

With the revision field assigned to each sources, the menu file works as a manifest for build reproducibility. If the menu file is in a project git repository the menu file can track releases if the revision fields are updated and the changes committed. A changelog text can then be generated from the menu file history by comparing the sources revisions and thus the changes from git commit history.

Rather than use an external script with jq to manipulate the JSON menu file and git to get the commit history, it would be convenient to add this feature to cooker like repo does.

Scope

The scope of the changelog feature are:

• Update the revisions fields (and branch if assigned) in the menu file for each sources found in the layer directory.
• Generate a changelog text of the sources changes for a given build between two menu file version.

Out of scope

The changelog feature should not make assumptions, what is out of the scope:

• Create or delete new sources in the menu file. These operations are considered manual.
• Create the release commit or the release tag. The changelog feature is not made to automate the release but only to generate the changelog text from the cooker context.

Implementation

Two new commands can be added to the codebase.

Release

For each defined layer in the layer directory update the revision field and branch if assigned. This command does not take any parameters.

The menu structure of the CookerCommands class is updated and written to the JSON menu file (config.menu()).

Example:

cooker release

Changelog

Generate and print the changelog text for a given build of the menu file. The changelog text describes the sources changes and/or detailed commit history between two revisions of the menu file, for example:

### Added projects
- meta-foo at revision ed785a

### Changed projects
- meta-toto changed from 3f7e89 to f7fe92
  - f7fe92 commit message
  - 669110 commit message

- meta-tata changed from cea381 to 590471

### Removed projects
- meta-bar at revision 8db9e4

By default the changelog is generated between the current menu file and the previous state of the menu file (HEAD~1). Specific revisions can be set as parameters. The command uses git show with the revision to get the menu file.

This command takes the build name as argument, the optional revisions and the detailed sources.

Example:

# Print changes from previous menu file state
cooker changelog rpi4
# Print changes from old menu file state
cooker changelog rpi4 HEAD~2
# Print changes from specific previous tag with detailed meta-toto
cooker changelog rpi4 v1.0.0 --history meta-toto
# Print changes between two specific revisions with detailed meta-toto and meta-tata
cooker changelog rpi4 v1.0.0 v2.0.0 -H meta-toto meta-tata

Usage

To create a new release the user can update the menu file easily with the release command and generate the changelog text with changelog command. The changelog text can then be used for the release commit or added to a changelog file.

Feedback and suggestions are welcome.

[pboettch]

If I understand correctly, you are proposing two functionalities:

1) Creating a changelog comparing revisions of sources between two versions of a menu. 2) Updating a menu with the currently checked out revision of a source.

Both are good ideas. But there some buts:

For 1) your example shows that cooker would use a git revision to get the previous menu-version. That is not a good idea, cooker does not know, that the menu is handled with git, and it should not know. I think the only acceptable way is to give two menu-files, which are physically present to the the changelog command.

For 2) cooker is not allowed to write the menu-file. We would lose comments and the order of the key will change as well, this is something not acceptable. However, having a command which shows the current differences, in revision and changes, of a source via cooker, is something totally feasible (for git this might be the output of describe of HEAD and the rev from the menu)

cooker diff

[PierreLoupG]

I agree with both of your points. With your suggestions, functionalities 1) and 2) are very similar since release will simply print the changes as changelog does but in an another way. I think one command can do the job, cooker diff.

The diff command is similar to changelog, except how output text is formatted. Diff can prints the changes with a markdown format to be added to a changelog or as a simple text easy to read and parse.

Like changelog, diff takes a build as parameter and optional menu file (rather than revision) to compare the changes. By default the diff command prints the changes between the current menu file sources revision and the local sources revisions.

For example:

usage: cooker diff [-h] [-H [HISTORY [HISTORY ...]]] [-o FORMAT] build [menu_from] [menu_to]

positional arguments:
  build                 show differences of the build
  menu_from             compare changes from the menu file
  menu_to               compare changes from the menu file

optional arguments:
  -h, --help            show this help message and exit
  -H [HISTORY [HISTORY ...]], --history [HISTORY [HISTORY ...]]
  -o FORMAT, --format FORMAT      output format 'md', 'text'

From the text format output the user can manually edit the menu file before a release and use the markdown format for a changelog.

Is this implementation better suited ?

[pboettch]

No, this diff-command only shows current gitref-changes of all sources (independently of the build) compared to the menu's referenced gitref, e.g.:

$ cooker diff <no-arguments-needed>
source: <git describe --tags of rev in menu> .. <git describe --dirty --tags HEAD>

nothing more.

It is your changelog-command which can be more or less elaborate based on given options and why not reduce its name to log?

[PierreLoupG]

Ok, I better understand your point. I will work on this implementation, diff and log commands. Thank you for your feedback.