Support publishing of a depot-user's index

[jschlatow]

Updating a user's depot index with packages published with Goa for a new Sculpt release is still a manual task. With an increasing number of Goa packages to be managed and published, this becomes a significant effort. Adding support for publishing an index to a user's depot, will automate this task and eliminate the need to switch to the depot tools within the Genode repo.

Here is a collection of ideas regarding this:

• Similar to how packages are exported/published, an index is exported/published by the to-be-added commands goa export-index and goa publish-index.
• Both commands take a version string (e.g. 23.04) as a positional argument. Moreover, the commands evaluate the depot_user and depot_overwrite variables.
• The index is built from a index file in the working directory that is similar to the exported/published file but lacks username and version information.
• The archive versions are taken from the goarc. Optionally, it might be possible to look up versions from the subdirectories that contain the Goa projects.
• It may be worth thinking about whether goa publish-index should call goa publish for not-yet-published archives.

Related to this, a command goa bump-versions <version> could be helpful for setting the versions of all Goa projects present in the subdirectories.

[nfeske]

That is very much in line with ambitions I had in the back of my head for a long time as well.

Similar to how packages are exported/published, an index is exported/published by the to-be-added commands goa export-index and goa publish-index.

I think that the mere presence of an index file could actually be enough to steer the regular publish and export commands into the right direction. An "index project" would be a Goa project without src/import/raw/... files, but with an index file. There could exist other Goa projects in sub directories of such an "index project". In my envisioned usual case, the top-level directory of a goa-projects repo would be an index project that holds all the parts hosted in various sub directories together. By using this convention (presence in index file -> index project), we won't need to add new top-commands to Goa.

Both commands take a version string (e.g. 23.04) as a positional argument.

Since Goa is tied to a concrete Sculpt version, wouldn't that argument be somewhat redundant? We may get away with merely adding the sculpt version to share/goa/goarc, keeping the UI as simple as possible. (when hypothetically targeting multiple Sculpt versions, one would need to switch to a different Goa branch for each Sculpt version anyway)

It may be worth thinking about whether goa publish-index should call goa publish for not-yet-published archives.

That is certainly a tempting direction. Down the road, we could go this road even further, letting Goa discover inter project dependencies (e.g., a pkg in one part of the tree depends on a raw or a src project in another part of the tree) and execute the build/export steps in the right order. That would rock because, ideally, a switch to a new Sculpt version could become as easy as updating goa via the update-goa command, followed by a goa -C publish in the top-level goa-projects directory (yes, I'd dreaming, but it's a nice dream).

[jschlatow]

Thanks for your immediate feedback. I very much agree with all your thoughts.

[jschlatow]

I just pushed a commit series to my issue039_index_projects branch that implements the depot export and publish, including resolution of project dependencies.

@nfeske If you're keen on trying it out, you may give it a go.

[nfeske]

That looks freaking fantastic! I'll be offline until Tuesday. Can't wait to try it out once I'm back!

[nfeske]

Today, I finally had the chance to explore this new feature using my goa-playground repository.

Initially, I was wondering why not much happened on my attempt of goa export - until I also specified the -r argument, which kicked off the actual build. During my experiments, I noticed a few minor things:

• When using -r, Goa seems to build everything, not only the pkgs mentioned in the index. For example, I happen to have a (incomplete) port of less lurking around, which I don't reference in any the pkgs of my index. It would be nice of Goa to skip it.

• Some of my projects have a version file, which seems to get ignored.

  Error: no version defined for depot archive 'nfeske/pkg/hatari-tt'

  When specifying the version in the .goarc, it works.

• When Goa encounters a src project without a license, it complains like this:

  Error: cannot export src archive because the license is undefined

  This message is clear when using Goa for single project, but when exporting multiple projects at once, one needs to specify --verbose to see what project the error is about. It would be nice of Goa to print the project name as part of the message.

• I happened to have a stale directory tree with a bunch of *~ files spilled around (residuals from another git branch). Goa still detected this directory as a valid project. Maybe we could tighten the detection of project directories to let Goa disregard directories without any real files.

The index file contains the <supports> nodes. I wondered, could this information be picked up by Goa to automatically build the binaries for the listed architectures?

I'm already very happy about the convenience of the index feature. I can clearly see how it can make huge difference when moving Goa projects from one Sculpt version to another.

[jschlatow]

Today, I finally had the chance to explore this new feature using my goa-playground repository.

Initially, I was wondering why not much happened on my attempt of goa export - until I also specified the -r argument, which kicked off the actual build. During my experiments, I noticed a few minor things:

This is odd. If Goa detects an index project in the current working directory, it is supposed to scan the subdirectories for the mentioned pkgs. I suppose Goa misses the index file in your working directory.

• When using -r, Goa seems to build everything, not only the pkgs mentioned in the index. For example, I happen to have a (incomplete) port of less lurking around, which I don't reference in any the pkgs of my index. It would be nice of Goa to skip it.

The -r switch actually overrides this behaviour and tells Goa to export every project found in any subdirectory.

• Some of my projects have a version file, which seems to get ignored.

  Error: no version defined for depot archive 'nfeske/pkg/hatari-tt'

  When specifying the version in the .goarc, it works.

Could you point me to the branch of your repository so that I can have a look at the structure?

• When Goa encounters a src project without a license, it complains like this:

  Error: cannot export src archive because the license is undefined

  This message is clear when using Goa for single project, but when exporting multiple projects at once, one needs to specify --verbose to see what project the error is about. It would be nice of Goa to print the project name as part of the message.

That'd be helpful indeed.

• I happened to have a stale directory tree with a bunch of *~ files spilled around (residuals from another git branch). Goa still detected this directory as a valid project. Maybe we could tighten the detection of project directories to let Goa disregard directories without any real files.

I'll think about that.

The index file contains the <supports> nodes. I wondered, could this information be picked up by Goa to automatically build the binaries for the listed architectures?

This should actually be the case. However, as mentioned above there seems to be something wrong with the detection of your index file.

I'm already very happy about the convenience of the index feature. I can clearly see how it can make huge difference when moving Goa projects from one Sculpt version to another.

Thanks for testing.

[nfeske]

Interestingly, Goa actually picks up my index file for publishing.

I have pushed everything to https://github.com/nfeske/goa-playground

As a prerequisite for building hatari, which requires sdl2, I'm using commit https://github.com/nfeske/goa/commit/86cdc99d8911022692d8b63b86428cb503f7a708.

I'm specifying --depot-dir and --public-dir at the command line.

[jschlatow]

@nfeske I just pushed a few fixup commits.

Most significantly, I misread tcl's glob command and was assuming it to recurse into all subdirectories. Commit 6be1219 fixes that. This fixes the issue with the version detection.

Commit bd33261 improves the detection of project directories by ignoring ~ files, .orig files and incomplete pkg/ directories.

Commit fe7948d adds a prefix to all error messages reported by Goa.

I noticed that your atari projects sometimes reference explicit archive versions as dependencies and even reference themselves (e.g. atari/m68k-sdk/pkg/m68k-sdk/archives mentions nfeske/raw/m68k-sdk). Since Goa adds the raw and src archives of a project automatically, you can remove the self-references.

[nfeske]

Thanks @jschlatow for following up so quickly.

WIth the new version, goa export w/o -r works as intended. I also noticed that both CPU architectures are covered automatically - very cool!

I noticed that your atari projects sometimes reference explicit archive versions as dependencies and even reference themselves (e.g. atari/m68k-sdk/pkg/m68k-sdk/archives mentions nfeske/raw/m68k-sdk). Since Goa adds the raw and src archives of a project automatically, you can remove the self-references.

This was out of desperation. The manual addition of the raw archive was needed because Goa adds the implicit raw archive at the beginning of the archives list whereas Sculpt lets subsequent entries override previous ones. In my case, the unix/raw/vfs.config is pulled in by pkg/unix. But I want to override it by the version m68k-sdk/raw/vfs.config. When deploying the pkg in Sculpt, I found the m68k-binutils missing in the vfs because unix/raw/vfs.config was used. I think we ultimately need make the shadowing of raw content consistent between Sculpt and Goa. The manual addition of the m68k-sdk/raw archive after the pkg/unix is just temporary a work-around.

[nfeske]

Can you try removing the line nfeske/src/hatari/2023-06-16 from atari/hatari/pkg/hatari/archives and do a goa run from within the hatari directory? I get the following error:

[hatari] Error: Unable to find content ROM module 'hatari'. You either need to add it to the 'raw/' directory or add the corresponding dependency to the 'archives' file.

The hatari ROM should be implicitly available because it is the build artifact created from hatari/src/.

[jschlatow]

The hatari ROM should be implicitly available because it is the build artifact created from hatari/src/.

There is actually a fixup commit (b8a05b2) already on staging.

[jschlatow]

This was out of desperation. The manual addition of the raw archive was needed because Goa adds the implicit raw archive at the beginning of the archives list whereas Sculpt lets subsequent entries override previous ones.

Thanks for this clarification. I recently thought about the shadowing policy in Goa in another context (which I don't remember) and assumed that Sculpt does the same. Sure thing that we should make this consistent. Is the shadowing in Sculpt only applied for raw content or for ROM modules in general?

[jschlatow]

@nfeske I checked the code and noticed that Goa applies the same shadowing policy as Sculpt. It just fails to add the local src and raw archives at the correct position in the archives file. Commit 0d04cad fixes this.

[nfeske]

Is the shadowing in Sculpt only applied for raw content or for ROM modules in general?

Albeit I don't remember ever having shadowed any binary-archive content so far, I would intuitively expect this to be possible as well. This might be useful for e.g., overriding an existing library by a patched version.

[jschlatow]

I merged the commit series to staging.