Version pruning in DAML compatibility tests

[cocreature]

Currently we test every single snapshot and stable release. This is clearly not feasible in the long run so we need to prune. I would propose that as a first step, we prune all snapshots from before the latest or maybe the latest two stable releases.

[garyverhaegen-da]

Would it make sense to just test the ones in the LATEST file? If so, could that be automated so we have only one place where we keep a list of "current" versions?

[cocreature]

I kind of like the idea of having all snapshots between the latest release and HEAD.

[cocreature]

But it doesn’t sound entirely unreasonable. It’s still tricky to automate since you need sha256sums of the released artifacts.

[garyverhaegen-da]

Do we actually need the checksums or is that a safety you added? If the latter, would it be reasonable to instead rely on the asc files we publish?

[cocreature]

Having thought about it a bit more, here are a few points:

1. The LATEST file is not sufficient since there is a delay between updating that file and the release being uploaded.
2. We need checksums for release artifacts to make Bazel caching work properly. Otherwise it refetches things occasionally which is pretty expensive for SDK releases. Github releases does not provide that info. I think it would be nice to host this somewhere. Maybe we can add it to the docs cronjob. This would be immensily useful even in the current state since the script for updating the versions file is super slow only because it has to download all SDK releases to get their checksums.
3. Trying to automatically fetch the current version from the snapshots.json or something like that is a bit problematic. It makes builds non-deterministic which I’m hesitant to do in general but here it’s especially problematic since we might need new exclusions or other version-specific changes in certain tests.

[garyverhaegen-da]

All releases back to 1.0.0 now have a sha256sums file; future releases will have it too.

[cocreature]

To summarize the current state: We updated the script to make use of the sha256sums and we now prune versions as follows:

A snapshot X-snapshot… is only kept if X does not exist. This means that if we have a snapshot for 1.2.0 we keep it until we release 1.2.1. If we make snapshots for 1.2.1 we keep those as well until we release 1.2.1 even if 1.3 gets released before. We also have some exceptions currently only for SDK 1.1.1 since the snapshots there were for 1.1.0 but for technical reasons we ended up releasing 1.1.1.

What is still missing atm is automating the addition of new versions:

• Before we do that we probably need ranges on the outer map of the exclusions for the ledger api test tool as well. Otherwise, they basically end up having to be modified every time which makes automation harder.
• Once that is done I see two options:
  1. Have a PR bot that discovers new versions and makes a PR for it. That would basically keep the state we have atm but make it easier to update.
  2. Come up with some way to avoid having to make a commit to add a new version. This would imply some sort of autodiscovery of available versions. While possible this goes towards making those tests non-reproducible which I’m somewhat hesitant to do but I’m happy to be convinced here.

[garyverhaegen-da]

Small update to reflect current status:

• As of #6479 (and to a lesser extent #6706) we now have a CI process that automatically opens a PR to update the list of versions on release.
• As of #6709, CI should also trigger a full daily build on that PR, so we should catch missing exclusions etc. before merging.

That should cover the "update" part of the issue. The pruning part is not quite resolved yet. At the moment, we do all stable releases + all snapshots since last stable. The number of snapshots is already causing serious timing issues, but that resets every month or so. Long-term, the set of stable releases as currently defined is ever-growing so that is going to become problematic too.

I'm not aware that we have reached a decision yet, but there has been a discussion #6689 that I think is worth linking to this issue.