Add information about revisions to `packages.json`

[michaelpj]

At the moment there's a timestamp field, but that's the time the package version was added. It would be nice to know about revisions. I can see a few possible designs here:

• Change timestamp to last-modified, and have it be the timestamp of the latest revision (or the original creation).
• Change timestamp to added, and add revised as the timestamp of the latest revision.
• Change timestamp to added, and add revised as a list of timestamps for the revisions.

The reason I care is that the CHaP CI diffs the metadata to tell when packages have changed, so it knows what to build. At the moment it can only notice new package additions because revisions don't affect packages.json. Making them change it in any way would suffice for my purposes!

[michaelpj]

See https://github.com/input-output-hk/cardano-haskell-packages/issues/170

[andreabedini]

My worry here is that we are basically building a toml to json converter :P I am not against this but I wonder if the CI could look at _sources?

Edit: This is slow but seems to do get you there already.

$ nix shell nixpkgs#yq
$ for m in $(find _sources/ -name meta.toml); do IFS=/ read -ra PARTS <<< "$m"; tomlq --arg name ${PARTS[1]} --arg version ${PARTS[2]} '{ "pkg-name": $name, "pkg-version": $version, timestamp: .timestamp, "revisions": [.revisions[]? | .timestamp] }' $m; done
...
{
  "pkg-name": "plutus-core",
  "pkg-version": "1.1.0.0",
  "timestamp": "2022-10-21T00:00:00+00:00",
  "revisions": []
}
{
  "pkg-name": "plutus-core",
  "pkg-version": "1.1.1.0",
  "timestamp": "2022-10-31T00:00:00+00:00",
  "revisions": [
    "2023-02-22T19:09:40+00:00",
    "2023-03-21T14:21:44+00:00"
  ]
}
...

btw, I initially used toml because nix suports it natively, but then it tuned out that nix cannot read toml timestamps :-/ Edit: maybe we can fix that https://github.com/NixOS/nix/pull/8120

[michaelpj]

Well, that was what I was doing originally and then you made the metadata output and I thought I was supposed to do that :p also tomlq looks nice! I can probably work with that indeed.

[andreabedini]

:thinking: I guess one difference is who is using that information. If it's in CI we have the input metadata, so we don't really need anything else. If it's the end consumer (whoever is using cabal) they don't have access to the input metadata anymore, which is why we made package.sjon.