Add tool to check for new upstream versions and autoupdate if possible

[bgilbert]

Use upstream release data from Anitya (release-monitoring.org). Anitya has a database of known software projects, a polling mechanism for identifying new releases, and a mechanism to map downstream package names to Anitya-tracked projects.

I've populated Anitya with WrapDB package names (click the names in the package name column), manually verifying in each case that the Anitya project matches the software actually shipped in WrapDB. In cases where Anitya doesn't know about a project, I have mostly not added a new record, and am not sure whether or not we should add records in all such cases.

With Anitya populated, we can query a single HTTPS endpoint to get the current versions of all our projects it knows about.

Add a tool to use this data. It can:

• List wraps, their versions in WrapDB, and their versions upstream. Command-line options can select ports with or without official Meson support upstream, ports with or without updates available, and ports where Anitya doesn't know the current upstream version (because Anitya doesn't know about the project or doesn't know it's packaged in WrapDB).

  tools/versions.py list (click for sample run)

  abseil-cpp                     20230802.1  =>      20240116.2
  arduinocore-avr                     1.8.2  =>           1.8.6
  argparse                              3.0
  argtable3                           3.2.2
  * argu-parser                         0.0.1  =>|
  asio                               1.30.2
  backward-cpp                          1.6  =>|
  bdwgc                               8.2.2  =>           8.2.6
  * blueprint-compiler                 0.10.0  =>          0.12.0
  box2d                               2.4.1
  bshoshany-thread-pool               4.1.0  =>|
  c-ares                             1.24.0  =>          1.31.0
  * c-flags                             1.5.8  =>|
  * cairo                              1.18.0
  catch                               2.2.2  =>         2.13.10
  * catch2                              3.6.0
  catchorg-clara                      1.1.5  =>|
  centurion                           7.3.0
  cereal                              1.3.2
  * cexception                          1.3.3  =>|
  * cglm                                0.9.2  =>           0.9.4
  check                              0.15.2
  chipmunk                            7.0.3
  cjson                              1.7.17  =>          1.7.18
  * cli11                               2.4.1  =>           2.4.2
  cmark-gfm                   0.29.0.gfm.13
  * cmock                               2.5.3  =>|
  cmocka                              1.1.7
  concurrentqueue                     1.0.3  =>           1.0.4
  * cpp-httplib                        0.15.3  =>          0.16.0
  cpp-semver                          0.3.3  =>|
  cpputest                              4.0
  cppzmq                             4.10.0
  cpr                                1.10.5
  croaring                            1.3.0  =>           4.0.0
  curl                                8.8.0
  cxxopts                             3.1.1  =>           3.2.1
  debug_assert                        1.3.3  =>           1.3.4
  directxmath                         3.1.9  =>|
  dlfcn-win32                         1.3.1  =>           1.4.1
  docopt                              0.6.3
  * doctest                            2.4.11
  dragonbox                           1.1.2  =>           1.1.3
  eigen                               3.4.0
  emilk-loguru                        2.1.0
  enet                               1.3.17  =>          1.3.18
  entt                               3.11.0  =>          3.13.2
  * epoxy                              1.5.10
  * exiv2                              0.28.1  =>          0.28.2
  expat                               2.6.0  =>           2.6.2
  facil                               0.7.6
  fdk-aac                             2.0.2  =>           2.0.3
  ff-nvcodec-headers               11.1.5.1  =>       12.2.72.0
  flac                                1.4.3
  flatbuffers                        24.3.6  =>         24.3.25
  fluidsynth                          2.3.3  =>           2.3.5
  fmt                                10.2.0  =>          10.2.1
  * fontconfig                         2.14.2  =>          2.15.0
  freeglut                            3.4.0  =>           3.6.0
  * freetype2                          2.13.2
  * fribidi                            1.0.13  =>          1.0.15
  frozen                              1.1.1
  ftxui                               5.0.0
  fuse                                2.9.9
  gdbm                                 1.23
  * gdk-pixbuf                        2.42.12
  gee                                0.20.6
  giflib                              5.2.2
  glbinding                           3.3.0  =>|
  glew                                2.2.0
  glfw                               3.3.10  =>             3.4
  * glib                               2.80.3
  * glib-networking                    2.78.0  =>          2.80.0
  glm                                 1.0.1
  godot-cpp                             4.2  =>           4.2.2
  google-benchmark                    1.8.4
  google-brotli                       1.1.0
  google-snappy                       1.1.9  =>           1.2.1
  google-woff2                        1.0.2
  graphite2                          1.3.14
  grpc                               1.59.1  =>          1.64.2
  gtest                              1.14.0
  gumbo-parser                       0.12.1  =>|
  * harfbuzz                            8.3.1  =>           8.5.0
  hedley                                 15
  hinnant-date                        3.0.1
  htslib                               1.17  =>            1.20
  icu                                  73.2  =>            75.1
  iir                                 1.9.3  =>           1.9.4
  imgui                              1.89.9  =>          1.90.8
  imgui-sfml                            2.6  =>|
  imguizmo                             1.83  =>|
  implot                               0.16
  indicators                            2.3  =>|
  * inih                                  r57  =>             r58
  * irepeat                               0.6  =>|
  jansson                              2.14
  jbig2dec                             0.20
  jbigkit                               2.1
  json-c                               0.17
  * json-glib                           1.6.6  =>    1.8.0-actual
  * jsoncpp                             1.9.5
  lame                                3.100
  * lcms2                                2.16
  leptonica                          1.84.1
  libarchive                          3.7.4
  libccp4c                            8.0.0  =>|
  * libdicom                            1.1.0
  * libdrm                            2.4.121
  libebml                             1.4.5
  libexif                            0.6.24
  libffi                              3.4.4  =>           3.4.6
  libffmpegthumbnailer                2.2.2
  libgpiod                            1.6.3  =>           2.1.2
  libjpeg-turbo                       3.0.3
  libkqueue                           2.6.2
  liblangtag                          0.6.7
  * liblastfm                           1.0.1  =>|
  liblbfgs                             1.10  =>|
  * libliftoff                          0.3.0  =>           0.5.0
  liblzma                            5.2.12  =>           5.6.2
  libmatroska                         1.7.1
  libmicrohttpd                      0.9.77  =>           1.0.1
  * libnpupnp                           6.1.2  =>           6.1.3
  * libobsd                             1.1.1  =>|
  libopenjp2                          2.5.2
  libpng                             1.6.43
  * libpsl                             0.21.5
  libsass                             3.6.4  =>           3.6.6
  libsignal-protocol-c                2.3.3
  libsndfile                          1.2.2
  * libsrtp2                            2.5.0  =>           2.6.0
  libssh2                            1.11.0
  libtiff                             4.6.0
  libtirpc                            1.3.3  =>           1.3.4
  libtomcrypt                        1.18.2
  libunibreak                           6.1
  libupnp                           1.14.19
  liburing                              2.5  =>             2.6
  libusb                             1.0.27
  libuv                              1.48.0
  libwebp                             1.3.2  =>           1.4.0
  libwebsockets                       4.3.3
  libxcursor                          1.2.1  =>           1.2.2
  libxext                             1.3.5  =>           1.3.6
  libxinerama                         1.1.4  =>           1.1.5
  * libxkbcommon                        1.7.0
  * libxml2                            2.13.1
  * libxmlpp                            5.2.0
  libxrandr                           1.5.2  =>           1.5.4
  libxrender                         0.9.10  =>          0.9.11
  libxslt                            1.1.39  =>          1.1.41
  libxv                              1.0.11  =>          1.0.12
  libxxf86vm                          1.1.4  =>           1.1.5
  libyaml                             0.2.5
  littlefs                            2.9.0  =>           2.9.3
  lmdb                               0.9.29  =>|
  lua                                 5.4.6  =>           5.4.7
  luajit                     2.1.1713773202  =>           2.0.5
  ludocode-mpack                        1.1  =>|
  lz4                                 1.9.4
  lzo2                                 2.10
  m4                                 1.4.19
  * magic_enum                          0.9.5
  mdds                                2.1.1
  microsoft-gsl                       4.0.0
  miniz                               3.0.2
  minizip-ng                          4.0.4  =>           4.0.7
  mocklibc                              1.0  =>|
  mpdecimal                           2.5.1  =>           4.0.0
  msgpackc-cxx                        6.1.1
  mt32emu                             2.7.1
  muellan-clipp                       1.2.3  =>|
  mujs                                1.3.3  =>           1.3.5
  * nanoarrow                           0.5.0
  nativefiledialog-extended           1.1.1  =>           1.2.0
  netstring-c                         0.0.0  =>|
  nghttp2                            1.58.0  =>          1.62.1
  * nlohmann_json                      3.11.2  =>          3.11.3
  nng                                 1.5.2  =>           1.8.0
  nonstd-any-lite                     0.2.0  =>|
  nonstd-byte-lite                    0.2.0  =>|
  nonstd-expected-lite                0.3.0  =>|
  nonstd-observer-ptr-lite            0.4.0  =>|
  nonstd-optional-lite                3.2.0  =>|
  nonstd-span-lite                    0.5.0  =>|
  nonstd-status-value-lite            1.1.0  =>|
  nonstd-string-view-lite             1.3.0  =>|
  nowide                             11.3.0  =>|
  oatpp                               1.3.0  =>    1.3.0-latest
  oatpp-openssl                       1.3.0  =>|
  oatpp-sqlite                        1.3.0  =>|
  oatpp-swagger                       1.3.0  =>|
  oatpp-websocket                     1.3.0  =>|
  oatpp-zlib                          1.3.0  =>|
  ogg                                 1.3.5
  openal-soft                        1.23.1
  opencl-headers                 2023.04.17  =>      2024.05.08
  * openh264                            2.3.1  =>           2.4.1
  openssl                             3.0.8  =>           3.3.1
  * opus                                  1.4  =>           1.5.2
  orcus                              0.19.2
  orocos_kdl                          1.5.1
  * pango                              1.51.0  =>          1.54.0
  pcg                                0.98.1
  * pciaccess                            0.18  =>          0.18.1
  pcre                                 8.45
  pcre2                               10.44
  physfs                              3.2.0
  * pixman                             0.43.4
  * pkgconf                             2.1.1  =>           2.2.0
  protobuf                             25.2  =>            27.1
  protobuf-c                          1.4.1  =>           1.5.0
  * proxy-libintl                         0.4
  pugixml                              1.14
  pv                                 1.8.10
  pybind11                           2.12.0
  * qarchive                            2.2.7  =>           2.2.9
  qrencode                            4.1.1
  quazip                                1.4
  * quill                               4.4.0  =>           4.4.1
  rang                                  3.2
  range-v3                           0.12.0
  rapidjson                           1.1.0
  rdkafka                             2.3.0  =>         2.4.0-3
  re2                              20230301  =>        20240601
  reflex                             3.2.11  =>           4.4.0
  robin-map                           1.2.1  =>           1.3.0
  * rtaudio                             6.0.1
  * rubberband                          2.0.2  =>           3.3.0
  rxcpp                               4.1.1  =>|
  sassc                               3.6.2
  sdl2                               2.30.3  =>          2.30.4
  sdl2_image                          2.6.3  =>           2.8.2
  sdl2_mixer                          2.6.2  =>           2.8.0
  sdl2_net                            2.2.0
  sdl2_ttf                           2.20.1  =>          2.22.0
  sds                                 2.0.0  =>|
  sergiusthebest-plog                1.1.10
  sfml                                2.6.1
  simdjson                            3.3.0  =>           3.9.4
  * slirp                               4.7.0  =>           4.8.0
  soundtouch                          2.3.2  =>           2.3.3
  sparsehash-c11                     2.11.1  =>|
  spdlog                             1.14.1
  speexdsp                            1.2.1
  * spng                                0.7.4
  sqlite3                            3.46.0
  sqlitecpp                           3.3.1
  sqlpp11                              0.64  =>|
  stduuid                             1.2.3
  tabulate                              1.5
  taglib                                2.0  =>           2.0.1
  tclap                               1.2.4  =>           1.2.5
  termbox                             1.1.2  =>|
  theora                              1.1.1
  tinyfsm                             0.3.3
  tinyply                             2.3.4
  * tinyxml2                           10.0.0
  tl-expected                         1.1.0
  tl-optional                         1.0.0  =>|
  * tomlplusplus                        3.4.0
  * tracy                                0.10
  tree-sitter                        0.22.5  =>          0.22.6
  trompeloeil                            47
  tronkko-dirent                     1.23.2  =>|
  turtle                              1.3.2
  type_safe                           0.2.3  =>|
  uchardet                            0.0.8
  * unit-system                         0.7.1  =>|
  unittest-cpp                        2.0.0
  * unity                               2.5.2  =>|
  utf8proc                            2.9.0
  utfcpp                              4.0.5
  uthash                              2.3.0
  vo-aacenc                           0.1.3
  vorbis                              1.3.7
  vulkan-headers                    1.3.283  =>         1.3.288
  vulkan-memory-allocator             3.1.0
  vulkan-validationlayers           1.2.158  =>         1.3.288
  * wayland                            1.20.0  =>          1.23.0
  * wayland-protocols                    1.35  =>            1.36
  websocketpp                         0.8.2
  wren                                0.4.0
  x-plane-sdk                         4.0.1  =>|
  xtensor                            0.21.5  =>          0.25.0
  xtl                                0.6.12  =>           0.7.7
  xxhash                              0.8.2
  yajl                                2.1.0
  zlib                                1.3.1
  zstd                                1.5.6

• Attempt to update wraps automatically. This only works for upstreams that support Meson directly, and may fail e.g. due to URL changes.

Add a semiweekly GitHub Actions workflow that submits individual PRs for wraps that can be autoupdated. Sample workflow run. (The wayland failure is fixed by #1554.) Sample PRs.

The update workflow doesn't run sanity checks on the updated wrap; it assumes PR CI will do that. However, that means we can't use GITHUB_TOKEN to push the PR branches, since GitHub doesn't allow actions by the token to invoke other workflows. This PR assumes the following workaround:

1. Create a wrapdb-bot GitHub account to hold the PR branches. That account should not be given any special privileges on mesonbuild/wrapdb. Fork mesonbuild/wrapdb into the bot account.
2. Create a Personal Access Token for the bot account and add it to mesonbuild/wrapdb as a GitHub Actions secret named BOT_TOKEN.
3. The peter-evans/create-pull-request Action will create PRs from the bot's fork, using the bot's access token.

[neheb]

this is...incredible.

I tried this locally. Works great. Although something's wrong with json-glib

./tools/versions.py autoupdate
Traceback (most recent call last):
  File "/home/mangix/devstuff/wrapdb/./tools/versions.py", line 324, in <module>
    main()
  File "/home/mangix/devstuff/wrapdb/./tools/versions.py", line 320, in main
    args.func(args)
  File "/home/mangix/devstuff/wrapdb/./tools/versions.py", line 185, in do_autoupdate
    update_wrap(name, cur_ver, upstream_ver)
  File "/home/mangix/devstuff/wrapdb/./tools/versions.py", line 140, in update_wrap
    resp.raise_for_status()
  File "/usr/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://download.gnome.org/sources/json-glib/1.8/json-glib-1.8.0-actual.tar.xz

[neheb]

There's also this issue: https://github.com/mesonbuild/wrapdb/issues/158

[bgilbert]

Yeah, there's a data error for json-glib. Upstream tagged 1.8.0 twice, as 1.8.0 and 1.8.0-actual, and there's no 1.8.0-actual tarball. I've flagged the Anitya record for admin review. Meanwhile, I changed the autoupdate subcommand to report errors at the end, rather than failing immediately.

For json (and onqtam-doctest) I've added a hardcoded list of deprecated wraps to ignore. This can be switched to use a releases.json deprecation mechanism if one is eventually added.

I've also made a few other tweaks, including to the list output format, and added the ability to produce Markdown tables (for the Actions workflow).

[bgilbert]

Updated the workflow to automatically delete unneeded branches created by previous runs. This closes the corresponding PR if still open, ensuring we don't leave PRs lying around for wraps that were updated manually.

Also added the infrastructure to push to a bot account's fork, specified by the HEAD_REPO variable in the workflow. Its Personal Access Token is assumed to be stored in the BOT_TOKEN repo secret.

Also added the previous version number to PR titles and commit messages, and made a couple other minor improvements.

[bgilbert]

I think this is in pretty good shape now. Lifting draft.

[bgilbert]

json-glib had a new upstream release, so that data error is no longer an issue.

Rebased.

[bgilbert]

I've seen transient gateway timeouts from Anitya a couple times, so added some retry logic for that case.

[bgilbert]

What's the plan for setting up the bot account as described in https://github.com/mesonbuild/wrapdb/pull/1555#issue-2366099390? This PR assumes the account is named wrapdb-bot, and no such account currently exists on GitHub, so the Actions workflow will fail on every run. I could create the account, but it seems better for the account to be controlled by someone with administrative access to this repo. After the account is created, a repo administrator will need to add its Personal Access Token as a repository secret here.

[neheb]

Probably @jpakkane needs to handle that.

[jpakkane]

I'll look into that once the 1.5 release is out.

[bgilbert]

Cool, thanks! This PR currently assumes the bot account has the email wrapdb@mesonbuild.com, though we can change that if need be. I believe the needed PAT scopes are just public_repo.

[jpakkane]

That can't work. There is no email service at that name. Or even MX records FWICR.

[bgilbert]

The GitHub account will need some email address, and the author/committer of the automated commits should be set to the same email address so GitHub can attribute them properly. The actual address can be anything; we'll just need to update the workflow here.

[bgilbert]

Another option is to use your-personal-email+wrapdb@yourdomain.com as the GitHub account email, if your email provider treats + extensions as still referring to your mailbox. Then the commit author/committer emails could use the email address GitHub autogenerates for its email privacy feature. That's not as pretty visually, but it avoids having to set up a new email address.

[bgilbert]

@jpakkane Do you have some time to look at this? I think it shouldn't take more than 10-15 minutes to set up, and the resulting automation would be very useful in this repo. I can PR the updated email address if you let me know what address to use.

[bgilbert]

Since there doesn't seem to be much interest here, I've posted https://github.com/mesonbuild/wrapdb/pull/1618 to remove the GitHub Actions workflow from this repo.