doesn't download "new" packages when getting dependencies

[jeremyd2019]

[MINGW-packages] [clangarm64] mingw-w64-python-pep517...
  [1/2] mingw-w64-clang-aarch64-python-pytoml-0.1.21-2-any.pkg.tar.zst
  [2/2] mingw-w64-clang-aarch64-python-setuptools-59.1.1-1-any.pkg.tar.zst
  ==> Adding package '/C/_REPO/clangarm64/mingw-w64-clang-aarch64-python-pytoml-0.1.21-2-any.pkg.tar.zst'
    -> Computing checksums...
    -> Creating 'desc' db entry...
    -> Creating 'files' db entry...
  ==> Adding package '/C/_REPO/clangarm64/mingw-w64-clang-aarch64-python-setuptools-59.1.1-1-any.pkg.tar.zst'
    -> Computing checksums...
    -> Creating 'desc' db entry...
    -> Creating 'files' db entry...
  ==> Creating updated database file '/C/_REPO/clangarm64/autobuild-clangarm64.db.tar.gz'
  :: Synchronizing package databases...
   autobuild-clangarm64 downloading...
   mingw32 downloading...
   mingw64 downloading...
   ucrt64 downloading...
   clang64 downloading...
   clang32 downloading...
   clangarm64 downloading...
   msys downloading...
  :: Starting core system upgrade...
   there is nothing to do
  :: Starting full system upgrade...
   there is nothing to do
  HEAD is now at eb5a8c7c9 enable package tree required to build tomli for clangarm64
  gpg: "gpg_KEY" not a key ID: skipping
  gpg: "gpg_KEY" not a key ID: skipping
  gpg: "gpg_KEY" not a key ID: skipping
  gpg: "gpg_KEY" not a key ID: skipping
  ==>  MINGW_ARCH: clangarm64
    -> Building clangarm64...
  ==> WARNING: PACKAGER should have the format 'Example Name <email@address.invalid>'
  ==> Making package: mingw-w64-python-pep517 0.12.0-1 (Wed, Dec  1, 2021  1:48:35 PM)
  ==> Checking runtime dependencies...
  ==> Installing missing dependencies...
  error: target not found: mingw-w64-clang-aarch64-python-tomli
  ==> ERROR: 'pacman' failed to install missing dependencies.
  ==> Missing dependencies:
    -> mingw-w64-clang-aarch64-python-tomli

[jeremyd2019]

a) this seems to be in msys2-web, not autobuild, and b) it may be some sort of confusion about circular dependencies? Some packages are getting tomli and some are not.

[jeremyd2019]

https://github.com/msys2-arm/msys2-autobuild/runs/4387514100?check_suite_focus=true

pep517 should have depended on tomli, but didn't. virtualenv should have depended on setuptools-scm but didn't (and was actually attempted to be built before setuptools-scm as a result).

[jeremyd2019]

If my underlying assumptions are correct, maybe it would make more sense to put breaking dependency loops in autobuild instead of msys2-web. autobuild's get_buildqueue_with_status would be able to know if some of them are already built and thus the "solution" would be obvious.

[lazka]

For context, it's that way currently so in case the cycle is already in the repo (the common case) it will fall back to the outdated package in the repo which usually works out if there is no API/ABI change.

I agree that autobuild could make smarter decisions if it had that info.

I'll try to move the same logic into autobuild, for starters.

[raedrizqie]

i think this only happens to python extensions right?

[jeremyd2019]

i think this only happens to python extensions right?

No, any dependency cycle could result in this.

[lazka]

I've removed this in https://github.com/msys2/msys2-web/commit/3c41af895f1734 now, so cycles will result in autobuild getting stuck if all of the cycle is missing.

But better than randomly missing deps I guess.

[lazka]

To prevent things getting stuck I've now added a manual list for breaking cycles:

https://github.com/msys2/msys2-autobuild/blob/a8d63e2852a11430d1be7c1fc5248f2bdf373f13/autobuild.py#L92

Not sure if this is the right approach.. we'll see. Other ideas welcome.

[jeremyd2019]

Ugh, it looks like maybe packages that are dependencies of -cc wind up in cycles. I threw in + "mingw-w64-openssl": ["mingw-w64-libxml2"], for the arm fork for now, but I think the libxml2 is coming in via cc, because it's only breaking on clang*.

[lazka]

Yeah.. Not really feasable...

[lazka]

• I'll try to see how many cycles there are for a full repo rebuild.
• The API currently only exposes all transitive dependencies, which isn't ideal for this. I might have to pull in all that dependency logic into autobuild.

[lazka]

There are 881 unique package pairs that are in cycles, so, no this doesn't scale.

[lazka]

Continuing in #52