Travis Python3.7 build broken

[spigo900]

Our Travis CI build for CPython 3.7 has been broken for a few months now. Example. (The PyPy3.7 builds work fine. Example.) Writing this up here as a place to put "things we've learned/eliminated."

• It's not the build system stuff that prints at the top, because those are identical between the PyPy3.7 and CPy3.7 builds
  • It's not Travis version
    • last time I compared, both had the same runtime kernel version (4.15.0-1098-gcp) and travis-build version (29f53f14)
  • It's not OS version
    • both used Ubuntu (Xenial) 16.04.7 LTS
  • It's not the other tool versions
• It's probably not (just) the precise version of Python used/imitated
  • the PyPy3.7 build says Python 3.7.10, while the CPy3.7 build prints Python 3.7.1
  • however, building locally on my Mac with CPy 3.7.1 doesn't reproduce the problem
• It's not the pip/setuptools/wheel version
  • Confirmed that both builds upgrade to pip==22.1.1, setuptools==62.3.2, wheel==0.37.1.
• There is a requirements difference which might explain why one is crashing and not the other
  • PyPy3.7 is installing requirements from requirements_pypy.txt, which means it is not installing towncrier or its dependency, click-default-group(==1.2.2).
  • CPy3.7 is installing requirements from requirements_lock.txt, which means it is installing towncrier==21.9.0 plus click-default-group==1.2.2.
  • So, maybe click-default-group==1.2.2 is broken on this combination of Ubuntu/Python/pip/setuptools version? I've only tested locally on Mac OS and it doesn't break in that context.
  • But it seems unlikely that click-default-group is the root problem.
  • The logs are complaining about the importlib-metadata version used. They say it's incompatible with the installed version of setuptools.
  • From the traceback there seems to be something weird going on. importlib-metadata has already been installed separately, because the trace references a path for it inside site-packages, but it must not be the version one we locked, because it's incompatible with the setuptools version, and it's also not using the "vendored" one... except at the end of the stack, which seems pretty strange.
  • I'm guessing importlib-metadata might be one of the "basic" packages Travis installs in the Python virtualenv? I'd have to check on this.
  • So, there are two things to look at here:
    • Does it work if we tell pip to upgrade importlib_metadata as well as pip, setuptools and wheel? I'll try committing this change on the m5-objects branch and see if it helps.
    • Does Travis install importlib-metadata by default? I think it must -- the raw CPy3.7 log doesn't mention installing it at at any point.

[spigo900]

Yep, looks like it was importlib_metadata! I added it to the upgrade command in 990a8eb35fe365f148817872b155cb848615b814 and it is working again. Apparently the default is really damn old:

  Attempting uninstall: importlib_metadata
    Found existing installation: importlib-metadata 0.18
    Uninstalling importlib-metadata-0.18:
      Successfully uninstalled importlib-metadata-0.18

(MyPy is still unhappy, but that's a separate issue. 🙃)

[lichtefeld]

@spigo900 Maybe we should update the base OS version to something other than Xenial as well :) LTS 16.04.07 is well old...

[spigo900]

@lichtefeld Sure, that might be worthwhile. It looks like Travis also supports Bionic (18.04) and Focal (20.04) which are newer. Same dist naming scheme, we'd specify bionic or focal.

[spigo900]

Updated the OS to focal as well.

It looks like the importlib version was because of using an old version of virtualenv:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

virtualenv 20.0.20 requires importlib-metadata<2,>=0.12; python_version < "3.8", but you have importlib-metadata 4.2.0 which is incompatible.

Fortunately things seem to work okay despite virtualenv's warning.