Releases are broken by Linux aarch64

[jsirois]

The pantsbuild.pants wheels generated by the Linux aarch64 shard are not manylinux and this is a no-go:

$ ./build-support/bin/release.sh publish
...
Uploading distributions to https://upload.pypi.org/legacy/
Signing pantsbuild.pants.testutil-2.16.0.dev5-py3-none-any.whl
Signing pantsbuild.pants-2.16.0.dev5-cp37-cp37m-manylinux2014_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_11_0_arm64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp38-cp38-manylinux2014_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-linux_aarch64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp37-cp37m-linux_aarch64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp37-cp37m-macosx_10_15_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp38-cp38-macosx_10_15_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp38-cp38-linux_aarch64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_10_15_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-manylinux2014_x86_64.whl
Uploading pantsbuild.pants.testutil-2.16.0.dev5-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.2/52.2 kB • 00:00 • 81.0 MB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp37-cp37m-manylinux2014_x86_64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.2/60.2 MB • 02:31 • 391.7 kB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_11_0_arm64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.4/10.4 MB • 00:21 • 385.6 kB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp38-cp38-manylinux2014_x86_64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.2/60.2 MB • 02:30 • 403.1 kB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp39-cp39-linux_aarch64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.8/59.8 MB • 02:28 • 404.2 kB/s
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
         Binary wheel 'pantsbuild.pants-2.16.0.dev5-cp39-cp39-linux_aarch64.whl' has an unsupported platform tag 'linux_aarch64'.
Traceback (most recent call last):
  File "/tmp/pants-sandbox-8EEcEk/./.cache/pex_root/venvs/371f239c6e05f3b21b727016c72f786b4240472e/c118e6ce068494080810f1d842becda796e2eddb/pex", line 270, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 1247, in <module>
    main()
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 1223, in main
    publish()
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 897, in publish
    upload_wheels_via_twine()
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 1020, in upload_wheels_via_twine
    check=True,
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/jsirois/dev/pantsbuild/pants/build-support/twine-deps.venv/bin/twine', 'upload', '--sign', '--sign-with=gpg2', '--identity=93E55CB567B5C626', '--skip-existing', '/home/jsirois/dev/pantsbuild/pants/dist/deploy/wheels/pantsbuild.pants/7766b9b3cedbfd9a90d5bc934b62e03ee7020ccd/2.16.0.dev5/*.whl']' returned non-zero exit status 1.

[jsirois]

@benjyw you have the context right now for the new Linux aarch64 shards and what you went through to arrive where things are at. I poked around a bit and we do not use the manylinux repair ... tool in either x86_64 or aarch64 which is surprising. So it seems like there is some magic in our Linux x86_64 manylinux image that somehow lets us get away without doing this?

[benjyw]

Yeah, I just did whatever we did for x86_64. I guess I was under the impression that building in a manylinux image was sufficient. I'll dig further.

[jsirois]

Ah, right. I once knew this. We do something incredibly hokey ... in the release script. We re-write the wheels and do this ourselves, but ~incorrectly and do not use auditwheel. I can fix the hokey to finish the release - it will still be hokey.

[benjyw]

Any reason not to use auditwheel? I know we do something weird with the stable_version, am hoping we can stop doing that and just build with the stable version up front.

[jsirois]

There is no reason not to. I will not be doing that though to solve this, I'm just fixing the broken jank with working jank for the immediate purpose.

[benjyw]

Sounds good.