feat: add aarch64 manylinux wheels

mayeut commented 5 months ago

Summary

OS: Linux
Bug fix: no
Type: wheels
Fixes: fixes #1966 fixes #1972 fixes #2206 mitigates #2090 mitigates #2175 mitigates #2293 closes #2178 closes #2224

Description

This PR continues work from #2103 (I've never removed that branch and rebased on master once in a while so that's the same branch in this PR).

cherry-picked commits from #2126 for missing test dependency in manylinux.

main differences between this PR and #2224:

This PR does not build ppc64le & s390x.
This PR does not build musllinux wheels.
This PR adds tests for architectures running QEMU: While the tests needed some modifications in order to pass, IMHO, it's best to check than what can be tested does work as intended. Running with QEMU however hides some bugs (thinking about /proc/cpuinfo and cpu related stuff here) that actually do exist or comes with its own bugs (s390x and /proc endianness issue is most likely a qemu issue). aarch64 will most likely be able to run natively by the end of the year.
This PR does not add Windows ARM64 wheel It could be added without any tests but I'd rather see Microsoft add Windows ARM64 runners to GitHub if they want to get some traction for their new platform (hopefully by the end of the year).
This PR does not add macOS universal2 wheel My personal opinion is that it's not useful as long as you provide both x86_64 & arm64 wheels (which is the case). They do take more space for the end user, pip will always choose the most specific one so publishing 3 will change nothing for almost all users. Remain people packaging universal2 python apps, and, IMHO, this should be handled by the tools creating such apps.
This PR feels simpler regarding the change of workflow

Regarding tests that are failing on real arm64 hardware but hidden by QEMU testing:

Using docker on macOS M1:


======================================================================
FAIL: psutil.tests.test_system.TestCpuAPIs.test_cpu_freq
----------------------------------------------------------------------
Traceback (most recent call last):
File "/project/psutil/tests/test_system.py", line 609, in test_cpu_freq
assert ls, ls
AssertionError: []

====================================================================== FAIL: psutil.tests.test_linux.TestSystemCPUCountCores.test_method_2

Traceback (most recent call last): File "/project/psutil/tests/test_linux.py", line 820, in test_method_2 self.assertEqual(meth_1, meth_2) AssertionError: 10 != None

====================================================================== FAIL: psutil.tests.test_linux.TestSystemCPUFrequency.test_emulate_use_second_file

Traceback (most recent call last): File "/project/psutil/tests/test_linux.py", line 845, in test_emulate_use_second_file assert psutil.cpu_freq() AssertionError

====================================================================== FAIL: psutil.tests.test_linux.TestSystemNetIfAddrs.test_ips

Traceback (most recent call last): File "/project/psutil/tests/test_linux.py", line 1020, in test_ips self.assertEqual(addr.address, get_mac_address(name)) AssertionError: '00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00' != '00:00:00:00:00:00'

00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
00:00:00:00:00:00

Using CirrusCI (Neoverse-N1):

======================================================================
FAIL: psutil.tests.test_system.TestCpuAPIs.test_cpu_freq
----------------------------------------------------------------------
Traceback (most recent call last):
File "/project/psutil/tests/test_system.py", line 609, in test_cpu_freq
assert ls, ls
AssertionError: []
======================================================================
FAIL: psutil.tests.test_system.TestSensorsAPIs.test_sensors_temperatures
----------------------------------------------------------------------
Traceback (most recent call last):
File "/project/psutil/tests/test_system.py", line 942, in test_sensors_temperatures
self.assertGreaterEqual(entry.high, 0)
AssertionError: -273.15 not greater than or equal to 0
======================================================================
FAIL: psutil.tests.test_linux.TestSystemCPUCountCores.test_method_2
----------------------------------------------------------------------
Traceback (most recent call last):
File "/project/psutil/tests/test_linux.py", line 820, in test_method_2
self.assertEqual(meth_1, meth_2)
AssertionError: 4 != None
======================================================================
FAIL: psutil.tests.test_linux.TestSystemCPUFrequency.test_emulate_use_second_file
----------------------------------------------------------------------
Traceback (most recent call last):
File "/project/psutil/tests/test_linux.py", line 845, in test_emulate_use_second_file
assert psutil.cpu_freq()
AssertionError

giampaolo commented 5 months ago

Thank you Matthieu.

I added other platforms as well (ppc64le & s390x) [...] and musllinux

Please, let's just add aarch64 wheels for now (the ones with the highest demand). There's too many skipIf(...) which seems are due to emulation. Just so you know: my main concern here is keeping things as simple and maintainable as possible. Above all, we shouldn't introduce new random test failures (we already have a lot), and producing wheels / releases should be a smooth process.

mayeut commented 5 months ago

Please, let's just add aarch64 wheels for now (the ones with the highest demand)

I removed ppc64le & s390x (there's one demand from IBM #2210 for ppc64le).

I kept musllinux which are I think the second highest demand with 1 related issue & 3 opened PR. If you don't want to keep musllinux, just drop the commit 445b35d. Please note that 6f55304 is related to the addition of musllinux which comes with an up-to-date version of free/vmstat. Even if not building musllinux, this one will eventually come back as an issue once glibc distros get updated.

There's too many skipIf(...) which seems are due to emulation. Just so you know: my main concern here is keeping things as simple and maintainable as possible.

There are now less with s390x being dropped. I however understand your concern. Given GHA will most likely provide a native aarch64 before the end of the year, the QEMU test commit might be dropped completely 6aa80d3.

Above all, we shouldn't introduce new random test failures (we already have a lot), and producing wheels / releases should be a smooth process.

Agreed, that's why I disabled test_memleaks on those architectures. It was the only flaky one as far as I saw (and memleaks should mostly be independent of the processor architecture and tested otherwise).

Please feel free to force-push, drop commits, ... as you see fit. I have backups / alternatives to this work in my fork.

giampaolo commented 5 months ago

If you don't want to keep musllinux, just drop the commit 445b35d.

It seems I cannot modify your PR (I can push but can't see the changes reflected in here). Can you please remove musl wheels and update the README? The rest looks good. Also one minor thing. Is there a reason why it's called QEMU_USER instead of QEMU?

mayeut commented 5 months ago

It seems I cannot modify your PR (I can push but can't see the changes reflected in here).

weird... anyway

Also one minor thing. Is there a reason why it's called QEMU_USER instead of QEMU?

QEMU provides full-system emulation or user-mode emulation. Here, we only use user-mode emulation.

mayeut commented 5 months ago

update the README?

I guess you meant HISTORY ?

giampaolo commented 5 months ago

Yes, sorry. 😄

giampaolo / psutil