aleaxit / gmpy

General Multi-Precision arithmetic for Python 2.6+/3+ (GMP, MPIR, MPFR, MPC)
https://gmpy2.readthedocs.io/en/latest/
GNU Lesser General Public License v3.0
521 stars 86 forks source link

Linux wheels in the latest release (2.2.0a1) are broken #438

Closed skirpichev closed 1 year ago

skirpichev commented 1 year ago

I did tests for CPython 3.11 and 3.12. But I suspect - all versions are affected. Maybe MacOS wheels are broken too.

An example:

$ python3.11 -m venv test11
$ . test11/bin/activate
(test11) $ pip install gmpy2==2.2.0a1
Collecting gmpy2==2.2.0a1
  Downloading gmpy2-2.2.0a1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 4.3 MB/s eta 0:00:00
Installing collected packages: gmpy2
Successfully installed gmpy2-2.2.0a1

[notice] A new release of pip is available: 23.1.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
(test11) $ python
Python$ python
Python 3.11.3+ (heads/3.11:9fbb614c4e, Apr 29 2023, 14:18:05) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gmpy2
>>> gmpy2.mpz(2)
Illegal instruction

This is on the Debian stable:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm
$ uname -a
Linux note 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux

I would guess, that this is because uploaded wheels aren't whose which tested in the GA CI. E.g. if I take wheel from the GA artifact (e.g. https://github.com/aleaxit/gmpy/actions/runs/6388473596#artifacts) - this works:

(test11) $ unzip -x wheels.zip gmpy2-2.2.0a2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Archive:  wheels.zip
  inflating: gmpy2-2.2.0a2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(test11) $ pip uninstall -y gmpy2
Found existing installation: gmpy2 2.2.0a1
Uninstalling gmpy2-2.2.0a1:
  Successfully uninstalled gmpy2-2.2.0a1
$ pip install gmpy2-2.2.0a2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Processing ./gmpy2-2.2.0a2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing collected packages: gmpy2
Successfully installed gmpy2-2.2.0a2

[notice] A new release of pip is available: 23.1.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
(test11) $ python
Python 3.11.3+ (heads/3.11:9fbb614c4e, Apr 29 2023, 14:18:05) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gmpy2
>>> gmpy2.mpz(2)
mpz(2)
>>>

I think the proper fix could be - a partial automation of the release process (windows wheels will be uploaded manually): we build wheels in GA jobs, thus we could upload them to the PyPI and/or to the Github releases page. (See how I did this in the Diofant project.)

Edit:

casevh commented 1 year ago

Thanks for catching this. We may be able to create windows wheels automatically. I will open an issue with the status later today.

skirpichev commented 1 year ago

@casevh, maybe this is related to fat-enabled builds? It seems that the Debian gmp package adds --disable-fat option on amd64.

casevh commented 1 year ago

IIRC, some virtualization platforms reported a CPU-ID that didn't precisely correspond to the actual instructions available to the VM.

Does Debian specify a target CPU architecture when compiling GMP?

skirpichev commented 1 year ago

IIRC, some virtualization platforms reported a CPU-ID that didn't precisely correspond to the actual instructions available to the VM.

On another hand, wheels from gmpy2's CI job - works fine for me...

Does Debian specify a target CPU architecture when compiling GMP?

Yes, I think so: https://salsa.debian.org/science-team/gmp/-/blob/master/debian/rules

On another hand, --enable-fat was reverted here in 11 years ago, for v5.0.4. Not sure why: https://salsa.debian.org/science-team/gmp/-/commit/63ceefd6c6bec68004288c84f880a2a6a1424ee6

casevh commented 1 year ago

Possibly related...

https://gmplib.org/list-archives/gmp-bugs/2021-January/004989.html

Unfortunately, I'm not sure what we should do. I think --enable-fat was used for most of the 2.1.x binary wheels. I'm oaky with skipping --enable-fat for the a2 release and watching for any feedback.

skirpichev commented 1 year ago

I think --enable-fat was used for most of the 2.1.x binary wheels.

You did these wheels by hand or they were uploaded to PyPI from the CI artifact?

casevh commented 1 year ago

I've only created the Windows wheels manually. The Linux and Mac wheels were artifacts from CI.

skirpichev commented 1 year ago

Indeed, I'm able to reproduce this issue with wheel from the CI. These seems to be jobs for the tagged commit (v2.2.0a1): https://github.com/aleaxit/gmpy/actions/runs/6294471115

I have a different md5 for gmpy2-2.2.0a1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from this artifact (515ec9c66067d2d1bca21c8fd34ade57 vs 72b20c4686d331a2b0bcc10697176e17 on PyPI), but the issue is valid for this wheel.

casevh commented 1 year ago

Are you running it on a bare-metal Linux host or in a virtual environment?

What is the physical CPU?

On Sun, Oct 29, 2023 at 6:34 PM Sergey B Kirpichev @.***> wrote:

Indeed, I'm able to reproduce this issue with wheel from the CI. These seems to be jobs for the tagged commit (v2.2.0a1): https://github.com/aleaxit/gmpy/actions/runs/6294471115

I have a different md5 for gmpy2-2.2.0a1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from this artifact (515ec9c66067d2d1bca21c8fd34ade57 vs 72b20c4686d331a2b0bcc10697176e17 on PyPI), but the issue is valid for this wheel.

— Reply to this email directly, view it on GitHub https://github.com/aleaxit/gmpy/issues/438#issuecomment-1784341544, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMR234XKBPCLEASMW7DGE3YB373FAVCNFSM6AAAAAA5Z54QRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUGM2DCNJUGQ . You are receiving this because you were mentioned.Message ID: @.***>

skirpichev commented 1 year ago

Are you running it on a bare-metal Linux host or in a virtual environment?

real

What is the physical CPU?

$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 55
model name      : Intel(R) Celeron(R) CPU  N2840  @ 2.16GHz
stepping        : 8
microcode       : 0x813
cpu MHz         : 693.584
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer rdrand lahf_lm 3dnowprefetch epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown
bogomips        : 4333.33
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 55
model name      : Intel(R) Celeron(R) CPU  N2840  @ 2.16GHz
stepping        : 8
microcode       : 0x813
cpu MHz         : 1470.579
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer rdrand lahf_lm 3dnowprefetch epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown
bogomips        : 4333.33
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management: