minio-mc error: Fatal glibc error: CPU does not support x86-64-v2

[rikukissa]

Root cause

MinIO MC is defined without a fixed version in docker-compose.

  minio-mc:
    image: minio/mc

Impact

• Basic processes like data reset doesn't work on servers
• Might also impact backups and other secondary processes
• Security: we might accidentally download a version of the library with a vulnerability planted

Tasks

• [ ] Define a fixed version for minio-mc. minio/mc:RELEASE.2023-09-13T23-08-58Z is pretty close to the version of MinIO we use
• [ ] Notice that we also refer to a version in infrastructure/clear-all-data.sh. Hardcode a specific image version there as well

More details

riku@e2e-64-3:~$ docker service logs -f opencrvs_minio-mc
opencrvs_minio-mc.1.lrs315zqt5ew@e2e-64-3    | Fatal glibc error: CPU does not support x86-64-v2
opencrvs_minio-mc.1.beil5xx5zagq@e2e-64-3    | Fatal glibc error: CPU does not support x86-64-v2
opencrvs_minio-mc.1.db3qco5tsssp@e2e-64-3    | Fatal glibc error: CPU does not support x86-64-v2
opencrvs_minio-mc.1.ohm5lp4n7yfv@e2e-64-3    | Fatal glibc error: CPU does not support x86-64-v2
opencrvs_minio-mc.1.bdjeo94j1rxh@e2e-64-3    | Fatal glibc error: CPU does not support x86-64-v2

riku@e2e-64-3:~$ uname -a
Linux e2e-64-3 6.0.0-060000-generic #202210022231 SMP PREEMPT_DYNAMIC Sun Oct 2 22:35:09 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Another machine affected

Linux IMM-FASO-SRV06-DGMEC-SERVER 6.8.0-48-generic #48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

riku@e2e-64-3:~$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               GenuineIntel
  Model name:            QEMU Virtual CPU version 2.5+
    CPU family:          6
    Model:               13
    Thread(s) per core:  1
    Core(s) per socket:  1
    Socket(s):           16
    Stepping:            3
    BogoMIPS:            5786.39
    Flags:               fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm pti
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   512 KiB (16 instances)
  L1i:                   512 KiB (16 instances)
  L2:                    64 MiB (16 instances)
  L3:                    256 MiB (16 instances)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:
  Itlb multihit:         KVM: Mitigation: VMX unsupported
  L1tf:                  Mitigation; PTE Inversion
  Mds:                   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Unknown: No mitigations
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[n1koo]

This seems to be caused issue discussed in https://github.com/minio/minio/issues/18365 and https://github.com/minio/minio/pull/18370

tl;dr older CPUs (or emulated ones) doesn't support latest extensions causing the failure.

There is a specific image (-cpuv1 postfix) that is build against older extensions but is likely to some extent slower on newer hardware.

Now the question is do we want to use those v1 images by default or should it be something we document to be editable in niche cases? Probably latter?

However we definitely should lock versions