Falaise from Docker not running on old CPUs

[pfranchini]

On this sort of CPUs I cannot run Falaise 3.3.0 (or 4.0.0) pulled from the docker (SIGILL: illegal instruction). Basically this sort of CPU constitutes all the nodes of the "long" queue of the Imperial cluster. Have you built using -march=native ? Thanks. Paolo

[pfranchi@lxa05 ~]$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 1 Core(s) per socket: 6 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Model name: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz Stepping: 2 CPU MHz: 2660.085 BogoMIPS: 5320.17 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0,2,4,6,8,10 NUMA node1 CPU(s): 1,3,5,7,9,11 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1d

[pfranchi@lxa05 ~]$ uname -a Linux lxa05.hep.ph.ic.ac.uk 3.10.0-957.21.2.el7.x86_64 SuperNEMO-DBD/Falaise#1 SMP Wed Jun 5 14:26:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[drbenmorgan]

Hi @pfranchini, could you post the exact sequence of commands used, and the full error reported please?

Things should be building with core2, but possible I missed something or that one of the packages is activating something it shouldn't.

[pfranchini]

Hi @drbenmorgan, basically the bare-bone command throwing the error is singularity exec -C /vols/build/snemo/falaise/falaise_latest.sif flreconstruct --version Here the exact error: singularity.log

[drbenmorgan]

Hi @pfranchini, are you able to run a simple shell into the container, e.g.

$ singularity shell /vols/build/snemo/falaise/falaise_latest.sif
... Now test ruby...
> brew snemo-shell
... test of python is usually reliable ...
snemo-shell> python -c "print('hello world')" 

I suspect that it is an Arch issue but just want to confirm the basic functionality of the container itself rather than built software inside it.

[pfranchini]

Here you are. Apparently the container has some issues on that specific node.

alias snemo-sing='singularity exec --home /home/hep/pfranchi --bind /vols/build/snemo/ -C /vols/build/snemo/falaise/falaise_latest.sif'

[pfranchi@lxa05 ~]$ snemo-sing ls
MICE  SuperNEMO  bzr-2.7.0  dirac  ncdu  public_html  tmp

[pfranchi@lxa05 ~]$ singularity shell /vols/build/snemo/falaise/falaise_latest.sif
Singularity falaise_latest.sif:~> python -c "print('hello world')" 
hello world

[pfranchi@lxa05 ~]$ snemo-sing brew --version
/opt/supernemo/Homebrew/Library/Homebrew/brew.sh: line 184:    76 Illegal instruction     "$HOMEBREW_PREFIX/opt/curl/bin/curl" --version > /dev/null
Homebrew >=1.7.1 (shallow or no git repository)
Supernemo-dbd/homebrew-core (git revision 42ec; last commit 2019-07-15)

[pfranchi@lxa05 ~]$ snemo-sing brew snemo-shell
/opt/supernemo/Homebrew/Library/Homebrew/brew.sh: line 184:    75 Illegal instruction     "$HOMEBREW_PREFIX/opt/curl/bin/curl" --version > /dev/null
/opt/supernemo/Homebrew/Library/Homebrew/brew.sh: line 184:   192 Illegal instruction     "$HOMEBREW_PREFIX/opt/curl/bin/curl" --version > /dev/null
Homebrew >=1.7.1 (shallow or no git repository)
Supernemo-dbd/homebrew-core (git revision 42ec; last commit 2019-07-15)
Type "brew ls --versions" to list available software
Type "exit" to deactivate the session
snemo-shell> python -c "print('hello world')" 
Illegal instruction

snemo-shell> ls -l /opt/supernemo/opt/curl/bin/curl
-rwxr-xr-x. 1 root root 219960 Aug  1 21:07 /opt/supernemo/opt/curl/bin/curl
snemo-shell> $HOMEBREW_PREFIX/opt/curl/bin/curl --version
Illegal instruction

[drbenmorgan]

Just transferring for reference. Looks like an arch issue, so will need to rebuild and redeploy containers. There's an incoming patch for Falaise, so should be in next day or two.

Note however that these are very old CPUs, so there will be a limit to how long we can continue to support these if we want to maximise performance of the containers.

[pfranchini]

Thanks for that. I have been running MC at Imperial just for my personal convenience, but of course is not really mandatory me for (or for the experiment). I will try to report this to the computing group, but I imagine same concerns might have been raised by other experiment.

[drbenmorgan]

Resolved here by patch to always build against core2 cpus.