search

caddyserver / caddy

Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
https://caddyserver.com
Apache License 2.0
55.71k stars 3.92k forks source link

Arm64 version 2.1.0 and 2.1.1 run the same error #3537

Closed IDSSC closed 4 years ago

IDSSC commented 4 years ago

Arm64 version 2.1.0 and 2.1.1 run the same error

SIGILL: illegal instruction PC=0x427ca0 m=0 sigcode=1

goroutine 1 [running, locked to thread]: github.com/klauspost/cpuid.getProcFeatures(0xfaaf60, 0x4000104990, 0x11dab7f, 0xc, 0x4000069e18, 0x421cd4, 0x216e6c0, 0x421cd0, 0x4000069e28, 0x51a5c, ...) github.com/klauspost/cpuid@v1.3.0/cpuid_arm64.s:15 fp=0x4000069dd0 sp=0x4000069dd0 pc=0x427ca0 github.com/klauspost/cpuid.addInfo(0x216e6c0) github.com/klauspost/cpuid@v1.3.0/detect_arm64.go:42 +0x20 fp=0x4000069e00 sp=0x4000069dd0 pc=0x422fd0 github.com/klauspost/cpuid.Detect() github.com/klauspost/cpuid@v1.3.0/cpuid.go:289 +0x70 fp=0x4000069e20 sp=0x4000069e00 pc=0x421d60 github.com/klauspost/cpuid.init.0() github.com/klauspost/cpuid@v1.3.0/cpuid.go:272 +0x24 fp=0x4000069e30 sp=0x4000069e20 pc=0x421cd4 runtime.doInit(0x20d4720) runtime/proc.go:5420 +0x9c fp=0x4000069e70 sp=0x4000069e30 pc=0x51a5c runtime.doInit(0x20e9620) runtime/proc.go:5415 +0x58 fp=0x4000069eb0 sp=0x4000069e70 pc=0x51a18 runtime.doInit(0x20e6820) runtime/proc.go:5415 +0x58 fp=0x4000069ef0 sp=0x4000069eb0 pc=0x51a18 runtime.doInit(0x20e6940) runtime/proc.go:5415 +0x58 fp=0x4000069f30 sp=0x4000069ef0 pc=0x51a18 runtime.doInit(0x20d3a60) runtime/proc.go:5415 +0x58 fp=0x4000069f70 sp=0x4000069f30 pc=0x51a18 runtime.main() runtime/proc.go:190 +0x1b8 fp=0x4000069fd0 sp=0x4000069f70 pc=0x44828 runtime.goexit() runtime/asm_arm64.s:1148 +0x4 fp=0x4000069fd0 sp=0x4000069fd0 pc=0x74604

r0 0x216e6c0 r1 0x4000068380 r2 0x4000069e00 r3 0x7 r4 0xfaaf60 r5 0x0 r6 0x11dab7f r7 0xcf r8 0xf r9 0x4 r10 0x0 r11 0xd r12 0x1 r13 0x802aaaa00aaaa r14 0xf8 r15 0x0 r16 0x34 r17 0x68 r18 0x0 r19 0xd0 r20 0x4000069ca8 r21 0x40001203a0 r22 0x4000002000 r23 0x0 r24 0x0 r25 0x0 r26 0x20d4748 r27 0x216e748 r28 0x4000000180 r29 0x4000069dc8 lr 0x422fd0 sp 0x4000069dd0 pc 0x427ca0 fault 0x0

mholt commented 4 years ago

Oh, fun. That looks like a bug in github.com/klauspost/cpuid then.

/cc @klauspost - have you seen this before? Are we "holding it wrong"? 😁

klauspost commented 4 years ago

It must be a problem with this: https://github.com/klauspost/cpuid/pull/44

@fwessels could you take a look?

fwessels commented 4 years ago

Looks like the following instruction is failing:

    mrs x0, id_aa64pfr0_el1  /* Processor Feature Register 0 */

What arm64 chip is this being run on?

(maybe paste the output from lscpu ?)

IDSSC commented 4 years ago

lscpu

Architecture: aarch64 Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: ARM Model: 4 Model name: Cortex-A53 Stepping: r0p4 CPU max MHz: 1512.0000 CPU min MHz: 100.0000 BogoMIPS: 48.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32

fwessels commented 4 years ago

Thanks, what sort of server is this? Something like Pine64 perhaps?

IDSSC commented 4 years ago

Hardware model: Amlogic s905d2

fwessels commented 4 years ago

@IDSSC can you clone https://github.com/fwessels/investigate-arm64 and run go test -v?

Eg. on an AWS Graviton2 instance I am getting:

$ go test -v 
=== RUN   TestMidr
 implementer:  0x41
     variant:   0x3
architecture:   0xf
    part num: 0xd0c
    revision:   0x1
--- PASS: TestMidr (0.00s)
=== RUN   TestProcFeatures
Processor Feature Register 0: 0x0000000000110011
--- PASS: TestProcFeatures (0.00s)
=== RUN   TestInstAttributess
Instruction Set Attribute Register 0: 0x0000100010211120
Instruction Set Attribute Register 1: 0x0000000000100001
--- PASS: TestInstAttributess (0.00s)
PASS
IDSSC commented 4 years ago

Docker container golang:1.14.4-alpine3.11 $ go test -v fork/exec /tmp/go-build217936674/b001/investigate-arm64.test: permission denied FAIL _/media/EXTHD/ext/src/investigate-arm64 0.001s

fwessels commented 4 years ago

Looks like the program was not executed, can you try running it with sudo ?

IDSSC commented 4 years ago

Neither root permissions nor normal permissions can be run

klauspost commented 4 years ago

Possibly Related: https://github.com/golang/go/issues/37345

klauspost commented 4 years ago

yes, id_aa64isar0_el1 is privileged. The FreeBSD kernel traps the exception and returns correct values to userland (at least on "recent" version of FreeBSD, I think it appears in FreeBSD 12.0, not sure it works in 11.x)

@IDSSC What OS are you running?

The Go runtime will likely have the same problems.

IDSSC commented 4 years ago

The host system is similar to Ubuntu 18.04

Caddy2.0.0 can be started before, and errors start to be reported after 2.1.0

  1. Running in a self-built docker container based on alpine3.11, using official or self-compiled caddy arm64 file to run error
  2. Running in a self-built Docker container based on Ubuntu 18.04.4, using the official or self-compiled caddy arm64 file to run an error
  3. Running on the host system, using the caddy arm64 file compiled by the official or self-run error

All three methods report the same error

root@ubuntu:/# uname -a Linux ubuntu 4.9.76 #1 SMP PREEMPT Thu Apr 11 20:39:46 CST 2019 aarch64 aarch64 aarch64 GNU/Linux root@ubuntu:/# caddy -version SIGILL: illegal instruction PC=0x427ca0 m=0 sigcode=1

goroutine 1 [running, locked to thread]: github.com/klauspost/cpuid.getProcFeatures(0xfaaf60, 0x400007e930, 0x11dab7f, 0xc, 0x4000063e18, 0x421cd4, 0x216e6c0, 0x421cd0, 0x4000063e28, 0x51a5c, ...) github.com/klauspost/cpuid@v1.3.0/cpuid_arm64.s:15 fp=0x4000063dd0 sp=0x4000063dd0 pc=0x427ca0 github.com/klauspost/cpuid.addInfo(0x216e6c0) github.com/klauspost/cpuid@v1.3.0/detect_arm64.go:42 +0x20 fp=0x4000063e00 sp=0x4000063dd0 pc=0x422fd0 github.com/klauspost/cpuid.Detect() github.com/klauspost/cpuid@v1.3.0/cpuid.go:289 +0x70 fp=0x4000063e20 sp=0x4000063e00 pc=0x421d60 github.com/klauspost/cpuid.init.0() github.com/klauspost/cpuid@v1.3.0/cpuid.go:272 +0x24 fp=0x4000063e30 sp=0x4000063e20 pc=0x421cd4 runtime.doInit(0x20d4720) runtime/proc.go:5420 +0x9c fp=0x4000063e70 sp=0x4000063e30 pc=0x51a5c runtime.doInit(0x20e9620) runtime/proc.go:5415 +0x58 fp=0x4000063eb0 sp=0x4000063e70 pc=0x51a18 runtime.doInit(0x20e6820) runtime/proc.go:5415 +0x58 fp=0x4000063ef0 sp=0x4000063eb0 pc=0x51a18 runtime.doInit(0x20e6940) runtime/proc.go:5415 +0x58 fp=0x4000063f30 sp=0x4000063ef0 pc=0x51a18 runtime.doInit(0x20d3a60) runtime/proc.go:5415 +0x58 fp=0x4000063f70 sp=0x4000063f30 pc=0x51a18 runtime.main() runtime/proc.go:190 +0x1b8 fp=0x4000063fd0 sp=0x4000063f70 pc=0x44828 runtime.goexit() runtime/asm_arm64.s:1148 +0x4 fp=0x4000063fd0 sp=0x4000063fd0 pc=0x74604

r0 0x216e6c0 r1 0x4000062380 r2 0x4000063e00 r3 0x5 r4 0xfaaf60 r5 0x0 r6 0x11dab7f r7 0xeb r8 0xf r9 0x4 r10 0x0 r11 0xd r12 0x1 r13 0x802aaaa00aaaa r14 0xf8 r15 0x0 r16 0x34 r17 0x68 r18 0x0 r19 0xd0 r20 0x4000063ca8 r21 0x400019a3a0 r22 0x4000002000 r23 0x0 r24 0x0 r25 0x0 r26 0x20d4748 r27 0x216e748 r28 0x4000000180 r29 0x4000063dc8 lr 0x422fd0 sp 0x4000063dd0 pc 0x427ca0 fault 0x0

klauspost commented 4 years ago

What do you mean "similar to Ubuntu 18.04"?

The reason I am asking is that the OS seems to lack support for intercepting these instructions, so I would like to know more about it to see if this is a general problem. The main issue is that there is no way of testing for this AFAICT.

The ticket above indicates that this would also affect the Go runtime. from v1.14 and going forward.

IDSSC commented 4 years ago

I tested the version of 2.1.0 and 2.1.1, this error can be reproduced in arm64 Android, Ubuntu, alpine But the 2.0.0 program is running normally, it can be determined that this error was introduced in 2.1.0

mholt commented 4 years ago

Caddy 2.0.0 used cpuid v1.2.3; Caddy 2.1.0 (and 2.1.1) uses cpuid v1.3.0.

klauspost commented 4 years ago

@mholt You can revert to v1.2.5 for now. It should be fine for the use you have. I will work with Frank on getting the issue resolved outside of here.