search

openshift / microshift

A small form factor OpenShift/Kubernetes optimized for edge computing
https://microshift.io
Apache License 2.0
670 stars 196 forks source link

[BUG] runtime.sigpanic while running F34 on aarch64 #219

Closed jbpratt closed 2 years ago

jbpratt commented 3 years ago

What happened:

Upon running microshift, I'm met with a large cgo trace, with what looks like an issue loading the boringssl DSO.

What you expected to happen:

A nice help command would be presented :smiley_cat:

How to reproduce it (as minimally and precisely as possible):

  1. Install FCOS34 on a Raspberry Pi (details below)
  2. Once booted, follow the install.sh script (mainly install cri-{o,tools})
  3. Download a build of microshift
  4. execute it!

Anything else we need to know?:

Booting FCOS on an RPi4 can be done roughly like so:

  1. Download RPi firemware
  2. Download aarch64 FCOS image (https://builds.coreos.fedoraproject.org/browser?stream=testing-devel&arch=aarch64)
  3. Create ignition file and generate image 
    podman run --pull=always \
--privileged --rm \
-v /dev:/dev \
-v /run/udev:/run/udev \
-v .:/data \
-w /data quay.io/coreos/coreos-installer:release install ${DISK} -i ${IGN} -f ${IMAGE} --offline --insecure
  4. Mount the second partition on the disk and unzip firmware
  5. Boot! There is a 3GB memory limit that will need to be removed on restart.

Environment:

[core@kitkat ~]$ cat /etc/os-release
NAME=Fedora
VERSION="34.20210810.dev.0 (CoreOS)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora CoreOS 34.20210810.dev.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='34.20210810.dev.0'
DEFAULT_HOSTNAME=localhost

Relevant Logs

[core@kitkat ~]$ rpm-ostree status
State: idle
Deployments:
* fedora:fedora/aarch64/coreos/testing-devel
                   Version: 34.20210810.dev.0 (2021-08-10T01:22:06Z)
                BaseCommit: 6e1fb34b883d1f3b8ff413a5819978c2fa7721868606301826be44a3d6e2c83e
              GPGSignature: (unsigned)
           LayeredPackages: openssl-devel cri-tools cri-o

  fedora:fedora/aarch64/coreos/testing-devel
                   Version: 34.20210810.dev.0 (2021-08-10T01:22:06Z)
                BaseCommit: 6e1fb34b883d1f3b8ff413a5819978c2fa7721868606301826be44a3d6e2c83e
              GPGSignature: (unsigned)
           LayeredPackages: cri-tools cri-o
[core@kitkat ~]$ curl -LO https://github.com/redhat-et/microshift/releases/download/4.7.0-0.microshift-2021-08-10-163759/microshift-linux-arm64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   630  100   630    0     0   2985      0 --:--:-- --:--:-- --:--:--  2985
100  110M  100  110M    0     0   9.8M      0  0:00:11  0:00:11 --:--:-- 19.9M
[core@kitkat ~]$ sudo chmod +x microshift-linux-arm64
[core@kitkat ~]$ ./microshift-linux-arm64
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38288f70 pc=0xffff7e2ace94]

runtime stack:
runtime.throw(0x44322ce, 0x2a)
/usr/lib/golang/src/runtime/panic.go:1116 +0x54
runtime.sigpanic()
/usr/lib/golang/src/runtime/signal_unix.go:726 +0x444

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x38480e0, 0x400009ed38, 0x3e034c0)
/usr/lib/golang/src/runtime/cgocall.go:133 +0x50 fp=0x400009ed00 sp=0x400009ecc0 pc=0x403b40
crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL(0x0)
_cgo_gotypes.go:287 +0x44 fp=0x400009ed30 sp=0x400009ed00 pc=0x66e564
crypto/internal/boring.init.0()
/usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x40 fp=0x400009ed70 sp=0x400009ed30 pc=0x674400
runtime.doInit(0x6eb5a80)
/usr/lib/golang/src/runtime/proc.go:5652 +0x9c fp=0x400009edb0 sp=0x400009ed70 pc=0x442ffc
runtime.doInit(0x6eb1c80)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009edf0 sp=0x400009edb0 pc=0x442fb8
runtime.doInit(0x6edb280)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ee30 sp=0x400009edf0 pc=0x442fb8
runtime.doInit(0x6edc680)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ee70 sp=0x400009ee30 pc=0x442fb8
runtime.doInit(0x6e9ed20)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009eeb0 sp=0x400009ee70 pc=0x442fb8
runtime.doInit(0x6ed7720)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009eef0 sp=0x400009eeb0 pc=0x442fb8
runtime.doInit(0x6ed2d20)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ef30 sp=0x400009eef0 pc=0x442fb8
runtime.doInit(0x6eae480)
/usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ef70 sp=0x400009ef30 pc=0x442fb8
runtime.main()
/usr/lib/golang/src/runtime/proc.go:191 +0x1b8 fp=0x400009efd0 sp=0x400009ef70 pc=0x435d08
runtime.goexit()
/usr/lib/golang/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400009efd0 sp=0x400009efd0 pc=0x468904

goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x71cd1a0)
/opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:1164 +0x70
created by k8s.io/klog/v2.init.0
/opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:418 +0x124

looping in @dustymabe since we chatted about this

rootfs commented 3 years ago

@jbpratt thanks for test driving arm64, would you please get the entire back trace, especially goroutine 19?

jbpratt commented 3 years ago

Hi @rootfs! Sadly this is the entire trace that is shown. Here are 2 executions just now, though it doesn't seem very helpful. If there is a way to get more info out of this, please let me know.

Details ``` [core@kitkat ~]$ sudo ./microshift-linux-arm64 fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x1f6f9f70 pc=0xffff603a6e94] runtime stack: runtime.throw(0x44322ce, 0x2a) /usr/lib/golang/src/runtime/panic.go:1116 +0x54 runtime.sigpanic() /usr/lib/golang/src/runtime/signal_unix.go:726 +0x444 goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x38480e0, 0x400009ad38, 0x3e034c0) /usr/lib/golang/src/runtime/cgocall.go:133 +0x50 fp=0x400009ad00 sp=0x400009acc0 pc=0x403b40 crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL(0x0) _cgo_gotypes.go:287 +0x44 fp=0x400009ad30 sp=0x400009ad00 pc=0x66e564 crypto/internal/boring.init.0() /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x40 fp=0x400009ad70 sp=0x400009ad30 pc=0x674400 runtime.doInit(0x6eb5a80) /usr/lib/golang/src/runtime/proc.go:5652 +0x9c fp=0x400009adb0 sp=0x400009ad70 pc=0x442ffc runtime.doInit(0x6eb1c80) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009adf0 sp=0x400009adb0 pc=0x442fb8 runtime.doInit(0x6edb280) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ae30 sp=0x400009adf0 pc=0x442fb8 runtime.doInit(0x6edc680) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ae70 sp=0x400009ae30 pc=0x442fb8 runtime.doInit(0x6e9ed20) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009aeb0 sp=0x400009ae70 pc=0x442fb8 runtime.doInit(0x6ed7720) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009aef0 sp=0x400009aeb0 pc=0x442fb8 runtime.doInit(0x6ed2d20) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009af30 sp=0x400009aef0 pc=0x442fb8 runtime.doInit(0x6eae480) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009af70 sp=0x400009af30 pc=0x442fb8 runtime.main() /usr/lib/golang/src/runtime/proc.go:191 +0x1b8 fp=0x400009afd0 sp=0x400009af70 pc=0x435d08 runtime.goexit() /usr/lib/golang/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400009afd0 sp=0x400009afd0 pc=0x468904 goroutine 19 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x71cd1a0) /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:1164 +0x70 created by k8s.io/klog/v2.init.0 /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:418 +0x124 [core@kitkat ~]$ sudo ./microshift-linux-arm64 fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x3475cf70 pc=0xffff9103ce94] runtime stack: runtime.throw(0x44322ce, 0x2a) /usr/lib/golang/src/runtime/panic.go:1116 +0x54 runtime.sigpanic() /usr/lib/golang/src/runtime/signal_unix.go:726 +0x444 goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x38480e0, 0x400009ad38, 0x3e034c0) /usr/lib/golang/src/runtime/cgocall.go:133 +0x50 fp=0x400009ad00 sp=0x400009acc0 pc=0x403b40 crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL(0x0) _cgo_gotypes.go:287 +0x44 fp=0x400009ad30 sp=0x400009ad00 pc=0x66e564 crypto/internal/boring.init.0() /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x40 fp=0x400009ad70 sp=0x400009ad30 pc=0x674400 runtime.doInit(0x6eb5a80) /usr/lib/golang/src/runtime/proc.go:5652 +0x9c fp=0x400009adb0 sp=0x400009ad70 pc=0x442ffc runtime.doInit(0x6eb1c80) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009adf0 sp=0x400009adb0 pc=0x442fb8 runtime.doInit(0x6edb280) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ae30 sp=0x400009adf0 pc=0x442fb8 runtime.doInit(0x6edc680) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009ae70 sp=0x400009ae30 pc=0x442fb8 runtime.doInit(0x6e9ed20) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009aeb0 sp=0x400009ae70 pc=0x442fb8 runtime.doInit(0x6ed7720) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009aef0 sp=0x400009aeb0 pc=0x442fb8 runtime.doInit(0x6ed2d20) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009af30 sp=0x400009aef0 pc=0x442fb8 runtime.doInit(0x6eae480) /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009af70 sp=0x400009af30 pc=0x442fb8 runtime.main() /usr/lib/golang/src/runtime/proc.go:191 +0x1b8 fp=0x400009afd0 sp=0x400009af70 pc=0x435d08 runtime.goexit() /usr/lib/golang/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400009afd0 sp=0x400009afd0 pc=0x468904 goroutine 6 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x71cd1a0) /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:1164 +0x70 created by k8s.io/klog/v2.init.0 /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:418 +0x124 ```
rootfs commented 3 years ago

@jbpratt this is interesting: boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL, i think microshift is built without external library dependent, but can you run ldd microshift-linux-arm64 to confirm? thanks

jbpratt commented 3 years ago

@jbpratt this is interesting: boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL, i think microshift is built without external library dependent, but can you run ldd microshift-linux-arm64 to confirm? thanks

[core@kitkat ~]$ ldd microshift-linux-arm64
    not a dynamic executable

here you go!

rootfs commented 3 years ago

this is similar to this BZ, the solution to the BZ is to drop the static build. cc @copejon

rootfs commented 3 years ago

@jbpratt tested on my rpi4 (running ubuntu 20.04), no issue. there are still library loaded but it looks your os-tree build already picks them up (libcrypto)

# ps -ef |grep microshift
root       14217    5774 86 16:41 pts/1    00:03:42 ./microshift-linux-arm64 run
root       15135   14868  0 16:45 pts/2    00:00:00 grep --color=auto microshift
# ls -l /proc/14217/map_files/ |awk '{print $NF}'|uniq
0
/home/hchen/microshift-linux-arm64
/var/lib/microshift/etcd/member/snap/db
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
/usr/lib/aarch64-linux-gnu/ld-2.33.so
/usr/lib/aarch64-linux-gnu/libc-2.33.so
/usr/lib/aarch64-linux-gnu/libpthread-2.33.so
/usr/lib/aarch64-linux-gnu/libdl-2.33.so
rootfs commented 3 years ago

to narrow down the openssl lib issue, please build a binary with openssl static library

# install openssl static first
# on fedora
# yum install openssl-static
# on ubuntu
# apt-get install libssl-dev

# build microshift binary using static openssl library:
 CGO_ENABLED=1 go build -ldflags "-extldflags '-Wl,-Bstatic -ldl -lc -lpthread -lcrypto -static' -w -s" -mod vendor -tags "osusergo netgo providerless cgo"  -o _output/bin/microshift-static cmd/microshift/main.go
jbpratt commented 3 years ago

thanks for the info @rootfs, I was able to get it built and executing with the ld flags you provided. I built the executable in a fedora container then copied it out to the fcos host and it starts up! Added some more info below. Let me know if this looks correct, if so I can open up a small patch with including the additional ld flags.

[root@kitkat core]# ps -ef | grep microshift
root       27915   27893 99 00:14 pts/1    00:00:30 ./microshift-static run
root       27977   26907  0 00:14 pts/0    00:00:00 grep --color=auto microshift
[root@kitkat core]# ls -l /proc/27915/map_files/ | awk '{print $NF}'| uniq
0
/var/home/core/microshift-static
/var/lib/microshift/etcd/member/snap/db

It should be noted we also had success running with a dynamic executable. CGO_ENABLED=1 go build -ldflags "-w -s" -mod vendor -tags "osusergo netgo providerless cgo" -o _output/bin/microshift-dyn cmd/microshift/main.go

[root@kitkat core]# ldd ./microshift-dyn
    linux-vdso.so.1 (0x0000ffff8410b000)
    libgpgme.so.11 => /lib64/libgpgme.so.11 (0x0000ffff8405c000)
    libassuan.so.0 => /lib64/libassuan.so.0 (0x0000ffff8402b000)
    libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x0000ffff83fea000)
    libdl.so.2 => /lib64/libdl.so.2 (0x0000ffff83fc9000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffff83f94000)
    libc.so.6 => /lib64/libc.so.6 (0x0000ffff83e09000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff840ce000)
[root@kitkat core]# ./microshift-dyn --help
Microshift, a minimal OpenShift

Usage:
  microshift [flags]
  microshift [command]

Available Commands:
  help        Help about any command
  run         Run Microshift
  version     Print Microshift version information

Flags:
      --alsologtostderr      log to standard error as well as files
  -h, --help                 help for microshift
      --log-dir string       If non-empty, write log files in this directory
  -v, --v Level              number for the log level verbosity
      --vmodule moduleSpec   comma-separated list of pattern=N settings for file-filtered logging

Use "microshift [command] --help" for more information about a command.

There is still an issue with some of the images being pulled aren't the correct arch, but it is unrelated to this and from the #arm channel you are already aware :smiley_cat: .

rootfs commented 3 years ago

@jbpratt that's great! please create a PR with your static build changes and send to @copejon and yes, the images are not quite aarch64 ready, @carlosedp is working on multiarch images.

jbpratt commented 2 years ago

testing on Fedora Server 34 aarch64

[autumn@kitkat ~]$ uname -a
Linux kitkat 5.11.12-300.fc34.aarch64 #1 SMP Wed Apr 7 16:12:21 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
[autumn@kitkat ~]$ journalctl -u microshift.service
-- Journal begins at Mon 2021-04-05 19:00:00 CDT, ends at Sat 2021-09-18 08:57:54 CDT. --
Sep 18 08:54:22 kitkat systemd[1]: Started Microshift.
Sep 18 08:54:22 kitkat microshift[48269]: fatal error: unexpected signal during runtime execution
Sep 18 08:54:22 kitkat microshift[48269]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xc978f70 pc=0xffff59fdae94]
Sep 18 08:54:22 kitkat microshift[48269]: runtime stack:
Sep 18 08:54:22 kitkat microshift[48269]: runtime.throw(0x441668a, 0x2a)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/panic.go:1116 +0x54
Sep 18 08:54:22 kitkat microshift[48269]: runtime.sigpanic()
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/signal_unix.go:726 +0x444
Sep 18 08:54:22 kitkat microshift[48269]: goroutine 1 [syscall, locked to thread]:
Sep 18 08:54:22 kitkat microshift[48269]: runtime.cgocall(0x3832b48, 0x400009bd38, 0x3dea100)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/cgocall.go:133 +0x50 fp=0x400009bd00 sp=0x400009bcc0 pc=0x403b40
Sep 18 08:54:22 kitkat microshift[48269]: crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL(0x0)
Sep 18 08:54:22 kitkat microshift[48269]:         _cgo_gotypes.go:287 +0x44 fp=0x400009bd30 sp=0x400009bd00 pc=0x66ea64
Sep 18 08:54:22 kitkat microshift[48269]: crypto/internal/boring.init.0()
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x40 fp=0x400009bd70 sp=0x400009bd30 pc>
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6e85600)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5652 +0x9c fp=0x400009bdb0 sp=0x400009bd70 pc=0x442ffc
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6e81800)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009bdf0 sp=0x400009bdb0 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6eaa540)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009be30 sp=0x400009bdf0 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6eaba80)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009be70 sp=0x400009be30 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6e6e8a0)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009beb0 sp=0x400009be70 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6ea6d80)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009bef0 sp=0x400009beb0 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6ea2580)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009bf30 sp=0x400009bef0 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.doInit(0x6e7e000)
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:5647 +0x58 fp=0x400009bf70 sp=0x400009bf30 pc=0x442fb8
Sep 18 08:54:22 kitkat microshift[48269]: runtime.main()
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/proc.go:191 +0x1b8 fp=0x400009bfd0 sp=0x400009bf70 pc=0x435d08
Sep 18 08:54:22 kitkat microshift[48269]: runtime.goexit()
Sep 18 08:54:22 kitkat microshift[48269]:         /usr/lib/golang/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400009bfd0 sp=0x400009bfd0 pc=0x468904
Sep 18 08:54:22 kitkat microshift[48269]: goroutine 6 [chan receive]:
Sep 18 08:54:22 kitkat microshift[48269]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x712f740)
Sep 18 08:54:22 kitkat microshift[48269]:         /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:1164 +0x70
Sep 18 08:54:22 kitkat microshift[48269]: created by k8s.io/klog/v2.init.0
Sep 18 08:54:22 kitkat microshift[48269]:         /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/klog/v2/klog.go:418 +0x124
Sep 18 08:54:22 kitkat systemd[1]: microshift.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 18 08:54:22 kitkat systemd[1]: microshift.service: Failed with result 'exit-code'.
Sep 18 08:54:22 kitkat systemd[1]: microshift.service: Scheduled restart job, restart counter is at 1.
Sep 18 08:54:22 kitkat systemd[1]: Stopped Microshift.
Sep 18 08:54:22 kitkat systemd[1]: Started Microshift.
Sep 18 08:54:22 kitkat microshift[48280]: fatal error: unexpected signal during runtime execution
Sep 18 08:54:22 kitkat microshift[48280]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x32314f70 pc=0xffff91f05e94]
Sep 18 08:54:22 kitkat microshift[48280]: runtime stack:
Sep 18 08:54:22 kitkat microshift[48280]: runtime.throw(0x441668a, 0x2a)
Sep 18 08:54:22 kitkat microshift[48280]:         /usr/lib/golang/src/runtime/panic.go:1116 +0x54
Sep 18 08:54:22 kitkat microshift[48280]: runtime.sigpanic()
Sep 18 08:54:22 kitkat microshift[48280]:         /usr/lib/golang/src/runtime/signal_unix.go:726 +0x444
Sep 18 08:54:22 kitkat microshift[48280]: goroutine 1 [syscall, locked to thread]:
Sep 18 08:54:22 kitkat microshift[48280]: runtime.cgocall(0x3832b48, 0x400009bd38, 0x3dea100)
Sep 18 08:54:22 kitkat microshift[48280]:         /usr/lib/golang/src/runtime/cgocall.go:133 +0x50 fp=0x400009bd00 sp=0x400009bcc0 pc=0x403b40
Sep 18 08:54:22 kitkat microshift[48280]: crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL(0x0)
Sep 18 08:54:22 kitkat microshift[48280]:         _cgo_gotypes.go:287 +0x44 fp=0x400009bd30 sp=0x400009bd00 pc=0x66ea64
Sep 18 08:54:22 kitkat microshift[48280]: crypto/internal/boring.init.0()
Sep 18 08:54:22 kitkat microshift[48280]:         /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x40 fp=0x400009bd70 sp=0x400009bd30 pc>