search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.8k stars 2.42k forks source link

`podman` dumping core during RHCOS 4.11 + RHEL 8.6 operation on `aarch64` #14503

Closed miabbott closed 2 years ago

miabbott commented 2 years ago

/kind bug

Description

The RHCOS team is rebasing RHCOS 4.11 to use RHEL 8.6; as part of our normal build + test operation on aarch64 we noticed that podman was dumping core.

Steps to reproduce the issue:

Not yet entirely clear, but my guess:

  1. Build RHCOS 4.11 w/ RHEL 8.6 on aarch64
  2. Run resulting image
  3. Start container with podman

Describe the results you received:

podman dumped core (cat journal.txt | grep podman):

Jun  6 18:30:39.815499 podman[1414]: time="2022-06-06T18:30:39Z" level=info msg="/usr/bin/podman filtering at log level info"                                  
Jun  6 18:30:39.844412 podman[1410]: time="2022-06-06T18:30:39Z" level=info msg="/usr/bin/podman filtering at log level info"
Jun  6 18:30:40.320937 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"                                                                                               
Jun  6 18:30:40.321224 podman[1410]: time="2022-06-06T18:30:40Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Jun  6 18:30:40.332304 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="Setting parallel job count to 4"                               
Jun  6 18:30:40.332500 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="Using systemd socket activation to determine API endpoint"                    
Jun  6 18:30:40.339074 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="API service listening on \"/run/user/0/podman/podman.sock\""                                                                                                                                                                                 
Jun  6 18:30:40.340034 podman[1410]: time="2022-06-06T18:30:40Z" level=info msg="Setting parallel job count to 4"
Jun  6 18:30:42.615329 podman[1686]: time="2022-06-06T18:30:42Z" level=info msg="/usr/bin/podman filtering at log level info"                   
Jun  6 18:30:42.625888 podman[1685]: time="2022-06-06T18:30:42Z" level=info msg="/usr/bin/podman filtering at log level info"                                  
Jun  6 18:30:43.394662 podman[1766]: time="2022-06-06T18:30:43Z" level=info msg="/usr/bin/podman filtering at log level info"
Jun  6 18:30:43.419905 podman[1751]: time="2022-06-06T18:30:43Z" level=info msg="/usr/bin/podman filtering at log level info"                                  
Jun  6 18:30:43.568481 podman[1751]: time="2022-06-06T18:30:43Z" level=info msg="Setting parallel job count to 4"                                   
Jun  6 18:30:43.580300 podman[1766]: time="2022-06-06T18:30:43Z" level=info msg="Setting parallel job count to 4"                                                                                                                                                                                                             
Jun  6 18:30:43.581047 podman[1766]: time="2022-06-06T18:30:43Z" level=info msg="Using systemd socket activation to determine API endpoint"
Jun  6 18:30:43.584367 podman[1766]: panic: runtime error: invalid memory address or nil pointer dereference                                    
Jun  6 18:30:43.584367 podman[1766]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xaaaae9ca647c]                                             
Jun  6 18:30:43.584367 podman[1766]: goroutine 1 [running]:                                                                                                                                                                                                                                                                   
Jun  6 18:30:43.584367 podman[1766]: panic({0xaaaaea1a9dc0, 0xaaaaeb042e60})
Jun  6 18:30:43.584367 podman[1766]:         /usr/lib/golang/src/runtime/panic.go:1147 +0x41c fp=0x400093f520 sp=0x400093f440 pc=0xaaaae8bf024c 
Jun  6 18:30:43.584367 podman[1766]: runtime.panicmem()                                                                                                                                                                                                                                                                       
Jun  6 18:30:43.584367 podman[1766]:         /usr/lib/golang/src/runtime/panic.go:221 +0x58 fp=0x400093f540 sp=0x400093f520 pc=0xaaaae8bede48
Jun  6 18:30:43.584367 podman[1766]: runtime.sigpanic()                                                                                                        
Jun  6 18:30:43.584367 podman[1766]:         /usr/lib/golang/src/runtime/signal_unix.go:735 +0x1fc fp=0x400093f580 sp=0x400093f540 pc=0xaaaae8c0935c
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/pkg/api/server.newServer(0x4000929ba0, 0x0, {{0x0, 0x0}, {0x0, 0x0}, 0x12a05f200, {0x0, 0x0}})                                                                                                                                                              
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/api/server/server.go:89 +0x1ac fp=0x400093fa00 sp=0x400093f590 pc=0xaaaae9ca647c
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/pkg/api/server.NewServerWithSettings(...)                                  
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/api/server/server.go:64                                                                                                                                                              
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/cmd/podman/system.restService(0x40008a5000, 0xaaaaeb114a80, {{0x0, 0x0}, {0x0, 0x0}, 0x12a05f200, {0x0, 0x0}})
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/system/service_abi.go:91 +0x578 fp=0x400093fc20 sp=0x400093fa00 pc=0xaaaae9cb0bd8
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/cmd/podman/system.service(0xaaaaeb072ba0, {0xaaaaeb148dc8, 0x0, 0x0})
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/system/service.go:102 +0x280 fp=0x400093fd10 sp=0x400093fc20 pc=0xaaaae9cb0270                                                                                                
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).execute(0xaaaaeb072ba0, {0x400003c0b0, 0x0, 0x0})
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:856 +0x668 fp=0x400093fdd0 sp=0x400093fd10 pc=0xaaaae91c0c78
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xaaaaeb076020)
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:974 +0x410 fp=0x400093feb0 sp=0x400093fdd0 pc=0xaaaae91c13f0
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:902
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...) 
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:895                                              
Jun  6 18:30:43.586574 podman[1766]: main.Execute()
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/root.go:100 +0x100 fp=0x400093ff30 sp=0x400093feb0 pc=0xaaaae9d14cb0
Jun  6 18:30:43.586574 podman[1766]: main.main()
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/main.go:39 +0xbc fp=0x400093ff70 sp=0x400093ff30 pc=0xaaaae9d143ec
Jun  6 18:30:43.586574 podman[1766]: runtime.main()
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:255 +0x284 fp=0x400093ffd0 sp=0x400093ff70 pc=0xaaaae8bf2fb4
Jun  6 18:30:43.586574 podman[1766]: runtime.goexit()
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x400093ffd0 sp=0x400093ffd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.586574 podman[1766]: goroutine 2 [force gc (idle)]:
Jun  6 18:30:43.586574 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb10f2e0, 0x11, 0x14, 0x1)
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000056fa0 sp=0x4000056f80 pc=0xaaaae8bf340c
Jun  6 18:30:43.586574 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.586574 podman[1766]: runtime.forcegchelper()
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:306 +0xd0 fp=0x4000056fd0 sp=0x4000056fa0 pc=0xaaaae8bf3290
Jun  6 18:30:43.586574 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000056fd0 sp=0x4000056fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.init.8
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:294 +0x30
Jun  6 18:30:43.587672 podman[1766]: goroutine 3 [GC sweep wait]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb111aa0, 0xc, 0x14, 0x1)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x40000577a0 sp=0x4000057780 pc=0xaaaae8bf340c
Jun  6 18:30:43.587672 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.587672 podman[1766]: runtime.bgsweep()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgcsweep.go:182 +0x150 fp=0x40000577d0 sp=0x40000577a0 pc=0xaaaae8bdc670
Jun  6 18:30:43.587672 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000577d0 sp=0x40000577d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.gcenable
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:181 +0x78
Jun  6 18:30:43.587672 podman[1766]: goroutine 4 [GC scavenge wait]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb112680, 0xd, 0x14, 0x1)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000057f70 sp=0x4000057f50 pc=0xaaaae8bf340c
Jun  6 18:30:43.587672 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.587672 podman[1766]: runtime.bgscavenge()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgcscavenge.go:314 +0x2c0 fp=0x4000057fd0 sp=0x4000057f70 pc=0xaaaae8bda940
Jun  6 18:30:43.587672 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000057fd0 sp=0x4000057fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.gcenable
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:182 +0x8c
Jun  6 18:30:43.587672 podman[1766]: goroutine 5 [finalizer wait]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb148cc0, 0x10, 0x14, 0x1)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000056710 sp=0x40000566f0 pc=0xaaaae8bf340c
Jun  6 18:30:43.587672 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.587672 podman[1766]: runtime.runfinq()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mfinal.go:177 +0xd8 fp=0x40000567d0 sp=0x4000056710 pc=0xaaaae8bd1398
Jun  6 18:30:43.587672 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000567d0 sp=0x40000567d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.createfing
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mfinal.go:157 +0x94
Jun  6 18:30:43.587672 podman[1766]: goroutine 8 [GC worker (idle)]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39f9b0, 0x400037d1c0, 0x18, 0x14, 0x0)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000058750 sp=0x4000058730 pc=0xaaaae8bf340c
Jun  6 18:30:43.590239 podman[1766]: runtime.gcBgMarkWorker()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:1200 +0x114 fp=0x40000587d0 sp=0x4000058750 pc=0xaaaae8bd4414
Jun  6 18:30:43.590239 podman[1766]: runtime.goexit()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000587d0 sp=0x40000587d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.590239 podman[1766]: created by runtime.gcBgMarkStartWorkers
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:1124 +0x34
Jun  6 18:30:43.590239 podman[1766]: goroutine 9 [chan receive]:
Jun  6 18:30:43.590239 podman[1766]: runtime.gopark(0xaaaaea39f930, 0x400007b558, 0xe, 0x17, 0x2)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000053ec0 sp=0x4000053ea0 pc=0xaaaae8bf340c
Jun  6 18:30:43.590239 podman[1766]: runtime.chanrecv(0x400007b500, 0x4000053fb0, 0x1)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/chan.go:576 +0x5f4 fp=0x4000053f50 sp=0x4000053ec0 pc=0xaaaae8bbf2c4
Jun  6 18:30:43.590239 podman[1766]: runtime.chanrecv2(0x400007b500, 0x4000053fb0)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/chan.go:444 +0x28 fp=0x4000053f80 sp=0x4000053f50 pc=0xaaaae8bbecb8
Jun  6 18:30:43.590239 podman[1766]: github.com/containers/podman/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0xaaaaeb114960)
Jun  6 18:30:43.590239 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:1169 +0x70 fp=0x4000053fd0 sp=0x4000053f80 pc=0xaaaae9730370
Jun  6 18:30:43.590239 podman[1766]: runtime.goexit()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000053fd0 sp=0x4000053fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.590239 podman[1766]: created by github.com/containers/podman/vendor/k8s.io/klog/v2.init.0
Jun  6 18:30:43.590239 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:420 +0x150
Jun  6 18:30:43.590239 podman[1766]: goroutine 10 [select, locked to thread]:
Jun  6 18:30:43.590239 podman[1766]: runtime.gopark(0xaaaaea39fb78, 0x0, 0x9, 0x18, 0x1)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000053610 sp=0x40000535f0 pc=0xaaaae8bf340c
Jun  6 18:30:43.590239 podman[1766]: runtime.selectgo(0x40000537a0, 0x4000053798, 0x0, 0x0, 0x2, 0x1)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/select.go:327 +0x6ec fp=0x4000053730 sp=0x4000053610 pc=0xaaaae8c050ac
Jun  6 18:30:43.590239 podman[1766]: runtime.ensureSigM.func1()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/signal_unix.go:890 +0x1b4 fp=0x40000537d0 sp=0x4000053730 pc=0xaaaae8c1f744
Jun  6 18:30:43.590239 podman[1766]: runtime.goexit()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000537d0 sp=0x40000537d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.591184 podman[1766]: created by runtime.ensureSigM
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/signal_unix.go:873 +0x10c
Jun  6 18:30:43.591184 podman[1766]: goroutine 11 [syscall]:
Jun  6 18:30:43.591184 podman[1766]: runtime.notetsleepg(0xaaaaeb149aa0, 0xffffffffffffffff)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/lock_futex.go:236 +0x30 fp=0x4000054780 sp=0x4000054750 pc=0xaaaae8bc41d0
Jun  6 18:30:43.591184 podman[1766]: os/signal.signal_recv()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/sigqueue.go:169 +0x110 fp=0x40000547b0 sp=0x4000054780 pc=0xaaaae8c23840
Jun  6 18:30:43.591184 podman[1766]: os/signal.loop()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/os/signal/signal_unix.go:24 +0x20 fp=0x40000547d0 sp=0x40000547b0 pc=0xaaaae922f070
Jun  6 18:30:43.591184 podman[1766]: runtime.goexit()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000547d0 sp=0x40000547d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.591184 podman[1766]: created by os/signal.Notify.func1.1
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/os/signal/signal.go:151 +0x38
Jun  6 18:30:43.591184 podman[1766]: goroutine 12 [select]:
Jun  6 18:30:43.591184 podman[1766]: runtime.gopark(0xaaaaea39fb78, 0x0, 0x9, 0x18, 0x1)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000067d10 sp=0x4000067cf0 pc=0xaaaae8bf340c
Jun  6 18:30:43.591184 podman[1766]: runtime.selectgo(0x4000067fa8, 0x4000054e90, 0x0, 0x0, 0x2, 0x1)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/select.go:327 +0x6ec fp=0x4000067e30 sp=0x4000067d10 pc=0xaaaae8c050ac
Jun  6 18:30:43.591184 podman[1766]: github.com/containers/podman/libpod/shutdown.Start.func1()
Jun  6 18:30:43.591184 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:47 +0x80 fp=0x4000067fd0 sp=0x4000067e30 pc=0xaaaae99aa2a0
Jun  6 18:30:43.591184 podman[1766]: runtime.goexit()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000067fd0 sp=0x4000067fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.591184 podman[1766]: created by github.com/containers/podman/libpod/shutdown.Start
Jun  6 18:30:43.591184 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:46 +0x134
Jun  6 18:30:43.591184 podman[1766]: goroutine 14 [sleep]:
Jun  6 18:30:43.591184 podman[1766]: runtime.gopark(0xaaaaea39fb58, 0x400007c7d0, 0x13, 0x13, 0x1)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000055600 sp=0x40000555e0 pc=0xaaaae8bf340c
Jun  6 18:30:43.591184 podman[1766]: time.Sleep(0x5f5e100)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/time.go:193 +0x138 fp=0x4000055640 sp=0x4000055600 pc=0xaaaae8c24218
Jun  6 18:30:43.601735 podman[1766]: github.com/containers/podman/libpod.(*Runtime).libimageEvents.func2(0x40001fff20, 0xaaaaea39ad38, 0x4000929ba0)
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/runtime.go:790 +0x38 fp=0x40000557c0 sp=0x4000055640 pc=0xaaaae9a7b268
Jun  6 18:30:43.601735 podman[1766]: runtime.goexit()
Jun  6 18:30:43.601735 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000557c0 sp=0x40000557c0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.601735 podman[1766]: created by github.com/containers/podman/libpod.(*Runtime).libimageEvents
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/runtime.go:767 +0xdc
Jun  6 18:30:43.601735 podman[1766]: goroutine 15 [runnable]:
Jun  6 18:30:43.601735 podman[1766]: github.com/containers/podman/pkg/servicereaper.reaper(0x400007a2a0)
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/servicereaper/service.go:38 fp=0x4000054fd0 sp=0x4000054fd0 pc=0xaaaae9992cc0
Jun  6 18:30:43.601735 podman[1766]: runtime.goexit()
Jun  6 18:30:43.601735 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000054fd0 sp=0x4000054fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.601735 podman[1766]: created by github.com/containers/podman/pkg/servicereaper.Start
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/servicereaper/service.go:35 +0x94
Jun  6 18:30:43.601735 podman[1766]: goroutine 16 [runnable]:
Jun  6 18:30:43.601735 podman[1766]: github.com/containers/podman/pkg/domain/infra.StartWatcher.func1(0x400007a360, 0x4000929ba0)
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:374 fp=0x4000055fc0 sp=0x4000055fc0 pc=0xaaaae9bd2e30
Jun  6 18:30:43.601735 podman[1766]: runtime.goexit()
Jun  6 18:30:43.601735 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000055fc0 sp=0x4000055fc0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.601735 podman[1766]: created by github.com/containers/podman/pkg/domain/infra.StartWatcher
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:374 +0x9c
Jun  6 18:30:43.843939 systemd[1622]: Started podman-pause-22df4033.scope.
Jun  6 18:30:43.847904 systemd[1622]: Started podman-pause-e1fe66be.scope.
Jun  6 18:30:44.958107 systemd-coredump[1866]: Resource limits disable core dumping for process 1766 (podman).
Jun  6 18:30:44.974946 systemd-coredump[1866]: Process 1766 (podman) of user 1000 dumped core.
Jun  6 18:30:44.995688 systemd[1622]: Started podman-pause-ff152fc7.scope.
Jun  6 18:30:44.999221 systemd[1622]: podman.service: Main process exited, code=exited, status=134/n/a
Jun  6 18:30:44.999612 systemd[1622]: podman.service: Failed with result 'exit-code'.

Describe the results you expected:

podman does not dump core

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman --version
podman version 4.0.2

Output of podman info --debug:

need to get access to aarch64 platform to reproduce

TBD

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-4.0.2-4.rhaos4.11.el8.aarch64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Sorta; we are limited to using a specific version of podman. Testing with newer versison is not feasible at this time.

Skimming the Troubleshooting Guide did not show any actionable advice for this situation.

Additional environment details (AWS, VirtualBox, physical, etc.):

aarch64 VM running in qemu via coreos-assembler in a privilged pod on the RHT internal OSBS cluster

xref: https://github.com/openshift/os/issues/820

mheon commented 2 years ago

Looking latest main, seems fixed already (pointer deref was removed). Looks like https://github.com/containers/podman/commit/13c6fd067cab2fd21f96837c6b6105ebb740fd1d did it. Is a backport desirable? If so, we'll probably need a BZ to justify.

miabbott commented 2 years ago

Looking latest main, seems fixed already (pointer deref was removed). Looks like 13c6fd0 did it. Is a backport desirable? If so, we'll probably need a BZ to justify.

Yeah, I think we will want this as part of OCP/RHCOS 4.11. Since we are maintaining separate branches for OCP 4.11 downstream, I believe we have a bit more leeway being able to get the backport in before OCP 4.11 code freeze.

Will create a BZ for this; thanks for the quick reply, Matt!

miabbott commented 2 years ago

https://bugzilla.redhat.com/show_bug.cgi?id=2094101

Prashanth684 commented 2 years ago

so will this need a backport or should podman be bumped to the v4.1.0 for ocp 4.11 ?

vrothberg commented 2 years ago

Closing as it's a downstream issue. We can continue the conversation for sure.

rhatdan commented 2 years ago

You should bump to podman 4.1 if possible.