search

fluxcd / image-automation-controller

GitOps Toolkit controller that patches container image tags in Git
https://fluxcd.io
Apache License 2.0
175 stars 71 forks source link

image-automation-controller crashing on fresh install #753

Closed myst3k closed 1 month ago

myst3k commented 1 month ago

Hello,

I have a fresh install, talos linux, with version 38.0 is in a crash loop, however 37.1 works just fine. I have tried removing any resource constraints cpu/mem, and giving privileged security admission. changing to log level debug doesn't show much more. Please let me know what else is needed.

Thanks

❯ kubectl describe pod -n flux-system image-automation-controller-fb6c9df74-dm8hj
Name:             image-automation-controller-fb6c9df74-dm8hj
Namespace:        flux-system
Priority:         0
Service Account:  image-automation-controller
Node:             node01/192.168.1.41
Start Time:       Fri, 20 Sep 2024 04:07:40 +0000
Labels:           app=image-automation-controller
                  pod-template-hash=fb6c9df74
Annotations:      prometheus.io/port: 8080
                  prometheus.io/scrape: true
Status:           Running
IP:               10.69.0.59
IPs:
  IP:           10.69.0.59
Controlled By:  ReplicaSet/image-automation-controller-fb6c9df74
Containers:
  manager:
    Container ID:    containerd://6075fae919b8efa9c95aa3b52b6786f26840f0cbb36b5c23aca444e8ee09f368
    Image:           ghcr.io/fluxcd/image-automation-controller:v0.38.0
    Image ID:        ghcr.io/fluxcd/image-automation-controller@sha256:ab5097213194f3cd9f0e68d8a937d94c4fc7e821f6544453211e94815b282aa2
    Ports:           8080/TCP, 9440/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      --events-addr=http://notification-controller.flux-system.svc.cluster.local./
      --watch-all-namespaces=true
      --log-level=info
      --log-encoding=json
      --enable-leader-election
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Fri, 20 Sep 2024 04:13:26 +0000
      Finished:     Fri, 20 Sep 2024 04:13:26 +0000
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   64Mi
    Liveness:   http-get http://:healthz/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:healthz/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      RUNTIME_NAMESPACE:  flux-system (v1:metadata.namespace)
      GOMAXPROCS:         1 (limits.cpu)
      GOMEMLIMIT:         1073741824 (limits.memory)
    Mounts:
      /tmp from temp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7fxk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  temp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-s7fxk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m56s                  default-scheduler  Successfully assigned flux-system/image-automation-controller-fb6c9df74-dm8hj to node01
  Normal   Pulled     4m27s (x5 over 5m55s)  kubelet            Container image "ghcr.io/fluxcd/image-automation-controller:v0.38.0" already present on machine
  Normal   Created    4m27s (x5 over 5m55s)  kubelet            Created container manager
  Normal   Started    4m27s (x5 over 5m55s)  kubelet            Started container manager
  Warning  BackOff    46s (x26 over 5m54s)   kubelet            Back-off restarting failed container manager in pod image-automation-controller-fb6c9df74-dm8hj_flux-system(4cd869f6-a2db-4b86-a0c8-05069d324d62)

{"level":"info","ts":"2024-09-20T04:08:01.991Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-09-20T04:08:01.991Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-09-20T04:08:01.991Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-09-20T04:08:01.991Z","msg":"starting server","name":"health probe","addr":"[::]:9440"}
SIGSEGV: segmentation violation
PC=0x1a2c268 m=5 sigcode=1 addr=0x7fdacc1f87ac

goroutine 0 gp=0xc000295180 m=5 mp=0xc00049e008 [idle]:
runtime.clearDeletedTimers(0xc00006c008)
        runtime/time.go:968 +0xe8 fp=0x7f1acc18c7e8 sp=0x7f1acc18c780 pc=0x1a2c268
runtime.checkTimers(0xc00006c008, 0xc0006c5800?)
        runtime/proc.go:3993 +0x17c fp=0x7f1acc18c830 sp=0x7f1acc18c7e8 pc=0x1a12d3c
runtime.findRunnable()
        runtime/proc.go:3160 +0x65 fp=0x7f1acc18c9a8 sp=0x7f1acc18c830 pc=0x1a10c05
runtime.schedule()
        runtime/proc.go:3868 +0xb1 fp=0x7f1acc18c9e0 sp=0x7f1acc18c9a8 pc=0x1a129d1
runtime.park_m(0xc000175500)
        runtime/proc.go:4036 +0x1ec fp=0x7f1acc18ca38 sp=0x7f1acc18c9e0 pc=0x1a12fac
runtime.mcall()
        runtime/asm_amd64.s:458 +0x4e fp=0x7f1acc18ca50 sp=0x7f1acc18ca38 pc=0x1a42a8e

goroutine 1 gp=0xc0000061c0 m=nil [select]:
runtime.gopark(0xc00042d490?, 0x2?, 0x18?, 0x25?, 0xc00042d42c?)
        runtime/proc.go:402 +0xce fp=0xc00042d2d0 sp=0xc00042d2b0 pc=0x1a0b68e
runtime.selectgo(0xc00042d490, 0xc00042d428, 0xc0005ee8f0?, 0x0, 0x10?, 0x1)
        runtime/select.go:327 +0x725 fp=0xc00042d3f0 sp=0xc00042d2d0 pc=0x1a1da05
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:158 +0x227 fp=0xc00042d4e8 sp=0xc00042d3f0 pc=0x2c36007
sync.(*Once).doSlow(0x30?, 0x572120?)
        sync/once.go:74 +0xc2 fp=0xc00042d548 sp=0xc00042d4e8 pc=0x1a66d82
sync.(*Once).Do(...)
        sync/once.go:65
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start(0xa105a0?, {0xa105a0?, 0xc000306500?})
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:135 +0x6c fp=0xc00042d5a0 sp=0xc00042d548 pc=0x2c35d8c
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start(0xc0002ff040, {0xa105a0, 0xc000306410})
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/internal.go:405 +0x714 fp=0xc00042d768 sp=0xc00042d5a0 pc=0x2c32974
main.main()
        ./main.go:217 +0x1093 fp=0xc00042df50 sp=0xc00042d768 pc=0x3133293
runtime.main()
        runtime/proc.go:271 +0x29d fp=0xc00042dfe0 sp=0xc00042df50 pc=0x1a0b23d
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00042dfe8 sp=0xc00042dfe0 pc=0x1a44ac1

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000084fa8 sp=0xc000084f88 pc=0x1a0b68e
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.forcegchelper()
        runtime/proc.go:326 +0xb3 fp=0xc000084fe0 sp=0xc000084fa8 pc=0x1a0b4f3
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x1a44ac1
created by runtime.init.6 in goroutine 1
        runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000006e00 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000085780 sp=0xc000085760 pc=0x1a0b68e
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.bgsweep(0xc00007e000)
        runtime/mgcsweep.go:318 +0xdf fp=0xc0000857c8 sp=0xc000085780 pc=0x19f2e9f
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x19e75a5
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x1a44ac1
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000006fc0 m=nil [GC scavenge wait]:
runtime.gopark(0xc00007e000?, 0x9dfff0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0x1a0b68e
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.(*scavengerState).park(0x32f1a80)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0x19f0689
runtime.bgscavenge(0xc00007e000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0x19f0c59
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x19e7545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x1a44ac1
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007880 m=nil [finalizer wait]:
runtime.gopark(0xc000084648?, 0x19d98a5?, 0xa8?, 0x1?, 0xc0000061c0?)
        runtime/proc.go:402 +0xce fp=0xc000084620 sp=0xc000084600 pc=0x1a0b68e
runtime.runfinq()
        runtime/mfinal.go:194 +0x107 fp=0xc0000847e0 sp=0xc000084620 pc=0x19e65e7
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x1a44ac1
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:164 +0x3d

goroutine 8 gp=0xc0002e8e00 m=nil [GC worker (idle)]:
runtime.gopark(0x1a033862b62?, 0x19d198b?, 0x97?, 0xd?, 0x1?)
        runtime/proc.go:402 +0xce fp=0xc000086750 sp=0xc000086730 pc=0x1a0b68e
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0000867e0 sp=0xc000086750 pc=0x19e9685
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x1a44ac1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 45 gp=0xc0002941c0 m=nil [select, locked to thread]:
runtime.gopark(0xc000081fa8?, 0x2?, 0xc0?, 0xc9?, 0xc000081f94?)
        runtime/proc.go:402 +0xce fp=0xc000081e38 sp=0xc000081e18 pc=0x1a0b68e
runtime.selectgo(0xc000081fa8, 0xc000081f90, 0x0?, 0x0, 0xa10201?, 0x1)
        runtime/select.go:327 +0x725 fp=0xc000081f58 sp=0xc000081e38 pc=0x1a1da05
runtime.ensureSigM.func1()
        runtime/signal_unix.go:1034 +0x19f fp=0xc000081fe0 sp=0xc000081f58 pc=0x1a3ae5f
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0x1a44ac1
created by runtime.ensureSigM in goroutine 1
        runtime/signal_unix.go:1017 +0xc8

goroutine 46 gp=0xc000294000 m=3 mp=0xc00007b008 [syscall]:
runtime.notetsleepg(0x3355f20, 0xffffffffffffffff)
        runtime/lock_futex.go:246 +0x29 fp=0xc0000817a0 sp=0xc000081778 pc=0x19d7ec9
os/signal.signal_recv()
        runtime/sigqueue.go:152 +0x29 fp=0xc0000817c0 sp=0xc0000817a0 pc=0x1a41229
os/signal.loop()
        os/signal/signal_unix.go:23 +0x13 fp=0xc0000817e0 sp=0xc0000817c0 pc=0x2ca3ab3
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000817e8 sp=0xc0000817e0 pc=0x1a44ac1
created by os/signal.Notify.func1.1 in goroutine 1
        os/signal/signal.go:151 +0x1f

goroutine 36 gp=0xc000294700 m=nil [IO wait]:
runtime.gopark(0x77ed7d97098fcdd6?, 0x71b06371fb16605a?, 0xd6?, 0xcd?, 0xb?)
        runtime/proc.go:402 +0xce fp=0xc0004cb6f8 sp=0xc0004cb6d8 pc=0x1a0b68e
runtime.netpollblock(0x1aa2638?, 0x19ceba6?, 0x0?)
        runtime/netpoll.go:573 +0xf7 fp=0xc0004cb730 sp=0xc0004cb6f8 pc=0x1a03577
internal/poll.runtime_pollWait(0x7f1bcc4d1eb0, 0x72)
        runtime/netpoll.go:345 +0x85 fp=0xc0004cb750 sp=0xc0004cb730 pc=0x1a3e4c5
internal/poll.(*pollDesc).wait(0xc0005f8580?, 0xc000670000?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004cb778 sp=0xc0004cb750 pc=0x1ac0d67
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005f8580, {0xc000670000, 0x5000, 0x5000})
        internal/poll/fd_unix.go:164 +0x27a fp=0xc0004cb810 sp=0xc0004cb778 pc=0x1ac219a
net.(*netFD).Read(0xc0005f8580, {0xc000670000?, 0x7f1bcc34d738?, 0xc000492300?})
        net/fd_posix.go:55 +0x25 fp=0xc0004cb858 sp=0xc0004cb810 pc=0x1b36845
net.(*conn).Read(0xc0005ee3c8, {0xc000670000?, 0xc0004cb938?, 0x19d94fb?})
        net/net.go:179 +0x45 fp=0xc0004cb8a0 sp=0xc0004cb858 pc=0x1b492a5
net.(*TCPConn).Read(0xc0005781c0?, {0xc000670000?, 0x19d94fb?, 0x1a230fe?})
        <autogenerated>:1 +0x25 fp=0xc0004cb8d0 sp=0xc0004cb8a0 pc=0x1b5b585
crypto/tls.(*atLeastReader).Read(0xc000492300, {0xc000670000?, 0x0?, 0xc000492300?})
        crypto/tls/conn.go:806 +0x3b fp=0xc0004cb918 sp=0xc0004cb8d0 pc=0x1d69f9b
bytes.(*Buffer).ReadFrom(0xc0005782b0, {0x9f1f40, 0xc000492300})
        bytes/buffer.go:211 +0x98 fp=0xc0004cb970 sp=0xc0004cb918 pc=0x1afc478
crypto/tls.(*Conn).readFromUntil(0xc000578008, {0x9f0f20, 0xc0005ee3c8}, 0xc0004cb980?)
        crypto/tls/conn.go:828 +0xde fp=0xc0004cb9a8 sp=0xc0004cb970 pc=0x1d6a17e
crypto/tls.(*Conn).readRecordOrCCS(0xc000578008, 0x0)
        crypto/tls/conn.go:626 +0x3cf fp=0xc0004cbc28 sp=0xc0004cb9a8 pc=0x1d6728f
crypto/tls.(*Conn).readRecord(...)
        crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0xc000578008, {0xc0005fb000, 0x1000, 0xc0002941c0?})
        crypto/tls/conn.go:1370 +0x156 fp=0xc0004cbc98 sp=0xc0004cbc28 pc=0x1d6db36
bufio.(*Reader).Read(0xc0000530e0, {0xc0005643c0, 0x9, 0x31401e0?})
        bufio/bufio.go:241 +0x197 fp=0xc0004cbcd0 sp=0xc0004cbc98 pc=0x1b62057
io.ReadAtLeast({0x9f0740, 0xc0000530e0}, {0xc0005643c0, 0x9, 0x9}, 0x9)
        io/io.go:335 +0x90 fp=0xc0004cbd18 sp=0xc0004cbcd0 pc=0x1a97510
io.ReadFull(...)
        io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0005643c0, 0x9, 0x4cbdc0?}, {0x9f0740?, 0xc0000530e0?})
        golang.org/x/net@v0.24.0/http2/frame.go:237 +0x65 fp=0xc0004cbd68 sp=0xc0004cbd18 pc=0x1f305e5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000564380)
        golang.org/x/net@v0.24.0/http2/frame.go:498 +0x85 fp=0xc0004cbe10 sp=0xc0004cbd68 pc=0x1f31205
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0004cbfa8)
        golang.org/x/net@v0.24.0/http2/transport.go:2429 +0xd8 fp=0xc0004cbf60 sp=0xc0004cbe10 pc=0x1f471d8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00057a000)
        golang.org/x/net@v0.24.0/http2/transport.go:2325 +0x65 fp=0xc0004cbfc8 sp=0xc0004cbf60 pc=0x1f467c5
golang.org/x/net/http2.(*ClientConn).readLoop-fm()
        <autogenerated>:1 +0x25 fp=0xc0004cbfe0 sp=0xc0004cbfc8 pc=0x1f53a45
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004cbfe8 sp=0xc0004cbfe0 pc=0x1a44ac1
created by golang.org/x/net/http2.(*ClientConn).goRun in goroutine 35
        golang.org/x/net@v0.24.0/http2/transport.go:369 +0x2d

goroutine 47 gp=0xc000295340 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000082710 sp=0xc0000826f0 pc=0x1a0b68e
runtime.chanrecv(0xc00002aa20, 0x0, 0x1)
        runtime/chan.go:583 +0x3bf fp=0xc000082788 sp=0xc000082710 pc=0x19d1e9f
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:442 +0x12 fp=0xc0000827b0 sp=0xc000082788 pc=0x19d1ab2
sigs.k8s.io/controller-runtime/pkg/manager/signals.SetupSignalHandler.func1()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/signals/signal.go:38 +0x27 fp=0xc0000827e0 sp=0xc0000827b0 pc=0x2ca3d07
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000827e8 sp=0xc0000827e0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/manager/signals.SetupSignalHandler in goroutine 1
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/signals/signal.go:37 +0xc5

goroutine 48 gp=0xc000295500 m=nil [chan receive]:
runtime.gopark(0xc00054e900?, 0xc00035aa20?, 0x0?, 0x0?, 0x82f38?)
        runtime/proc.go:402 +0xce fp=0xc000082ee0 sp=0xc000082ec0 pc=0x1a0b68e
runtime.chanrecv(0xc0004319e0, 0xc000082fb0, 0x1)
        runtime/chan.go:583 +0x3bf fp=0xc000082f58 sp=0xc000082ee0 pc=0x19d1e9f
runtime.chanrecv2(0x0?, 0x0?)
        runtime/chan.go:447 +0x12 fp=0xc000082f80 sp=0xc000082f58 pc=0x19d1ad2
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0xc000490510)
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:186 +0x45 fp=0xc000082fc8 sp=0xc000082f80 pc=0x2c36325
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1.gowrap2()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:139 +0x25 fp=0xc000082fe0 sp=0xc000082fc8 pc=0x2c36245
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000082fe8 sp=0xc000082fe0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 1
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:139 +0xc8

goroutine 49 gp=0xc0002956c0 m=nil [IO wait]:
runtime.gopark(0x19d94fb?, 0xc0005ee998?, 0x0?, 0x0?, 0xc0004c9a48?)
        runtime/proc.go:402 +0xce fp=0xc0004c99a8 sp=0xc0004c9988 pc=0x1a0b68e
runtime.netpollblock(0x7f1bcc34c6f8?, 0x19ceba6?, 0x0?)
        runtime/netpoll.go:573 +0xf7 fp=0xc0004c99e0 sp=0xc0004c99a8 pc=0x1a03577
internal/poll.runtime_pollWait(0x7f1bcc4d1cc0, 0x72)
        runtime/netpoll.go:345 +0x85 fp=0xc0004c9a00 sp=0xc0004c99e0 pc=0x1a3e4c5
internal/poll.(*pollDesc).wait(0x8?, 0x10?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004c9a28 sp=0xc0004c9a00 pc=0x1ac0d67
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000209600)
        internal/poll/fd_unix.go:611 +0x2ac fp=0xc0004c9ad0 sp=0xc0004c9a28 pc=0x1ac624c
net.(*netFD).accept(0xc000209600)
        net/fd_unix.go:172 +0x29 fp=0xc0004c9b88 sp=0xc0004c9ad0 pc=0x1b38869
net.(*TCPListener).accept(0xc00026c120)
        net/tcpsock_posix.go:159 +0x1e fp=0xc0004c9bb0 sp=0xc0004c9b88 pc=0x1b5301e
net.(*TCPListener).Accept(0xc00026c120)
        net/tcpsock.go:327 +0x30 fp=0xc0004c9be0 sp=0xc0004c9bb0 pc=0x1b52030
net/http.(*onceCloseListener).Accept(0xa102c8?)
        <autogenerated>:1 +0x24 fp=0xc0004c9bf8 sp=0xc0004c9be0 pc=0x1e60ec4
net/http.(*Server).Serve(0xc0004fa2d0, {0xa037c0, 0xc00026c120})
        net/http/server.go:3255 +0x33e fp=0xc0004c9d28 sp=0xc0004c9bf8 pc=0x1e381de
sigs.k8s.io/controller-runtime/pkg/metrics/server.(*defaultServer).Start(0xc00043e180, {0xa105a0, 0xc000306050})
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/metrics/server/server.go:245 +0x838 fp=0xc0004c9f68 sp=0xc0004c9d28 pc=0x2c184f8
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0xc00026c0c0)
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:226 +0xc8 fp=0xc0004c9fc8 sp=0xc0004c9f68 pc=0x2c365e8
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.gowrap1()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:229 +0x24 fp=0xc0004c9fe0 sp=0xc0004c9fc8 pc=0x2c364e4
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c9fe8 sp=0xc0004c9fe0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 48
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:210 +0x19d

goroutine 56 gp=0xc000295880 m=nil [chan receive]:
runtime.gopark(0xc0000061c0?, 0xc00035aa20?, 0xc0?, 0x61?, 0x83f38?)
        runtime/proc.go:402 +0xce fp=0xc000083ee0 sp=0xc000083ec0 pc=0x1a0b68e
runtime.chanrecv(0xc000431b60, 0xc000083fb0, 0x1)
        runtime/chan.go:583 +0x3bf fp=0xc000083f58 sp=0xc000083ee0 pc=0x19d1e9f
runtime.chanrecv2(0xc000431980?, 0xc000492f50?)
        runtime/chan.go:447 +0x12 fp=0xc000083f80 sp=0xc000083f58 pc=0x19d1ad2
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0xc000490630)
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:186 +0x45 fp=0xc000083fc8 sp=0xc000083f80 pc=0x2c36325
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1.gowrap2()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:139 +0x25 fp=0xc000083fe0 sp=0xc000083fc8 pc=0x2c36245
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 1
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:139 +0xc8

goroutine 51 gp=0xc000295a40 m=nil [chan receive]:
runtime.gopark(0x3b34e0?, 0x0?, 0x50?, 0x37?, 0x1b14a19?)
        runtime/proc.go:402 +0xce fp=0xc0000836c0 sp=0xc0000836a0 pc=0x1a0b68e
runtime.chanrecv(0xc0000ac240, 0x0, 0x1)
        runtime/chan.go:583 +0x3bf fp=0xc000083738 sp=0xc0000836c0 pc=0x19d1e9f
runtime.chanrecv1(0xc000306050?, 0xc00043e180?)
        runtime/chan.go:442 +0x12 fp=0xc000083760 sp=0xc000083738 pc=0x19d1ab2
sigs.k8s.io/controller-runtime/pkg/metrics/server.(*defaultServer).Start.func1()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/metrics/server/server.go:233 +0x45 fp=0xc0000837e0 sp=0xc000083760 pc=0x2c18625
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000837e8 sp=0xc0000837e0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/metrics/server.(*defaultServer).Start in goroutine 49
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/metrics/server/server.go:232 +0x81b

goroutine 52 gp=0xc000295c00 m=nil [IO wait]:
runtime.gopark(0x19d94fb?, 0xc0004c5b28?, 0xfe?, 0x8c?, 0xc0004c5b88?)
        runtime/proc.go:402 +0xce fp=0xc0004c5ae8 sp=0xc0004c5ac8 pc=0x1a0b68e
runtime.netpollblock(0x7f1bcc34c6f8?, 0x19ceba6?, 0x0?)
        runtime/netpoll.go:573 +0xf7 fp=0xc0004c5b20 sp=0xc0004c5ae8 pc=0x1a03577
internal/poll.runtime_pollWait(0x7f1bcc4d1db8, 0x72)
        runtime/netpoll.go:345 +0x85 fp=0xc0004c5b40 sp=0xc0004c5b20 pc=0x1a3e4c5
internal/poll.(*pollDesc).wait(0x7?, 0x10?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004c5b68 sp=0xc0004c5b40 pc=0x1ac0d67
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000209180)
        internal/poll/fd_unix.go:611 +0x2ac fp=0xc0004c5c10 sp=0xc0004c5b68 pc=0x1ac624c
net.(*netFD).accept(0xc000209180)
        net/fd_unix.go:172 +0x29 fp=0xc0004c5cc8 sp=0xc0004c5c10 pc=0x1b38869
net.(*TCPListener).accept(0xc00007dd40)
        net/tcpsock_posix.go:159 +0x1e fp=0xc0004c5cf0 sp=0xc0004c5cc8 pc=0x1b5301e
net.(*TCPListener).Accept(0xc00007dd40)
        net/tcpsock.go:327 +0x30 fp=0xc0004c5d20 sp=0xc0004c5cf0 pc=0x1b52030
net/http.(*onceCloseListener).Accept(0xa102c8?)
        <autogenerated>:1 +0x24 fp=0xc0004c5d38 sp=0xc0004c5d20 pc=0x1e60ec4
net/http.(*Server).Serve(0xc0004fa1e0, {0xa037c0, 0xc00007dd40})
        net/http/server.go:3255 +0x33e fp=0xc0004c5e68 sp=0xc0004c5d38 pc=0x1e381de
sigs.k8s.io/controller-runtime/pkg/manager.(*Server).serve(0xa169f0?)
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/server.go:106 +0x25 fp=0xc0004c5e90 sp=0xc0004c5e68 pc=0x2c37505
sigs.k8s.io/controller-runtime/pkg/manager.(*Server).Start(0xc0002a2580, {0xa105a0, 0xc000306050})
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/server.go:84 +0x332 fp=0xc0004c5f68 sp=0xc0004c5e90 pc=0x2c37232
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0xc00026c100)
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:226 +0xc8 fp=0xc0004c5fc8 sp=0xc0004c5f68 pc=0x2c365e8
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.gowrap1()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:229 +0x24 fp=0xc0004c5fe0 sp=0xc0004c5fc8 pc=0x2c364e4
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c5fe8 sp=0xc0004c5fe0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 48
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:210 +0x19d

goroutine 55 gp=0xc000295dc0 m=nil [chan receive]:
runtime.gopark(0xc0000061c0?, 0x19d1500?, 0xc0?, 0x61?, 0x4?)
        runtime/proc.go:402 +0xce fp=0xc0000876e0 sp=0xc0000876c0 pc=0x1a0b68e
runtime.chanrecv(0xc000431aa0, 0xc0000877b0, 0x1)
        runtime/chan.go:583 +0x3bf fp=0xc000087758 sp=0xc0000876e0 pc=0x19d1e9f
runtime.chanrecv2(0xc000431980?, 0xc0004932b0?)
        runtime/chan.go:447 +0x12 fp=0xc000087780 sp=0xc000087758 pc=0x19d1ad2
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0xc0004905a0)
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:186 +0x45 fp=0xc0000877c8 sp=0xc000087780 pc=0x2c36325
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1.gowrap2()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:139 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x2c36245
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x1a44ac1
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 1
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/runnable_group.go:139 +0xc8

goroutine 54 gp=0xc000468000 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000086ea0 sp=0xc000086e80 pc=0x1a0b68e
runtime.chanrecv(0xc0000ac240, 0x0, 0x1)
        runtime/chan.go:583 +0x3bf fp=0xc000086f18 sp=0xc000086ea0 pc=0x19d1e9f
runtime.chanrecv1(0xc0004932a8?, 0xc000295c00?)
        runtime/chan.go:442 +0x12 fp=0xc000086f40 sp=0xc000086f18 pc=0x19d1ab2
sigs.k8s.io/controller-runtime/pkg/manager.(*Server).Start.func1()
        sigs.k8s.io/controller-runtime@v0.18.1/pkg/manager/server.go:67 +0x6e fp=0xc000086fe0 sp=0xc000086f40 pc=0x2c3732e
runtime.goexit({})
myst3k commented 1 month ago

Used a different talos image and it came right up. Weird that 37.1 worked fine though.

https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&extensions=siderolabs%2Famdgpu-firmware&extensions=siderolabs%2Famd-ucode&extensions=siderolabs%2Futil-linux-tools&platform=metal&target=metal&version=1.7.6

old image https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&platform=metal&target=metal&version=1.7.6