numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io
Apache License 2.0
1.01k stars 98 forks source link

error during start up of new pipeline #1775

Closed nagarajatantry closed 1 week ago

nagarajatantry commented 1 week ago
qal-usw2-eks git:(numaflow-pipeline) id=kafka-test-pipeline-1-cre-0d0d6a-zvlql
➜  qal-usw2-eks git:(numaflow-pipeline) kubectl logs -f $id --previous
{"level":"error","ts":"2024-06-26T22:26:32.634308912Z","logger":"numaflow.isbsvc-create","caller":"commands/isbsvc_create.go:84","msg":"Failed to create buffers, buckets and side inputs store.","pipeline":"kafka-test-pipeline-1","error":"failed to create side inputs KV \"secfraud-thirdparties-astraea-usw2-qal-kafka-test-pipeline-1_SIDE_INPUTS\", nats: no suitable peers for placement","stacktrace":"github.com/numaproj/numaflow/cmd/commands.NewISBSvcCreateCommand.func1\n\t/home/runner/work/numaflow/numaflow/cmd/commands/isbsvc_create.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/numaproj/numaflow/cmd/commands.Execute\n\t/home/runner/work/numaflow/numaflow/cmd/commands/root.go:32\nmain.main\n\t/home/runner/work/numaflow/numaflow/cmd/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/proc.go:267"}
Error: failed to create side inputs KV "secfraud-thirdparties-astraea-usw2-qal-kafka-test-pipeline-1_SIDE_INPUTS", nats: no suitable peers for placement
{"level":"error","ts":"2024-06-26T22:26:32.634352408Z","logger":"numaflow.isbsvc-create","caller":"nats/nats_client.go:68","msg":"Nats default: disconnected","pipeline":"kafka-test-pipeline-1","stacktrace":"github.com/numaproj/numaflow/pkg/shared/clients/nats.NewNATSClient.func3\n\t/home/runner/work/numaflow/numaflow/pkg/shared/clients/nats/nats_client.go:68\ngithub.com/nats-io/nats%2ego.(*Conn).close.func1\n\t/home/runner/go/pkg/mod/github.com/nats-io/nats.go@v1.33.1/nats.go:5122\ngithub.com/nats-io/nats%2ego.(*asyncCallbacksHandler).asyncCBDispatcher\n\t/home/runner/go/pkg/mod/github.com/nats-io/nats.go@v1.33.1/nats.go:2901"}
{"level":"info","ts":"2024-06-26T22:26:32.634498775Z","logger":"numaflow.isbsvc-create","caller":"nats/nats_client.go:62","msg":"Nats default: connection closed","pipeline":"kafka-test-pipeline-1"}
Usage:
  numaflow isbsvc-create [flags]

Flags:
      --buckets strings            Buckets to create
      --buffers strings            Buffers to create
  -h, --help                       help for isbsvc-create
      --isbsvc-type string         ISB Service type, e.g. jetstream
      --side-inputs-store string   Name of the side inputs store

panic: failed to create side inputs KV "secfraud-thirdparties-astraea-usw2-qal-kafka-test-pipeline-1_SIDE_INPUTS", nats: no suitable peers for placement

goroutine 1 [running]:
github.com/numaproj/numaflow/cmd/commands.Execute(...)
    /home/runner/work/numaflow/numaflow/cmd/commands/root.go:33
main.main()
    /home/runner/work/numaflow/numaflow/cmd/main.go:24 +0x3c
vigith commented 1 week ago

The error was transient and caused by a delay in the ISB coming up. The ISB delay was caused by the cluster autoscaler (no free nodes).

Looks like we always create SI Bucket even if the pipeline spec doesn't include one.