search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.47k stars 584 forks source link

Disable agent-service plugin in binary helm chart since service agent is disabled by default #5459

Closed Sovietaced closed 3 months ago

Sovietaced commented 3 months ago

Why are the changes needed?

This fixes an issue found where the Flyte sandbox workflow causes propeller to panic on startup. The cause of this panic is that propeller is attempting to initialize the agent plugin when the agent service is down (by default). Requests to the agent service to retrieve the list of supported tasks fails with a timeout.

{"json":{"src":"entrypoint.go:56"},"level":"info","msg":"Started propeller webhook","ts":"2024-06-08T00:43:17Z"}
panic: failed to initialize agent registry with error: failed to list agent: [&{dns:///flyteagent.flyte.svc.cluster.local:8000 true  map[gettask:10s] 10s}] with error: [rpc error: code = DeadlineExceeded desc = context deadline exceeded]

goroutine 32 [running]:
github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin()
        /flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:378 +0x45a
github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.RegisterAgentPlugin()
        /flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:403 +0xa5
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList.func1()
        /flyteorg/build/flytepropeller/pkg/controller/nodes/task/plugin_config.go:28 +0xf
sync.(*Once).doSlow(0x0?, 0x1000100?)
        /usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList({0x4302f80, 0xc00089ad70}, 0xc000773980, {0x7fc2084a7118, 0x5f3af40}, {0x43359c8?, 0xc00137f520})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/task/plugin_config.go:28 +0xac
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.(*Handler).Setup(0xc000903450, {0x4302f80, 0xc00089ad70}, {0x42f6930?, 0xc0009d2600})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/task/handler.go:231 +0x191
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/factory.(*handlerFactory).Setup(0xc000157b80, {0x4302f80, 0xc00089ad70}, {0x430f480?, 0xc000816320}, {0x42f6930, 0xc0009d2600})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/factory/handler_factory.go:73 +0x81f
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).Initialize(0xc000816320, {0x4302f80, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/executor.go:464 +0xbf
github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).Initialize(0xc000482500, {0x4302f80, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/workflow/executor.go:385 +0x59
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).Initialize(0x4302f80?, {0x4302f80?, 0xc00089ad70?})
        /flyteorg/build/flytepropeller/pkg/controller/handler.go:83 +0x26
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Initialize(...)
        /flyteorg/build/flytepropeller/pkg/controller/workers.go:121
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).run(0xc0005871f0, {0x4302f80, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/controller.go:109 +0x82
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).Run(0xc0005871f0, {0x4302f80?, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/controller.go:96 +0x119
github.com/flyteorg/flyte/flytepropeller/pkg/controller.StartController({0x4302f80?, 0xc00089acd0?}, 0xc000830000, {0x343e1f4, 0x3}, {0x431e3c0?, 0xc00137e680}, 0xc001408000)
        /flyteorg/build/flytepropeller/pkg/controller/controller.go:612 +0x5b4
github.com/flyteorg/flyte/cmd/single.startPropeller.func2()
        /flyteorg/build/cmd/single/start.go:167 +0x94
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 82
        /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96

What changes were proposed in this pull request?

This change remove the agent service from the enabled plugin list considering that the flyte agent service is disabled by default.

How was this patch tested?

See generated Kubernetes manifests.

Setup process

Screenshots

Check all the applicable boxes

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 61.07%. Comparing base (25c3596) to head (52f13a8). Report is 141 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #5459 +/- ## ========================================== - Coverage 61.07% 61.07% -0.01% ========================================== Files 793 793 Lines 51226 51226 ========================================== - Hits 31288 31287 -1 - Misses 17062 17063 +1 Partials 2876 2876 ``` | [Flag](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | Coverage Δ | | |---|---|---| | [unittests-datacatalog](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `69.31% <ø> (ø)` | | | [unittests-flyteadmin](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `58.90% <ø> (ø)` | | | [unittests-flytecopilot](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `17.79% <ø> (ø)` | | | [unittests-flytectl](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `67.97% <ø> (ø)` | | | [unittests-flyteidl](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `79.04% <ø> (ø)` | | | [unittests-flyteplugins](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `61.94% <ø> (ø)` | | | [unittests-flytepropeller](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `57.32% <ø> (ø)` | | | [unittests-flytestdlib](https://app.codecov.io/gh/flyteorg/flyte/pull/5459/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `65.80% <ø> (-0.03%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Future-Outlier commented 3 months ago

After this PR is merged, we can enable the agent-service plugin in the binary helm chart by default. https://github.com/flyteorg/flyte/pull/5460 So I think we only need to fix the typo.

Sovietaced commented 3 months ago

So I think we only need to fix the typo.

Fair enough, I will back out the helm chart change.