search

garden-io / garden

Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
https://garden.io
Mozilla Public License 2.0
3.39k stars 275 forks source link

Crash: toKubernetesError encountered an unknown error unexpectedly during Kub #5623

Closed brkwokglx closed 9 months ago

brkwokglx commented 10 months ago

Crash report

Error message

` Validate

ℹ garden → Initializing... ℹ garden-dashboard → Connecting project... ✔ garden-dashboard → Ready Garden v0.13 (Bonsai) is a major release with significant changes. Please help us improve it by reporting any issues/bugs here: https://go.garden.io/report-bonsai → Run garden util hide-warning 0.13-bonsai to disable this message. ℹ garden → Running in environment local.vote-demo-quickstart-bryankwok ℹ providers → Resolving providers... ℹ exec → Configuring provider... ✔ exec → Provider configured ✔ exec → Provider status cached ℹ local-kubernetes → Configuring provider... ℹ local-kubernetes → Using current context: kind-kind ✔ local-kubernetes → Provider configured ℹ local-kubernetes → Ensuring nginx Ingress Controller... ℹ local-kubernetes → nginx Ingress Controller ready ℹ local-kubernetes → Not ready, will initialize ✖ providers → Failed processing resolve provider local-kubernetes (took 0.46 sec). This is what happened:

──────────────────────────────────────────────────────────────────────────────────────────────────────── Encountered an unexpected Garden error. This is likely a bug 🍂

You can help by reporting this on GitHub: https://github.com/garden-io/garden/issues/new?labels=bug,crash&template=CRASH.md&title=Crash%3A%20toKubernetesError%20encountered%20an%20unknown%20error%20unexpectedly%20during%20Kub...

Please attach the following information to the bug report after making sure that the error message does not contain sensitive information:

Failed processing resolve provider local-kubernetes (took 0.46 sec). This is what happened: Error: Cannot parse content. No Content-Type defined. at ObjectSerializer.parse (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:571760:21) at CoreV1ApiResponseProcessor.readNamespacedPodLogWithHttpInfo (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:605869:111) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async retry (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:702394:20) at async requestWithRetry (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:702420:20) at async file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706237:19 at async Promise.all (index 0) at async file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706300:25 at async Promise.all (index 0) at async getFormattedPodLogs (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706295:21) ────────────────────────────────────────────────────────────────────────────────────────────────────────

toKubernetesError encountered an unknown error unexpectedly during Kubernetes API: readNamespacedPodLog: toKubernetesError encountered an unknown error unexpectedly during readNamespacedPodLog: Cannot parse content. No Content-Type defined. Failed resolving one or more providers:

See .garden/error.log for detailed error message `

` [omitted_user_id]:~/garden/quickstart-example$ garden validate -l=silly [verbose] garden version: 0.13.23 Validate

ℹ garden → Initializing... ℹ garden-dashboard → [debug] Initializing Garden Cloud API client. [silly] Retrieving client auth token from config store [debug] Checking client auth token with the Garden dashboard: https://app.garden.io/token/verify [silly] Calling Cloud API with GET token/verify [silly] Retrieving client auth token from config store [debug] Checked client auth token with the Garden dashboard - valid: true ℹ garden-dashboard → [debug] Authorizing... ℹ garden-dashboard → [debug] Starting refresh interval. [debug] Will run refresh function every 4500 ms. [silly] Calling git with args 'rev-parse --abbrev-ref HEAD' in /home/bryankwok/garden/quickstart-example [silly] Calling git with args 'rev-parse HEAD' in /home/bryankwok/garden/quickstart-example [silly] Calling git with args 'config --get remote.origin.url' in /home/bryankwok/garden/quickstart-example ℹ garden-dashboard → Connecting project... ℹ garden-dashboard → [debug] Fetching or creating project vote-demo-quickstart from https://app.garden.io [silly] Calling Cloud API with GET /projects [silly] Retrieving client auth token from config store ✔ garden-dashboard → Ready [silly] No OTEL collector configured, setting no-op exporter [silly] Calling git with args 'rev-parse --show-toplevel' in /home/bryankwok/garden/quickstart-example Garden v0.13 (Bonsai) is a major release with significant changes. Please help us improve it by reporting any issues/bugs here: https://go.garden.io/report-bonsai → Run garden util hide-warning 0.13-bonsai to disable this message. ℹ garden → Running in environment local.vote-demo-quickstart-bryankwok [silly] Scanning for configs (force=false) [silly] Scanning for configs in /home/bryankwok/garden/quickstart-example [silly] TreeCache: No cached value for key ["git-repo-files","/home/bryankwok/garden/quickstart-example"] [silly] TreeCache: No cached value for key ["git-repo-files","/home/bryankwok/garden/quickstart-example"] [silly] Scanning repository at /home/bryankwok/garden/quickstart-example ℹ git → [debug] Scanning project root at /home/bryankwok/garden/quickstart-example → Includes: (none) → Excludes: .garden//*,.git,.gitmodules,.garden//,debug-info/,/.garden//* ℹ git → [silly] Calling git with args 'diff-index --name-only HEAD /home/bryankwok/garden/quickstart-example' in /home/bryankwok/garden/quickstart-example ℹ git → [silly] Calling git with args '--glob-pathspecs ls-files --ignored --cached --exclude /home/bryankwok/garden/quickstart-example/.garden --exclude .garden//* --exclude .git --exclude .gitmodules --exclude .garden// --exclude debug-info/ --exclude /.garden//* --exclude-per-directory .gitignore' in /home/bryankwok/garden/quickstart-example ℹ git → [silly] Calling git with args '--glob-pathspecs ls-files -s --others --cached --exclude /home/bryankwok/garden/quickstart-example/.garden --exclude .garden//* --exclude .git --exclude .gitmodules --exclude .garden// --exclude debug-info/ --exclude /.garden//* --exclude-per-directory .gitignore' in /home/bryankwok/garden/quickstart-example ℹ git → [debug] Found 67 files in project root /home/bryankwok/garden/quickstart-example [silly] TreeCache: Setting value for key ["git-repo-files","/home/bryankwok/garden/quickstart-example"] [debug] Found 67 files in module path, filtering by 1 include and 6 exclude globs [silly] Include globs: /garden.yml [silly] Exclude globs: .garden//*, .git/*/, .gitmodules, .garden//, debug-info/, /.garden/*/ [silly] Checking if api/garden.yml matches include/exclude globs [silly] Checking if postgres/garden.yml matches include/exclude globs [silly] Checking if redis/garden.yml matches include/exclude globs [silly] Checking if result/garden.yml matches include/exclude globs [silly] Checking if vote/garden.yml matches include/exclude globs [silly] Checking if worker/garden.yml matches include/exclude globs [silly] Checking if project.garden.yml matches include/exclude globs [debug] Found 7 files in module path after glob matching [silly] Load configs from /home/bryankwok/garden/quickstart-example/api/garden.yml [silly] Load configs from /home/bryankwok/garden/quickstart-example/postgres/garden.yml [silly] Load configs from /home/bryankwok/garden/quickstart-example/redis/garden.yml [silly] Load configs from /home/bryankwok/garden/quickstart-example/result/garden.yml [silly] Load configs from /home/bryankwok/garden/quickstart-example/vote/garden.yml [silly] Load configs from /home/bryankwok/garden/quickstart-example/worker/garden.yml [silly] Load configs from /home/bryankwok/garden/quickstart-example/project.garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/api/garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/postgres/garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/vote/garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/worker/garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/project.garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/redis/garden.yml [silly] Loaded configs from /home/bryankwok/garden/quickstart-example/result/garden.yml [silly] Adding Build action api [silly] Adding Build action result [silly] Adding Build action vote [silly] Adding Build action e2e-runner [silly] Adding Build action worker [silly] Adding Deploy action api [silly] Adding Deploy action db [silly] Adding Deploy action redis [silly] Adding Deploy action result [silly] Adding Deploy action vote [silly] Adding Deploy action worker [silly] Adding Run action db-init [silly] Adding Run action db-clear [silly] Adding Test action unit [silly] Adding Test action result-integ [silly] Adding Test action unit-vote [silly] Adding Test action e2e-vote [debug] Scanned and found 17 actions, 0 workflows and 0 modules [silly] Calling Cloud API with GET /profile [silly] Retrieving client auth token from config store [silly] BufferedEventStream: Connected [silly] TreeCache: Invalidating caches down from context ["path"] [silly] Starting command 'validate' action ℹ providers → Resolving providers... [silly] Loading plugins [silly] Loading plugin container [silly] Loading plugin exec [silly] Loading plugin hadolint [silly] Loading plugin kubernetes [silly] Loading plugin local-kubernetes [silly] Loading plugin ephemeral-kubernetes [silly] Loading plugin openshift [silly] Loading plugin octant [silly] Loading plugin otel-collector [silly] Loading plugin templated [silly] Loading plugin conftest [silly] Loading plugin conftest-container [silly] Loading plugin conftest-kubernetes [silly] Loading plugin jib [silly] Loading plugin terraform [silly] Loading plugin pulumi [silly] Loaded plugin ephemeral-kubernetes [silly] Loaded plugin hadolint [silly] Loaded plugin octant [silly] Loaded plugin otel-collector [silly] Loaded plugin templated [silly] Loaded plugin conftest-container [silly] Loaded plugin conftest [silly] Loaded plugin conftest-kubernetes [silly] Loaded plugin terraform [silly] Loaded plugin pulumi [silly] Loaded plugin container [silly] Loaded plugin jib [silly] Loaded plugin local-kubernetes [silly] Loaded plugin openshift [silly] Loaded plugin kubernetes [silly] Loaded plugin exec [silly] Validating plugin container [silly] Done loading plugin container [silly] Validating plugin hadolint [silly] Done loading plugin hadolint [silly] Validating plugin jib [silly] Done loading plugin jib [silly] Validating plugin kubernetes [silly] Done loading plugin kubernetes [silly] Validating plugin local-kubernetes [silly] Done loading plugin local-kubernetes [silly] Validating plugin ephemeral-kubernetes [silly] Done loading plugin ephemeral-kubernetes [silly] Validating plugin openshift [silly] Done loading plugin openshift [silly] Validating plugin octant [silly] Done loading plugin octant [silly] Validating plugin exec [silly] Done loading plugin exec [silly] Validating plugin otel-collector [silly] Done loading plugin otel-collector [silly] Validating plugin templated [silly] Done loading plugin templated [silly] Validating plugin conftest [silly] Done loading plugin conftest [silly] Validating plugin conftest-container [silly] Done loading plugin conftest-container [silly] Validating plugin conftest-kubernetes [silly] Done loading plugin conftest-kubernetes [silly] Validating plugin terraform [silly] Done loading plugin terraform [silly] Validating plugin pulumi [silly] Done loading plugin pulumi [silly] Loaded plugins: container, hadolint, jib, kubernetes, local-kubernetes, ephemeral-kubernetes, openshift, octant, exec, otel-collector, templated, conftest, conftest-container, conftest-kubernetes, terraform, pulumi [silly] GraphSolver: Starting batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 (4 tasks) [silly] GraphSolver: start [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 is missing its status. [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 is missing its status. [silly] Request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 is missing its status. [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 is missing its status. [silly] Task nodes in progress: (none) ℹ graph-solver → [silly] Processing node resolve-provider.exec:status ℹ providers → [silly] Executing node resolve-provider.exec:status ℹ graph-solver → [silly] Processing node resolve-provider.container:status ℹ providers → [silly] Executing node resolve-provider.container:status ℹ graph-solver → [silly] Processing node resolve-provider.templated:status ℹ providers → [silly] Executing node resolve-provider.templated:status ℹ graph-solver → [silly] Processing node resolve-provider.local-kubernetes:status ℹ providers → [silly] Executing node resolve-provider.local-kubernetes:status ℹ providers → [silly] Completing node resolve-provider.exec:status. aborted=false, error=null ℹ providers → [silly] Completing node resolve-provider.container:status. aborted=false, error=null ℹ providers → [silly] Completing node resolve-provider.templated:status. aborted=false, error=null ℹ providers → [silly] Completing node resolve-provider.local-kubernetes:status. aborted=false, error=null [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider exec","key":"resolve-provider.exec","name":"exec","aborted":false,"startedAt":"2024-01-10T02:38:34.796Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider container","key":"resolve-provider.container","name":"container","aborted":false,"startedAt":"2024-01-10T02:38:34.797Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider templated","key":"resolve-provider.templated","name":"templated","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: (none) ℹ graph-solver → [silly] Processing node resolve-provider.exec:process ℹ providers → [silly] Executing node resolve-provider.exec:process ℹ providers → [silly] Resolving template strings for provider exec ℹ exec → Configuring provider... ℹ providers → [silly] Validating exec config ℹ providers → [silly] Calling configureProvider on exec ℹ graph-solver → [silly] Processing node resolve-provider.templated:process ℹ providers → [silly] Executing node resolve-provider.templated:process ℹ providers → [silly] Resolving template strings for provider templated ℹ templated → [debug] Configuring provider... ℹ providers → [silly] Validating templated config ℹ providers → [silly] Calling configureProvider on templated ℹ graph-solver → [silly] Processing node resolve-provider.container:process ℹ providers → [silly] Executing node resolve-provider.container:process ℹ providers → [silly] Resolving template strings for provider container ℹ container → [debug] Configuring provider... ℹ providers → [silly] Validating container config ℹ providers → [silly] Calling configureProvider on container [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider exec","key":"resolve-provider.exec","name":"exec","aborted":false,"startedAt":"2024-01-10T02:38:34.796Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider container","key":"resolve-provider.container","name":"container","aborted":false,"startedAt":"2024-01-10T02:38:34.797Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider templated","key":"resolve-provider.templated","name":"templated","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: processing resolve provider exec, processing resolve provider templated, processing resolve provider container [silly] GraphSolver: loop [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider exec","key":"resolve-provider.exec","name":"exec","aborted":false,"startedAt":"2024-01-10T02:38:34.796Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider container","key":"resolve-provider.container","name":"container","aborted":false,"startedAt":"2024-01-10T02:38:34.797Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider templated","key":"resolve-provider.templated","name":"templated","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: processing resolve provider exec, processing resolve provider templated, processing resolve provider container [silly] GraphSolver: loop [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider exec","key":"resolve-provider.exec","name":"exec","aborted":false,"startedAt":"2024-01-10T02:38:34.796Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider container","key":"resolve-provider.container","name":"container","aborted":false,"startedAt":"2024-01-10T02:38:34.797Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider templated","key":"resolve-provider.templated","name":"templated","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: processing resolve provider exec, processing resolve provider templated, processing resolve provider container [silly] GraphSolver: loop [silly] Registering Build type exec [silly] Registering Build type container [silly] Registering Deploy type exec [silly] Registering Deploy type container [silly] Registering Deploy type kubernetes [silly] Registering Deploy type helm [silly] Registering Deploy type configmap [silly] Registering Deploy type persistentvolumeclaim [silly] Registering Run type exec [silly] Registering Run type container [silly] Registering Run type kubernetes-exec [silly] Registering Run type kubernetes-pod [silly] Registering Run type helm-pod [silly] Registering Test type exec [silly] Registering Test type container [silly] Registering Test type kubernetes-exec [silly] Registering Test type kubernetes-pod [silly] Registering Test type helm-pod [silly] Creating ActionRouter with 4 configured providers [silly] Registering Build type exec [silly] Registering Build type container [silly] Registering Deploy type exec [silly] Registering Deploy type container [silly] Registering Deploy type kubernetes [silly] Registering Deploy type helm [silly] Registering Deploy type configmap [silly] Registering Deploy type persistentvolumeclaim [silly] Registering Run type exec [silly] Registering Run type container [silly] Registering Run type kubernetes-exec [silly] Registering Run type kubernetes-pod [silly] Registering Run type helm-pod [silly] Registering Test type exec [silly] Registering Test type container [silly] Registering Test type kubernetes-exec [silly] Registering Test type kubernetes-pod [silly] Registering Test type helm-pod [silly] Creating ActionRouter with 4 configured providers [silly] Registering Build type exec [silly] Registering Build type container [silly] Registering Deploy type exec [silly] Registering Deploy type container [silly] Registering Deploy type kubernetes [silly] Registering Deploy type helm [silly] Registering Deploy type configmap [silly] Registering Deploy type persistentvolumeclaim [silly] Registering Run type exec [silly] Registering Run type container [silly] Registering Run type kubernetes-exec [silly] Registering Run type kubernetes-pod [silly] Registering Run type helm-pod [silly] Registering Test type exec [silly] Registering Test type container [silly] Registering Test type kubernetes-exec [silly] Registering Test type kubernetes-pod [silly] Registering Test type helm-pod [silly] Creating ActionRouter with 4 configured providers [silly] Calling 'configureProvider' handler on 'exec' [silly] Calling 'configureProvider' handler on 'templated' [silly] Calling 'configureProvider' handler on 'container' [silly] Returned default 'configureProvider' handler for 'exec' [silly] Returned default 'configureProvider' handler for 'templated' [silly] Returned default 'configureProvider' handler for 'container' [silly] Called 'configureProvider' handler on 'exec' [silly] Called 'configureProvider' handler on 'templated' [silly] Called 'configureProvider' handler on 'container' ℹ providers → [silly] Validating exec config returned from configureProvider handler ℹ providers → [silly] Ensuring exec provider is ready ✔ exec → Provider configured ℹ providers → [silly] Validating templated config returned from configureProvider handler ℹ providers → [silly] Ensuring templated provider is ready ✔ templated → [debug] Provider configured ℹ providers → [silly] Validating container config returned from configureProvider handler ℹ providers → [silly] Ensuring container provider is ready ✔ container → [debug] Provider configured ℹ providers → [silly] Checking provider status cache for exec at /home/bryankwok/garden/quickstart-example/.garden/cache/provider-statuses/exec.json ℹ providers → [silly] Checking provider status cache for templated at /home/bryankwok/garden/quickstart-example/.garden/cache/provider-statuses/templated.json ℹ providers → [silly] Checking provider status cache for container at /home/bryankwok/garden/quickstart-example/.garden/cache/provider-statuses/container.json ✔ templated → [debug] Provider status cached ℹ providers → [verbose] resolve provider templated is ready. ℹ providers → [silly] Completing node resolve-provider.templated:process. aborted=false, error=null [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider exec","key":"resolve-provider.exec","name":"exec","aborted":false,"startedAt":"2024-01-10T02:38:34.796Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider container","key":"resolve-provider.container","name":"container","aborted":false,"startedAt":"2024-01-10T02:38:34.797Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 has been processed. ℹ providers → [silly] Completing node resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56. aborted=false, error=null [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 called with result resolve-provider.templated [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 matched with request resolve-provider.templated:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 [silly] Batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 has 3 result(s) still missing: resolve-provider.exec, resolve-provider.container, resolve-provider.local-kubernetes [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: processing resolve provider exec, processing resolve provider container [silly] GraphSolver: loop ✔ exec → Provider status cached ℹ providers → [verbose] resolve provider exec is ready. ℹ providers → [silly] Completing node resolve-provider.exec:process. aborted=false, error=null [silly] GraphSolver: loop [silly] Request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 has been processed. ℹ providers → [silly] Completing node resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56. aborted=false, error=null [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 called with result resolve-provider.exec [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 matched with request resolve-provider.exec:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 [silly] Batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 has 2 result(s) still missing: resolve-provider.container, resolve-provider.local-kubernetes [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider container","key":"resolve-provider.container","name":"container","aborted":false,"startedAt":"2024-01-10T02:38:34.797Z","completedAt":"2024-01-10T02:38:34.799Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: processing resolve provider container [silly] GraphSolver: loop ✔ container → [debug] Provider status cached ℹ providers → [verbose] resolve provider container is ready. ℹ providers → [silly] Completing node resolve-provider.container:process. aborted=false, error=null [silly] GraphSolver: loop [silly] Request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 has been processed. ℹ providers → [silly] Completing node resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56. aborted=false, error=null [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 called with result resolve-provider.container [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 matched with request resolve-provider.container:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 [silly] Batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 has 1 result(s) still missing: resolve-provider.local-kubernetes [silly] Request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 should be processed. Status: {"type":"resolve-provider","description":"resolve provider local-kubernetes","key":"resolve-provider.local-kubernetes","name":"local-kubernetes","aborted":false,"startedAt":"2024-01-10T02:38:34.798Z","completedAt":"2024-01-10T02:38:34.800Z","error":null,"inputVersion":"0.13.23","processed":false,"success":true,"attached":false} [silly] Task nodes in progress: (none) ℹ graph-solver → [silly] Processing node resolve-provider.local-kubernetes:process ℹ providers → [silly] Executing node resolve-provider.local-kubernetes:process ℹ providers → [silly] Resolving template strings for provider local-kubernetes ℹ local-kubernetes → Configuring provider... ℹ providers → [silly] Validating local-kubernetes config ℹ providers → [silly] Calling configureProvider on local-kubernetes [silly] Calling 'configureProvider' handler on 'local-kubernetes' [silly] Found 'configureProvider' handler on 'local-kubernetes' ℹ local-kubernetes → [silly] Execing '/home/bryankwok/.garden/tools/kubectl/49eb930aa565a80f/kubectl config view --raw' in /home/bryankwok/.garden/tools/kubectl/49eb930aa565a80f ℹ local-kubernetes → Using current context: kind-kind [silly] Called 'configureProvider' handler on 'local-kubernetes' ℹ providers → [silly] Validating local-kubernetes config returned from configureProvider handler ℹ providers → [silly] Validating 'local-kubernetes' config against 'kubernetes' schema ℹ providers → [silly] Ensuring local-kubernetes provider is ready ✔ local-kubernetes → Provider configured ℹ providers → [silly] Checking provider status cache for local-kubernetes at /home/bryankwok/garden/quickstart-example/.garden/cache/provider-statuses/local-kubernetes.json ℹ providers → [silly] Unable to find or read provider status from /home/bryankwok/garden/quickstart-example/.garden/cache/provider-statuses/local-kubernetes.json: Error: ENOENT: no such file or directory, open '/home/bryankwok/garden/quickstart-example/.garden/cache/provider-statuses/local-kubernetes.json' [silly] Found 'getEnvironmentStatus' handler on 'local-kubernetes' ℹ local-kubernetes → [silly] Execing '/home/bryankwok/.garden/tools/kubectl/49eb930aa565a80f/kubectl --context=kind-kind config view --raw' in /home/bryankwok/.garden/tools/kubectl/49eb930aa565a80f ℹ local-kubernetes → [silly] GET https://127.0.0.1:38685/version ℹ local-kubernetes → Ensuring nginx Ingress Controller... ℹ local-kubernetes → nginx Ingress Controller ready ℹ local-kubernetes → Not ready, will initialize [silly] Found 'prepareEnvironment' handler on 'local-kubernetes' ℹ local-kubernetes → [debug] Found kind with the following version details kind v0.20.0 go1.20.5 linux/amd64 ℹ local-kubernetes → [silly] Fetching Kubernetes resource apps/v1/DaemonSet/kindnet ℹ local-kubernetes → [debug] Kubernetes: Getting API resource info for group apps/v1 ℹ local-kubernetes → [silly] GET https://127.0.0.1:38685/apis/apps/v1 ℹ local-kubernetes → [silly] GET https://127.0.0.1:38685/apis/apps/v1/namespaces/kube-system/daemonsets/kindnet ℹ local-kubernetes → [silly] Fetching Kubernetes resource apps/v1/Deployment/ingress-nginx-controller ℹ local-kubernetes → [silly] GET https://127.0.0.1:38685/apis/apps/v1/namespaces/garden-system/deployments/ingress-nginx-controller ℹ providers → [silly] Completing node resolve-provider.local-kubernetes:process. aborted=false, error=toKubernetesError encountered an unknown error unexpectedly during Kubernetes API: readNamespacedPodLog: toKubernetesError encountered an unknown error unexpectedly during readNamespacedPodLog: Cannot parse content. No Content-Type defined. ℹ providers → [silly] Completing node resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56. aborted=false, error=resolve provider local-kubernetes failed: Error: toKubernetesError encountered an unknown error unexpectedly during Kubernetes API: readNamespacedPodLog: toKubernetesError encountered an unknown error unexpectedly during readNamespacedPodLog: Cannot parse content. No Content-Type defined. [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 called with result resolve-provider.local-kubernetes [silly] GraphSolver: Complete handler for batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 matched with request resolve-provider.local-kubernetes:request:2a17afaa-a05c-44a8-9468-b036d64f0b56 [silly] Batch 2a17afaa-a05c-44a8-9468-b036d64f0b56 failed: Failed to complete 1/4 tasks: ↳ resolve provider local-kubernetes: resolve provider local-kubernetes failed: Error: toKubernetesError encountered an unknown error unexpectedly during Kubernetes API: readNamespacedPodLog: toKubernetesError encountered an unknown error unexpectedly during readNamespacedPodLog: Cannot parse content. No Content-Type defined. ✖ providers → Failed processing resolve provider local-kubernetes (took 0.57 sec). This is what happened:

──────────────────────────────────────────────────────────────────────────────────────────────────────── Encountered an unexpected Garden error. This is likely a bug 🍂

You can help by reporting this on GitHub: https://github.com/garden-io/garden/issues/new?labels=bug,crash&template=CRASH.md&title=Crash%3A%20toKubernetesError%20encountered%20an%20unknown%20error%20unexpectedly%20during%20Kub...

Please attach the following information to the bug report after making sure that the error message does not contain sensitive information:

Failed processing resolve provider local-kubernetes (took 0.57 sec). This is what happened: Error: Cannot parse content. No Content-Type defined. at ObjectSerializer.parse (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:571760:21) at CoreV1ApiResponseProcessor.readNamespacedPodLogWithHttpInfo (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:605869:111) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async retry (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:702394:20) at async requestWithRetry (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:702420:20) at async file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706237:19 at async Promise.all (index 0) at async file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706300:25 at async Promise.all (index 0) at async getFormattedPodLogs (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706295:21) ────────────────────────────────────────────────────────────────────────────────────────────────────────

toKubernetesError encountered an unknown error unexpectedly during Kubernetes API: readNamespacedPodLog: toKubernetesError encountered an unknown error unexpectedly during readNamespacedPodLog: Cannot parse content. No Content-Type defined. ℹ providers → [silly] Full error with stack trace and wrapped errors: ──────────────────────────────────────────────────────────────────────────────────────────────────────── Error: Cannot parse content. No Content-Type defined. at ObjectSerializer.parse (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:571760:21) at CoreV1ApiResponseProcessor.readNamespacedPodLogWithHttpInfo (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:605869:111) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async retry (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:702394:20) at async requestWithRetry (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:702420:20) at async file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706237:19 at async Promise.all (index 0) at async file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706300:25 at async Promise.all (index 0) at async getFormattedPodLogs (file:///home/bryankwok/.local/share/garden/1704780732-ScZIxCW.r/rollup/garden.mjs:706295:21) Error type: crash ──────────────────────────────────────────────────────────────────────────────────────────────────────── [silly] GraphSolver: loop [silly] Task nodes in progress: (none) [debug] Done flushing all events and log entries. [silly] Tracking Run Command event. Payload: {"userId":[omitted_user_id],"anonymousId":"a1f545dd-f8b3-4e9b-bb4a-6284e40890d3","event":"Run Command","properties":{"projectId":"87ec0a05c08b0fde0b1b8fb4f743a23de957a001928162791e9a116abf515606c7d4e42bcb488ca9025ec88ce75e2a8b2ba543bfbf629565699b05a42d727018","projectIdV2":"strong-deeply-president_87ec0a05c08b0fde0b1b8fb4f743a23d","projectName":"1a1828ac00d21a1f0a11111eb44300cef05967137dde75bee4ebfe87fd472f640a38be8991d643c7f65b9992b3955a38479390a08d774e6589f57fe81e7bb69f","projectNameV2":"painstaking-elfin-home_1a1828ac00d21a1f0a11111eb44300ce","enterpriseProjectId":"0bd5f7e9cf5bdeff500431cb2fbed2d533861609fd6ac73db466026fc53422e35590edc3b76dcf9c7cfbaf32129f51f6e2047a05395837244ed73e9784f6b085","enterpriseProjectIdV2":"unbiased-dependent-dress_0bd5f7e9cf5bdeff500431cb2fbed2d5","enterpriseDomain":"2769c2abae62151b2ebb8658628f7c5f5d0dc0c29fdefdd19a23dec9cb0a7b96d74d82512d1f6906bef65b24a29d84685dfd2fd66964a56fbdaff39fabd69206","enterpriseDomainV2":"ratty-willing-science_2769c2abae62151b2ebb8658628f7c5f","isLoggedIn":true,"ciName":null,"customer":"[username_omitted]","system":{"platform":"linux","platformVersion":"5.15.133.1-microsoft-standard-WSL2","gardenVersion":"0.13.23"},"isCI":false,"sessionId":"5cbf0a96-7cb8-4ec9-bbce-4531a1417b0a","parentSessionId":"5cbf0a96-7cb8-4ec9-bbce-4531a1417b0a","projectMetadata":{"modulesCount":0,"moduleTypes":[],"tasksCount":0,"servicesCount":0,"testsCount":0,"actionsCount":17,"buildActionCount":5,"testActionCount":4,"deployActionCount":6,"runActionCount":2},"firstRunAt":"2023-12-18T19:30:22.711Z","latestRunAt":"2024-01-10T02:38:34.493Z","isRecurringUser":true,"name":"validate"}} [silly] Tracking Command Result event. Payload: {"userId":[omitted_user_id],"anonymousId":"a1f545dd-f8b3-4e9b-bb4a-6284e40890d3","event":"Command Result","properties":{"projectId":"87ec0a05c08b0fde0b1b8fb4f743a23de957a001928162791e9a116abf515606c7d4e42bcb488ca9025ec88ce75e2a8b2ba543bfbf629565699b05a42d727018","projectIdV2":"strong-deeply-president_87ec0a05c08b0fde0b1b8fb4f743a23d","projectName":"1a1828ac00d21a1f0a11111eb44300cef05967137dde75bee4ebfe87fd472f640a38be8991d643c7f65b9992b3955a38479390a08d774e6589f57fe81e7bb69f","projectNameV2":"painstaking-elfin-home_1a1828ac00d21a1f0a11111eb44300ce","enterpriseProjectId":"0bd5f7e9cf5bdeff500431cb2fbed2d533861609fd6ac73db466026fc53422e35590edc3b76dcf9c7cfbaf32129f51f6e2047a05395837244ed73e9784f6b085","enterpriseProjectIdV2":"unbiased-dependent-dress_0bd5f7e9cf5bdeff500431cb2fbed2d5","enterpriseDomain":"2769c2abae62151b2ebb8658628f7c5f5d0dc0c29fdefdd19a23dec9cb0a7b96d74d82512d1f6906bef65b24a29d84685dfd2fd66964a56fbdaff39fabd69206","enterpriseDomainV2":"ratty-willing-science_2769c2abae62151b2ebb8658628f7c5f","isLoggedIn":true,"ciName":null,"customer":"[omitted_user_id]","system":{"platform":"linux","platformVersion":"5.15.133.1-microsoft-standard-WSL2","gardenVersion":"0.13.23"},"isCI":false,"sessionId":"5cbf0a96-7cb8-4ec9-bbce-4531a1417b0a","parentSessionId":"5cbf0a96-7cb8-4ec9-bbce-4531a1417b0a","projectMetadata":{"modulesCount":0,"moduleTypes":[],"tasksCount":0,"servicesCount":0,"testsCount":0,"actionsCount":17,"buildActionCount":5,"testActionCount":4,"deployActionCount":6,"runActionCount":2},"firstRunAt":"2023-12-18T19:30:22.711Z","latestRunAt":"2024-01-10T02:38:34.493Z","isRecurringUser":true,"errors":["plugin"],"lastError":{"error":{"errorType":"plugin","stackTrace":{"functionName":"","lineNumber":820734}},"wrapped":{"errorType":"graph","taskType":"resolve-provider","stackTrace":{"functionName":"ProcessTaskNode.complete","lineNumber":793684}},"leaf":{"errorType":"crash","stackTrace":{"functionName":"ObjectSerializer.parse","lineNumber":571760}}},"result":"failure","name":"validate","durationMsec":583,"exitCode":1}} Failed resolving one or more providers:

See .garden/error.log for detailed error message bryankwok@DESKTOP-I8HNE8I:~/garden/quickstart-example$ `

What did you do?

Your environment

Frequency

Reproducable. When testing with the Quickstart example, it appears that the error occurs whenever defaultEnvironment is set to local on the project.garden.yml file.

Example:

Workaround

Additional context

10ko commented 10 months ago

Hi @brkwokglx, thanks for the detailed report! We'll take a look.