search

werf / nelm

Nelm is a Helm 3 alternative and werf deployment engine
Apache License 2.0
243 stars 1 forks source link

werf dismiss stuck after deploy custom chart with minio dependency #60

Open kozharskyad opened 3 months ago

kozharskyad commented 3 months ago

Before proceeding

Version

2.3.3

How to reproduce

  1. Environment: Hardware: Apple MacBook Pro, M3 Pro, 18 Gb RAM Software: macOS Sonoma 14.5

    $ k3d --version|head -1
k3d version v5.6.3
    $ kubectl version|tail -1
Server Version: v1.29.5+k3s1

  2. Add custom hosts record in your OS:

    $ echo '127.0.0.1 k3s-default' | sudo tee -a /etc/hosts

  3. Run k3s cluster via k3d:

    k3d cluster create --api-port 6550 -p '9080:80@loadbalancer' -p '9443:443@loadbalancer' -a 3 -s 3 -i rancher/k3s:v1.29.5-k3s1 --registry-create k3s-default:0.0.0.0:6000 --k3s-arg '--disable=traefik@server:*'

  4. Unpack werf-stuck-bug.tgz archive and change current working directory to directory that unpacked from archive

  5. Export environment variables:

    export WERF_DEV="true"
export WERF_INSECURE_REGISTRY="true"
export WERF_PLATFORM="linux/amd64"
export WERF_REPO="k3s-default:6000/werf-stuck-bug"
export WERF_TELEMETRY="0"

  6. Run werf converge

  7. After converge success, wait for MinIO Tenant is green healthStatus:

    kubectl get -n werf-stuck-bug tenant/minio -o jsonpath='{ .status.healthStatus } ' -w

    Cltr+C after green status appears.

  8. Run werf dismiss. Adding --debug flag possible, because it not affects result.

Result

$ werf dismiss --debug
146.409375ms Version: v2.3.3
162.783417ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 version"
165.75725ms  git version 2.45.2
166.14175ms  Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --git-dir /Users/a.kozharsky/Develop/comita/werf-stuck-bug/.git      ↵
166.15775ms  rev-parse"
168.896833ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --git-dir /Users/a.kozharsky/Develop/comita/werf-stuck-bug/.git      ↵
168.904458ms rev-parse --git-dir"
171.773667ms /Users/a.kozharsky/Develop/comita/werf-stuck-bug/.git
171.88875ms  Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 show-ref --head"
174.988ms    c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb HEAD
174.995ms    c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb refs/heads/_werf-dev
175.001458ms c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb refs/heads/master
175.494417ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 worktree list --porcelain"
179.782292ms worktree /Users/a.kozharsky/Develop/comita/werf-stuck-bug
179.786167ms HEAD c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
179.79525ms  branch refs/heads/master
179.799708ms 
179.836167ms worktree /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree
179.8455ms   HEAD c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
179.848375ms branch refs/heads/_werf-dev
179.853667ms 
179.925708ms ┌ Switch work tree /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22 ...
179.999667ms │ Work tree dir:                                                                                                                                ↵
180.007167ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree
180.03975ms  │ Commit: c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
180.092042ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 checkout --force --detach c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb"
189.729542ms │ HEAD сейчас на c2438c4 Initial commit
190.237333ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 reset --hard c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb"
195.869958ms │ Указатель HEAD сейчас на коммите c2438c4 Initial commit
196.087417ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree                                                        ↵
196.094667ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree clean    ↵
196.110792ms │ -d -f -f -x"
200.376ms    │ ┌ Sync submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97 ...
200.441417ms │ │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 submodule sync --recursive"
239.399625ms │ └ Sync submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713 ... (0.04 seconds)
239.44475ms  │ 
239.450208ms │ ┌ Update submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b ...
239.529708ms │ │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 submodule update --checkout --force --init --recursive"
277.515958ms │ └ Update submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b7 ... (0.04 seconds)
277.972417ms │ 
277.974417ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree                                                        ↵
277.985958ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree          ↵
278.003375ms │ submodule foreach --recursive git -c core.autocrlf=false -c gc.auto=0 reset --hard"
315.508458ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree                                                        ↵
315.515792ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree          ↵
315.533167ms │ submodule foreach --recursive git -c core.autocrlf=false -c gc.auto=0 clean -d -f -f -x"
350.3195ms   └ Switch work tree /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97 ... (0.17 seconds)
350.606375ms 
350.607167ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 branch --list _werf-dev"
354.74675ms    _werf-dev
354.895ms    Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 checkout _werf-dev"
359.78125ms  Переключились на ветку «_werf-dev»
360.215708ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 merge-base --is-ancestor c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb    ↵
360.235667ms _werf-dev"
364.097917ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 rev-parse _werf-dev"
367.441ms    c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
367.621208ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree /Users/a.kozharsky/Develop/comita/werf-stuck-bug add     ↵
367.628125ms --dry-run --ignore-missing --pathspec-from-file=- --pathspec-file-nul"
376.218958ms -- giterminism_manager.NewManager: projectDir="/Users/a.kozharsky/Develop/comita/werf-stuck-bug"                                                ↵
376.231042ms localGitRepo.WorkTreeDir="/Users/a.kozharsky/Develop/comita/werf-stuck-bug"
383.879792ms Using werf config render file: /private/var/folders/k7/l591ptpx0lz3kgd5cvhtcm3c0000gn/T/werf-config-render-653589138
384.933125ms 
384.933625ms Using namespace: werf-stuck-bug
384.948708ms Using release: werf-stuck-bug
426.596708ms uninstall: Deleting werf-stuck-bug
509.210458ms Starting delete for "console" ServiceStarting delete for "operator" ServiceStarting delete for "sts" ServiceStarting delete for "minio-operator" DeploymentStarting delete for "console" DeploymentStarting delete for "console-sa-binding" ClusterRoleBindingStarting delete for "minio-operator-binding" ClusterRoleBindingStarting delete for "console-sa-role" ClusterRoleStarting delete for "minio-operator-role" ClusterRoleStarting delete for "policybindings.sts.min.io" CustomResourceDefinitionStarting delete for "miniojobs.job.min.io" CustomResourceDefinitionStarting delete for "tenants.minio.min.io" CustomResourceDefinitionStarting delete for "console-env" ConfigMapStarting delete for "console-sa-secret" SecretStarting delete for "minio" SecretStarting delete for "minio-config" SecretStarting delete for "console-sa" ServiceAccountStarting delete for "minio-operator" ServiceAccountStarting delete for "minio" TenantWaiting for resources elimination: services/console, services/operator, services/sts, deployments/minio-operator, deployments/console, clus ...

Output stucks in this moment. All resources successfully removed, but Werf not going to stop and continue waiting.

Result stability: 9/10

Expected result

Werf successfully undeploy all resources and exit.

Additional information

Not sure if this may be cause stuck situation:

$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n minio
...
NAME                                            HOLDER                           AGE
lease.coordination.k8s.io/minio-operator-lock   minio-operator-9799b649c-4phlp   5m14s
...
ilya-lesikov commented 3 months ago

Thank you, will be solved as part of werf/nelm#68