search

GoogleContainerTools / skaffold

Easy and Repeatable Kubernetes Development
https://skaffold.dev/
Apache License 2.0
15.04k stars 1.62k forks source link

Skaffold render does not update helm dependencies #5445

Closed rccrdpccl closed 1 year ago

rccrdpccl commented 3 years ago

Expected behavior

when running skaffold render with helm deploy, I would expect the render to include helm dependencies. If not possible, it would be good to have a switch to only update deploy dependencies

Actual behavior

when running skaffold render with helm deploy, no dependencies are update, incurring in an error when the command is run from a fresh checkout (for example from CI)

Information

apiVersion: skaffold/v2beta12
kind: Config
metadata:
  name: test
deploy:
  helm:
    releases:
      - name: test-v2
        chartPath: ./charts/test
        valuesFiles:
          - ./charts/test/values.yaml
        createNamespace: true
        skipBuildDependencies: false
        namespace: test-v2

Steps to reproduce the behavior

  1. git clone git@github.com:rccrdpccl/skaffold-debug.git
  2. skaffold render
  3. got rendering manifests: std out err: Error: found in Chart.yaml, but missing in charts/ directory: mysql, however I'd expect it to render
tejal29 commented 3 years ago

adding #5442

tejal29 commented 3 years ago

Was able to reproduce this on the latest master.

Here are the debug logs

 ../skaffold/out/skaffold render -v=debug
INFO[0000] Skaffold &{Version:v1.23.0-41-g01feff8c4-dirty ConfigVersion:skaffold/v2beta16 GitVersion: GitCommit:01feff8c4dccaa6d76072abac1fe9ed6494da866 BuildDate:2021-05-13T12:31:10Z GoVersion:go1.16.3 Compiler:gc Platform:darwin/amd64 User:} 
DEBU[0000] Disable housekeeping messages for command explicitly 
DEBU[0000] config version out of date: upgrading to latest "skaffold/v2beta16" 
DEBU[0000] parsed 1 configs from configuration file /Users/tejaldesai/workspace/skaffold-debug/skaffold.yaml 
DEBU[0000] Defaulting build type to local build         
INFO[0000] Using kubectl context: gke_tejal-gke1_us-central1-c_dump-apis 
INFO[0000] Loaded Skaffold defaults from "/Users/tejaldesai/.skaffold/config" 
DEBU[0000] Running command: [minikube version --output=json] 
DEBU[0000] Command output: [{"commit":"043bdca07e54ab6e4fc0457e3064048f34133d7e","minikubeVersion":"v1.17.1"}
] 
DEBU[0000] Running command: [/Users/tejaldesai/workspace/google-cloud-sdk/bin/minikube profile list -o json] 
DEBU[0001] Command output: [{"invalid":[],"valid":[{"Name":"gcloud-local-dev","Status":"Running","Config":{"Name":"gcloud-local-dev","KeepContext":true,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e","Memory":8100,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","SSHIPAddress":"","SSHUser":"root","SSHKey":"","SSHPort":22,"KubernetesConfig":{"KubernetesVersion":"v1.20.2","ClusterName":"gcloud-local-dev","Namespace":"default","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","CustomIngressCert":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"192.168.59.2","Port":8443,"KubernetesVersion":"v1.20.2","ControlPlane":true,"Worker":true}],"Addons":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":[],"Network":"","MultiNodeRequested":false}},{"Name":"minikube","Status":"Running","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e","Memory":8100,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","SSHIPAddress":"","SSHUser":"root","SSHKey":"","SSHPort":22,"KubernetesConfig":{"KubernetesVersion":"v1.20.2","ClusterName":"minikube","Namespace":"default","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","CustomIngressCert":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"192.168.49.2","Port":8443,"KubernetesVersion":"v1.20.2","ControlPlane":true,"Worker":true}],"Addons":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":[],"Network":"","MultiNodeRequested":false}}]}] 
DEBU[0001] Running command: [/Users/tejaldesai/workspace/google-cloud-sdk/bin/minikube profile list -o json] 
DEBU[0003] Command output: [{"invalid":[],"valid":[{"Name":"gcloud-local-dev","Status":"Running","Config":{"Name":"gcloud-local-dev","KeepContext":true,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e","Memory":8100,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","SSHIPAddress":"","SSHUser":"root","SSHKey":"","SSHPort":22,"KubernetesConfig":{"KubernetesVersion":"v1.20.2","ClusterName":"gcloud-local-dev","Namespace":"default","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","CustomIngressCert":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"192.168.59.2","Port":8443,"KubernetesVersion":"v1.20.2","ControlPlane":true,"Worker":true}],"Addons":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":[],"Network":"","MultiNodeRequested":false}},{"Name":"minikube","Status":"Running","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e","Memory":8100,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","SSHIPAddress":"","SSHUser":"root","SSHKey":"","SSHPort":22,"KubernetesConfig":{"KubernetesVersion":"v1.20.2","ClusterName":"minikube","Namespace":"default","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","CustomIngressCert":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"192.168.49.2","Port":8443,"KubernetesVersion":"v1.20.2","ControlPlane":true,"Worker":true}],"Addons":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":[],"Network":"","MultiNodeRequested":false}}]}] 
DEBU[0003] setting Docker user agent to skaffold-v1.23.0-41-g01feff8c4-dirty 
DEBU[0003] Using builder: local                         
DEBU[0003] push value not present in NewBuilder, defaulting to true because cluster.PushImages is true 
INFO[0003] build concurrency first set to 0 parsed from *local.Builder[0] 
INFO[0003] final build concurrency value is 0           
DEBU[0003] Running command: [helm version --client]     
DEBU[0003] Command output: [version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"dirty", GoVersion:"go1.16.3"}
] 
INFO[0003] Tags generated in 1.494µs                    
DEBU[0003] Executing template &{envTemplate 0xc000c82d80 0xc000d864c0  } with environment map[DISPLAY:/private/tmp/com.apple.launchd.42hcsqMQdi/org.xquartz:0 HOME:/Users/tejaldesai LANG:en_US.UTF-8 LESS:-R LOGNAME:tejaldesai LSCOLORS:Gxfxcxdxbxegedabagacad LaunchInstanceID:5209FE7B-121C-48CB-936C-55708CB3E126 OLDPWD:/Users/tejaldesai/workspace PAGER:less PATH:/Users/tejaldesai/workspace/google-cloud-sdk/bin:/usr/local/opt/python@3.8/bin:/Users/tejaldesai/mdproxy/bin:/Users/tejaldesai/workspace/google-cloud-sdk/bin:/usr/local/opt/python@3.8/bin:/Users/tejaldesai/mdproxy/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/X11/bin:/Library/Apple/usr/bin PWD:/Users/tejaldesai/workspace/skaffold-debug SECURITYSESSIONID:186ab SHELL:/bin/zsh SHLVL:1 SK_SIGNING_PLUGIN:gnubbyagent SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.m795MDvENI/Listeners TERM:xterm-256color TERM_PROGRAM:Apple_Terminal TERM_PROGRAM_VERSION:440 TERM_SESSION_ID:21CD0E26-1AF9-4720-8484-F6B4AB24F9F8 TMPDIR:/var/folders/gk/s778hvkj0lb0zl95ywvw0snr00cfc0/T/ USER:tejaldesai XPC_FLAGS:0x0 XPC_SERVICE_NAME:0 ZSH:/Users/tejaldesai/.oh-my-zsh _:/Users/tejaldesai/workspace/skaffold-debug/../skaffold/out/skaffold __CFBundleIdentifier:com.apple.Terminal] 
DEBU[0003] EnvVarMap: map[]                             
DEBU[0003] Executing template &{envTemplate 0xc000c83440 0xc000d86640  } with environment map[DISPLAY:/private/tmp/com.apple.launchd.42hcsqMQdi/org.xquartz:0 HOME:/Users/tejaldesai LANG:en_US.UTF-8 LESS:-R LOGNAME:tejaldesai LSCOLORS:Gxfxcxdxbxegedabagacad LaunchInstanceID:5209FE7B-121C-48CB-936C-55708CB3E126 OLDPWD:/Users/tejaldesai/workspace PAGER:less PATH:/Users/tejaldesai/workspace/google-cloud-sdk/bin:/usr/local/opt/python@3.8/bin:/Users/tejaldesai/mdproxy/bin:/Users/tejaldesai/workspace/google-cloud-sdk/bin:/usr/local/opt/python@3.8/bin:/Users/tejaldesai/mdproxy/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/X11/bin:/Library/Apple/usr/bin PWD:/Users/tejaldesai/workspace/skaffold-debug SECURITYSESSIONID:186ab SHELL:/bin/zsh SHLVL:1 SK_SIGNING_PLUGIN:gnubbyagent SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.m795MDvENI/Listeners TERM:xterm-256color TERM_PROGRAM:Apple_Terminal TERM_PROGRAM_VERSION:440 TERM_SESSION_ID:21CD0E26-1AF9-4720-8484-F6B4AB24F9F8 TMPDIR:/var/folders/gk/s778hvkj0lb0zl95ywvw0snr00cfc0/T/ USER:tejaldesai XPC_FLAGS:0x0 XPC_SERVICE_NAME:0 ZSH:/Users/tejaldesai/.oh-my-zsh _:/Users/tejaldesai/workspace/skaffold-debug/../skaffold/out/skaffold __CFBundleIdentifier:com.apple.Terminal] 
DEBU[0003] Executing template &{envTemplate 0xc000c839e0 0xc000d86740  } with environment map[DISPLAY:/private/tmp/com.apple.launchd.42hcsqMQdi/org.xquartz:0 HOME:/Users/tejaldesai LANG:en_US.UTF-8 LESS:-R LOGNAME:tejaldesai LSCOLORS:Gxfxcxdxbxegedabagacad LaunchInstanceID:5209FE7B-121C-48CB-936C-55708CB3E126 OLDPWD:/Users/tejaldesai/workspace PAGER:less PATH:/Users/tejaldesai/workspace/google-cloud-sdk/bin:/usr/local/opt/python@3.8/bin:/Users/tejaldesai/mdproxy/bin:/Users/tejaldesai/workspace/google-cloud-sdk/bin:/usr/local/opt/python@3.8/bin:/Users/tejaldesai/mdproxy/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/X11/bin:/Library/Apple/usr/bin PWD:/Users/tejaldesai/workspace/skaffold-debug SECURITYSESSIONID:186ab SHELL:/bin/zsh SHLVL:1 SK_SIGNING_PLUGIN:gnubbyagent SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.m795MDvENI/Listeners TERM:xterm-256color TERM_PROGRAM:Apple_Terminal TERM_PROGRAM_VERSION:440 TERM_SESSION_ID:21CD0E26-1AF9-4720-8484-F6B4AB24F9F8 TMPDIR:/var/folders/gk/s778hvkj0lb0zl95ywvw0snr00cfc0/T/ USER:tejaldesai XPC_FLAGS:0x0 XPC_SERVICE_NAME:0 ZSH:/Users/tejaldesai/.oh-my-zsh _:/Users/tejaldesai/workspace/skaffold-debug/../skaffold/out/skaffold __CFBundleIdentifier:com.apple.Terminal] 
DEBU[0003] Running command: [helm --kube-context gke_tejal-gke1_us-central1-c_dump-apis template test-v2 ./charts/test -f ./charts/test/values.yaml --namespace test-v2] 
DEBU[0003] Running command: [tput colors]               
DEBU[0003] Command output: [256
]                       
std out err: Error: found in Chart.yaml, but missing in charts/ directory: mysql

skaffold relies on helm template to render. Running helm --kube-context gke_tejal-gke1_us-central1-c_dump-apis template test-v2 ./charts/test -f ./charts/test/values.yaml --namespace test-v2 Error: found in Chart.yaml, but missing in charts/ directory: mysql itself errors out.

Would you happen to know how helm template handles dependencies.

rccrdpccl commented 3 years ago

I believe the issue here is that the flag skipBuildDependencies: false is not honored. Helm updates its dependencies by helm dependency upgrade command. Both helm commands install/upgrade and templates rely on dependencies to be already there when run.

This case is handled OK in skaffold run command, however in skaffold render is not.

Just for context, in my case I am trying to use skaffold render to generate the manifest to feed it to ArgoCD, however due to this limitation and this I am now running skaffold run --render-only ....

briandealwis commented 3 years ago

This is difficult: render shouldn't actually apply changes to the cluster, but then using helm install or helm upgrade would change the cluster.

rccrdpccl commented 3 years ago

There is no need to run helm install/upgrade: by running helm dependencies update helm will just download all dependencies, but not interact with the cluster. Then helm template could be run as it is currently done.

tejal29 commented 3 years ago

Subtask of #5442

dsiebel commented 2 years ago

Stumbled upon this issue when testing v2.0.0-beta3. Is there anything happening in this area? Any updates?

helm template has a flag --dependency-update nowadays, that could be tied to skipBuildDependencies. We'd be happy to try and contribute a PR.

dsiebel commented 2 years ago

For now we managed to get around it by implementing a before-render hook, but this is less than ideal. Ideally render should update chart dependencies the same way run or deploy does.

before-render hook

manifests:
  helm:
    releases:
      - name: my-chart
        chartPath: kubernetes/helm/charts/myChart
  hooks:
    before:
      - host:
          # `skaffold render` does not update chart dependencies like `run` or `deploy` does:
          # https://github.com/GoogleContainerTools/skaffold/issues/5445
          # So we update the dependencies ourselves.
          command:
            - 'bash'
            - '-c'
            - >-
              for f in kubernetes/helm/charts/*; do
                helm dep build $f;
              done
          os: [darwin, linux]
ericzzzzzzz commented 1 year ago

Hi @dsiebel thank you for providing your workaround! If you can contribute a pr to fix this issue, that would be great!

dsiebel commented 1 year ago

Happy to! CDA should already be signed as well. I'll have to see when I get around to it, though.

gsquared94 commented 1 year ago

Hi @dsiebel are you still intending to send a fix for this?

dsiebel commented 1 year ago

Yes, I still have this on my list. So far this year has been crazy busy.. If someone else wants to pick this up, that would be fine by me as well

dsiebel commented 1 year ago

Finally managed to implement the changes. See the linked Pull Request. The unit tests currently only cover the bare minimum, so I'd appreciate any feedback there.

dsiebel commented 1 year ago

Hey @gsquared94, any feedback?