Remote debugger disconnecting with Helm deploy in skaffold.yaml

gautamr commented 3 years ago

I'm trying to debug java-hello-world from here with our private GCR. Everything is working fine when we use kubectl skaffold deployment as in the above code. But when I change the deployment to Helm the remote debugger got disconnected with the following log

Connected to the target VM, address: '127.0.0.1:5005', transport: 'socket'
Disconnected from the target VM, address: '127.0.0.1:5005', transport: 'socket'

My modified skaffold.yaml is as follows:

# To learn more about the skaffold.yaml schema visit
# https://skaffold.dev/docs/references/yaml/
apiVersion: skaffold/v2beta13
kind: Config
build:
  tagPolicy:
    sha256: {}
  # defines where to find the code at build time and where to push the resulting image
  artifacts:
    - image: gcr.io/<PROJECT_ID>/java-hello-world
      # To learn more about how Jib builds Java containers visit
      # https://github.com/GoogleContainerTools/jib
      jib:
        args:
        - -Dmaven.test.skip=true

# defines the Kubernetes manifests to deploy on each run
deploy:
  helm:
    releases:
      - name: java-hello-world
        namespace: default
        chartPath: "./charts/java-hello-world"
        valuesFiles:
          - "./charts/java-hello-world/values.yaml"
        artifactOverrides:
          image: gcr.io/gintaa-cloud-develop/java-hello-world
  #kubectl:
    #manifests:
    #- ./kubernetes-manifests/**.yaml

Expected Behavior

Will be able to remote debug with Kubernetes

Actual Behavior

Remote debugger got disconnected

Additional Information

IDE type - IntelliJ
IDE version: IntelliJ IDEA 2021.1 (April 7)
Cloud Code version (Settings > Plugins > Cloud Code): 21.3.1-202
Cloud SDK (Settings > Cloud Code > Cloud SDK)
- Are you allowing the plugin to manage the Cloud SDK: Yes
- Version of the Cloud SDK: Google Cloud SDK 332.0.0
Skaffold version (If you are using Kubernetes features; Settings > Cloud Code > Kubernetes): v1.21.0
Operating System: macOS Big Sur, Version 11.2.3

etanshaul commented 3 years ago

@gautamr thanks for reporting. Just in case - what version of Helm are you using? Helm 3.1+ is required: https://skaffold.dev/docs/workflows/debug/#supported-deployers (we should add this req. to the Cloud Code docs too.).

etanshaul commented 3 years ago

@gautamr also, to help us debug faster, would you mind providing the full verbose logs (redacting any sensitive information) that are output from you session? To toggle verbosity, edit the Kubernetes run config and under advanced, select DEBUG:

thanks

gautamr commented 3 years ago

@etanshaul Helm version is Ok I hope

❯ helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.16"}

gautamr commented 3 years ago

@etanshaul I'm trying to give the DEBUG verbose log without any sensitive data, please excuse the huge log

One observation that might help is that the debugger got disconnecting when port-forward is failing like in log when this is printed

failed to port forward pod-java-hello-world-5c744464d4-4n9vl-default-8080, port 8080 is taken, retrying...

The full log

Validating deployment settings...

/Users/gautamr/Library/Application Support/cloud-code/bin/versions/9c41f7c2bb0f357e51c85690102e2c4e78d336c2083f9f1a9a555858131b5189/skaffold debug --filename skaffold.yaml --label ide=Idea --label ideVersion=2021.1.0.0 --label ijPluginVersion=unknown --rpc-port 50051 --port-forward=true --status-check=true --enable-rpc=true --verbosity debug
time="2021-04-10T22:56:38+05:30" level=info msg="starting gRPC server on port 50051"
time="2021-04-10T22:56:38+05:30" level=info msg="starting gRPC HTTP server on port 50052"
time="2021-04-10T22:56:38+05:30" level=info msg="Skaffold &{Version:v1.21.0 ConfigVersion:skaffold/v2beta13 GitVersion: GitCommit:4830337932d53185445812d29f078667b3b74fa5 BuildDate:2021-03-19T01:50:34Z GoVersion:go1.14.14 Compiler:gc Platform:darwin/amd64}"
time="2021-04-10T22:56:38+05:30" level=debug msg="Update check, survey prompt and telemetry disabled in non-interactive mode"
time="2021-04-10T22:56:38+05:30" level=debug msg="parsed 1 configs from configuration file /Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/skaffold.yaml"
time="2021-04-10T22:56:38+05:30" level=debug msg="Defaulting build type to local build"
time="2021-04-10T22:56:38+05:30" level=info msg="Using kubectl context: gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME>"
time="2021-04-10T22:56:38+05:30" level=info msg="Loaded Skaffold defaults from \"/Users/gautamr/.skaffold/config\""
time="2021-04-10T22:56:38+05:30" level=debug msg="Running command: [/Users/gautamr/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube profile list -o json]"
time="2021-04-10T22:56:39+05:30" level=debug msg="Command output: [{\"invalid\":[],\"valid\":[{\"Name\":\"minikube\",\"Status\":\"Unknown\",\"Config\":{\"Name\":\"minikube\",\"KeepContext\":false,\"EmbedCerts\":false,\"MinikubeISO\":\"https://storage.googleapis.com/minikube/iso/minikube-v1.18.0.iso\",\"KicBaseImage\":\"gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e\",\"Memory\":4000,\"CPUs\":2,\"DiskSize\":20000,\"VMDriver\":\"\",\"Driver\":\"hyperkit\",\"HyperkitVpnKitSock\":\"\",\"HyperkitVSockPorts\":[],\"DockerEnv\":null,\"ContainerVolumeMounts\":null,\"InsecureRegistry\":null,\"RegistryMirror\":null,\"HostOnlyCIDR\":\"192.168.99.1/24\",\"HypervVirtualSwitch\":\"\",\"HypervUseExternalSwitch\":false,\"HypervExternalAdapter\":\"\",\"KVMNetwork\":\"default\",\"KVMQemuURI\":\"qemu:///system\",\"KVMGPU\":false,\"KVMHidden\":false,\"KVMNUMACount\":1,\"DockerOpt\":null,\"DisableDriverMounts\":false,\"NFSShare\":[],\"NFSSharesRoot\":\"/nfsshares\",\"UUID\":\"\",\"NoVTXCheck\":false,\"DNSProxy\":false,\"HostDNSResolver\":true,\"HostOnlyNicType\":\"virtio\",\"NatNicType\":\"virtio\",\"SSHIPAddress\":\"\",\"SSHUser\":\"root\",\"SSHKey\":\"\",\"SSHPort\":22,\"KubernetesConfig\":{\"KubernetesVersion\":\"v1.20.2\",\"ClusterName\":\"minikube\",\"Namespace\":\"default\",\"APIServerName\":\"minikubeCA\",\"APIServerNames\":null,\"APIServerIPs\":null,\"DNSDomain\":\"cluster.local\",\"ContainerRuntime\":\"docker\",\"CRISocket\":\"\",\"NetworkPlugin\":\"\",\"FeatureGates\":\"\",\"ServiceCIDR\":\"10.96.0.0/12\",\"ImageRepository\":\"\",\"LoadBalancerStartIP\":\"\",\"LoadBalancerEndIP\":\"\",\"CustomIngressCert\":\"\",\"ExtraOptions\":null,\"ShouldLoadCachedImages\":true,\"EnableDefaultCNI\":false,\"CNI\":\"\",\"NodeIP\":\"\",\"NodePort\":8443,\"NodeName\":\"\"},\"Nodes\":[{\"Name\":\"\",\"IP\":\"\",\"Port\":8443,\"KubernetesVersion\":\"v1.20.2\",\"ControlPlane\":true,\"Worker\":true}],\"Addons\":null,\"VerifyComponents\":{\"apiserver\":true,\"system_pods\":true},\"StartHostTimeout\":360000000000,\"ScheduledStop\":null,\"ExposedPorts\":[],\"Network\":\"\",\"MultiNodeRequested\":false}}]}]"
time="2021-04-10T22:56:39+05:30" level=debug msg="Running command: [/Users/gautamr/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube profile list -o json]"
time="2021-04-10T22:56:39+05:30" level=debug msg="Command output: [{\"invalid\":[],\"valid\":[{\"Name\":\"minikube\",\"Status\":\"Unknown\",\"Config\":{\"Name\":\"minikube\",\"KeepContext\":false,\"EmbedCerts\":false,\"MinikubeISO\":\"https://storage.googleapis.com/minikube/iso/minikube-v1.18.0.iso\",\"KicBaseImage\":\"gcr.io/k8s-minikube/kicbase:v0.0.18@sha256:ddd0c02d289e3a6fb4bba9a94435840666f4eb81484ff3e707b69c1c484aa45e\",\"Memory\":4000,\"CPUs\":2,\"DiskSize\":20000,\"VMDriver\":\"\",\"Driver\":\"hyperkit\",\"HyperkitVpnKitSock\":\"\",\"HyperkitVSockPorts\":[],\"DockerEnv\":null,\"ContainerVolumeMounts\":null,\"InsecureRegistry\":null,\"RegistryMirror\":null,\"HostOnlyCIDR\":\"192.168.99.1/24\",\"HypervVirtualSwitch\":\"\",\"HypervUseExternalSwitch\":false,\"HypervExternalAdapter\":\"\",\"KVMNetwork\":\"default\",\"KVMQemuURI\":\"qemu:///system\",\"KVMGPU\":false,\"KVMHidden\":false,\"KVMNUMACount\":1,\"DockerOpt\":null,\"DisableDriverMounts\":false,\"NFSShare\":[],\"NFSSharesRoot\":\"/nfsshares\",\"UUID\":\"\",\"NoVTXCheck\":false,\"DNSProxy\":false,\"HostDNSResolver\":true,\"HostOnlyNicType\":\"virtio\",\"NatNicType\":\"virtio\",\"SSHIPAddress\":\"\",\"SSHUser\":\"root\",\"SSHKey\":\"\",\"SSHPort\":22,\"KubernetesConfig\":{\"KubernetesVersion\":\"v1.20.2\",\"ClusterName\":\"minikube\",\"Namespace\":\"default\",\"APIServerName\":\"minikubeCA\",\"APIServerNames\":null,\"APIServerIPs\":null,\"DNSDomain\":\"cluster.local\",\"ContainerRuntime\":\"docker\",\"CRISocket\":\"\",\"NetworkPlugin\":\"\",\"FeatureGates\":\"\",\"ServiceCIDR\":\"10.96.0.0/12\",\"ImageRepository\":\"\",\"LoadBalancerStartIP\":\"\",\"LoadBalancerEndIP\":\"\",\"CustomIngressCert\":\"\",\"ExtraOptions\":null,\"ShouldLoadCachedImages\":true,\"EnableDefaultCNI\":false,\"CNI\":\"\",\"NodeIP\":\"\",\"NodePort\":8443,\"NodeName\":\"\"},\"Nodes\":[{\"Name\":\"\",\"IP\":\"\",\"Port\":8443,\"KubernetesVersion\":\"v1.20.2\",\"ControlPlane\":true,\"Worker\":true}],\"Addons\":null,\"VerifyComponents\":{\"apiserver\":true,\"system_pods\":true},\"StartHostTimeout\":360000000000,\"ScheduledStop\":null,\"ExposedPorts\":[],\"Network\":\"\",\"MultiNodeRequested\":false}}]}]"
time="2021-04-10T22:56:39+05:30" level=debug msg="setting Docker user agent to skaffold-v1.21.0"
time="2021-04-10T22:56:39+05:30" level=debug msg="Using builder: local"
time="2021-04-10T22:56:39+05:30" level=debug msg="push value not present, defaulting to true because cluster.PushImages is true"
time="2021-04-10T22:56:39+05:30" level=debug msg="Running command: [helm version --client]"
time="2021-04-10T22:56:40+05:30" level=debug msg="Command output: [version.BuildInfo{Version:\"v3.5.3\", GitCommit:\"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622\", GitTreeState:\"dirty\", GoVersion:\"go1.16\"}
]"
Listing files to watch...
time="2021-04-10T22:56:40+05:30" level=debug msg="Using wrapper for mvnw: mvn"
 - gcr.io/<GCP_PROJECT_ID>/java-hello-world
time="2021-04-10T22:56:40+05:30" level=debug msg="Running command: [/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/mvnw jib:_skaffold-fail-if-jib-out-of-date -Djib.requiredVersion=1.4.0 -Dmaven.test.skip=true --non-recursive jib:_skaffold-files-v2 --quiet --batch-mode]"
time="2021-04-10T22:56:54+05:30" level=debug msg="Command output: [
BEGIN JIB JSON
{\"build\":[\"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/pom.xml\"],\"inputs\":[\"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/java\",\"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/resources\",\"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/resources\",\"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/jib\"],\"ignore\":[]}
]"
time="2021-04-10T22:56:54+05:30" level=debug msg="could not stat dependency: stat /Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/jib: no such file or directory"
time="2021-04-10T22:56:54+05:30" level=debug msg="Found dependencies for jib maven artifact: [pom.xml src/main/java/cloudcode/helloworld/HelloWorldApplication.java src/main/java/cloudcode/helloworld/web/HelloWorldController.java src/main/resources/application.properties src/main/resources/application.properties src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/lightbulb_icon.svg src/main/resources/static/img/lightbulb_icon.svg src/main/resources/templates/index.html src/main/resources/templates/index.html]"
time="2021-04-10T22:56:54+05:30" level=info msg="List generated in 13.729 seconds"
Generating tags...
time="2021-04-10T22:56:54+05:30" level=info msg="Tags generated in 194.756µs"
 - gcr.io/<GCP_PROJECT_ID>/java-hello-world -> gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest
Checking cache...
time="2021-04-10T22:56:54+05:30" level=debug msg="Using wrapper for mvnw: mvn"
time="2021-04-10T22:56:54+05:30" level=debug msg="could not stat dependency: stat /Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/jib: no such file or directory"
time="2021-04-10T22:56:54+05:30" level=debug msg="Found dependencies for jib maven artifact: [pom.xml src/main/java/cloudcode/helloworld/HelloWorldApplication.java src/main/java/cloudcode/helloworld/web/HelloWorldController.java src/main/resources/application.properties src/main/resources/application.properties src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/lightbulb_icon.svg src/main/resources/static/img/lightbulb_icon.svg src/main/resources/templates/index.html src/main/resources/templates/index.html]"
time="2021-04-10T22:56:54+05:30" level=debug msg="push value not present, defaulting to true because cluster.PushImages is true"
 - gcr.io/<GCP_PROJECT_ID>/java-hello-world: Found Remotely
time="2021-04-10T22:56:58+05:30" level=debug msg="push value not present, defaulting to true because cluster.PushImages is true"
time="2021-04-10T22:56:58+05:30" level=debug msg="push value not present, defaulting to true because cluster.PushImages is true"
time="2021-04-10T22:56:58+05:30" level=info msg="Cache check completed in 4.937 seconds"
Starting test...
time="2021-04-10T22:56:58+05:30" level=info msg="Test completed in 14.443µs"
time="2021-04-10T22:56:58+05:30" level=debug msg="push value not present, defaulting to true because cluster.PushImages is true"
Tags used in deployment:
time="2021-04-10T22:56:58+05:30" level=debug msg="getting client config for kubeContext: ``"
 - gcr.io/<GCP_PROJECT_ID>/java-hello-world -> gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7
time="2021-04-10T22:56:59+05:30" level=info msg="Deploying with helm v3.5.3 ..."
Starting deploy...
time="2021-04-10T22:56:59+05:30" level=debug msg="Executing template &{envTemplate 0xc0004baa00 0xc000af8740  } with environment map[ANDROID_AVD_HOME:/Users/gautamr/.android/avd ANDROID_HOME:/Users/gautamr/Library/Android/sdk ANDROID_SDK_ROOT:/Users/gautamr/Library/Android/sdk ANDROID_STUDIO_HOME:/Applications/Android Studio.app/Contents ASDF_DIR:/usr/local/opt/asdf CLOUDSDK_CONFIG:/Users/gautamr/.config/gcloud COMMAND_MODE:unix2003 DART_HOME:/usr/local/Caskroom/flutter/latest/flutter/bin/cache/dart-sdk FLUTTER_HOME:/usr/local/Caskroom/flutter/latest/flutter FPATH:/usr/local/Cellar/zplug/2.4.2/autoload:/usr/local/Cellar/zplug/2.4.2/misc/completions:/usr/local/Cellar/zplug/2.4.2/base/sources:/usr/local/Cellar/zplug/2.4.2/base/utils:/usr/local/Cellar/zplug/2.4.2/base/job:/usr/local/Cellar/zplug/2.4.2/base/log:/usr/local/Cellar/zplug/2.4.2/base/io:/usr/local/Cellar/zplug/2.4.2/base/core:/usr/local/Cellar/zplug/2.4.2/base/base:/usr/local/Cellar/zplug/2.4.2/autoload/commands:/usr/local/Cellar/zplug/2.4.2/autoload/options:/usr/local/Cellar/zplug/2.4.2/autoload/tags:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.8/functions HOME:/Users/gautamr JAVA_HOME:/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home JDK_HOME:/Users/gautamr/.jenv/versions/11.0.6 JENV_FORCEJAVAHOME:true JENV_FORCEJDKHOME:true JENV_LOADED:1 JENV_SHELL:zsh LC_CTYPE:en_IN.UTF-8 LOGNAME:gautamr LaunchInstanceID:3877928B-9D3C-4EDD-9AFA-58B23C77A18B OLDPWD:/ PATH:/Users/gautamr/Library/Application Support/cloud-code/bin/versions/9c41f7c2bb0f357e51c85690102e2c4e78d336c2083f9f1a9a555858131b5189:/Users/gautamr/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin:/usr/local/Cellar/zplug/2.4.2/bin:/Users/gautamr/.cabal/bin:/Users/gautamr/.ghcup/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/gautamr/.jenv/shims:/Users/gautamr/.jenv/bin:/Users/gautamr/.asdf/shims:/usr/local/opt/asdf/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/gautamr/Library/Android/sdk/cmdline-tools/latest/bin:/Users/gautamr/Library/Android/sdk/emulator:/Users/gautamr/Library/Android/sdk/platform-tools/platform-tools:/usr/local/Caskroom/flutter/latest/flutter/bin/cache/dart-sdk/bin PERIOD:30 PWD:/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world SECURITYSESSIONID:186a9 SHELL:/bin/zsh SKAFFOLD_FORCE_COLORS:true SKAFFOLD_INTERACTIVE:false SKAFFOLD_UPDATE_CHECK:false SKAFFOLD_XXENABLEBUILDPACKINIT:true SKAFFOLD_XXENABLEBUILDPACKSINIT:true SKAFFOLD_XXENABLENEWINITFORMAT:true SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.tWeFKDDQNA/Listeners STARSHIP_SHELL:zsh TMPDIR:/var/folders/1k/f8_94y4s4912kw8f8fxwpyfc0000gn/T/ USER:gautamr VERSIONER_PYTHON_VERSION:2.7 VIRTUALENVWRAPPER_HOOK_DIR:/Users/gautamr/.virtualenvs VIRTUALENVWRAPPER_PROJECT_FILENAME:.project VIRTUALENVWRAPPER_PYTHON:/usr/local/bin/python3 VIRTUALENVWRAPPER_SCRIPT:/usr/local/bin/virtualenvwrapper.sh VIRTUALENVWRAPPER_VIRTUALENV:/usr/local/bin/virtualenv VIRTUALENVWRAPPER_WORKON_CD:1 WORKON_HOME:/Users/gautamr/.virtualenvs XPC_FLAGS:0x0 XPC_SERVICE_NAME:application.com.jetbrains.intellij.ce.38622466.59626763 ZPLUG_BIN:/usr/local/opt/zplug/bin ZPLUG_CACHE_DIR:/usr/local/opt/zplug/cache ZPLUG_ERROR_LOG:/usr/local/opt/zplug/.error_log ZPLUG_FILTER:fzf-tmux:fzf:peco:percol:fzy:zaw ZPLUG_HOME:/usr/local/opt/zplug ZPLUG_LOADFILE:/usr/local/opt/zplug/packages.zsh ZPLUG_PROTOCOL:HTTPS ZPLUG_REPOS:/usr/local/opt/zplug/repos ZPLUG_ROOT:/usr/local/Cellar/zplug/2.4.2 ZPLUG_THREADS:16 ZPLUG_USE_CACHE:true _ZPLUG_AWKPATH:/usr/local/Cellar/zplug/2.4.2/misc/contrib _ZPLUG_CONFIG_SUBSHELL:: _ZPLUG_OHMYZSH:robbyrussell/oh-my-zsh _ZPLUG_PREZTO:sorin-ionescu/prezto _ZPLUG_URL:https://github.com/zplug/zplug _ZPLUG_VERSION:2.4.2 __CFBundleIdentifier:com.jetbrains.intellij.ce __CF_USER_TEXT_ENCODING:0x1F5:0:0]"
time="2021-04-10T22:56:59+05:30" level=debug msg="Executing template &{envTemplate 0xc0004bac00 0xc000af88c0  } with environment map[ANDROID_AVD_HOME:/Users/gautamr/.android/avd ANDROID_HOME:/Users/gautamr/Library/Android/sdk ANDROID_SDK_ROOT:/Users/gautamr/Library/Android/sdk ANDROID_STUDIO_HOME:/Applications/Android Studio.app/Contents ASDF_DIR:/usr/local/opt/asdf CLOUDSDK_CONFIG:/Users/gautamr/.config/gcloud COMMAND_MODE:unix2003 DART_HOME:/usr/local/Caskroom/flutter/latest/flutter/bin/cache/dart-sdk FLUTTER_HOME:/usr/local/Caskroom/flutter/latest/flutter FPATH:/usr/local/Cellar/zplug/2.4.2/autoload:/usr/local/Cellar/zplug/2.4.2/misc/completions:/usr/local/Cellar/zplug/2.4.2/base/sources:/usr/local/Cellar/zplug/2.4.2/base/utils:/usr/local/Cellar/zplug/2.4.2/base/job:/usr/local/Cellar/zplug/2.4.2/base/log:/usr/local/Cellar/zplug/2.4.2/base/io:/usr/local/Cellar/zplug/2.4.2/base/core:/usr/local/Cellar/zplug/2.4.2/base/base:/usr/local/Cellar/zplug/2.4.2/autoload/commands:/usr/local/Cellar/zplug/2.4.2/autoload/options:/usr/local/Cellar/zplug/2.4.2/autoload/tags:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.8/functions HOME:/Users/gautamr JAVA_HOME:/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home JDK_HOME:/Users/gautamr/.jenv/versions/11.0.6 JENV_FORCEJAVAHOME:true JENV_FORCEJDKHOME:true JENV_LOADED:1 JENV_SHELL:zsh LC_CTYPE:en_IN.UTF-8 LOGNAME:gautamr LaunchInstanceID:3877928B-9D3C-4EDD-9AFA-58B23C77A18B OLDPWD:/ PATH:/Users/gautamr/Library/Application Support/cloud-code/bin/versions/9c41f7c2bb0f357e51c85690102e2c4e78d336c2083f9f1a9a555858131b5189:/Users/gautamr/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin:/usr/local/Cellar/zplug/2.4.2/bin:/Users/gautamr/.cabal/bin:/Users/gautamr/.ghcup/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/gautamr/.jenv/shims:/Users/gautamr/.jenv/bin:/Users/gautamr/.asdf/shims:/usr/local/opt/asdf/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/gautamr/Library/Android/sdk/cmdline-tools/latest/bin:/Users/gautamr/Library/Android/sdk/emulator:/Users/gautamr/Library/Android/sdk/platform-tools/platform-tools:/usr/local/Caskroom/flutter/latest/flutter/bin/cache/dart-sdk/bin PERIOD:30 PWD:/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world SECURITYSESSIONID:186a9 SHELL:/bin/zsh SKAFFOLD_FORCE_COLORS:true SKAFFOLD_INTERACTIVE:false SKAFFOLD_UPDATE_CHECK:false SKAFFOLD_XXENABLEBUILDPACKINIT:true SKAFFOLD_XXENABLEBUILDPACKSINIT:true SKAFFOLD_XXENABLENEWINITFORMAT:true SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.tWeFKDDQNA/Listeners STARSHIP_SHELL:zsh TMPDIR:/var/folders/1k/f8_94y4s4912kw8f8fxwpyfc0000gn/T/ USER:gautamr VERSIONER_PYTHON_VERSION:2.7 VIRTUALENVWRAPPER_HOOK_DIR:/Users/gautamr/.virtualenvs VIRTUALENVWRAPPER_PROJECT_FILENAME:.project VIRTUALENVWRAPPER_PYTHON:/usr/local/bin/python3 VIRTUALENVWRAPPER_SCRIPT:/usr/local/bin/virtualenvwrapper.sh VIRTUALENVWRAPPER_VIRTUALENV:/usr/local/bin/virtualenv VIRTUALENVWRAPPER_WORKON_CD:1 WORKON_HOME:/Users/gautamr/.virtualenvs XPC_FLAGS:0x0 XPC_SERVICE_NAME:application.com.jetbrains.intellij.ce.38622466.59626763 ZPLUG_BIN:/usr/local/opt/zplug/bin ZPLUG_CACHE_DIR:/usr/local/opt/zplug/cache ZPLUG_ERROR_LOG:/usr/local/opt/zplug/.error_log ZPLUG_FILTER:fzf-tmux:fzf:peco:percol:fzy:zaw ZPLUG_HOME:/usr/local/opt/zplug ZPLUG_LOADFILE:/usr/local/opt/zplug/packages.zsh ZPLUG_PROTOCOL:HTTPS ZPLUG_REPOS:/usr/local/opt/zplug/repos ZPLUG_ROOT:/usr/local/Cellar/zplug/2.4.2 ZPLUG_THREADS:16 ZPLUG_USE_CACHE:true _ZPLUG_AWKPATH:/usr/local/Cellar/zplug/2.4.2/misc/contrib _ZPLUG_CONFIG_SUBSHELL:: _ZPLUG_OHMYZSH:robbyrussell/oh-my-zsh _ZPLUG_PREZTO:sorin-ionescu/prezto _ZPLUG_URL:https://github.com/zplug/zplug _ZPLUG_VERSION:2.4.2 __CFBundleIdentifier:com.jetbrains.intellij.ce __CF_USER_TEXT_ENCODING:0x1F5:0:0]"
time="2021-04-10T22:56:59+05:30" level=debug msg="Running command: [helm --kube-context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> get all --namespace default java-hello-world]"
Helm release java-hello-world not installed. Installing...
time="2021-04-10T22:56:59+05:30" level=info msg="Building helm dependencies..."
time="2021-04-10T22:56:59+05:30" level=debug msg="Running command: [helm --kube-context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> dep build ./charts/java-hello-world]"
time="2021-04-10T22:57:00+05:30" level=debug msg="EnvVarMap: map[DIGEST:gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 DIGEST_ALGO:gcr.io/<GCP_PROJECT_ID>/java-hello-world DIGEST_HEX:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 IMAGE_DIGEST:sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 IMAGE_NAME:gcr.io/<GCP_PROJECT_ID>/java-hello-world IMAGE_REPO:gcr.io/<GCP_PROJECT_ID>/java-hello-world IMAGE_TAG:latest]
"
time="2021-04-10T22:57:00+05:30" level=debug msg="Executing template &{envTemplate 0xc00061e500 0xc0009bf080  } with environment map[ANDROID_AVD_HOME:/Users/gautamr/.android/avd ANDROID_HOME:/Users/gautamr/Library/Android/sdk ANDROID_SDK_ROOT:/Users/gautamr/Library/Android/sdk ANDROID_STUDIO_HOME:/Applications/Android Studio.app/Contents ASDF_DIR:/usr/local/opt/asdf CLOUDSDK_CONFIG:/Users/gautamr/.config/gcloud COMMAND_MODE:unix2003 DART_HOME:/usr/local/Caskroom/flutter/latest/flutter/bin/cache/dart-sdk DIGEST:gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 DIGEST_ALGO:gcr.io/<GCP_PROJECT_ID>/java-hello-world DIGEST_HEX:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 FLUTTER_HOME:/usr/local/Caskroom/flutter/latest/flutter FPATH:/usr/local/Cellar/zplug/2.4.2/autoload:/usr/local/Cellar/zplug/2.4.2/misc/completions:/usr/local/Cellar/zplug/2.4.2/base/sources:/usr/local/Cellar/zplug/2.4.2/base/utils:/usr/local/Cellar/zplug/2.4.2/base/job:/usr/local/Cellar/zplug/2.4.2/base/log:/usr/local/Cellar/zplug/2.4.2/base/io:/usr/local/Cellar/zplug/2.4.2/base/core:/usr/local/Cellar/zplug/2.4.2/base/base:/usr/local/Cellar/zplug/2.4.2/autoload/commands:/usr/local/Cellar/zplug/2.4.2/autoload/options:/usr/local/Cellar/zplug/2.4.2/autoload/tags:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.8/functions HOME:/Users/gautamr IMAGE_DIGEST:sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 IMAGE_NAME:gcr.io/<GCP_PROJECT_ID>/java-hello-world IMAGE_REPO:gcr.io/<GCP_PROJECT_ID>/java-hello-world IMAGE_TAG:latest JAVA_HOME:/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home JDK_HOME:/Users/gautamr/.jenv/versions/11.0.6 JENV_FORCEJAVAHOME:true JENV_FORCEJDKHOME:true JENV_LOADED:1 JENV_SHELL:zsh LC_CTYPE:en_IN.UTF-8 LOGNAME:gautamr LaunchInstanceID:3877928B-9D3C-4EDD-9AFA-58B23C77A18B OLDPWD:/ PATH:/Users/gautamr/Library/Application Support/cloud-code/bin/versions/9c41f7c2bb0f357e51c85690102e2c4e78d336c2083f9f1a9a555858131b5189:/Users/gautamr/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin:/usr/local/Cellar/zplug/2.4.2/bin:/Users/gautamr/.cabal/bin:/Users/gautamr/.ghcup/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/gautamr/.jenv/shims:/Users/gautamr/.jenv/bin:/Users/gautamr/.asdf/shims:/usr/local/opt/asdf/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/gautamr/Library/Android/sdk/cmdline-tools/latest/bin:/Users/gautamr/Library/Android/sdk/emulator:/Users/gautamr/Library/Android/sdk/platform-tools/platform-tools:/usr/local/Caskroom/flutter/latest/flutter/bin/cache/dart-sdk/bin PERIOD:30 PWD:/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world SECURITYSESSIONID:186a9 SHELL:/bin/zsh SKAFFOLD_FORCE_COLORS:true SKAFFOLD_INTERACTIVE:false SKAFFOLD_UPDATE_CHECK:false SKAFFOLD_XXENABLEBUILDPACKINIT:true SKAFFOLD_XXENABLEBUILDPACKSINIT:true SKAFFOLD_XXENABLENEWINITFORMAT:true SSH_AUTH_SOCK:/private/tmp/com.apple.launchd.tWeFKDDQNA/Listeners STARSHIP_SHELL:zsh TMPDIR:/var/folders/1k/f8_94y4s4912kw8f8fxwpyfc0000gn/T/ USER:gautamr VERSIONER_PYTHON_VERSION:2.7 VIRTUALENVWRAPPER_HOOK_DIR:/Users/gautamr/.virtualenvs VIRTUALENVWRAPPER_PROJECT_FILENAME:.project VIRTUALENVWRAPPER_PYTHON:/usr/local/bin/python3 VIRTUALENVWRAPPER_SCRIPT:/usr/local/bin/virtualenvwrapper.sh VIRTUALENVWRAPPER_VIRTUALENV:/usr/local/bin/virtualenv VIRTUALENVWRAPPER_WORKON_CD:1 WORKON_HOME:/Users/gautamr/.virtualenvs XPC_FLAGS:0x0 XPC_SERVICE_NAME:application.com.jetbrains.intellij.ce.38622466.59626763 ZPLUG_BIN:/usr/local/opt/zplug/bin ZPLUG_CACHE_DIR:/usr/local/opt/zplug/cache ZPLUG_ERROR_LOG:/usr/local/opt/zplug/.error_log ZPLUG_FILTER:fzf-tmux:fzf:peco:percol:fzy:zaw ZPLUG_HOME:/usr/local/opt/zplug ZPLUG_LOADFILE:/usr/local/opt/zplug/packages.zsh ZPLUG_PROTOCOL:HTTPS ZPLUG_REPOS:/usr/local/opt/zplug/repos ZPLUG_ROOT:/usr/local/Cellar/zplug/2.4.2 ZPLUG_THREADS:16 ZPLUG_USE_CACHE:true _ZPLUG_AWKPATH:/usr/local/Cellar/zplug/2.4.2/misc/contrib _ZPLUG_CONFIG_SUBSHELL:: _ZPLUG_OHMYZSH:robbyrussell/oh-my-zsh _ZPLUG_PREZTO:sorin-ionescu/prezto _ZPLUG_URL:https://github.com/zplug/zplug _ZPLUG_VERSION:2.4.2 __CFBundleIdentifier:com.jetbrains.intellij.ce __CF_USER_TEXT_ENCODING:0x1F5:0:0]"
time="2021-04-10T22:57:00+05:30" level=debug msg="Running command: [helm --kube-context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> install java-hello-world --post-renderer /Users/gautamr/Library/Application Support/cloud-code/bin/versions/9c41f7c2bb0f357e51c85690102e2c4e78d336c2083f9f1a9a555858131b5189/skaffold ./charts/java-hello-world --namespace default --set-string image=gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7 -f ./charts/java-hello-world/values.yaml]"
NAME: java-hello-world
LAST DEPLOYED: Sat Apr 10 22:57:03 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
http://java-hello-world. to access your application
time="2021-04-10T22:57:10+05:30" level=debug msg="Running command: [helm --kube-context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> get all --namespace default java-hello-world]"
time="2021-04-10T22:57:10+05:30" level=debug msg="NAME: java-hello-world
LAST DEPLOYED: Sat Apr 10 22:57:03 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
image: gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7
imageConfig:
  pullPolicy: Always
replicaCount: 1
service:
  externalPort: 80
  internalPort: 8080
  name: java-hello-world
  protocol: TCP
  type: ClusterIP

COMPUTED VALUES:
image: gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7
imageConfig:
  pullPolicy: Always
replicaCount: 1
service:
  externalPort: 80
  internalPort: 8080
  name: java-hello-world
  protocol: TCP
  type: ClusterIP

HOOKS:
MANIFEST:
---
# Source: java-hello-world/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: java-hello-world
spec:
  #type: LoadBalancer
  selector:
    app: java-hello-world
  ports:
    - name: java-hello-world
      port: 80
      protocol: TCP
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  name: java-hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: java-hello-world
  strategy: {}
  template:
    metadata:
      annotations:
        debug.cloud.google.com/config: '{\"java-hello-world\":{\"artifact\":\"gcr.io/<GCP_PROJECT_ID>/java-hello-world\",\"runtime\":\"jvm\",\"workingDir\":\"/\",\"ports\":{\"jdwp\":5005}}}'
      creationTimestamp: null
      labels:
        app: java-hello-world
    spec:
      containers:
      - env:
        - name: PORT
          value: \"8080\"
        - name: JAVA_TOOL_OPTIONS
          value: -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
        image: gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:98267b3e0ee50f9508bc924c937bd7d20e2fc950bbfa5e9a4a7093bb0d4628c7
        name: java-hello-world
        ports:
        - containerPort: 8080
        - containerPort: 5005
          name: jdwp
        readinessProbe:
          initialDelaySeconds: 5
          tcpSocket:
            port: 8080
        resources: {}
status: {}
NOTES:
http://java-hello-world. to access your application
"
time="2021-04-10T22:57:10+05:30" level=debug msg="found deployed object: &Service{ObjectMeta:{java-hello-world      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:java-hello-world,Protocol:TCP,Port:80,TargetPort:{0 8080 },NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app: java-hello-world,},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,IPFamily:nil,TopologyKeys:[],},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},},}"
time="2021-04-10T22:57:10+05:30" level=info msg="error parsing object from string: collecting namespaces: reading Kubernetes YAML: yaml: line 40: could not find expected ':'"
time="2021-04-10T22:57:10+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:10+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:10+05:30" level=debug msg="Patching java-hello-world in namespace default"
time="2021-04-10T22:57:11+05:30" level=info msg="Deploy completed in 11.729 seconds"
Waiting for deployments to stabilize...
time="2021-04-10T22:57:11+05:30" level=debug msg="getting client config for kubeContext: ``"
Deployments stabilized in 138.503739ms
time="2021-04-10T22:57:11+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:11+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:11+05:30" level=debug msg="found open port: 4503"
time="2021-04-10T22:57:11+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:11+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:11+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:11+05:30" level=debug msg="Forwarding service java-hello-world/80 to pod java-hello-world-5c744464d4-4n9vl/8080"
time="2021-04-10T22:57:11+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 4503:8080]"
Press Ctrl+C to exit
time="2021-04-10T22:57:11+05:30" level=debug msg="Using wrapper for mvnw: mvn"
time="2021-04-10T22:57:11+05:30" level=debug msg="Change detected <nil>"
time="2021-04-10T22:57:11+05:30" level=debug msg="could not stat dependency: stat /Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/jib: no such file or directory"
time="2021-04-10T22:57:11+05:30" level=debug msg="Found dependencies for jib maven artifact: [pom.xml src/main/java/cloudcode/helloworld/HelloWorldApplication.java src/main/java/cloudcode/helloworld/web/HelloWorldController.java src/main/resources/application.properties src/main/resources/application.properties src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/lightbulb_icon.svg src/main/resources/static/img/lightbulb_icon.svg src/main/resources/templates/index.html src/main/resources/templates/index.html]"
Not watching for changes...
time="2021-04-10T22:57:12+05:30" level=debug msg="port forwarding service-java-hello-world-default-80 got terminated: exit status 1, output: error: unable to forward port because pod is not running. Current status=Pending
"
port forwarding service-java-hello-world-default-80 got terminated: output: error: unable to forward port because pod is not running. Current status=Pending

time="2021-04-10T22:57:12+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:12+05:30" level=debug msg="Using wrapper for mvnw: mvn"
time="2021-04-10T22:57:12+05:30" level=debug msg="could not stat dependency: stat /Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/jib: no such file or directory"
time="2021-04-10T22:57:12+05:30" level=debug msg="Found dependencies for jib maven artifact: [pom.xml src/main/java/cloudcode/helloworld/HelloWorldApplication.java src/main/java/cloudcode/helloworld/web/HelloWorldController.java src/main/resources/application.properties src/main/resources/application.properties src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/lightbulb_icon.svg src/main/resources/static/img/lightbulb_icon.svg src/main/resources/templates/index.html src/main/resources/templates/index.html]"
time="2021-04-10T22:57:12+05:30" level=debug msg="Forwarding service java-hello-world/80 to pod java-hello-world-5c744464d4-4n9vl/8080"
time="2021-04-10T22:57:12+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 4503:8080]"
time="2021-04-10T22:57:13+05:30" level=debug msg="port forwarding service-java-hello-world-default-80 got terminated: exit status 1, output: error: unable to forward port because pod is not running. Current status=Pending
"
time="2021-04-10T22:57:13+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:13+05:30" level=info msg="Streaming logs from pod: java-hello-world-5c744464d4-4n9vl container: java-hello-world"
time="2021-04-10T22:57:13+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> logs --since=15s -f java-hello-world-5c744464d4-4n9vl -c java-hello-world --namespace default]"
time="2021-04-10T22:57:13+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:13+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 8080:8080]"
time="2021-04-10T22:57:13+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:14+05:30" level=debug msg="Forwarding service java-hello-world/80 to pod java-hello-world-5c744464d4-4n9vl/8080"
time="2021-04-10T22:57:14+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 4503:8080]"
[java-hello-world] Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
Port forwarding pod/java-hello-world-5c744464d4-4n9vl in namespace default, remote port 8080 -> 127.0.0.1:8080
time="2021-04-10T22:57:14+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 5005:5005]"
Port forwarding pod/java-hello-world-5c744464d4-4n9vl in namespace default, remote port 5005 -> 127.0.0.1:5005
[java-hello-world] 
[java-hello-world]   .   ____          _            __ _ _
[java-hello-world]  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
[java-hello-world] ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
[java-hello-world]  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
[java-hello-world]   '  |____| .__|_| |_|_| |_\__, | / / / /
[java-hello-world]  =========|_|==============|___/=/_/_/_/
[java-hello-world]  :: Spring Boot ::        (v2.1.4.RELEASE)
[java-hello-world] 
[java-hello-world] 2021-04-10 17:27:18.552  INFO 1 --- [           main] c.helloworld.HelloWorldApplication       : Starting HelloWorldApplication on java-hello-world-5c744464d4-4n9vl with PID 1 (/app/classes started by root in /)
[java-hello-world] 2021-04-10 17:27:18.566  INFO 1 --- [           main] c.helloworld.HelloWorldApplication       : No active profile set, falling back to default profiles: default
[java-hello-world] 2021-04-10 17:27:21.325  INFO 1 --- [           main] org.eclipse.jetty.util.log               : Logging initialized @7631ms to org.eclipse.jetty.util.log.Slf4jLog
[java-hello-world] 2021-04-10 17:27:21.610  INFO 1 --- [           main] o.s.b.w.e.j.JettyServletWebServerFactory : Server initialized with port: 8080
[java-hello-world] 2021-04-10 17:27:21.621  INFO 1 --- [           main] org.eclipse.jetty.server.Server          : jetty-9.4.15.v20190215; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826fa0b; jvm 1.8.0_275-8u275-b01-1~deb9u1-b01
[java-hello-world] 2021-04-10 17:27:21.702  INFO 1 --- [           main] o.e.j.s.h.ContextHandler.application     : Initializing Spring embedded WebApplicationContext
[java-hello-world] 2021-04-10 17:27:21.702  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2985 ms
[java-hello-world] 2021-04-10 17:27:22.006  INFO 1 --- [           main] org.eclipse.jetty.server.session         : DefaultSessionIdManager workerName=node0
[java-hello-world] 2021-04-10 17:27:22.008  INFO 1 --- [           main] org.eclipse.jetty.server.session         : No SessionScavenger set, using defaults
[java-hello-world] 2021-04-10 17:27:22.015  INFO 1 --- [           main] org.eclipse.jetty.server.session         : node0 Scavenging every 660000ms
[java-hello-world] 2021-04-10 17:27:22.047  INFO 1 --- [           main] o.e.jetty.server.handler.ContextHandler  : Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@433ffad1{application,/,[file:///tmp/jetty-docbase.7387114566817215112.8080/],AVAILABLE}
[java-hello-world] 2021-04-10 17:27:22.053  INFO 1 --- [           main] org.eclipse.jetty.server.Server          : Started @8360ms
[java-hello-world] 2021-04-10 17:27:23.605  INFO 1 --- [           main] o.s.c.g.a.c.GcpContextAutoConfiguration  : The default project ID is <GCP_PROJECT_ID>
[java-hello-world] 2021-04-10 17:27:23.874  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
[java-hello-world] 2021-04-10 17:27:24.278  INFO 1 --- [           main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page template: index
[java-hello-world] 2021-04-10 17:27:24.693  INFO 1 --- [           main] o.s.c.g.core.DefaultCredentialsProvider  : Default credentials provider for Google Compute Engine.
[java-hello-world] 2021-04-10 17:27:24.694  INFO 1 --- [           main] o.s.c.g.core.DefaultCredentialsProvider  : Scopes in use by default credentials: [https://www.googleapis.com/auth/pubsub, https://www.googleapis.com/auth/spanner.admin, https://www.googleapis.com/auth/spanner.data, https://www.googleapis.com/auth/datastore, https://www.googleapis.com/auth/sqlservice.admin, https://www.googleapis.com/auth/devstorage.read_only, https://www.googleapis.com/auth/devstorage.read_write, https://www.googleapis.com/auth/cloudruntimeconfig, https://www.googleapis.com/auth/trace.append, https://www.googleapis.com/auth/cloud-platform, https://www.googleapis.com/auth/cloud-vision, https://www.googleapis.com/auth/bigquery, https://www.googleapis.com/auth/monitoring.write]
[java-hello-world] 2021-04-10 17:27:24.786  INFO 1 --- [           main] o.e.j.s.h.ContextHandler.application     : Initializing Spring DispatcherServlet 'dispatcherServlet'
[java-hello-world] 2021-04-10 17:27:24.787  INFO 1 --- [           main] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
[java-hello-world] 2021-04-10 17:27:24.804  INFO 1 --- [           main] o.s.web.servlet.DispatcherServlet        : Completed initialization in 16 ms
[java-hello-world] 2021-04-10 17:27:24.874  INFO 1 --- [           main] o.e.jetty.server.AbstractConnector       : Started ServerConnector@799e0c83{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
[java-hello-world] 2021-04-10 17:27:24.887  INFO 1 --- [           main] o.s.b.web.embedded.jetty.JettyWebServer  : Jetty started on port(s) 8080 (http/1.1) with context path '/'
[java-hello-world] 2021-04-10 17:27:24.899  INFO 1 --- [           main] c.helloworld.HelloWorldApplication       : Started HelloWorldApplication in 7.77 seconds (JVM running for 11.208)
time="2021-04-10T22:57:34+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:34+05:30" level=debug msg="getting client config for kubeContext: ``"
time="2021-04-10T22:57:34+05:30" level=debug msg="Terminating port-forward pod-java-hello-world-5c744464d4-4n9vl-default-8080"
failed to port forward pod-java-hello-world-5c744464d4-4n9vl-default-8080, port 8080 is taken, retrying...
time="2021-04-10T22:57:34+05:30" level=debug msg="terminated pod-java-hello-world-5c744464d4-4n9vl-default-8080 due to context cancellation"
port forwarding pod-java-hello-world-5c744464d4-4n9vl-default-8080 recovered on port 8080
time="2021-04-10T22:57:39+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 8080:8080]"
Port forwarding pod/java-hello-world-5c744464d4-4n9vl in namespace default, remote port 8080 -> 127.0.0.1:8080
time="2021-04-10T22:57:40+05:30" level=debug msg="Terminating port-forward pod-java-hello-world-5c744464d4-4n9vl-default-5005"
failed to port forward pod-java-hello-world-5c744464d4-4n9vl-default-5005, port 5005 is taken, retrying...
time="2021-04-10T22:57:40+05:30" level=debug msg="terminated pod-java-hello-world-5c744464d4-4n9vl-default-5005 due to context cancellation"
port forwarding pod-java-hello-world-5c744464d4-4n9vl-default-5005 recovered on port 5005
time="2021-04-10T22:57:45+05:30" level=debug msg="Running command: [kubectl --context gke_<GCP_PROJECT_ID>_asia-southeast1-a_<GKE_CLUSTER_NAME> port-forward --pod-running-timeout 1s --namespace default pod/java-hello-world-5c744464d4-4n9vl 5005:5005]"
Port forwarding pod/java-hello-world-5c744464d4-4n9vl in namespace default, remote port 5005 -> 127.0.0.1:5005
time="2021-04-10T22:58:28+05:30" level=debug msg="Change detected notify.Write: \"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/.idea/workspace.xml\""
time="2021-04-10T22:58:28+05:30" level=debug msg="Change detected notify.Write: \"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/.idea/workspace.xml~\""
time="2021-04-10T22:58:28+05:30" level=debug msg="Change detected notify.Remove: \"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/.idea/workspace.xml~\""
time="2021-04-10T22:58:28+05:30" level=debug msg="Change detected notify.Create: \"/Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/.idea/workspace.xml~\""
time="2021-04-10T22:58:29+05:30" level=debug msg="Using wrapper for mvnw: mvn"
time="2021-04-10T22:58:29+05:30" level=debug msg="could not stat dependency: stat /Users/gautamr/dev-projects-asconsoft/cloud-code-samples/java/java-hello-world/src/main/jib: no such file or directory"
time="2021-04-10T22:58:29+05:30" level=debug msg="Found dependencies for jib maven artifact: [pom.xml src/main/java/cloudcode/helloworld/HelloWorldApplication.java src/main/java/cloudcode/helloworld/web/HelloWorldController.java src/main/resources/application.properties src/main/resources/application.properties src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world-dark.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/KE-hello-world.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/cloud_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/dark_bg.svg src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/kubernetes-engine-icon.png src/main/resources/static/img/lightbulb_icon.svg src/main/resources/static/img/lightbulb_icon.svg src/main/resources/templates/index.html src/main/resources/templates/index.html]"

gautamr commented 3 years ago

@etanshaul I guess the debugger is disconnecting when the following is happening

failed to port forward pod-java-hello-world-5c744464d4-sgqsf-default-5005, port 5005 is taken, retrying...

gautamr commented 3 years ago

@etanshaul any luck on this ?

briandealwis commented 3 years ago

@gautamr is it possible your helm-spawned pods are being terminated or configured differently from your kubernetes manifests? Can you report back on the results of kubectl describe pod/xxx for your pods? I'm particularly interested in the events.

etanshaul commented 3 years ago

@gautamr are you still facing this issue? If you could follow up with the details that Brian requested above we can continue to help you debug through this. Thanks.

gautamr commented 3 years ago

@briandealwis @etanshaul I'm extremely sorry for being late, I will update on this in full details very soon

gautamr commented 3 years ago

Hey @briandealwis @etanshaul The k8s configurations are same in helm and plain Kubernetes configurations, in fact I tried with other deployment also, same thing is happening, still if you need the entire YAML I can share that.

Now I'm putting the output for kubectl describe pod/java-hello-world-799bd8c566-x72rh -n gintaa, hope this will help you

Name:         java-hello-world-799bd8c566-x72rh
Namespace:    gintaa
Priority:     0
Node:         gke-gintaa-services--gintaa-services--391f3b2a-8n60/10.0.0.15
Start Time:   Tue, 27 Apr 2021 20:21:37 +0530
Labels:       app=java-hello-world
              istio.io/rev=default
              pod-template-hash=799bd8c566
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=java-hello-world
              service.istio.io/canonical-revision=latest
Annotations:  cni.projectcalico.org/podIP: 10.1.0.39/32
              debug.cloud.google.com/config:
                {"java-hello-world":{"artifact":"gcr.io/<PROJECT_ID>/java-hello-world","runtime":"jvm","workingDir":"/","ports":{"jdwp":5005}}}
              kubectl.kubernetes.io/default-logs-container: java-hello-world
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/status:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
Status:       Running
IP:           10.1.0.39
IPs:
  IP:           10.1.0.39
Controlled By:  ReplicaSet/java-hello-world-799bd8c566
Init Containers:
  istio-init:
    Container ID:  docker://9f6c6d21c8045ffb481ef20e7b31893740c14b6316a05d6c72c8db6d61adfbcf
    Image:         docker.io/istio/proxyv2:1.9.2
    Image ID:      docker-pullable://istio/proxyv2@sha256:ee9c153e2f973937befb8af61e7269ab368020b6e7d91b5d891bbbeba55eb266
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 27 Apr 2021 20:21:39 +0530
      Finished:     Tue, 27 Apr 2021 20:21:41 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        10m
      memory:     40Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s8x2k (ro)
Containers:
  java-hello-world:
    Container ID:   docker://bd443035f0df54f301c63f2dad21bd249b3c0b1e9e21fdd4418e91a74fcdd1cc
    Image:          gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:b4c786a3d3e4ec2bc66053863decaab8d9873fcec4f419ada149938593dd6358
    Image ID:       docker-pullable://gcr.io/<GCP_PROJECT_ID>/java-hello-world@sha256:b4c786a3d3e4ec2bc66053863decaab8d9873fcec4f419ada149938593dd6358
    Ports:          8080/TCP, 5005/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 27 Apr 2021 20:21:46 +0530
    Ready:          True
    Restart Count:  0
    Readiness:      tcp-socket :8080 delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      PORT:               8080
      JAVA_TOOL_OPTIONS:  -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s8x2k (ro)
  istio-proxy:
    Container ID:  docker://1a3135ab63977027a35c1bc78bd2440844bdbba14635d6a431494faa015a9cea
    Image:         docker.io/istio/proxyv2:1.9.2
    Image ID:      docker-pullable://istio/proxyv2@sha256:ee9c153e2f973937befb8af61e7269ab368020b6e7d91b5d891bbbeba55eb266
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      java-hello-world.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --concurrency
      2
    State:          Running
      Started:      Tue, 27 Apr 2021 20:21:47 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      10m
      memory:   40Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      java-hello-world-799bd8c566-x72rh (v1:metadata.name)
      POD_NAMESPACE:                 gintaa (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])
      PROXY_CONFIG:                  {}

      ISTIO_META_POD_PORTS:          [
                                         {"containerPort":8080,"protocol":"TCP"}
                                         ,{"name":"jdwp","containerPort":5005,"protocol":"TCP"}
                                     ]
      ISTIO_META_APP_CONTAINERS:     java-hello-world
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_METAJSON_ANNOTATIONS:    {"debug.cloud.google.com/config":"{\"java-hello-world\":{\"artifact\":\"gcr.io/<GCP_PROJECT_ID>/java-hello-world\",\"runtime\":\"jvm\",\"workingDir\":\"/\",\"ports\":{\"jdwp\":5005}}}"}

      ISTIO_META_WORKLOAD_NAME:      java-hello-world
      ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/gintaa/deployments/java-hello-world
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s8x2k (ro)
      /var/run/secrets/tokens from istio-token (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
      limits.cpu -> cpu-limit
      requests.cpu -> cpu-request
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  default-token-s8x2k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-s8x2k
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  5m46s  default-scheduler  Successfully assigned gintaa/java-hello-world-799bd8c566-x72rh to gke-gintaa-services--gintaa-services--391f3b2a-8n60
  Normal   Pulling    5m45s  kubelet            Pulling image "docker.io/istio/proxyv2:1.9.2"
  Normal   Pulled     5m44s  kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.2"
  Normal   Created    5m44s  kubelet            Created container istio-init
  Normal   Started    5m44s  kubelet            Started container istio-init
  Normal   Pulling    5m41s  kubelet            Pulling image "gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:b4c786a3d3e4ec2bc66053863decaab8d9873fcec4f419ada149938593dd6358"
  Normal   Created    5m37s  kubelet            Created container java-hello-world
  Normal   Pulled     5m37s  kubelet            Successfully pulled image "gcr.io/<GCP_PROJECT_ID>/java-hello-world:latest@sha256:b4c786a3d3e4ec2bc66053863decaab8d9873fcec4f419ada149938593dd6358"
  Normal   Started    5m37s  kubelet            Started container java-hello-world
  Normal   Pulling    5m37s  kubelet            Pulling image "docker.io/istio/proxyv2:1.9.2"
  Normal   Pulled     5m36s  kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.2"
  Normal   Created    5m36s  kubelet            Created container istio-proxy
  Normal   Started    5m36s  kubelet            Started container istio-proxy
  Warning  Unhealthy  5m34s  kubelet            Readiness probe failed: Get http://10.1.0.39:15021/healthz/ready: dial tcp 10.1.0.39:15021: connect: connection refused

gautamr commented 3 years ago

For your kind information please note that I'm able to debug the exact stuffs (with same helm configuration) in VSCode with Cloud Code plugin. So I'm almost sure that it's related to this plugin i.e. IntelliJ

etanshaul commented 3 years ago

thanks for the extra info @gautamr

It looks like your readiness probe is failing (from looking at the kubectl describe events). Is your app also not loading properly, or is it just the debugger that is not working? I'm also curious what happens if you just run the app in normal run mode - e.g. if the app starts up as expected.

etanshaul commented 3 years ago

anything jump out at you @briandealwis , the debug annotations look normal:

debug.cloud.google.com/config:
                {"java-hello-world":{"artifact":"gcr.io/<PROJECT_ID>/java-hello-world","runtime":"jvm","workingDir":"/","ports":{"jdwp":5005}}}

briandealwis commented 3 years ago

The readiness probe failing shouldn't account for this problem since we're connecting directly to the pod via the port-forward.

@gautamr is your kubectl deploy still using Istio/Calico in the cluster? Does it work if you try without Istio?

It's curious that it works with CC for VS Code, which uses the Eclipse JDT debugger under the hood, but is failing with IntelliJ. Maybe there is something happening in the IntelliJ side?

etanshaul commented 3 years ago

@gautamr one more request :) could you share your IDE logs? You can find them under Help -> Show logs... A snapshot of the logs around when you are launching the session and debugging fails would be great, in case there is any info in there. To configure the logging to be more verbose see: https://intellij-support.jetbrains.com/hc/en-us/articles/207241085-Locating-IDE-log-files

gautamr commented 3 years ago

@etanshaul @briandealwis

I can start the app in normal run mode, no issue with that, problem is debugger is disconnected if I start in debug mode in CC in IntelliJ
Regarding istio, we are using istio in the cluster but definitely we are not mentioning anything related to istio in that app deployment configuration, that is observed by istio automatically by labelling namespace istio-injection=enabled. I did not tried uninstalling istio.

briandealwis commented 3 years ago

Thank you for your patience @gautamr in trying to narrow this down. This is baffling! The IDE logs may provide some insight.

Would it be possible for you to attach your Helm project here so that we could try to reproduce it ourselves? Have you been able to reproduce this with o other clusters, like Minikube?

briandealwis commented 3 years ago

The other thing that might prove useful is to do a kubectl get -oyaml pod/xxx for the pod when deployed via kubectl and helm, and take a diff.

briandealwis commented 3 years ago

I’m getting a sense of déjà vu — I vaguely remember digging into a situation like this involving a side-car proxy like we have here. The side-car proxies are containers in the same pod and so start in parallel with the app, and there is a window where the proxy hasn’t yet performed its magic. Since the JVM opens the debugging port early, perhaps that’s accounting for the drop.

Ah and now that I think about it, I think CC-VSC retries connections with a small back off. Is that right @quoctruong: CC-VSC will retry JVM connections too?

(This wouldn’t account for difference in behaviour with kubectl vs helm.)

@gautamr one last thing: can you wait a couple of seconds and then try connecting as a normal Java process remote connection to the local side of the forwarded debugger port (jdwp port 5005)?

gautamr commented 3 years ago

@briandealwis i will try to give you the details soon, currently having issue in our cluster

DanVisan commented 3 years ago

I am not sure if this is right but this is my guess on this issue:

The inteliJ debugger will connect to the remote container via the TCP port named jdwp (exposed at 5005 in this example). When deploying using kubectl, the cloud-code plugin will add this port to your deployment.yaml configuration. However, when deploying with helm, this port will not be added.

Example:

kubectl deployment


apiVersion: skaffold/v2beta4
kind: Config
build:
tagPolicy:
sha256: {}
artifacts:
- image: java-hello-world
  jib:
    args:
    - -Dmaven.test.skip=true
deploy:
kubectl:
manifests: ["kubernetes-manifests/*.yaml"]


Run the debugger from the InteliJ IDE

`$ kubectl get pod POD_NAME -oyaml`

apiVersion: v1 kind: Pod name: java-hello-world ... ports:

containerPort: 8080 protocol: TCP
containerPort: 5005 name: jdwp protocol: TCP

helm deployment

apiVersion: skaffold/v2beta4
kind: Config
build:
tagPolicy:
sha256: {}
artifacts:
- image: nexus.example.com/java-hello-world
  jib:
    args:
    - -Dmaven.test.skip=true
deploy:
helm:
releases:
- name: hello-java-release
  chartPath: kubernetes-manifests

$ kubectl get pod POD_NAME -oyaml

apiVersion: v1
kind: Pod
name: java-hello-world
...
    ports:
    - containerPort: 8080
      protocol: TCP
# NO jdwp port here ???

Obviously the debugger will crash.. because it can't run kubectl port-forward pod/POD_NAME 5005:5005 in the background

briandealwis commented 3 years ago

@DanVisan skaffold debug only applies the debug transformations to image references that correspond to images built by Skaffold. In your case, your Helm deploy in your skaffold.yaml does not specify an artifactOverrides and will not work. Please read the Skaffold documentation on properly configuring Helm to reference built images. Helm's templating is powerful but complicated.

I'm fairly certain that that your deployments likely seem to work as

you're using the sha256 tagger, which causes images to be tagged with :latest;
your Helm charts are likely using latest as their tag.

If you instead changed your tagging policy to be git- or datetime-based, you'll see that your deployments no longer update on changes as the deployment will not be using the newly-created tags.

Zweib commented 2 years ago

@briandealwis Any update on this? I have the same problem too, with the newest released intellij plugin(22.2.1-212).

It seems earlier releases(maybe before 2021, idk) work fine with this.

I think the pods running fine and I didn't use readinessProbe/livenessProbe, however, the plugin keeps retrying running command of port-forward, nearly every minutes. Every time it redo port-forward, the debugger disconnected.

There also is no disconnection if I execute the skaffold debug command with terminal, and then use another intellij remote JVM debug configuration, everything is ok.

etanshaul commented 2 years ago

hi @Zweib. I wonder if this is related to the original issue discussed here. Are you using Helm or kubectl deployer?

Could you share a (redacted as needed) output from your skaffold session we debug level verbosity (configurable in the run configuration under the advanced options); as well as the contents of your skaffold.yaml?

Zweib commented 2 years ago

hi @Zweib. I wonder if this is related to the original issue discussed here. Are you using Helm or kubectl deployer?

Could you share a (redacted as needed) output from your skaffold session we debug level verbosity (configurable in the run configuration under the advanced options); as well as the contents of your skaffold.yaml?

@etanshaul sorry for late reply. I used helm to deploy, and here is the skaffold debug log

Validating Kubernetes dependencies...
Validating image repository settings...

/Users/zweib/Library/Application Support/cloud-code/bin/versions/44ae7a4f2dfd5678febbcc10805a99c6f4f56413a3515db412fbf9709cab3ba5/skaffold debug --filename skaffold.yaml --profile noah-demo --label ide=idea --label ideVersion=2021.2.2.0.0 --label ijPluginVersion=22.2.1-212 --rpc-port 50051 --port-forward=true --wait-for-deletions-max=2m0s --status-check=true --verbosity debug
starting gRPC server on port 50051
Skaffold &{Version:v1.34.0 ConfigVersion:skaffold/v2beta25 GitVersion: GitCommit:22cfab75ffb305e7af220910af2f48d0a5c0e6af BuildDate:2021-10-27T00:29:22Z GoVersion:go1.14.14 Compiler:gc Platform:darwin/amd64 User:}
Skipping update check for flag `--update-check` set to false
Loaded Skaffold defaults from "/Users/zweib/.skaffold/config"
config version out of date: upgrading to latest "skaffold/v2beta25"
applying profile: noah-demo
overlaying profile on config for field Build
overlaying profile on config for field artifacts
overlaying profile on config for field insecureRegistries
overlaying profile on config for field tagPolicy
no values found in profile for field TagPolicy, using original config values
overlaying profile on config for field BuildType
no values found in profile for field BuildType, using original config values
overlaying profile on config for field Test
overlaying profile on config for field Deploy
overlaying profile on config for field DeployType
overlaying profile on config for field docker
overlaying profile on config for field helm
overlaying profile on config for field kpt
overlaying profile on config for field kubectl
overlaying profile on config for field kustomize
overlaying profile on config for field statusCheck
overlaying profile on config for field statusCheckDeadlineSeconds
overlaying profile on config for field kubeContext
overlaying profile on config for field logs
overlaying profile on config for field prefix
overlaying profile on config for field -
overlaying profile on config for field PortForward
Defaulting build type to local build
Using kubectl context: gke_noah-dev-1_asia-east1-b_noah-dev-1
Running command: [minikube version --output=json]
setting Docker user agent to skaffold-v1.34.0
Running command: [helm version --client]
Command output: [version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.17"}
]
Using builder: local
push value not present in NewBuilder, defaulting to true because cluster.PushImages is true
build concurrency first set to 1 parsed from *local.Builder[0]
final build concurrency value is 1
Listing files to watch...
 - <private_registry>/noahdev/noah-demo-server
Running command: [java -version]
Using wrapper for gradlew: gradle
Command output: [
BEGIN JIB JSON
List generated in 2.534 seconds
Generating tags...
 - <private_registry>/noahdev/noah-demo-server -> Running command: [git describe --tags --always]
Command output: [bfe4740
]
Running command: [git status . --porcelain]
Command output: []
<private_registry>/noahdev/noah-demo-server:bfe4740
Tags generated in 47.934222ms
Checking cache...
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
 - <private_registry>/noahdev/noah-demo-server: push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
Found Remotely
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
Cache check completed in 1.05 second
Tags used in deployment:
 - <private_registry>/noahdev/noah-demo-server -> <private_registry>/noahdev/noah-demo-server:bfe4740@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
Starting deploy...
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Deploying with helm v3.7.0 ...
Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 get all --namespace <namespace> noah-demo-udon]
Helm release noah-demo-udon not installed. Installing...
Building helm dependencies...
Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 dep build deploy_chart]
EnvVarMap: map[DIGEST:<private_registry>/noahdev/noah-demo-server:bfe4740@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 DIGEST_ALGO:<private_registry>/noahdev/noah-demo-server DIGEST_HEX:bfe4740@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 IMAGE_DIGEST:sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 IMAGE_NAME:<private_registry>/noahdev/noah-demo-server IMAGE_REPO:<private_registry>/noahdev/noah-demo-server IMAGE_TAG:bfe4740]

Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 install noah-demo-udon --post-renderer /Users/zweib/Library/Application Support/cloud-code/bin/versions/44ae7a4f2dfd5678febbcc10805a99c6f4f56413a3515db412fbf9709cab3ba5/skaffold deploy_chart --namespace <namespace> --set-string image.name=<private_registry>/noahdev/noah-demo-server:bfe4740@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 --set ingressroute.preferDev=udon]
NAME: noah-demo-udon
LAST DEPLOYED: Mon Feb 21 02:47:25 2022
NAMESPACE: <namespace>
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace <namespace> -l "app.kubernetes.io/name=noah-micro-service,app.kubernetes.io/instance=noah-demo-udon" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:80
Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 get all --namespace <namespace> noah-demo-udon --template {{.Release.Manifest}}]
---
# Source: noah-micro-service/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: noah-demo-udon-noah-micro-service
  labels:
    app.kubernetes.io/name: noah-micro-service
    helm.sh/chart: noah-micro-service-0.1.2
    app.kubernetes.io/instance: noah-demo-udon
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: noah-micro-service
    app.kubernetes.io/instance: noah-demo-udon
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/instance: noah-demo-udon
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: noah-micro-service
    app.kubernetes.io/version: "1.0"
    helm.sh/chart: noah-micro-service-0.1.2
  name: noah-demo-udon
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: noah-demo-udon
      app.kubernetes.io/name: noah-micro-service
  strategy: {}
  template:
    metadata:
      annotations:
        debug.cloud.google.com/config: '{"noah-micro-service":{"artifact":"<private_registry>/noahdev/noah-demo-server","runtime":"jvm","workingDir":"/","ports":{"jdwp":5005}}}'
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: noah-demo-udon
        app.kubernetes.io/name: noah-micro-service
    spec:
      containers:
      - env:
        - name: JAVA_TOOL_OPTIONS
          value: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y
        - name: SERVER_PORT
          value: "8080"
        envFrom:
        - configMapRef:
            name: java-server-conf
        - secretRef:
            name: spring-app-secret
        image: <private_registry>/noahdev/noah-demo-server:bfe4740@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035
        imagePullPolicy: Always
        name: noah-micro-service
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 5005
          name: jdwp
          protocol: TCP
        resources: {}
      imagePullSecrets:
      - name: gcp-gitlab-registry
status: {}
---
# Source: noah-micro-service/templates/ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: noah-demo-udon-noah-micro-service
  labels:
    app.kubernetes.io/name: noah-micro-service
    helm.sh/chart: noah-micro-service-0.1.2
    app.kubernetes.io/instance: noah-demo-udon
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Helm
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`<gateway>`) && PathPrefix(`/noah-demo`) && Headers(`prefer-dev`, `udon`)
    priority: 1000
    kind: Rule
    middlewares:
      - name: noah-demo-udon-noah-micro-service-stripprefix
    services:
    - name: noah-demo-udon-noah-micro-service
      port: 80
---
# Source: noah-micro-service/templates/ingressroute.yaml
# Strip prefix
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: noah-demo-udon-noah-micro-service-stripprefix
  labels:
    app.kubernetes.io/name: noah-micro-service
    helm.sh/chart: noah-micro-service-0.1.2
    app.kubernetes.io/instance: noah-demo-udon
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Helm
spec:
  stripPrefix:
    prefixes:
    - /noah-demo
found deployed object: &Service{ObjectMeta:{noah-demo-udon-noah-micro-service      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:noah-demo-udon app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:noah-micro-service app.kubernetes.io/version:1.0 helm.sh/chart:noah-micro-service-0.1.2] map[] [] []  []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:http,Protocol:TCP,Port:80,TargetPort:{1 0 http},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/instance: noah-demo-udon,app.kubernetes.io/name: noah-micro-service,},ClusterIP:,Type:ClusterIP,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,TopologyKeys:[],IPFamilyPolicy:nil,ClusterIPs:[],IPFamilies:[],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},}
found deployed object: &Deployment{ObjectMeta:{noah-demo-udon      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:noah-demo-udon app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:noah-micro-service app.kubernetes.io/version:1.0 helm.sh/chart:noah-micro-service-0.1.2] map[] [] []  []},Spec:DeploymentSpec{Replicas:*1,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/instance: noah-demo-udon,app.kubernetes.io/name: noah-micro-service,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:noah-demo-udon app.kubernetes.io/name:noah-micro-service] map[debug.cloud.google.com/config:{"noah-micro-service":{"artifact":"<private_registry>/noahdev/noah-demo-server","runtime":"jvm","workingDir":"/","ports":{"jdwp":5005}}}] [] []  []} {[] [] [{noah-micro-service <private_registry>/noahdev/noah-demo-server:bfe4740@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 [] []  [{http 0 8080 TCP } {jdwp 0 5005 TCP }] [{ ConfigMapEnvSource{LocalObjectReference:LocalObjectReference{Name:java-server-conf,},Optional:nil,} nil} { nil &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:spring-app-secret,},Optional:nil,}}] [{JAVA_TOOL_OPTIONS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y nil} {SERVER_PORT 8080 nil}] {map[] map[]} [] [] nil nil nil nil   Always nil false false false}] []  <nil> <nil>  map[]   <nil>  false false false <nil> nil [{gcp-gitlab-registry}]   nil  [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil>}},Strategy:DeploymentStrategy{Type:,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:nil,Paused:false,ProgressDeadlineSeconds:nil,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}
error decoding parsed yaml: no kind "IngressRoute" is registered for version "traefik.containo.us/v1alpha1" in scheme "pkg/runtime/scheme.go:100"
error decoding parsed yaml: no kind "Middleware" is registered for version "traefik.containo.us/v1alpha1" in scheme "pkg/runtime/scheme.go:100"
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Patchingnoah-demo-udon-noah-micro-servicein namespace<namespace>
Patchingnoah-demo-udonin namespace<namespace>
Deploy completed in 7.083 seconds
Waiting for deployments to stabilize...
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
checking status <namespace>:deployment/noah-demo-udon
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 rollout status deployment noah-demo-udon --namespace <namespace> --watch=false]
Command output: [Waiting for deployment "noah-demo-udon" rollout to finish: 0 of 1 updated replicas are available...
]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 rollout status deployment noah-demo-udon --namespace <namespace> --watch=false]
Command output: [Waiting for deployment "noah-demo-udon" rollout to finish: 0 of 1 updated replicas are available...
]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 rollout status deployment noah-demo-udon --namespace <namespace> --watch=false]
Command output: [deployment "noah-demo-udon" successfully rolled out
]
 - <namespace>:deployment/noah-demo-udon is ready.
Deployments stabilized in 3.868 seconds
found open port: 5005
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
Port forwarding deployment/noah-demo-udon in namespace <namespace>, remote port 5005 -> http://127.0.0.1:5005
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
found open port: 4503
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-f45cc97f9-rq8nv/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-f45cc97f9-rq8nv 4503:8080]
Port forwarding service/noah-demo-udon-noah-micro-service in namespace <namespace>, remote port 80 -> http://127.0.0.1:4503
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Press Ctrl+C to exit
Change detected<nil>
Not watching for changes...
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Streaming logs from pod: noah-demo-udon-f45cc97f9-rq8nv container: noah-micro-service
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 logs --since=15s -f noah-demo-udon-f45cc97f9-rq8nv -c noah-micro-service --namespace <namespace>]
selecting debug port for pod/noah-demo-udon-f45cc97f9-rq8nv/noah-micro-service: {jdwp 0 5005 TCP }
found open port: 5006
Forwarding container noah-demo-udon-f45cc97f9-rq8nv/noah-micro-service to local port 5006.
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-f45cc97f9-rq8nv 5006:5005]
[noah-micro-service] Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y
[noah-micro-service] 
[noah-micro-service]   .   ____          _            __ _ _
[noah-micro-service]  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
[noah-micro-service] ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
[noah-micro-service]  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
[noah-micro-service]   '  |____| .__|_| |_|_| |_\__, | / / / /
[noah-micro-service]  =========|_|==============|___/=/_/_/_/
[noah-micro-service]  :: Spring Boot ::                (v2.6.3)
[noah-micro-service] 
[noah-micro-service] 2022-02-20 18:47:33.992  INFO [,,] --- [           main] com.noahcustody.demo.DemoApplication     : Starting DemoApplication using Java 1.8.0_275 on noah-demo-udon-f45cc97f9-rq8nv with PID 1 (/app/classes started by root in /)
[noah-micro-service] 2022-02-20 18:47:33.997  INFO [,,] --- [           main] com.noahcustody.demo.DemoApplication     : No active profile set, falling back to default profiles: default
Using wrapper for gradlew: gradle
Found dependencies for jib-gradle artifact: [build.gradle noah-demo-sdk/build.gradle noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/ErrorCode.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Bar.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Foo.java noah-demo-server/build.gradle noah-demo-server/src/main/java/com/noahcustody/demo/DemoApplication.java noah-demo-server/src/main/java/com/noahcustody/demo/controller/FooController.java noah-demo-server/src/main/java/com/noahcustody/demo/service/DemoService.java noah-demo-server/src/main/resources/application.yml settings.gradle]
[noah-micro-service] 2022-02-20 18:47:36.802  INFO [,,] --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=e409916d-d36e-368c-af2b-0026172f9e2c
Port forwarding pod/noah-demo-udon-f45cc97f9-rq8nv in namespace <namespace>, remote port 5005 -> http://127.0.0.1:5006
[noah-micro-service] 2022-02-20 18:47:38.627  INFO [,,] --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
[noah-micro-service] Feb 20, 2022 6:47:38 PM org.apache.coyote.AbstractProtocol init
[noah-micro-service] INFO: Initializing ProtocolHandler ["http-nio-8080"]
[noah-micro-service] Feb 20, 2022 6:47:38 PM org.apache.catalina.core.StandardService startInternal
[noah-micro-service] INFO: Starting service [Tomcat]
[noah-micro-service] Feb 20, 2022 6:47:38 PM org.apache.catalina.core.StandardEngine startInternal
[noah-micro-service] INFO: Starting Servlet engine: [Apache Tomcat/9.0.56]
[noah-micro-service] Feb 20, 2022 6:47:38 PM org.apache.catalina.core.ApplicationContext log
[noah-micro-service] INFO: Initializing Spring embedded WebApplicationContext
[noah-micro-service] 2022-02-20 18:47:38.772  INFO [,,] --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4692 ms
[noah-micro-service] 2022-02-20 18:47:42.363  INFO [,,] --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
[noah-micro-service] Feb 20, 2022 6:47:42 PM org.apache.coyote.AbstractProtocol start
[noah-micro-service] INFO: Starting ProtocolHandler ["http-nio-8080"]
[noah-micro-service] 2022-02-20 18:47:42.491  INFO [,,] --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
[noah-micro-service] 2022-02-20 18:47:42.525  INFO [,,] --- [           main] com.noahcustody.demo.DemoApplication     : Started DemoApplication in 9.669 seconds (JVM running for 10.375)
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-f45cc97f9-rq8nv/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-f45cc97f9-rq8nv 4503:8080]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-f45cc97f9-rq8nv 5006:5005]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-f45cc97f9-rq8nv/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-f45cc97f9-rq8nv 4503:8080]

skaffold.yaml

#apiVersion: skaffold/v1
apiVersion: skaffold/v2beta19
kind: Config

profiles:
  - name: noah-demo
    build:
      artifacts:
        - image: <private_registry>/noahdev/noah-demo-server
          jib:
            project: noah-demo-server
            args:
              - -Djib.from.image=gcr.io/distroless/java:8
#            fromImage: gcr.io/distroless/java:8
    portForward:
    - namespace: <namespace>
      localPort: 5005
      port: 5005
      resourceType: deployment
      resourceName: noah-demo-{{ .username }}
    deploy:
      helm:
        releases:
          - name: noah-demo-{{ .username }}
            namespace: <namespace>
            chartPath: deploy_chart
            artifactOverrides:
              image.name: <private_registry>/noahdev/noah-demo-server
#            values:
#              image.name: <private_registry>/noahdev/noah-demo-server
            setValueTemplates:
              ingressroute.preferDev: "{{ .username }}"
        flags:
          upgrade:
            - --cleanup-on-fail
            - --install

the debugger disconnected after

Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-f45cc97f9-rq8nv 5006:5005]

came for the second time. It would repeat in every 1~2 minutes.

Zweib commented 2 years ago

More FYI: There is no difference whether I put "portForward" configuration on the skaffold.yaml file or not. It's just one of my attempt to fix this problem which doesn't work. And the pod runs smoothly with no restarting.

I also tried the cloud code java sample project (https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/v1/java/java-hello-world#getting-started-with-intellij), which uses kubectl as deployer, and I met the same problem.

briandealwis commented 2 years ago

@Zweib I noticed from your logs that you are using Traefik: can you try running without it?

As I mention further up there are problems with the way some of these meshes configure themselves.

briandealwis commented 2 years ago

@Zweib we saw similar issues with port interference with Linkerd from the set up of their sidecar container. In Kubernetes, all pod containers are started in parallel, so the main app may be running and have opened the port by the time the service mesh starts enacting its iptables shenanigans.

You may be able to configure Traefik to ignore port 5005?

Zweib commented 2 years ago

Validating Kubernetes dependencies...
Validating image repository settings...

/Users/zweib/Library/Application Support/cloud-code/bin/versions/44ae7a4f2dfd5678febbcc10805a99c6f4f56413a3515db412fbf9709cab3ba5/skaffold debug --filename skaffold.yaml --profile noah-demo --label ide=idea --label ideVersion=2021.2.2.0.0 --label ijPluginVersion=22.2.1-212 --rpc-port 50051 --port-forward=true --wait-for-deletions-max=2m0s --status-check=true --verbosity debug
starting gRPC server on port 50051
Skaffold &{Version:v1.34.0 ConfigVersion:skaffold/v2beta25 GitVersion: GitCommit:22cfab75ffb305e7af220910af2f48d0a5c0e6af BuildDate:2021-10-27T00:29:22Z GoVersion:go1.14.14 Compiler:gc Platform:darwin/amd64 User:}
Skipping update check for flag `--update-check` set to false
Loaded Skaffold defaults from "/Users/zweib/.skaffold/config"
config version out of date: upgrading to latest "skaffold/v2beta25"
applying profile: noah-demo
overlaying profile on config for field Build
overlaying profile on config for field artifacts
overlaying profile on config for field insecureRegistries
overlaying profile on config for field tagPolicy
no values found in profile for field TagPolicy, using original config values
overlaying profile on config for field BuildType
no values found in profile for field BuildType, using original config values
overlaying profile on config for field Test
overlaying profile on config for field Deploy
overlaying profile on config for field DeployType
overlaying profile on config for field docker
overlaying profile on config for field helm
overlaying profile on config for field kpt
overlaying profile on config for field kubectl
overlaying profile on config for field kustomize
overlaying profile on config for field statusCheck
overlaying profile on config for field statusCheckDeadlineSeconds
overlaying profile on config for field kubeContext
overlaying profile on config for field logs
overlaying profile on config for field prefix
overlaying profile on config for field -
overlaying profile on config for field PortForward
Defaulting build type to local build
Using kubectl context: gke_noah-dev-1_asia-east1-b_noah-dev-1
Running command: [minikube version --output=json]
Command output: [{"commit":"76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b","minikubeVersion":"v1.24.0"}
]
Running command: [/Users/zweib/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube profile list -o json --user=skaffold]
Command output: [{"invalid":[],"valid":[{"Name":"minikube","Status":"Unknown","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"https://storage.googleapis.com/minikube/iso/minikube-v1.24.0.iso","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.11@sha256:6fee59db7d67ed8ae6835e4bcb02f32056dc95f11cb369c51e352b62dd198aa0","Memory":4000,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"hyperkit","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"KVMNUMACount":0,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","SSHIPAddress":"","SSHUser":"","SSHKey":"","SSHPort":0,"KubernetesConfig":{"KubernetesVersion":"v1.18.3","ClusterName":"minikube","Namespace":"","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","CustomIngressCert":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"","Port":8443,"KubernetesVersion":"v1.18.3","ControlPlane":true,"Worker":true}],"Addons":null,"CustomAddonImages":null,"CustomAddonRegistries":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":null,"ListenAddress":"","Network":"","MultiNodeRequested":false,"ExtraDisks":0,"CertExpiration":94608000000000000,"Mount":false,"MountString":""}}]}]
Running command: [/Users/zweib/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube profile list -o json --user=skaffold]
Command output: [{"invalid":[],"valid":[{"Name":"minikube","Status":"Unknown","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"https://storage.googleapis.com/minikube/iso/minikube-v1.24.0.iso","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.11@sha256:6fee59db7d67ed8ae6835e4bcb02f32056dc95f11cb369c51e352b62dd198aa0","Memory":4000,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"hyperkit","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"KVMNUMACount":0,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","SSHIPAddress":"","SSHUser":"","SSHKey":"","SSHPort":0,"KubernetesConfig":{"KubernetesVersion":"v1.18.3","ClusterName":"minikube","Namespace":"","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","CustomIngressCert":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"","Port":8443,"KubernetesVersion":"v1.18.3","ControlPlane":true,"Worker":true}],"Addons":null,"CustomAddonImages":null,"CustomAddonRegistries":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":null,"ListenAddress":"","Network":"","MultiNodeRequested":false,"ExtraDisks":0,"CertExpiration":94608000000000000,"Mount":false,"MountString":""}}]}]
setting Docker user agent to skaffold-v1.34.0
Running command: [helm version --client]
Command output: [version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.17"}
]
Using builder: local
push value not present in NewBuilder, defaulting to true because cluster.PushImages is true
build concurrency first set to 1 parsed from *local.Builder[0]
final build concurrency value is 1
Listing files to watch...
 - <private_registry>/noahdev/noah-demo-server
Running command: [java -version]
Using wrapper for gradlew: gradle
Command output: [
BEGIN JIB JSON
]
Found dependencies for jib-gradle artifact: [build.gradle noah-demo-sdk/build.gradle noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/ErrorCode.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Bar.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Foo.java noah-demo-server/build.gradle noah-demo-server/src/main/java/com/noahcustody/demo/DemoApplication.java noah-demo-server/src/main/java/com/noahcustody/demo/controller/FooController.java noah-demo-server/src/main/java/com/noahcustody/demo/service/DemoService.java noah-demo-server/src/main/resources/application.yml settings.gradle]
List generated in 14.059 seconds
Generating tags...
 - <private_registry>/noahdev/noah-demo-server -> Running command: [git describe --tags --always]
Command output: [bfe4740
]
Running command: [git status . --porcelain]
Command output: [ M deploy_chart/values.yaml
]
<private_registry>/noahdev/noah-demo-server:bfe4740-dirty
Tags generated in 44.865666ms
Checking cache...
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
 - <private_registry>/noahdev/noah-demo-server: Found. Tagging
attempting to add tag <private_registry>/noahdev/noah-demo-server:bfe4740-dirty to src <private_registry>/noahdev/noah-demo-server:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
Cache check completed in 3.13 seconds
Tags used in deployment:
 - <private_registry>/noahdev/noah-demo-server -> <private_registry>/noahdev/noah-demo-server:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
Starting deploy...
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Deploying with helm v3.7.0 ...
Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 get all --namespace <namespace> noah-demo-udon]
Helm release noah-demo-udon not installed. Installing...
Building helm dependencies...
Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 dep build deploy_chart]
EnvVarMap: map[DIGEST:<private_registry>/noahdev/noah-demo-server:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 DIGEST_ALGO:<private_registry>/noahdev/noah-demo-server DIGEST_HEX:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 IMAGE_DIGEST:sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 IMAGE_NAME:<private_registry>/noahdev/noah-demo-server IMAGE_REPO:<private_registry>/noahdev/noah-demo-server IMAGE_TAG:bfe4740-dirty]

Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 install noah-demo-udon --post-renderer /Users/zweib/Library/Application Support/cloud-code/bin/versions/44ae7a4f2dfd5678febbcc10805a99c6f4f56413a3515db412fbf9709cab3ba5/skaffold deploy_chart --namespace <namespace> --set-string image.name=<private_registry>/noahdev/noah-demo-server:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 --set ingressroute.preferDev=udon]
NAME: noah-demo-udon
LAST DEPLOYED: Wed Feb 23 09:22:54 2022
NAMESPACE: <namespace>
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace <namespace> -l "app.kubernetes.io/name=noah-micro-service,app.kubernetes.io/instance=noah-demo-udon" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:80
Running command: [helm --kube-context gke_noah-dev-1_asia-east1-b_noah-dev-1 get all --namespace <namespace> noah-demo-udon --template {{.Release.Manifest}}]
---
# Source: noah-micro-service/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: noah-demo-udon-noah-micro-service
  labels:
    app.kubernetes.io/name: noah-micro-service
    helm.sh/chart: noah-micro-service-0.1.2
    app.kubernetes.io/instance: noah-demo-udon
    app.kubernetes.io/version: "1.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: noah-micro-service
    app.kubernetes.io/instance: noah-demo-udon
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/instance: noah-demo-udon
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: noah-micro-service
    app.kubernetes.io/version: "1.0"
    helm.sh/chart: noah-micro-service-0.1.2
  name: noah-demo-udon
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: noah-demo-udon
      app.kubernetes.io/name: noah-micro-service
  strategy: {}
  template:
    metadata:
      annotations:
        debug.cloud.google.com/config: '{"noah-micro-service":{"artifact":"<private_registry>/noahdev/noah-demo-server","runtime":"jvm","workingDir":"/","ports":{"jdwp":5005}}}'
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: noah-demo-udon
        app.kubernetes.io/name: noah-micro-service
    spec:
      containers:
      - env:
        - name: JAVA_TOOL_OPTIONS
          value: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y
        - name: SERVER_PORT
          value: "8080"
        envFrom:
        - configMapRef:
            name: java-server-conf
        - secretRef:
            name: spring-app-secret
        image: <private_registry>/noahdev/noah-demo-server:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035
        imagePullPolicy: Always
        name: noah-micro-service
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 5005
          name: jdwp
          protocol: TCP
        resources: {}
      imagePullSecrets:
      - name: gcp-gitlab-registry
status: {}
found deployed object: &Service{ObjectMeta:{noah-demo-udon-noah-micro-service      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:noah-demo-udon app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:noah-micro-service app.kubernetes.io/version:1.0 helm.sh/chart:noah-micro-service-0.1.2] map[] [] []  []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:http,Protocol:TCP,Port:80,TargetPort:{1 0 http},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/instance: noah-demo-udon,app.kubernetes.io/name: noah-micro-service,},ClusterIP:,Type:ClusterIP,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,TopologyKeys:[],IPFamilyPolicy:nil,ClusterIPs:[],IPFamilies:[],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},}
found deployed object: &Deployment{ObjectMeta:{noah-demo-udon      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:noah-demo-udon app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:noah-micro-service app.kubernetes.io/version:1.0 helm.sh/chart:noah-micro-service-0.1.2] map[] [] []  []},Spec:DeploymentSpec{Replicas:*1,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/instance: noah-demo-udon,app.kubernetes.io/name: noah-micro-service,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:noah-demo-udon app.kubernetes.io/name:noah-micro-service] map[debug.cloud.google.com/config:{"noah-micro-service":{"artifact":"<private_registry>/noahdev/noah-demo-server","runtime":"jvm","workingDir":"/","ports":{"jdwp":5005}}}] [] []  []} {[] [] [{noah-micro-service <private_registry>/noahdev/noah-demo-server:bfe4740-dirty@sha256:6249cabee940e24a6cce2aab63b68bd2a0519cc5eb32b4175e25542fef024035 [] []  [{http 0 8080 TCP } {jdwp 0 5005 TCP }] [{ ConfigMapEnvSource{LocalObjectReference:LocalObjectReference{Name:java-server-conf,},Optional:nil,} nil} { nil &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:spring-app-secret,},Optional:nil,}}] [{JAVA_TOOL_OPTIONS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y nil} {SERVER_PORT 8080 nil}] {map[] map[]} [] [] nil nil nil nil   Always nil false false false}] []  <nil> <nil>  map[]   <nil>  false false false <nil> nil [{gcp-gitlab-registry}]   nil  [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil>}},Strategy:DeploymentStrategy{Type:,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:nil,Paused:false,ProgressDeadlineSeconds:nil,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Patchingnoah-demo-udon-noah-micro-servicein namespace<namespace>
Patchingnoah-demo-udonin namespace<namespace>
Deploy completed in 8.329 seconds
Waiting for deployments to stabilize...
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
checking status <namespace>:deployment/noah-demo-udon
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 rollout status deployment noah-demo-udon --namespace <namespace> --watch=false]
Command output: [Waiting for deployment "noah-demo-udon" rollout to finish: 0 of 1 updated replicas are available...
]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 rollout status deployment noah-demo-udon --namespace <namespace> --watch=false]
Command output: [Waiting for deployment "noah-demo-udon" rollout to finish: 0 of 1 updated replicas are available...
]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 rollout status deployment noah-demo-udon --namespace <namespace> --watch=false]
Command output: [deployment "noah-demo-udon" successfully rolled out
]
 - <namespace>:deployment/noah-demo-udon is ready.
Deployments stabilized in 3.915 seconds
found open port: 5005
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
Port forwarding deployment/noah-demo-udon in namespace <namespace>, remote port 5005 -> http://127.0.0.1:5005
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
found open port: 4503
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-77fbf8f886-jmvk9/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 4503:8080]
Port forwarding service/noah-demo-udon-noah-micro-service in namespace <namespace>, remote port 80 -> http://127.0.0.1:4503
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Press Ctrl+C to exit
Change detected<nil>
Not watching for changes...
Streaming logs from pod: noah-demo-udon-77fbf8f886-jmvk9 container: noah-micro-service
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 logs --since=17s -f noah-demo-udon-77fbf8f886-jmvk9 -c noah-micro-service --namespace <namespace>]
selecting debug port for pod/noah-demo-udon-77fbf8f886-jmvk9/noah-micro-service: {jdwp 0 5005 TCP }
found open port: 5006
Forwarding container noah-demo-udon-77fbf8f886-jmvk9/noah-micro-service to local port 5006.
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 5006:5005]
[noah-micro-service] Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y
[noah-micro-service] 
[noah-micro-service]   .   ____          _            __ _ _
[noah-micro-service]  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
[noah-micro-service] ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
[noah-micro-service]  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
[noah-micro-service]   '  |____| .__|_| |_|_| |_\__, | / / / /
[noah-micro-service]  =========|_|==============|___/=/_/_/_/
[noah-micro-service]  :: Spring Boot ::                (v2.6.3)
[noah-micro-service] 
[noah-micro-service] 2022-02-23 01:23:03.519  INFO [,,] --- [           main] com.noahcustody.demo.DemoApplication     : Starting DemoApplication using Java 1.8.0_275 on noah-demo-udon-77fbf8f886-jmvk9 with PID 1 (/app/classes started by root in /)
[noah-micro-service] 2022-02-23 01:23:03.524  INFO [,,] --- [           main] com.noahcustody.demo.DemoApplication     : No active profile set, falling back to default profiles: default
[noah-micro-service] 2022-02-23 01:23:05.957  INFO [,,] --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=e409916d-d36e-368c-af2b-0026172f9e2c
Using wrapper for gradlew: gradle
Found dependencies for jib-gradle artifact: [build.gradle noah-demo-sdk/build.gradle noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/ErrorCode.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Bar.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Foo.java noah-demo-server/build.gradle noah-demo-server/src/main/java/com/noahcustody/demo/DemoApplication.java noah-demo-server/src/main/java/com/noahcustody/demo/controller/FooController.java noah-demo-server/src/main/java/com/noahcustody/demo/service/DemoService.java noah-demo-server/src/main/resources/application.yml settings.gradle]
[noah-micro-service] 2022-02-23 01:23:08.080  INFO [,,] --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
[noah-micro-service] Feb 23, 2022 1:23:08 AM org.apache.coyote.AbstractProtocol init
[noah-micro-service] INFO: Initializing ProtocolHandler ["http-nio-8080"]
[noah-micro-service] Feb 23, 2022 1:23:08 AM org.apache.catalina.core.StandardService startInternal
[noah-micro-service] INFO: Starting service [Tomcat]
[noah-micro-service] Feb 23, 2022 1:23:08 AM org.apache.catalina.core.StandardEngine startInternal
[noah-micro-service] INFO: Starting Servlet engine: [Apache Tomcat/9.0.56]
[noah-micro-service] Feb 23, 2022 1:23:08 AM org.apache.catalina.core.ApplicationContext log
[noah-micro-service] INFO: Initializing Spring embedded WebApplicationContext
[noah-micro-service] 2022-02-23 01:23:08.273  INFO [,,] --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4672 ms
Port forwarding pod/noah-demo-udon-77fbf8f886-jmvk9 in namespace <namespace>, remote port 5005 -> http://127.0.0.1:5006
[noah-micro-service] 2022-02-23 01:23:11.555  INFO [,,] --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
[noah-micro-service] Feb 23, 2022 1:23:11 AM org.apache.coyote.AbstractProtocol start
[noah-micro-service] INFO: Starting ProtocolHandler ["http-nio-8080"]
[noah-micro-service] 2022-02-23 01:23:11.666  INFO [,,] --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
[noah-micro-service] 2022-02-23 01:23:11.698  INFO [,,] --- [           main] com.noahcustody.demo.DemoApplication     : Started DemoApplication in 9.366 seconds (JVM running for 10.052)
Using wrapper for gradlew: gradle
Found dependencies for jib-gradle artifact: [build.gradle noah-demo-sdk/build.gradle noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/ErrorCode.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Bar.java noah-demo-sdk/src/main/java/com/noahcustody/demo/sdk/model/Foo.java noah-demo-server/build.gradle noah-demo-server/src/main/java/com/noahcustody/demo/DemoApplication.java noah-demo-server/src/main/java/com/noahcustody/demo/controller/FooController.java noah-demo-server/src/main/java/com/noahcustody/demo/service/DemoService.java noah-demo-server/src/main/resources/application.yml settings.gradle]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-77fbf8f886-jmvk9/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 4503:8080]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 5006:5005]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-77fbf8f886-jmvk9/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 4503:8080]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 5006:5005]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`
Forwarding service noah-demo-udon-noah-micro-service/80 to pod noah-demo-udon-77fbf8f886-jmvk9/8080
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 4503:8080]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> pod/noah-demo-udon-77fbf8f886-jmvk9 5006:5005]
Running command: [kubectl --context gke_noah-dev-1_asia-east1-b_noah-dev-1 port-forward --pod-running-timeout 1s --namespace <namespace> deployment/noah-demo-udon 5005:5005]
getting client config for kubeContext: `gke_noah-dev-1_asia-east1-b_noah-dev-1`

@briandealwis I disabled the traefik ingressroute, however result is the same

anthonyalayo commented 2 years ago

For what it's worth, I've been seeing this issue as well using Helm.

briandealwis commented 2 years ago

skaffold debug uses kubectl-based port-forwarding, which connects via the API server and then directly into the cluster and isn't affected by ingress.

So my suspicion here is that your mesh is seeing the debugging-related containerPort for the debugging port created by skaffold debug, and intercepting incoming traffic to those pods. In vanilla clusters, traffic connects directly to ports on running containers:

My understanding is that these service meshes use iptables skullduggery via sidecars to intercept socket connections to redirect incoming and outgoing traffic between pods through their proxies to enable features like mTLS, retries, and circuit breaking. That interception is fine for inter-pod communication as the outgoing connections are also intercepted and redirected through the proxies.

But kubectl port-forward's outgoing connection does not go through the mesh proxy and so they connect to the proxied port, which fails as the proxy does not "speak" the debugging protocol.

I don't know if there's a way to configure Traefik to disable proxying the debug ports.

But perhaps one of you can try patching the debug code that adds containerPort definitions with the following patch:

// exposePort adds a `types.ContainerPort` instance or amends an existing entry with the same port.
func exposePort(entries []types.ContainerPort, portName string, port int32) []types.ContainerPort {
    // experiment: don't add containerPorts to avoid traffic being intercepted by service meshes
    return entries
}

~It's a bit unfortunate as debug uses these containerPort definitions to identify whether debugging is already configured.~ (edit: I don't know what I meant by this.)

tejal29 commented 2 years ago

/cc @gsquared94

briandealwis commented 2 years ago

@etanshaul tried the exposePort() approach above and it doesn't work as the IDEs rely on Skaffold's port-forwarding, which relies on the presence of the containerPort definitions. Skaffold could add a new port-forwarder that uses the debug annotation on the pods to establish the port-forward.

Additional note for @gsquared94: the exposePort() is required for the Docker deployer so that it can open the requisite ports. I think the change would need to be made in the kubernetes-debug adapter.

gsquared94 commented 2 years ago

I was able to repro this issue with Skaffold v1.20.0 on IntelliJ as originally reported above. However that is a very old version of Skaffold. I couldn't repro it with the latest versions v1.39.3 and v2.0.1. I remember there being a bug for duplicate port forwards happening, that was fixed sometime ago that this seems related to. @anthonyalayo what Skaffold version are you using, can you try using the latest?

anthonyalayo commented 2 years ago

@gsquared94 I just testing out skaffold v2.0.1 and I didn't see an issue!

etanshaul commented 1 year ago

@anthonyalayo thanks for confirming. I am going to close this issue as it is resolved with Skaffold 2+.

Note to followers: the next Cloud Code release will officially support Skaffold V2 (including bundling it). However, in the meantime, you can point Cloud Code to use Skaffold v2 in the settings: Go to Settings / Preferences -> Tools -> Cloud Code -> Dependencies -> select "use custom specified dependencies" and browse to skaffold on your system (or let it pull from PATH).

Feel free to follow up with any questions or concerns.

GoogleCloudPlatform / cloud-code-intellij