Breakpoints not hit in debug mode on intellij

scottiedog45 commented 1 year ago

(Please ensure you are running the latest version of Cloud Cloud IntelliJ with Help > Check for Updates.)

(screenshots are often helpful)

Expected Behavior

Normal breakpoint behavior when running debug mode, like pausing at breakpoints.

Actual Behavior

Breakpoints skipped/not hit

Additional Information

Trying to run the guestbook application. I can see the output in the terminal indicating there might be a problem connecting with the debugger. Everything else in the project seems to work fine when running outside debug mode.

[backend] 2023-01-08T04:27:01Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:41584: use of closed network connection
[frontend] 2023-01-08T04:27:01Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:41570: use of closed network connection

I am able to hit breakpoints when running the project through VS Code.

IDE type (e.g. IntelliJ, Pycharm): Intellij
IDE version: 2022.2.5
Cloud Code version (Settings > Plugins > Cloud Code): 22-12-2.222
Cloud SDK (Settings > Cloud Code > Cloud SDK)
- Are you allowing the plugin to manage the Cloud SDK: Yes
- Version of the Cloud SDK: 412.0.0
Skaffold version (If you are using Kubernetes features; Settings > Cloud Code > Kubernetes): skaffold v2.0.4
Operating System: 13.1

Feel free to deviate from this template as needed, especially if you are submitting a feature request.

Here is the full log if it's helpful:

Validating Kubernetes dependencies...
Validating Docker settings...
Validating image repository settings...

Status of minikube api-server is STOPPED

minikube may take several minutes to start up...

Starting minikube

/Users/scottadmin/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube start --wait true --vm-driver docker --interactive false --delete-on-failure --user cloud-code-intellij
* minikube v1.28.0 on Darwin 13.1 (arm64)
  - MINIKUBE_WANTUPDATENOTIFICATION=false
* Using the docker driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Restarting existing docker container for "minikube" ...
* Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
* Verifying Kubernetes components...
  - Using image docker.io/kubernetesui/dashboard:v2.7.0
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
  - Using image docker.io/kubernetesui/metrics-scraper:v1.0.8
* Some dashboard features require the metrics-server addon. To enable all features please run:

    minikube addons enable metrics-server   

* Enabled addons: storage-provisioner, default-storageclass, dashboard

! /Users/scottadmin/Library/Application Support/cloud-code/bin/versions/0031dbdda2711c2792ffe249194dabbb8599cded7f7b28a0d9085f5455359ec5/kubectl is version 1.21.0, which may have incompatibilities with Kubernetes 1.25.3.
  - Want kubectl v1.25.3? Try 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

minikube started successfully.
Enabling GCP auth addon...

GCP auth addon successfully enabled

/Users/scottadmin/Library/Application Support/cloud-code/bin/versions/0031dbdda2711c2792ffe249194dabbb8599cded7f7b28a0d9085f5455359ec5/skaffold debug --filename skaffold.yaml --label ide=cloudcodeintellij --rpc-port 50051 --port-forward=true --wait-for-deletions-max=2m0s --status-check=true --verbosity warn
Listing files to watch...
 - go-guestbook-frontend
 - go-guestbook-backend
Generating tags...
 - go-guestbook-frontend -> go-guestbook-frontend:latest
 - go-guestbook-backend -> go-guestbook-backend:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
 - go-guestbook-frontend: Not found. Building
 - go-guestbook-backend: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Found [minikube] context, using local docker daemon.
Building [go-guestbook-backend]...
Target platforms: [linux/arm64]
Build [go-guestbook-backend] succeeded
Building [go-guestbook-frontend]...
Target platforms: [linux/arm64]
Build [go-guestbook-frontend] succeeded
Tags used in deployment:
 - go-guestbook-frontend -> go-guestbook-frontend:25a4170b96d9fabb0aef7a8f3edb57aeba42f8536198a6c983cadfe4e004c9c2
 - go-guestbook-backend -> go-guestbook-backend:0134a5cd0230a4a6b08b72f70e14c46b3cfaf1fcb5c6c538cf3424fe32b44acd
Starting deploy...
 - deployment.apps/go-guestbook-frontend created
 - service/go-guestbook-frontend created
 - deployment.apps/go-guestbook-backend created
 - service/go-guestbook-backend created
 - deployment.apps/go-guestbook-mongodb created
 - service/go-guestbook-mongodb created
Waiting for deployments to stabilize...
 - deployment/go-guestbook-mongodb is ready. [2/3 deployment(s) still pending]
 - pod/go-guestbook-frontend-8448b8bf8-c4nv7: waiting for init container install-go-debug-support to start
 - pod/go-guestbook-backend-8dcb95fcc-wn9n4: waiting for init container init-db-ready to start
 - pod/go-guestbook-frontend-8448b8bf8-c4nv7: running.
 - deployment/go-guestbook-frontend is ready. [1/3 deployment(s) still pending]
 - deployment/go-guestbook-backend: 
 - pod/go-guestbook-backend-8dcb95fcc-wn9n4: running.
 - deployment/go-guestbook-backend is ready.
Deployments stabilized in 11.136 seconds
Port forwarding service/go-guestbook-frontend in namespace default, remote port 80 -> http://127.0.0.1:4503
Port forwarding service/go-guestbook-mongodb in namespace default, remote port 27017 -> http://127.0.0.1:27017
Port forwarding service/go-guestbook-backend in namespace default, remote port 8080 -> http://127.0.0.1:8080
Forwarding container go-guestbook-frontend-8448b8bf8-c4nv7/frontend to local port 56269.
Forwarding container go-guestbook-frontend-8448b8bf8-c4nv7/frontend to local port 56269.
Press Ctrl+C to exit
Not watching for changes...
[frontend] API server listening at: [::]:56268
[backend] API server listening at: [::]:56268
[init-db-ready] Waiting for mongodb at go-guestbook-mongodb:27017 to go live before the BE...
[install-go-debug-support] Installing runtime debugging support files in /dbg
[backend] 2023-01-08T04:26:50Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
[install-go-debug-support] Installation complete
[backend] 2023/01/08 04:26:51 backend server listening on port 8080
[frontend] 2023-01-08T04:26:49Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
[frontend] 2023/01/08 04:26:50 frontend server listening on port 8080
[install-go-debug-support] Installing runtime debugging support files in /dbg
[install-go-debug-support] Installation complete
Port forwarding pod/go-guestbook-backend-8dcb95fcc-wn9n4 in namespace default, remote port 56268 -> http://127.0.0.1:56268
Port forwarding pod/go-guestbook-frontend-8448b8bf8-c4nv7 in namespace default, remote port 56268 -> http://127.0.0.1:56269
[backend] 2023-01-08T04:27:01Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:41584: use of closed network connection
[frontend] 2023-01-08T04:27:01Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:41570: use of closed network connection

Cleaning up...
 - deployment.apps "go-guestbook-backend" deleted
Disabling GCP auth addon...

 - service "go-guestbook-backend" deleted
 - deployment.apps "go-guestbook-mongodb" deleted
 - service "go-guestbook-mongodb" deleted
 - deployment.apps "go-guestbook-frontend" deleted
 - service "go-guestbook-frontend" deleted

GCP auth addon successfully disabled

Stopping minikube

/Users/scottadmin/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube stop --user cloud-code-intellij
* Stopping node "minikube"  ...
* Powering off "minikube" via SSH ...
* 1 node stopped.

minikube stopped successfully.

etanshaul commented 1 year ago

hi @scottiedog45 - I see you are using a Go application. Did you enable go modules support for your project? This is required for debugging Go containers with Cloud Code IntelliJ.

From that link:

To configure your application for debugging, your app must be a Go Module-based application and be identified as being Go-based by setting one of the standard Go runtime environment variables in the container, such as GODEBUG, GOGC, GOMAXPROCS, or GOTRACEBACK. GOTRACEBACK=single is the default setting for Go and GOTRACEBACK=all is a generally useful configuration.

To enable Go modules, go to Settings > Languages & Frameworks > Go > Go Modules, and select Enable Go modules integration:

scottiedog45 commented 1 year ago

Hey @etanshaul , thanks for the speedy reply.

Can confirm modules are enabled:

Screenshot 2023-01-09 at 10 16 23 AM

Also on Go version 1.19.4 if that helps. Also, I'm using Goland, specifically.

etanshaul commented 1 year ago

I see. I am not able to reproduce with the go-guestbook application.

Are you able to see the debug tabs as shown here:

Also, just in case (I've made this mistake before) - where did you set your breakpoint? You won't be able to debug the server start up code as-is (main()) as the debugger is attached later. Instead, try setting a breakpoint in the homeHandler:

Otherwise, is there anything you may have changed in your application?

scottiedog45 commented 1 year ago

Yep I can see the debug tabs, but there is no output when the application starts:

I made a fresh project, just to be sure I haven't changed anything. Still seeing the same output. Not sure if these lines from the terminal output are suspicious:

Some taggers failed. Rerun with -vdebug for errors.

[frontend] 2023-01-09T15:37:34Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:34416: use of closed network connection
[backend] 2023-01-09T15:37:34Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:34418: use of closed network connection

Made sure to put a breakpoint in the homeHandler, still not being hit.

etanshaul commented 1 year ago

fwiw, the console tab is also blank for me, and I also see the following messages in the verbose logs:

[frontend] 2023-01-09T16:39:31Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:51758: use of closed network connection [backend] 2023-01-09T16:39:31Z error layer=rpc writing response:write tcp 127.0.0.1:56268->127.0.0.1:48174: use of closed network connection

Yet, I can still hit the breakpoints.

You should be able to get a bunch of additional debug logs if you enable verbose debugging on Skaffold. To do this, in the Run Configuration, under advanced settings, set logging verbosity to DEBUG:

If you search for the keyword "debug" in the log, you should see several bits on information. For example this snippet showing successful application of the debug transformations:

Configuring "" for Go/Delve debugging
"frontend" requires debugging support image "go"
Configuring installation of debugging support files
Applied debugging transform:
 apiVersion: apps/v1
kind: Deployment

etanshaul commented 1 year ago

here is a gist containing my debug level logs for the go-guestbook app (with working breakpoint debugging), in case it helps for comparison.: https://gist.github.com/etanshaul/4d27a3244e17b0ad68d16d9f143e0553

If you share yours, I can take a look too

scottiedog45 commented 1 year ago

Thanks again for your help with this, and easing my mind about those logs before.

Here is a link to my gist of logs.

One difference I can see is regarding tags. Yours says

Tags used in deployment:
 - go-guestbook-frontend -> gcr.io/etan-gcp/go-guestbook-frontend:v2.0.1-1-gaa0bf348b-dirty@sha256:4186360accc42b675ca32aeb216d4a78881d2486e0d929d1b7d608115cc89f60
 - go-guestbook-backend -> gcr.io/etan-gcp/go-guestbook-backend:v2.0.1-1-gaa0bf348b-dirty@sha256:e9b59ccc2d56c0a8532175e888df09a23aacadb3ecdd4dbfed365d2f46acdd5c
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true
push value not present in isImageLocal(), defaulting to true because cluster.PushImages is true

while mine is

Tags used in deployment:
 - go-guestbook-frontend -> go-guestbook-frontend:3d6a7afe103cef8afbc064da54b654c69ceabbac8d1ee73d53b94038e94f8e35
 - go-guestbook-backend -> go-guestbook-backend:09130c2d49256a3c65c898d5fcdd90315874ea2a009c9b33ad1c8d1f93aa7c99
push value not present in isImageLocal(), defaulting to false because cluster.PushImages is false
push value not present in isImageLocal(), defaulting to false because cluster.PushImages is false
Local images can't be referenced by digest.
They are tagged and referenced by a unique, local only, tag instead.
See https://skaffold.dev/docs/pipeline-stages/taggers/#how-tagging-works

Here is a copy of my rc, to verify that I'm trying to deploy to a local cluster

etanshaul commented 1 year ago

strange. The only other difference I noticed here is that you are deploying to a minikube cluster (whereas I was using GKE). I tried minikube however, and it worked fine, so this is a bit perplexing.

I will give the logs another closer look and see if we have any ideas.

In the meantime - could you try the simple hello-world Go app (not the guestbook app)? It is in the New Project wizard as well. I wonder if it is that app specifically.

scottiedog45 commented 1 year ago

I can hit breakpoints in the hello-world app 🥳 .

I tried the guest book app after again, and noticed that several debugger tabs open instead of two (i.e. one for the backend and frontend). This time, two were opened for the front end? Also, the breakpoints shift to an "invalid" state when the front end tabs are selected, even though the breakpoints are in the frontend:

GoogleCloudPlatform / cloud-code-intellij