search

GoogleContainerTools / container-debug-support

Language-runtime support files for in-container debugging
Apache License 2.0
93 stars 25 forks source link

Revert dlv to 1.3.1 #36

Closed briandealwis closed 4 years ago

briandealwis commented 4 years ago

33 upgraded Delve to v1.14. Delve 1.14 introduces a new behaviour to enforce that when listening for connections are from the same user (https://github.com/go-delve/delve/pull/1764). This behaviour is enabled by default and breaks current uses of skaffold debug as the connections proxied via kubectl are local but from a different user.

Steps to reproduce:

Use skaffold debug on Skaffold's Getting Started example. The container will seem to start with no issue, but attempting to connect will fail:

$ dlv connect 127.0.0.1:56268
Type 'help' for list of commands.
(dlv) bt
Command failed: connection is shut down
(dlv) c
Command failed: connection is shut down
(dlv) exit
connection is shut down

Log output:

When enabling Delve's logging on the server (in the container):

[web-5f64f88986-nwl74 install-go-support] Installing runtime debugging support files in /dbg
[web-5f64f88986-nwl74 install-go-support] Installation complete
[web-5f64f88986-nwl74 web] API server listening at: 127.0.0.1:56268
[web-5f64f88986-nwl74 web] 2020-04-25T20:59:37Z info layer=debugger launching process with args: [/layers/google.go.build/bin/main]
[web-5f64f88986-nwl74 web] 2020-04-25T20:59:37Z debug layer=debugger continuing
[web-5f64f88986-nwl74 web] 2020/04/25 20:59:37 Listening on port :8080
[web-5f64f88986-nwl74 web] 2020/04/25 20:59:58 sameuser_linux.go:111: closing connection from different user (127.0.0.1:47726): connections to localhost are only accepted from the same UNIX user for security reasons

Additional Information:

We can work around this by having Delve listen on 0.0.0.0 rather than localhost, but as this breaks installations in the field, we need to revert to Delve 1.3.1 first.

briandealwis commented 4 years ago

Rebuilt and pushed fixed duct-tape images.

etanshaul commented 4 years ago

I just tested this out using the Cloud Code Go guestbook samples app (in GoLand - although issues do not appear to be IDE related): https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/master/golang/go-guestbook

And am getting all sorts of errors:

Note: when running in normal mode (skaffold dev) the app starts up fine. Everything below is for debug mode.

1) Running the sample as is, the app fails to start up: hitting the frontend endpoint: querying backend failed: Get "http://go-guestbook-backend:8080/messages": dial tcp 10.24.9.13:8080: connect: connection refused From the skaffold logs:

[go-guestbook-frontend-7b59bc45d4-nxhwt frontend] 2020-04-26T19:44:28Z warning layer=debugger reading debug_info: could not find abstract origin (0x1a9a9d) of inlined call at 0x173006
[go-guestbook-frontend-7b59bc45d4-nxhwt frontend] 2020-04-26T19:44:28Z debug layer=debugger continuing
[go-guestbook-frontend-7b59bc45d4-nxhwt frontend] 2020/04/26 19:44:28 frontend server listening on port 8080
[go-guestbook-frontend-7b59bc45d4-nxhwt frontend] 2020-04-26T19:44:29Z debug layer=debugger halting
[go-guestbook-frontend-7b59bc45d4-nxhwt frontend] 2020-04-26T19:44:29Z error layer=rpc writing response:write tcp 127.0.0.1:3000->127.0.0.1:55662: use of closed network connection

Ok maybe the app is misconfigured? The entrypoint is as follows: ENTRYPOINT ["dlv", "exec", "/app/backend", "--continue", "--accept-multiclient", "--api-version=2", "--headless", "--listen=:3000", "--log"]

Instead, I try just ENTRYPOINT ["/app/backend"] (same for the frontend) :

2) The app now starts up but no debug events.

I notice that the Dockerfile doesn't have an Go specific env vars that the debug heuristics can recognize, so I add the following to both frontend and backend Dockerfiles:

ENV GOTRACEBACK=single

3) Debuggers attach now, but I get the following errors in a loop. Port forwarding fails to stabilize:

Port forwarding pod/go-guestbook-frontend-66b75595cb-lnhkj in namespace default, remote port 8080 -> address 127.0.0.1 port 8081
Port forwarding pod/go-guestbook-frontend-66b75595cb-lnhkj in namespace default, remote port 3000 -> address 127.0.0.1 port 3000
Port forwarding pod/go-guestbook-frontend-66b75595cb-lnhkj in namespace default, remote port 56268 -> address 127.0.0.1 port 56268
failed to port forward pod-go-guestbook-frontend-66b75595cb-lnhkj-default-56268, port 56268 is taken, retrying...
failed to port forward pod-go-guestbook-frontend-66b75595cb-lnhkj-default-3000, port 3000 is taken, retrying...
failed to port forward pod-go-guestbook-frontend-66b75595cb-lnhkj-default-8080, port 8081 is taken, retrying...
[go-guestbook-frontend-66b75595cb-lnhkj frontend] API server listening at: 127.0.0.1:56268
[go-guestbook-frontend-66b75595cb-lnhkj frontend] Version of Delve is too old for this version of Go (maximum supported version 1.13, suppress this error with --check-go-version=false)

Summary:

Skaffold debug seems pretty broken right now fo the Go sample apps (the guestbook one at least).

I'm not sure if these issues are related to this dlv change. But I'm hoping you can help me work through these issues and try and reproduce.

I suspect its a combination of the Samples being poorly configured and Skaffold/container-debug-support issues.

briandealwis commented 4 years ago

Sorry I didn't see your update @etanshaul :-( This should now be fixed with #38.