Debugging Python in Docker via SSH, breakpoints will not be hit

eohlde commented 4 years ago

Issue Type: Bug

-> vscode-remote ssh into ubuntu remote machine -> docker-compose build [app] -> docker-compose up [app] -> CMD python -m ptvsd --host 0.0.0.0 --port 5678 --wait server.py -> attach debugger

I then expect the breakpoints I have defined in source (little red dots) to be stopped at. This does not happen. -> only 'breakpoint()' will be stopped at, then any red dot breakpoints will be honored in the same .py file

VS Code version: Code 1.41.1 (26076a4de974ead31f97692a0d32f90d735645c0, 2019-12-18T15:04:31.999Z) OS version: Linux x64 5.3.0-29-generic Remote OS version: Linux x64 4.15.0-72-generic

System Info

|Item|Value| |---|---| |CPUs|Intel(R) Xeon(R) CPU E3-1505M v5 @ 2.80GHz (2 x 2808)| |GPU Status|2d_canvas: unavailable_software
flash_3d: unavailable_off
flash_stage3d: unavailable_off
flash_stage3d_baseline: unavailable_off
gpu_compositing: unavailable_off
metal: disabled_off
multiple_raster_threads: disabled_off
oop_rasterization: unavailable_off
protected_video_decode: unavailable_off
rasterization: unavailable_off
skia_renderer: disabled_off
surface_control: disabled_off
surface_synchronization: enabled_on
video_decode: unavailable_off
viz_display_compositor: enabled_on
viz_hit_test_surface_layer: disabled_off
webgl: enabled_readback
webgl2: unavailable_off| |Load (avg)|0, 0, 0| |Memory (System)|7.78GB (2.78GB free)| |Process Argv|--no-sandbox --unity-launch| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: 10.0.1.252| |OS|Linux x64 4.15.0-72-generic| |CPUs|Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz (2 x 2500)| |Memory (System)|3.71GB (0.32GB free)| |VM|50%|

Extensions (8)

Extension|Author (truncated)|Version ---|---|--- vscode-docker|ms-|0.10.0 remote-containers|ms-|0.94.0 remote-ssh|ms-|0.48.0 remote-ssh-edit|ms-|0.49.0 remote-wsl|ms-|0.41.9 vscode-remote-extensionpack|ms-|0.19.0 vscode-docker|ms-|0.10.0 python|ms-|2020.1.58038

Environment data

VS Code version: Code 1.41.1 (26076a4de974ead31f97692a0d32f90d735645c0, 2019-12-18T15:04:31.999Z)
Extension version (available under the Extensions sidebar): 2020.1.58038
OS and version: Ubuntu 18.04
Python version (& distribution if applicable, e.g. Anaconda): 3.7.6 Alpine
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): docker from 3.7-alpine
Relevant/affected Python packages and their versions: tornado == 6.0.3
Relevant/affected Python-related VS Code extensions and their versions: vscode-docker|ms-|0.10.0 remote-containers|ms-|0.94.0 remote-ssh|ms-|0.48.0 remote-ssh-edit|ms-|0.49.0 remote-wsl|ms-|0.41.9 vscode-remote-extensionpack|ms-|0.19.0 vscode-docker|ms-|0.10.0 python|ms-|2020.1.58038

Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info microsoft/vscode-python#3977):

{
"python.testing.pytestArgs": [
    "tests",
    "--no-cov"
],
"python.testing.unittestEnabled": false,
"python.testing.nosetestsEnabled": false,
"python.testing.pytestEnabled": true,
"python.jediEnabled": true,
"python.linting.pylintEnabled": true,
"python.linting.enabled": true
}

Value of the python.languageServer setting: I'm not sure what this means?

Expected behaviour

I then expect the breakpoints I have defined in source (little red dots) to be stopped at.

Actual behaviour

if I do not include a 'breakpoint()' statement: The program executes as if there was no debugger attached (I do get output to the debug console) if I do include a 'breakpoint()' statement: The program stops at the first breakpoint() statement, then will honor other red-dot breakpoints only in the same file

Steps to reproduce:

-> vscode-remote ssh into ubuntu remote machine -> docker-compose build myapp -> docker-compose up myapp -> CMD python -m ptvsd --host 0.0.0.0 --port 5678 --wait server.py -> attach debugger

Dockerfile:

FROM python:3.7-alpine as base

COPY /src/requirements.txt /tmp/

RUN pip install -r /tmp/requirements.txt

RUN mkdir /opt/myapp
RUN ls -la /opt
RUN ls -la /opt/myapp

####### Start New Image : Debugger ####
FROM base as debug

RUN pip install ptvsd
RUN pip freeze | grep "ptvsd"

WORKDIR /opt/myapp

CMD python -m ptvsd --host 0.0.0.0 --port 5678 --wait --multiprocess server.py

docker-compose.yaml:

version: '3.4'
services:
  myapp:
    container_name: myapp
    image: 'myapp'
    build:
      context: ./
      target: debug
    ports:
      - 80:8090
      #- 443:8090
      - 5678:5678
    volumes:
      - ./src/myapp:/opt/myapp
    environment:
      - IP_NETWORK=10.0.1.0/24
      - SERVER_PORT=8090
      - PORTAL_SERVER_URI=http://10.0.1.209:80/myapi/endpoint
      - MYAPP-DEBUG=False
    restart: always

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python) The Output -> Python window contains no data

Output -> Log(Window)

[renderer5] [error] 'setBreakpoints' request must be issued after 'launch' or 'attach' request.: Error: 'setBreakpoints' request must be issued after 'launch' or 'attach' request.
    at t.RawDebugSession.handleErrorResponse (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2821:819)
    at file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2821:250
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at async R.sendBreakpoints (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5114:210)
    at async Y.sendToOneOrAllSessions (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5146:591)
    at async Promise.all (index 0)
    at async Y.sendAllBreakpoints (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5145:441)
    at async file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5120:850

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)


log.ts:196   ERR timeout after 500 ms: Error: timeout after 500 ms
    at t.RawDebugSession.handleErrorResponse (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2821:819)
    at file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2821:250
    at async t.RawDebugSession.shutdown (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2819:336)
log.ts:196   ERR 'setBreakpoints' request must be issued after 'launch' or 'attach' request.: Error: 'setBreakpoints' request must be issued after 'launch' or 'attach' request.
    at t.RawDebugSession.handleErrorResponse (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2821:819)
    at file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2821:250
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at async R.sendBreakpoints (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5114:210)
    at async Y.sendToOneOrAllSessions (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5146:591)
    at async Promise.all (index 0)
    at async Y.sendAllBreakpoints (file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5145:441)
    at async file:///usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:5120:850

eohlde commented 4 years ago

I am using Remote-Explorer to SSH into an Ubuntu 18.04 VM then trying to debug a docker container on that VM.

karthiknadig commented 4 years ago

@eohlde can you share your attach configuration?

eohlde commented 4 years ago

-edit: had an incorrect 'remoteRoot'- edit 2: I was correct the first time. Sorry, Monday morning, early, no coffee yet.

launch.json: I am using 'Python Attach"

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python Attach",
            "type": "python",
            "request": "attach",
            "justMyCode": false,
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/src",
                    "remoteRoot": "/opt"
                }
            ],
            "port": 5678,
            "host": "127.0.0.1"
        },
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal"
        },
        {
            "name": "Debug Tests",
            "type": "python",
            "request": "test",
            "console": "integratedTerminal",
            "justMyCode": false
        }
    ]
}

int19h commented 4 years ago

Can you try enabling logging for ptvsd, and share the logs? The command line switch is --log-dir ..., and it must be a path to some existing directory. A bunch of .log files should be created there; the one that I need is ptvsd.adapter*.log.

eohlde commented 4 years ago

Logs attached,

ptvsd-1.txt

fabioz commented 4 years ago

@int19h it seems that the issue is that the attach request is only sent after the setBreakpoints request (we need the attach first to get the path mappings), so, the adapter, is failing the request (which is correct for the debugger).

Now, I was reading https://microsoft.github.io/debug-adapter-protocol/overview and it seems it can receive setBreakpoints after the debugger sends the initialized event... right now we return the initialized event in response to the initialize event, but maybe we should return it in response to the attach or launch event? (if that's not the case, then the issue is in the client).

int19h commented 4 years ago

The DAP spec is unfortunately rather confusing on this. This diagram is much clearer on the relative ordering of "initialize", "initialized", "configurationDone", and "launch"/"attach".

But note that the adapter handles the initialization sequence itself, and it does so according to the diagram above - i.e. it only sends "initialized" after receiving "attach" or "launch". It also suppresses the "initialized" event from pydevd. So this should work properly.

I think the problem here is that this is ptvsd 4 on the server side, but PVSC is in the new-ptvsd experiment, and so assumes that it can just talk to it directly over a socket, without going through the old debug adapter (which took care of initialization order).

@eohlde, can you please try updating ptvsd in the container to the most recent pre-release (i.e. use pip install -U --pre ptvsd), and see if that resolves your issue? You can check the version number from within the app by printing ptvsd.__version__.

eohlde commented 4 years ago

edit: my home internet connection tanked. Thanks Comcast

I did receive: Error: unrecognized switch --multiprocess

so I changed my Dockerfile line to: CMD python -m ptvsd --host 0.0.0.0 --port 5678 --log-dir /opt/logs/ptvsd --wait ./server.py

Now when I try to attach with VSCode I get this:

The version of ptvsd in the container:

ptvsd==5.0.0a12

The version of ptvsd on the machine I'm ssh'd into (I don't know/think this has any effect):

ptvsd==5.0.0a12

The version of ptvsd on my machine:

ptvsd==5.0.0a12

int19h commented 4 years ago

@karthiknadig, can you think of anything that could possibly cause this? The error message looks like it's coming out of the debug adapter factory in PVSC, but how and why would either local or remote version of ptvsd affect that?

karthiknadig commented 4 years ago

I think something went wrong while debug adapter factory was being registered or when we trying to create the debug adapter factory. This error is possible when extension fails to activate fully.

eohlde commented 4 years ago

@int19h, upgrading to the pre released ptvsd fixes the problem. my debugger now stops at my defined breakpoints!

int19h commented 4 years ago

Perfect, thank you! I'll go ahead and close the issue.

microsoft / ptvsd