[Docker Labs Debug Tools] Issues/Bugs

[joe0BAB]

Tell us about your request The Docker Labs Debug Tools extension is an experimental feature to improve debugging with docker. It currently features:

• shell: Get a shell into any image or container, local or remote. An enhanced replacement of the docker exec with Nix based package manager.
• rdb: Easy remote debug setup for containers running a Go/C/C++/Rust binary (gdb and delve support)
• attach: Like the shell command but specialized for running containers.

For any issues or bugs, please report here.

Which service(s) is this request for? Docker Labs Debug Tools extension

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? This is a placeholder to collect issues and bugs.

Are you currently working around the issue? N/A

Additional context N/A

[justincormack]

bug - stops the new Docker Desktop sleep feature working as there is a persistent running container in the extension.

[justincormack]

bug / feature request - PATH does not have the original container path, just the path to the installed tools, I really need both available (unsure about optimal ordering, probably installed tools first though).

[justincormack]

bug - initial working directory is not the default working directory of the container - I expect it to be like exec in this case, as in some containers used to ending up eg where the app is installed.

[tminich]

bug - On Windows, installation hints are wrong. The extension asks you to add C:\Users\<user>\AppData\Local\Docker Labs Debug Tools\bin to your path. That is an empty folder. Either there is a symlink/copy step missing to get dld.exe there or the correct folder C:\Users\<user>\AppData\Roaming\Docker\extensions\docker_labs-debug-tools-extension\host should be explained to add to your path.

[joe0BAB]

@tminich thanks for reporting the issue! I could reproduce and just released 0.37 which should fix it. Could you update the extension and retry? (you might need to restart docker desktop before so the extension cache gets refreshed and you see the update)

With 0.37 installed, you should see a 'installed' label in the top right corner of the extension ui tab.

[tminich]

Worked, thanks!

[joe0BAB]

@justincormack

bug - initial working directory is not the default working directory of the container - I expect it to be like exec in this case, as in some containers used to ending up eg where the app is installed.

This should be fixed in version 0.38 published yesterday. Would you mind re-testing if that solves the issue?

[andrew-lozoya]

MacOS 14.0 Beta (23A5328b) - arm64 (M1) zsh 5.9 (arm-apple-darwin21.3.0)

Unable to use the debug tool:

Failed to install the CLI binary. Error: code: 1, stdout: exit status 1 , stdout: Error: exit status 1 Usage: install install [command] Available Commands: check Check if dld is installed and reachable from $PATH completion Generate the autocompletion script for the specified shell fix Fix the $PATH variable to include ~/.local/bin help Help about any command uninstall Remove ~/.local/bin/dld Flags: -h, --help help for install Use "install [command] --help" for more information about a command.

Status: not installed 0.0.41

ZSH has all the correct Paths

fpath=(~/.local/share/zsh/functions $fpath)
autoload -Uz compinit
compinit -u

export PATH="$PATH:/Users/alozoya/.local/bin" # Added by Docker Labs Debug Tools"

Nothing is added or installed to this location related to dld. Please advise...

[joe0BAB]

Thanks @andrew-lozoya for reporting the issue! I need a bit more context to reproduce/understand the problem. Could you tell me the output of docker context ls and ls -l /var/run/docker.sock (this file might not exist but then I need to know that instead)?

[joe0BAB]

@justincormack

bug / feature request - PATH does not have the original container path, just the path to the installed tools, I really need both available (unsure about optimal ordering, probably installed tools first though).

This should be fixed in version 0.0.41. The original container/image $PATH is appended for now (maybe there needs to be a convenient way to dynamically change the order).

[jamesaburger]

Install on Windows 11.
Verified path env variable is set correctly. Contents of that directory is solo file dld.exe

Running with no arguments shows usage info. So, OK

Running with version argument produces: Docker Labs Debug Tools Extension Version: 0.0.42 Commit: 5fe3c6850763b83e33a5f81cd2c9135c4ad287d3 So OK

Running with any other arguments produces: message "Seems like the extension is not installed. Open Docker Desktop and install the 'Docker Labs Debug Tools' extension to continue."

Uninstall and Re-install of extension, restart of docker desktop. No improvement. Any help is appreciated.

[joe0BAB]

@jamesaburger thanks for reporting the issue! I could reproduce it and already pushed a fix release. Could you update to version 0.0.43 an re-try? (you might need to restart Docker Desktop to see the update)

[jamesaburger]

Thanks for the rapid turn. Issue resolved. Cheers

[tminich]

It seems like I cannot use dld with the container used by Dagger for it's engine. I get

Error: unexpected end of JSON input

when I try dld attach or dld shell

[joe0BAB]

@tminich could you give more context? Which exact command were you running? I'm guessing you were using the --host flag. If so, what's the output of docker info on that machine? While it's possible to use--host against any socket, so far it's tested only with docker-ce+runc runtime.

[tminich]

@tminich could you give more context? Which exact command were you running? I'm guessing you were using the --host flag. If so, what's the output of docker info on that machine? While it's possible to use--host against any socket, so far it's tested only with docker-ce+runc runtime.

Sure. In this case I wasn't working remotely, this was local on my development machine running Docker Desktop. The 'target' service is the one automatically provisioned by the dagger command to run the ci pipeline. Basically I was only using dld attach dagger-engine-5097823f019aa092 (or dld shell ..., tried both)

[joe0BAB]

Interesting, any chance you could give some commands/instructions to try reproduce that on my machine? (maybe that's obvious but I've never used dagger)

[tminich]

I've just started to use it yesterday myself =) But in general, I think the basic introductory quickstart example should work. It provisions the engine service and leaves it running. If you have nix in your environment, it provides the binary (using that myself in WSL) as a package.

As a side note, the regular shell is accessible in Docker Desktop.

[joe0BAB]

I'll check! (might take a bit until I get to that) Thanks for reporting!

[tminich]

I'll check! (might take a bit until I get to that) Thanks for reporting!

Thanks, no rush. I'm mostly interested in what's going wrong there =)

[tminich]

Sorry to shove another issue on your plate. I'm having trouble with dld shell and containers I created via nix' buildLayeredImage. Not exactly sure what triggers it, maybe the amount of Layers? The error message I'm getting is

Error: overlay mount to /tmp/279bff3e-76dd-4146-b6c4-b9642eaacf5b/lower: mount options is too long

Here is a (somewhat) minimal flake file to create an image with this issue:

{
    inputs = {
        nixpkgs.url = "github:nixos/nixpkgs";
        systems.url = "github:nix-systems/x86_64-linux";
        flake-utils = {
            url = "github:numtide/flake-utils";
            inputs.systems.follows = "systems";
        };
    };

    outputs = { self, nixpkgs, flake-utils, ... }: 
        flake-utils.lib.eachDefaultSystem (system: 
            let 
                pkgs = nixpkgs.legacyPackages.${system};
                dt = pkgs.dockerTools;
            in {
                packages.default = pkgs.dockerTools.buildLayeredImage {
                    name = "hello";
                    tag = "latest";

                    contents = with pkgs; [
                        dt.usrBinEnv
                        dt.binSh
                        dt.caCertificates
                        dt.fakeNss
                        coreutils
                        nginx
                    ];

                    config.Cmd = [ "${pkgs.hello}/bin/hello" ];
                };
            }
        );
}

If you remove nginx from the contents, the issue goes away. Also if the image is running (in case of the one I'm building), dld attach works fine.

[joe0BAB]

This is very interesting! It makes me wonder if some architectural changes might be needed to the way we currently create the shell for images. The thing is there are like 10 ways to build a feature like this, but they don't tell you which is the right one.. Let me dig into this and get back to you (might take me a bit). Thanks for the comprehensive example!

[tminich]

You're welcome =) A minor thing I noticed, not sure if that is intentional or not. If I connect to a running nix-based container created in this way with dld attach, /bin is not in the path. Using the regular docker shell (e.g. in docker desktop) it is.

[jbonnier]

I'm always using my WSL terminal, never Powershell or other options in the windows file system. However, after installing the extension, when I type dld in my WSL Ubuntu I get Command 'dld' not found, did you mean: ... How can I fix that?

[CostaRica]

Hi. I have Docker Desktop 4.23.0 running on top of Windows 10 + WSL 1.2.5.0 + Ubuntu 22.04.2 LTS. I have installed and uninstalled the extension multiple times to try to get it to work, but when I run dld in my WSL/Ubuntu terminal, it just says "Command 'dld' not found, did you mean:" and a list of other binary names. The Troubleshooting tab says that "The dld binary has been successfully installed at ~/.local/bin/dld", but my user's home does not have a .local folder, and neither does root. docker images does show that docker/labs-debug-tools-extension 0.0.43 has been pulled. Any tips on where to look next?

[jbonnier]

Hi. I have Docker Desktop 4.23.0 running on top of Windows 10 + WSL 1.2.5.0 + Ubuntu 22.04.2 LTS. I have installed and uninstalled the extension multiple times to try to get it to work, but when I run dld in my WSL/Ubuntu terminal, it just says "Command 'dld' not found, did you mean:" and a list of other binary names. The Troubleshooting tab says that "The dld binary has been successfully installed at ~/.local/bin/dld", but my user's home does not have a .local folder, and neither does root. docker images does show that docker/labs-debug-tools-extension 0.0.43 has been pulled. Any tips on where to look next?

I'm always using my WSL terminal, never Powershell or other options in the windows file system. However, after installing the extension, when I type dld in my WSL Ubuntu I get Command 'dld' not found, did you mean: ... How can I fix that?

I reinstalled it on another computer and saw the command to add it to my path. Unfortunately, it's a command for the Windows path (using setx).

But I did add it to my path on WSL and it works well. Note that I added a dld alias in order to be able to use dld and not dld.exe in my linux terminal.

Also note that my Windows username is jbonn so adapt it for your user.

cat >> ~/.bashrc <<'EOT'
export PATH="/mnt/c/Users/jbonn/AppData/Local/Docker Labs Debug Tools/bin:$PATH"
alias dld='dld.exe'
EOT

[whalelines]

bug — On macOS, m2 MacBook Pro using Homebrew installed bash (/opt/homebrew/bin/bash) as my shell, I get

$ ~/.local/bin/dld shell reg/namespace/repo:tag
Failed to detect shell: reading exe path of PID 9064: missing txt data returned by lsof. Defaulting to zsh.
…

Docker Debug info:

$ ~/.local/bin/dld version
Docker Labs Debug Tools Extension
Version:    0.0.43
Commit: 556e9bb7051de6bd79be842265773cff487a0702

[joel-u410]

I'm not sure what happened, but dld stopped working for me today. It might be related to updating Docker Desktop.

Now, dld fails to run a shell or attach to a container, I only see a message like this:

$ dld shell c43a1e72ad25
Failed to detect shell: reading exe path of PID 84102: missing txt data returned by lsof. Defaulting to zsh.
Error: chown /tmp/845ca3d8-aa1e-4673-bf55-249ba9e7191f/tmp-home/nix/tmp/debug-tools/.nix-profile: no such file or directory

$ dld shell c43a1e72ad25 --shell bash
Error: chown /tmp/b79f6d80-8c34-4920-9615-a2a094bf850f/tmp-home/nix/tmp/debug-tools/.nix-profile: no such file or directory

I believe I have the latest version (also tried deleting / reinstalling as suggested in the troubleshooting page):

$ dld version
Docker Labs Debug Tools Extension
Version:    0.0.43
Commit: 556e9bb7051de6bd79be842265773cff487a0702

Docker Desktop version:

You're up to date
Docker Desktop 4.24.2 (124339) is currently the newest version available.

Is there anything I can do to gain more visibility into what's happening there? dld seems like a bit of a black box.

[whalelines]

bug — dld does not work with Docker Desktop 4.25.0 when the containerd backend is enabled. It seems the location of the containerd sock has changed in the underlying DD VM.

[whalelines]

I'm not sure if this is technically a bug, but since dld is always running a container, it effectively disables resource saving mode in Docker Desktop. Is there a way to only start the dld container when it is needed, or let DD know not to consider the dld container when determining whether to sleep or not?

[joel-u410]

I'm not sure what happened, but dld stopped working for me today. It might be related to updating Docker Desktop.

Update: I just realized that uninstalling/reinstalling the dld binary was not the same thing as uninstalling/reinstalling the extension itself. Just did that, via the Extensions tab, and now it's working again.

It would be helpful, if possible, for the "re-install the extension" text to link to the relevant Extensions page where that can be done.