Utilize Docker Socket/TCP API Instead of Local Binary

[ibnesayeed]

Currently, lazydocker supports only local docker management by directly interacting with the locally installed docker CLI binary. This also means, running it inside a container (as per #36) would require packaging the CLI binary in it. By utilizing the HTTP API exposed by the Docker the tool can easily be configured to work with both local and remote Docker instances.

[jesseduffield]

Some initial thoughts on this: 1) When you say use the HTTP API, what are you referring to? Do you mean that we'll be sending HTTP requests from within lazydocker to the api? Or do you mean we'll be making use of the docker SDK? Or something else?

2) Using the docker CLI supports easy customisation. Right now if somebody wants to stop the logs from only going back an hour they just override the command template by removing the --since=60m like so:

commandTemplates:
  containerLogs: docker logs --follow {{ .Container.ID }}

How would a user go about customising these things if everything went through the api?

3) Is there anything that the CLI can do that the api cannot? I have used the docker SDK in a few places, however if I recall correctly there were some instances using that I could not get the same functionality as I could from the CLI command. (Worth noting I may be recalling incorrectly here)

If we could get this to work in a way where the user still has the same amount of configurability, I'd be all for it, but if not, I think packaging the binary may be the better solution. What are your thoughts?

[ibnesayeed]

I used HTTP API in a broader sense here, but I meant using the Docker SDK which is nothing but a wrapper in various languages around the HTTP API exposed by the TCP/Unix Socket. Docker CLI itself uses the same API underneath to talk to the Docker engine/daemon, so theoretically everything that the CLI can do should be possible using the API and perhaps some additional local request/response handling.

As far as --since is concerned, the HTTP API certainly has a parameter for that and other means to filter the log. Also, the Go SDK seems to have container log options too.

[jesseduffield]

I suppose in that case my only remaining concern is that things are going to get much more complicated both for contributors and for users who want to configure things. Instead of just swapping out a command template with one the user already used on the command line, they'll now need to deal with the schema of the SDK in order to make changes, and we'll need to write the code that parses the config values and sends them to the SDK

[jesseduffield]

also to confirm, is there any benefit to making the switch here beyond reducing the size of the docker container thanks to not having to include the docker binary?

[ibnesayeed]

Well, technically it is possible to call the CLI binary with any Docker daemon host (default is local socket) as it is now, by modifying command templates. However, I personally do not think it is the most user friendly way to use this tool. Command templates are global in nature, but one might want to apply different choices per container. Having good defaults, taking control of config parsing (instead of making users know CLI commands) and applying them on the fly would make this tool much more friendly. I would envision more features added to this on top of what is returned from the engine. For example, applying local filters on the buffer of various outputs (so that I can type anything to live filter only matching rows in logs/container/images/volumes/or anything for that matter) or choosing from a given set of options to apply on only one instance of the command (so that I can stream live logs of one container, but have a time boundary for another). I see this tool as Portainer for the terminal.

[ibnesayeed]

also to confirm, is there any benefit to making the switch here beyond reducing the size of the docker container thanks to not having to include the docker binary?

I am not too keen to reduce the size of the Docker image really (though it is a plus). Taking control of options eliminates the need for users to know every flag of docker CLI (though it comes with more work for this tool). I believe this tool can do a lot more without switching the context and without relying on a static set of command templates to run.

[jesseduffield]

I just did a scan of our current docker CLI uses and luckily we've only got two that are currently exposed to the user config:

            ContainerLogs:            "docker logs --timestamps --follow --since=60m {{ .Container.ID }}",
            ViewContainerLogs:        "docker logs --timestamps --follow --since=60m {{ .Container.ID }}",

The other ones all look like it wouldn't be too hard to swap them out for the client, however: 1) Would you agree that the docker-compose commands should remain as-is? I haven't customised any of the docker commands on my end but I indeed have customised my docker-compose commands, in fact my workplace uses its own program in place of docker-compose which only has a CLI interface. It might be a bit awkward as a user to have direct CLI command templates for docker-compose but specific flags for docker (though I don't really have an issue with it)

2) I now recall the things I found hard via the SDK: a) docker attach. If I recall correctly, with the SDK's docker attach you can access the stdout of a container, but not the stdin. I think we would need to solve this problem in order to make the switch to the SDK (though I think everything else is simple enough to change) b) docker stats. I currently have this:

func (c *DockerCommand) MonitorContainerStats() {
    go c.MonitorCLIContainerStats()
    go c.MonitorClientContainerStats()
}

Because the stats returned from the two are different. It may be that if we dug into it deeper we could get everything we need from the SDK but we'd need to investigate

[dawidd6]

Regarding point 2a, I think I could try to do it. Since I've already made something similar here. Granted it is ExecAttach, not Attach, but it seems to be almost the same.

[jesseduffield]

@dawidd6 sounds good :)

[skanehira]

@jesseduffield About docker attach you can reference this. https://github.com/skanehira/docui/blob/master/docker/container.go#L186

About docker stats you can reference this. https://sourcegraph.com/github.com/docker/cli/-/blob/cli/command/container/stats_helpers.go#L53

Also I developed docker.vim and that collect cpu/mem percent using api. https://github.com/skanehira/docker.vim/blob/master/autoload/docker/monitor.vim#L89

If you want to know collect logic please reference this. https://www.datadoghq.com/blog/how-to-collect-docker-metrics/

I hope lazydocker becomes better OSS :)

[jesseduffield]

@skanehira thanks for those links!