Docker Engine v 25.0 API now returns virtualSize null when enumerating Images, which means containerized shinyproxy cannot start containers after docker engine upgrade to v 25.0

[mskyttner]

2024-01-19 Docker Engine v 25.0 was rolled out, see https://docs.docker.com/engine/api/version-history/ with some API deprecations that break backward compatibility, for example:

"The VirtualSize field in the GET /images/{name}/json, GET /images/json, and GET /system/df responses is now omitted. Use the Size field instead, which contains the same information."

If shinyproxy runs containerized and the docker engine is upgrade, the result can be that shinyproxy report for example "Container with index 0 failed to start".

I think it is because it seems https://github.com/spotify/docker-client/blob/master/src/main/java/com/spotify/docker/client/messages/Image.java#L76 will from now on be null which would mean that containers cannot start since the following would normally occur:

"eu.openanalytics.containerproxy.ProxyFailedToStartException: Container with index 0 failed to start"

shinyproxy can log for example:

Caused by: eu.openanalytics.containerproxy.ContainerFailedToStartException: Docker container failed to start
shinyproxy-1  |     at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.startContainer(DockerEngineBackend.java:153) ~[containerproxy-1.0.2.jar!/:1.0.2]
shinyproxy-1  |     at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:108) ~[containerproxy-1.0.2.jar!/:1.0.2]
shinyproxy-1  |     ... 8 common frames omitted
shinyproxy-1  | Caused by: com.spotify.docker.client.exceptions.DockerRequestException: Request error: GET unix://localhost:80/images/${IMAGE_SLUG_HERE)/json: 200
shinyproxy-1  |     at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2840) ~[docker-client-8.16.0.jar!/:8.16.0]
shinyproxy-1  |     at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2712) ~[docker-client-8.16.0.jar!/:8.16.0]
shinyproxy-1  |     at com.spotify.docker.client.DefaultDockerClient.inspectImage(DefaultDockerClient.java:1509) ~[docker-client-8.16.0.jar!/:8.16.0]
shinyproxy-1  |     at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.isImagePresent(DockerEngineBackend.java:243) ~[containerproxy-1.0.2.jar!/:1.0.2]
shinyproxy-1  |     at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.startContainer(DockerEngineBackend.java:81) ~[containerproxy-1.0.2.jar!/:1.0.2]
shinyproxy-1  |     ... 9 common frames omitted
shinyproxy-1  | Caused by: javax.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `com.spotify.docker.client.messages.ImageInfo`, problem: Null virtualSize

I discovered this on Friday when existing containerized shinyproxy services stopped being able to create containers.

[mskyttner]

It seems that if I roll back Docker Engine to use API v 24, things are working again. This varies by platform, but I used this command to see which engine versions were available:

apt-cache madison docker-ce | awk '{ print $3 }'

I then picked the latest one that was still lower than v 25:

VERSION_STRING=5:24.0.0-1~ubuntu.22.04~jammy
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

The containerized shinyproxy services are now running again, which seems to confirm that shinyproxy v 3.0.2 does not (yet) speak docker engine v 25 engine API (v 1.44) (?)

[LEDfan]

Hi Thanks for reporting this. As part of the next release, we planned to use an up to date fork of the Spotify docker client. For example, https://github.com/dmandalidis/docker-client/ . We will test whether this versions works with the latest docker version, or otherwise create a PR to fix it. So this issue will definitely be solved in the next release.

[mskyttner]

That library also seems to require a patch from the same reason it seems as it currently would not be able to enumerate images without a null pointer exception if docker engine v 25 or later is in play, judging from the latest PR there.

I guess it means that any systems that (automatically?) upgrade their docker engine would likely see that any shinyproxy services would stop working (when reading from the engine API over a unix socket ie docker engine api v 1.44).

So pinning or holding the docker packages is how I try to work around the issue currently. For example, on a linux system (ubuntu focal w docker ce):

# use commands such as these to discover available docker engine package versions on the host
apt policy containerd.io docker-ce docker-ce-cli docker-compose-plugin docker-buildx-plugin docker-ce-rootless-extras 
apt-cache madison docker-ce

# then create three files to pin the docker packages on the host 
# to the latest working version before v docker engine v 25

more /etc/apt/preferences.d/docker*

::::::::::::::
/etc/apt/preferences.d/docker-ce
::::::::::::::
Package: docker-ce
Pin: version 5:24.0.7-1~ubuntu.20.04~focal
Pin-Priority: 1001

::::::::::::::
/etc/apt/preferences.d/docker-ce-cli
::::::::::::::
Package: docker-ce-cli
Pin: version 5:24.0.7-1~ubuntu.20.04~focal
Pin-Priority: 1001
::::::::::::::
/etc/apt/preferences.d/docker-ce-rootless-extras
::::::::::::::
Package: docker-ce-rootless-extras
Pin: version 5:24.0.7-1~ubuntu.20.04~focal
Pin-Priority: 1001

[jkh1]

I have a VM with docker v25.0 and two instances of shinyproxy, one v3.0.2 and one v2.6.1. The former doesn't work with the error reported above while the later runs fine. So a temporary solution could be to downgrade either docker or shinyproxy.

[syedalizaib]

I was facing the same issue on RHEL 9. Rolling back the docker version to 24.x.x worked for me. 👍

sudo yum remove docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras

sudo rm -rf /var/lib/docker

sudo rm -rf /var/lib/containerd

sudo yum install -y yum-utils

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce-3:24.0.7-1.el9 docker-ce-cli-1:24.0.7-1.el9 containerd.io docker-buildx-plugin docker-compose-plugin

sudo systemctl start docker

[jkh1]

Or in my case, simply :smiley:

sudo yum downgrade docker-ce-24.0.9-1.el8 docker-ce-cli-24.0.9-1.el8
sudo systemctl restart docker

[ChristianRohde]

Hi, I think I have the same problem. I already downgraded from Docker v25 to Docker version 24.0.9, build 2936816. So far no docker image was loading in shinyproxy, although I am able to run it from docker command line without any problem. Please find my log attached. shinyproxy.log

How can I check if shinyproxy connects to docker? I have no idea how to continue troubleshooting otherwise.

Best, Christian

[mskyttner]

I think you can see events in docker with docker system events command so that log should come to life as shinyproxy starts up, I guess.

Not sure, but it could potentially be something else? For " java.lang.IllegalArgumentException: Schema specific part is opaque", a search turns up an old post where it seems like an ip:port setting is causing an error like that if used where an setting starting with the http_scheme is expected (http|https)... For example the setting is "{some_ip}:443" instead of "https://something.somewhere".

Perhaps the application.yaml file could give more clues and maybe there is someone over at the support forum that might have an idea.

[eastclintw00d]

@LEDfan Can you please give a status update on this issue? Will there be a solution for the docker v25 engine, anytime soon?

[tverbeke]

@LEDfan Can you please give a status update on this issue? Will there be a solution for the docker v25 engine, anytime soon?

@eastclintw00d: we are working hard on a new release that will - amongst many other things - include a solution. If you want predictable times for issues to be fixed, you can always subscribe to a ShinyProxy support contract.

[LEDfan]

Hi, this is now part of the ShinyProxy 3.1.0 we released today. Thanks again for reporting this issue!