Error "connection reset by peer" after docker load

[lucapatarnello]

Issue description

Having issues following the hello-world-python guide, at the "Install the image step", we run the command:

docker save $APP_NAME | docker --host tcp://$DEVICE_IP:$DOCKER_PORT load

And we get the following error:

error during connect: Post "[http://DEVICE_IP:DOCKER_PORT/v1.41/images/load?quiet=0":](http://DEVICE_IP:DOCKER_PORT/v1.41/images/load?quiet=0%22:) write tcp 10.0.0.4:43498->DEVICE_IP:DOCKER_PORT: write: connection reset by peer 

I've correctly set the DEVICE_IP and DOCKER_PORT with those of my device. The connection works fine running the command of the previous step:

docker --host tcp://$DEVICE_IP:$DOCKER_PORT system prune --all --force

As we get the following output:

Total reclaimed space: 0B 

Any clue on how to solve this? Thanks

Environment

• Axis device model: Q1656-LE camera
• Axis device firmware version: 11.4.63

[Corallo]

Hello @lucapatarnello

Do you have an SD card installed in the device? Is the docker acap app using the SD card? Even the hello-world example might be too large without an SD card.

Check the system log and the docker acap log to get more information about what's going on. (you can do it from the web gui).

[lucapatarnello]

Hi @Corallo

As you can see from the screenshot below, the SD card is mounted, used by docker and with plenty of space available

The logs do not report any information regarding the error that we are getting.

[Corallo]

Strange, are you checking the log from the app log? (pressing the three dots next to the application -> app log)

Pulling an image directly to the camera works? docker -H <camera_ip>:<docker_port> pull ubuntu

[lucapatarnello]

We cannot perform the image pull as the camera is not connected to the internet, all the files are being provided by a VM, which has access to the camera. We also tried to copy the saved .tar of the image and to transfer it to the camera using scp as following:

scp -P $SSH_PORT image.tar root@$CAMERA_IP:/root/image.tar

But we get the following error:

image.tar: Permission denied

Could it be that we do not have writing permissions on the camera?

[Corallo]

Can you try coping it in /tmp/image.tar

[lucapatarnello]

The permission denied error persists. By running "chmod -R 777 /tmp" it says that it is a read-only file system, same goes for other directories.

[Corallo]

Your camera must be in a bad state. I think the best solution might be a factory default. Try to install the latest firmware (11.5) and in doing so select factory default From the GUI Maintenance -> Firmware upgrade -> Upgrade -> Factory default

[lucapatarnello]

We updated the firmware to Q1656-LE_11_5_64.bin and docker to Docker_Daemon_1_4_1_aarch64_signed.eap. We cannot perform a FACTORY DEFAULT as we would lose important ip settings configuration. We opted for a RESTORE.

We noticed that by running shell docker -H <camera_ip>:<docker_port> pull ubuntu the event that the camera cannot pull the image as it is not connected to the internet is registered in the camera logs (docker daemon app log).

Still, the command shell docker save $APP_NAME | docker --host tcp://$DEVICE_IP:$DOCKER_PORT load returns the error reported at the beginning of this issue. Since the request does not reach the camera, nothing is registered in its logs this time.

One thing we did not mention is that, the network to which the camera is connected is protected and only reachable by white-listed IPs. The VM's IP is white-listed and working as we've tested running telnet commands.

[Corallo]

After the restore, do you still have the permission issue when running? scp -P $SSH_PORT image.tar root@$CAMERA_IP:/tmp/image.tar Are you sure the Permission denied is referring to the destination /tmp/image.tar on the camera and not the source?

Might it be that there's a size limit of the file that you can upload to the camera? (imposed by your network settings)

Let's try to tunnel the docker communication using ssh: In one shell, run sudo ssh -L 8080:<camera_ip>:<docker_port> root@<camera_ip> Then open another shell and try again with: docker save $APP_NAME | docker --host tcp://$DEVICE_IP:8080 load

(if 8080 is used by something else, pick another port)

[lucapatarnello]

scp worked and we were able to copy the .tar image in the tmp folder, but that is not useful as when connecting to the camera via ssh docker is not available. Indeed, running any docker command from the camera when connected via ssh results in docker: not found. We also tried ssh tunnelling but results in a connection timeout.

[lucapatarnello]

Some additional info, just to double check we run the command

docker --host tcp://$CAMERA_IP:$DOCKER_PORT info

We obtained an output, confirming that there is a connection:

Client:
Context:    default
Debug Mode: false
Plugins:
  buildx: Build with BuildKit (Docker Inc., 0.6.3+azure)

Server:
Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
Images: 0
Server Version: dev
Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 579a6380ec93ab92a6e7f26167fe4f18dfcf2a4b.m
runc version: v1.1.3-2-g1e7bb5b7-dirty
init version: de40ad0
Security Options:
  apparmor
  seccomp
   Profile: default
Kernel Version: 5.15.13-axis9
Operating System: AXIS OS (based on Poky from the Yocto Project) 11.5.64 (langdale)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 959.5MiB
Name: axis-b8a44f32d713
ID: BLOW:BXPF:ZW54:AKBD:HHTK:YYUW:YRMC:DXJV:QP62:EGMY:IGU6:LO52
Docker Root Dir: /var/spool/storage/SD_DISK/dockerd/data
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
  127.0.0.0/8
Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375/ without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/

So the connection to docker daemon works, the info command works, but the load command doesn't. Could it be some firewall rule on the docker port interfering with the load command?

[Corallo]

Yes, you are probably facing a network issue that is blocking the file transfer in that port. You can try with the docker compose acap Differently from the docker acap it should allow you to run docker command directly from the device.

[lucapatarnello]

We did as you suggested and we finally made it work, thank you. The question we have now, is there a way, after transfering the image via scp, to launch the load command from the VM instead of having to connect via ssh to the camera and then launching it from the device itself?

We tried running the following command (the path you'll see is where we placed the image):

sudo docker -H $CAMERA_IP:$DOCKER_PORT load -i /var/spool/storage/SD_DISK/images_tar/image.tar

But it cannot find the file. Any suggestion on how we could do it?

[Corallo]

@lucapatarnello It doesn't find the file because even if it is executing the load command remotely, docker looks for the image.tar in your local file system. As far as I know, it is not possible to run the docker load command from your VM giving a path in the camera file system.

The correct way to run this command from remote is the one described in the instruction, you will have to do some debugging of your network to find out the problem that prevent to send the docker image via the docker port.

We can't help you with that, as we can't reproduce your issue.

Feel free to get in touch again if you have more questions

[lucapatarnello]

Thank you for helping us figure it out. I'm closing the issue.