Error grabbing logs: invalid character '\x00' looking for beginning of value

[sabrine2020]

• [x] This is a bug report
• [ ] This is a feature request
• [x] I searched existing issues before opening this one

Expected behavior

reading logs from docker containers

Actual behavior

Can't read logs using:

docker-compose logs -f myservice

nor with:

docker logs -f 6f454c73ff9c

I am getting this error:

error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

Steps to reproduce the behavior

After some hours I can't read logs from my containers

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:18 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 11
 Running: 11
 Paused: 0
 Stopped: 0
Images: 8
Server Version: 17.09.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 76
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-97-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.695GiB
Name: egmintel-desktop
ID: VB22:IXWI:GY6D:QPM4:SPHX:HYUP:OQN7:ZM55:LLKE:P3UU:XK7F:26TH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

My docker-compose version: docker-compose version 1.16.1, build 6d1ac21

[degerrit]

Have a look at https://github.com/moby/moby/issues/35185#issuecomment-356738860

We used to have this issue but it was resolved years ago by no longer using external log rotation (logrotated) and using the (internal) docker log rotation provided by the "json-file" log-driver. It takes a while to work because it is only applied to new container, it seems.

[mrclschstr]

It seems that this problem does not only occur with log rotation. I have a VPS that has recently been affected by several host crashes at the hoster. After the crashes, the VPS was restarted automatically, but the log files were also corrupt afterwards.

I don't know how to tackle the problem without writing my own systemd service that periodically checks the logs.

[fukemy]

I've solved it using docker compose down docker compose up -d

worked to me

[XaratosS]

I've solved it using docker compose down docker compose up -d

worked to me

So the same error appears on normal docker container, which were not made with compose. How should we fix this?

[tompaz3]

I've solved it using docker compose down docker compose up -d

worked to me

So the same error appears on normal docker container, which were not made with compose. How should we fix this?

@XaratosS Most likely your log file is broken. You can verify where your problematic container stores logs on your host.

Find the file your container tries to write its logs to:

docker inspect --format='{{.LogPath}}' <container-name>

It should print you the log file path.

You can then search for lines in your file which are broken using for example this grep regex search:

grep -Pan '\x00' <log-file-path>

You can investigate these lines and remove them if you don't want to loose all the container logs. If you find your existing logs important, I recommend creating a backup before manipulating the log file.

Or if you don't care for your container logs, just reset the entire file content to empty:

cat /dev/null > <log-file-path>

[intech]

https://github.com/moby/moby/issues/29511#issuecomment-2051485559

[igloro]

I've solved it using docker compose down docker compose up -d

worked to me

Didnt work for me. in my case Docker restart works.

[declandragon]

Docker desktop still has this issue

docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:56 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 3.822GiB
 Name: docker-desktop
 ID: 78aeb0b9-27a8-4218-ab45-a11cb4edc226
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

docker compose version

Docker Compose version v2.26.1-desktop.1

screenshot

docker-compose.yml

# docker-compose up -d
version: '3'
services:
    bilihelper:
        container_name: bilihelper
        image: lkeme/bilihelper-personal:latest # my-app: x.x.x
        stdin_open: true # docker run -i 交互模式
        tty: true        # docker run -t 终端模式
        restart: always # 重启策略 / on-failure:5 重启次数5次 / always 一直重启 / unless-stopped 除非停止 / no 从不重启
        volumes:
            - ./profile/user:/app/profile/user  # /opt/bhp/profile/user替换为自己的路径
        network_mode: bridge # host
        ports:
            - "50001:50001" # 50001替换为自己的端口 user.ini->login_captcha->url
        environment:
            MIRRORS: 1 # 使用github镜像加速 docs/DOC.md#docker使用指南
            CAPTCHA: 1 # docs/CAPTCHA.md

[pooya-mohammadi]

I'm facing the following issue: error from daemon in stream: Error grabbing logs: invalid character 'l' after object key:value pair Is there any fix for this?

[mkywall]

I'm facing the following issue: error from daemon in stream: Error grabbing logs: invalid character 'l' after object key:value pair Is there any fix for this?

I also have this issue and then the computer freezes

[shareef945]

is there a permanent solution for this yet?

[Taruga]

is there a permanent solution for this yet?

I've added: "log-driver": "local" to the Docker Engine config. Since yesterday, I'm not seeing the error.

From the documentation , It looks like Docker uses default json-file to remain backward compatible with older docker versions. But json-file logging has no log-rotation compared to local.

More info about it: https://docs.docker.com/config/containers/logging/configure/

[Taruga]

After some days without errors, today I got this new error:

error from daemon in stream: Error grabbing logs: log message is too large (1937007717 > 1000000)

Looks like it was caused by the: "max-size": "100m" I added to the Docker Engine config. Will remove the log-opts part from the Docker Engine config and test again

"log-opts": {
   "max-size": "100m"
}

My curremt config:

[pablomendezroyo]

More than providing solutions could someone explain why this is happening?

[martinbaste]

Docker is writing logs to a file in json format (each line it’s own json object). If the process writing logs is stopped abruptly, it can write null bytes to the log, so the JSON parser errors out. Sometimes the process partially writes a log line before being killed, so the json line is unfinished and invalid.

At least in my case it happens when a raspberry pi running docker loses power, and not always.

A proper solution would be the parser just ignoring the invalid line, or deleting it from the log file.

On Wed, Jul 24, 2024 at 16.12 pablomendezroyo @.***> wrote:

More than providing solutions could someone explain why this is actually happening?

— Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/140#issuecomment-2248080448, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJCH4NFJNTXZNHWAM7LTY3ZN6Y4PAVCNFSM4D74AYN2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMRUHAYDQMBUGQ4A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[dexter74]

Edit file wih gedit

Type de document: Yaml Registration It's work

[Buggem]

fix issue plezzzz its bugged

[cpuguy83]

This issue should be partially fixed in the latest version of the engine. When the log reader finds corrupted data the rest of the log file is now discarded and moves on to the next file rather than erroring out.

There's not really anything docker can do to prevent null bytes from being written to the log file since that's happening due to the filesystem not doing full journaling by default (for performance reasons).

In the future we may enhance the local log driver to be able to delineate messages better in case of corrupted writes so instead of having to discard the rest of a file we only discard the corrupted bits but there is no one working on this AFAIK. The json driver would have to be opt-in to use such a change due to back compat.

[cpuguy83]

For reference, here is the change: https://github.com/moby/moby/pull/47983

[cpuguy83]

Actually this was not released in the latest release, it will go into v28.

[cpuguy83]

I'm going to go ahead and close this since the reported issue should not be able to be triggered anymore.

[Buggem]

I'm going to go ahead and close this since the reported issue should not be able to be triggered anymore.

@cpuguy83 congrats on achieving nothing. It's still here.

[woj-tek]

I'm going to go ahead and close this since the reported issue should not be able to be triggered anymore.

Why it was closed even though it's not (seemingly) released?

I just tried to figure out why one of the services was acting up and got greeted with the lovey:

$ sudo docker compose logs --since=30m -tf
error from daemon in stream: Error grabbing logs: invalid character '\x00' in string literal

If the issue is closed then either the fixed version should be released (and indicated which version has the fix) or provide a solution/workaround (switching logging engine?)

[cpuguy83]

@buggen such comments are uncalled for and the behavior is not acceptable on this repo.

@woj-tek The issue is closed because it is fixed. We do not keep issues open just because the fix is not in a release yet.

[woj-tek]

The issue is closed because it is fixed. We do not keep issues open just because the fix is not in a release yet.

Would it be possible to at least provide information in which version it will be released?

(in this particular case I find githubs nice simplicity actually as a shortcoming... it would be nice to have something akin to "release" that follows "fixed" :) )

[cpuguy83]

I mentioned this just before closing: https://github.com/docker/for-linux/issues/140#issuecomment-2380888200

This will be in the v28 release.

[Buggem]

@Buggen such comments are uncalled for and the behavior is not acceptable on this repo.

@woj-tek The issue is closed because it is fixed. We do not keep issues open just because the fix is not in a release yet.

Ok, sorry and please release it soon. FYI, you pinged a random person instead of me. (Buggen)

[Buggem]

The latest version hasn't fixed this. I am on Docker Desktop 4.35.1. I know this is the repo for the CLI but alas I would like help on how to update the CLI/engine.

[cpuguy83]

I've opened a backport to hopefully get this into the next 27.x release. https://github.com/moby/moby/pull/48842