Docker scout huge memory consumption

[ykazakov]

Description

After updating to the latest docker, I noticed that my system (MacBook Pro M1, 16 GB) runs out of working memory when building docker images. Checking Activity Monitor reveals a huge memory consumption by the process docker-scout:

This process starts shortly after the image is built, so it seems like docker-scout tries to analyse the built images. However I do not think I ever enabled docker-scout. Actually, I was not even aware about its existence. The docker scout configuration is empty, however there is clearly something in the cache:

% docker scout config       
Configuration empty. See docker scout config --help for details.
% docker scout cache df
...
Total: 5.0 GB

Other commands like docker scout quickview ask me for my Docker ID, which I do not have.

Please, make it possible to disable docker-scout completely!

Reproduce

1. Create a Dockerfile:

   
   FROM jupyter/minimal-notebook:2023-09-18 AS test

RUN touch /tmp/test

2. Run `docker build -t test .` from the same directory
3. Observe the memory consumption spikes *after* the image has been built:
<img width="344" alt="Screenshot 2023-09-21 at 09 51 25" src="https://github.com/docker/for-mac/assets/2140361/e2c24d40-1e7c-4229-b4ef-180f82ce45e6">

### Expected behavior

I expect that Docker should not start any new processes (or increase the memory consumption) after the image is built but not used.

### docker version

```bash
Client:
 Cloud integration: v1.0.35+desktop.4
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.4
    Path:     /Users/username/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0-desktop.1
    Path:     /Users/username/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/username/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/username/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.7
    Path:     /Users/username/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/username/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/username/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.24.1
    Path:     /Users/username/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.3.13-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 5.8GiB
 Name: docker-desktop
 ID: eb28bdcf-66a4-4963-bbcf-c58cf711d1a9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

6734B03B-C87B-4461-884C-139350A00371/20230921080327

Additional Info

Interestingly, I did not notice docker-scout starting if I use other base docker images such as ubuntu:latest, debian:latest or alpine:latest. Is this behaviour image/organization-specific?

[ykazakov]

The problem is exacerbated if several images are built at once, e.g., using docker compose. For example for this Dockerfile:

FROM jupyter/minimal-notebook:2023-09-18 AS test1

RUN touch /tmp/test

FROM test1 AS test2

RUN touch /tmp/test

FROM test2 AS test3

RUN touch /tmp/test

and this docker-compose.yaml:

version: '3'

services:

  test1:
    image: test1
    build:
      target: test1

  test2:
    image: test2
    build:
      target: test2

  test3:
    image: test3
    build:
      target: test3

after running docker compose build, two docker-scout processes start and memory consumption almost doubles:

I had a similar situation in my first screenshot.

Furthermore, when all system memory is consumed, docker crashes, leaving some temporary files in /private/var/folders/, which, in my case occupied 20+ GB. First I did not notice this and my system ran out of disk space.

[bsousaa]

You can disable the SBOM background indexer under Settings > Features in development > Experimental features > Enable background SBOM indexing

[ykazakov]

Thanks @bsousaa, that worked! Finally the memory is under control! Did this feature appear in the last version? Please consider disabling SBOM by default.

[bsousaa]

Thanks @ykazakov, I will pass the feedback to the team

[rfay]

Just had this disable a macOS machine. Why are experimental features enabled by default?

[MattJustMatt]

This absolutely destroyed my 8GB MBA (I know... my fault!) but would be good to have this disabled by default!

[GautamBose]

@MattJustMatt Same - knew this machine had little memory but have rarely seen it slow to the absolute crawl this feature caused!

[dmg0345]

Just to add that it also occurs in Docker Desktop for Windows, the experimental feature is activated by default and the same workaround as proposed here fixes it.

[cameron-hobbs]

Same I had this issue in Docker Desktop for Windows too

[NailKhalimov]

I've just installed the latest version of Docker Desktop 4.25.0 and this feature SBOM indexing has been enabled by default😒 Docker team, please disable it by default🙏 On my MacBook m1 building time increased 10x times! I've waited to make a build for around 35 minutes instead of 3 after I disable it.

[daneverson]

This is no longer found under "experimental features" but is still called "SBOM indexing" under the general settings tab. Until I disabled it, 2 docker-scout processes were consuming all 24GB of memory on my M2 macbook, and I had to shut it off to get any work done.

I agree that this should be disabled by default.

[bn-l]

Same issue here. Is docker desktop a botnet?

[Ericwln]

Dis enable SBOM indexing ？

[ykazakov]

I just updated to docker desktop 4.25.2 (129061) to observe that SBOM indexing has been reenabled after I manually disabled it in the settings:

What is going on??

[mmiguerodriguez]

➕ on this issue. Had the same problem when building a simple postgres image. Memory consumption going to its limits with the container unused. Lots of memory usage also happening with docker-scout process.

[nullbtb]

This was using 15GB of RAM out of a 16GB Intel Mac on 4.25.0. This needs attention.

[alexbalchin]

For anyone still finding this issue, the setting in Docker Desktop has now moved to Settings > General > SBOM Indexing (tested on 4.25.4).

[jay-depot]

Chiming in here:

I have disabled SBOM indexing, and Usage Statistics and restarted docker desktop. docker-scout is still being run aggressively, as if on a timer even if no new containers are pulled or built, and it is trying to use up to 32 GB (!!!) of memory in one of two processes on a computer with only 16.

The only way I can somewhat attempt to use my work computer for work right now is running killall -9 docker-scout repeatedly on the command line to buy myself 15 minutes of actually using my computer.

This is unacceptable.

[OsirisMaeluma]

Didn't realize this was happening till my machine was running slow. Docker Scout is throttling entire machine performance. Is there a timeline on this fix since it seems like a High priority bug since it affects entire machines.

[bsousaa]

@OsirisMaeluma memory consumption will depend on the size of the images in your machine. You can disable indexing in Settings > General > SBOM Indexing

[foxt]

What's the point of a memory usage limit if Docker is simply going to blow past it, with a 4GB memory limit, and Docker's scout component alone using 6GB.

[jay-depot]

Chiming in here:

I have disabled SBOM indexing, and Usage Statistics and restarted docker desktop. docker-scout is still being run aggressively, as if on a timer even if no new containers are pulled or built, and it is trying to use up to 32 GB (!!!) of memory in one of two processes on a computer with only 16.

The only way I can somewhat attempt to use my work computer for work right now is running killall -9 docker-scout repeatedly on the command line to buy myself 15 minutes of actually using my computer.

This is unacceptable.

So, this is interesting. After a few system restarts, it seems like Docker desktop is now honoring those settings and not firing up docker-scout. Restarting Docker Desktop isn't enough though, despite the UI implying it is. Apparently these settings in particular only take on a full reboot.

[foxt]

FWIW, it stopped immediately after unchecking the box for me

On Tue, 6 Feb 2024 at 7:26 pm, Jason DePeaux @.***> wrote:

Chiming in here:

I have disabled SBOM indexing, and Usage Statistics and restarted docker desktop. docker-scout is still being run aggressively, as if on a timer even if no new containers are pulled or built, and it is trying to use up to 32 GB (!!!) of memory in one of two processes on a computer with only 16.

The only way I can somewhat attempt to use my work computer for work right now is running killall -9 docker-scout repeatedly on the command line to buy myself 15 minutes of actually using my computer.

This is unacceptable.

So, this is interesting. After a few system restarts, it seems like Docker desktop is now honoring those settings and not firing up docker-scout. Restarting Docker Desktop isn't enough though, despite the UI implying it is. Apparently these settings in particular only take on a full reboot.

— Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/6987#issuecomment-1930611091, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5QOWC4JQGVMTDASF4SRATYSJ7WFAVCNFSM6AAAAAA5BDYNXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZQGYYTCMBZGE . You are receiving this because you commented.Message ID: @.***>

[GetAHat]

Oh god, I've just discovered that thing. It's bloody awful. I have quite big build image (around 50gb of code) and decided to build it on Docker Desktop (usually do it in Engine on some Linux distribution like fedora). Docker Engine on Linux used 9 gigs entirely (with the system!). And build on Desktop with scout ballooned to... 110gigs. Of ram. When pulling an image. This is atrocious and it should be disabled by default.

[iamronr]

@GetAHat AGREED! 100%. WTH!

[nepp95]

This is still a massive problem. Computers just grind to a halt because of 97% memory usage because of docker scout. That, and the fact that disabling it often does not work at all makes Docker Desktop at this moment pretty unusable unless you have task manager open to terminate docker scout every few minutes.

Devs, please look at this and at least DISABLE it by default since it is causing so much problems.

[thgla]

It is wild that this is still enabled by default. I have a 32 GB machine and it almost froze up completely yesterday, until I discovered that docker-scout.exe was taking up multiple GBs of RAM and I found this issue here. How could an experimental feature like this even be promoted to a stable feature, when it is clearly broken?

[bhurlow]

Also was hit by this with about 9GB of ram used

[tr7zw]

Just managed to catch it after last time it wasn't visible in the task manager. This started eating over 16gb of ram on a 32gb machine, nearly crashing it. This issue also has been open for 8 months. At this point, this is just malicious.

[Pyroluk]

No joke, it just used up all of my 64GB(!!!!!) of RAM on my windows machine. And I am running the normal Nvidia PyTorch containers (nvcr.io/nvidia/pytorch). SBOM Indexing in its current form is a hazardous feature! Fix it, disable it, or get rid of it!

[matt-gibbs]

On a fresh install of Docker Desktop, the indexing is no longer enabled by default. When upgrading an existing install, the current setting is preserved and it was originally introduced as enabled by default. In Settings->General at the bottom of the list, you can disable it explicitly. Sorry for the frustration.

[tr7zw]

But why was the experimental feature on by default, and that then migrated? It's clearly a broken feature that should still be in experimental and also not turned on by default. An update should turn it off for everyone, and people actually wanting it can turn it back on. In the meantime, Docker Desktop is just crashing PCs of people who can't figure out what's happening.

[roldengarm]

I just found this thread as I found docker-scout.exe using 6GB of RAM. I've never enabled it, and I agree such an experimental feature should stay disabled. Even after disabling, and clicking Apply & Restart, it's still there doing stuff. Had to quit and re-start Docker Desktop before it was really gone

[Courier-Britva]

same here, my 16gm RAM was overflow in a few minutes after each docker-compose up --build so i have to turn off pc and restart it. totally should be disabled by default

[dciszewski]

Just had the same issue — docker-scout sucked up all my RAM with SBOM indexing enabled, completely paralysed my machine with memory pressure being completely red. Please disable it by default or put some sane limits on this feature as it clearly shouldn't behave like that.