docker / compose

Define and run multi-container applications with Docker
https://docs.docker.com/compose/
Apache License 2.0
33.51k stars 5.18k forks source link

[BUG] commit 163cdfd31 breaks interpolation in extended files #11994

Closed jolmg closed 1 month ago

jolmg commented 1 month ago

Description

Current behavior is the following error, with a stacktrace (included in reproduction steps):

panic: interface conversion: interface {} is string, not map[string]interface {}

I also get a different error if I remove a mount option next to the interpolation. Both configs and errors included in reproduction steps.

Expected containers to come up when I did docker compose up. Error(s) also happens with docker compose config, and it's got something to do with combining interpolation and extended files.

It seems the bug was introduced in 163cdfd31, which adds a test and updates the compose-go dependency.

Checking out compose-go, with compose set to 163cdfd31~, it seems the bug was introduced there in compose-spec/compose-go@65600ce.

Adding report here rather than there because I'm observing and replicating the error from compose.

Steps To Reproduce

Script to reproduce:

d=/tmp/docker-compose-interpolation-test-extended-with-ro

set -e
mkdir "$d"
cd "$d"

cat > docker-compose.yml <<EOF
services:
  foo:
    extends: { file: foo.yml, service: foo }
EOF

cat > foo.yml <<EOF
services:
  foo:
    image: bash
    volumes:
      - \${FOO:-/dev/null}:/tmp/foo:ro
EOF

docker compose config

Result:

panic: interface conversion: interface {} is string, not map[string]interface {}

goroutine 1 [running]:
github.com/compose-spec/compose-go/v2/paths.(*relativePathsResolver).absVolumeMount(0x2?, {0x5ea29c98d9c0?, 0xc0007137a0?})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:123 +0x1be
github.com/compose-spec/compose-go/v2/paths.(*relativePathsResolver).resolveRelativePaths(0xc00047e6c0, {0x5ea29c98d9c0, 0xc0007137a0}, {0xc00041f770, 0x17})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:74 +0xf0
github.com/compose-spec/compose-go/v2/paths.(*relativePathsResolver).resolveRelativePaths(0xc00047e6c0, {0x5ea29c959c80, 0xc00073aa98}, {0xc00041f758, 0x14})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:88 +0x339
github.com/compose-spec/compose-go/v2/paths.(*relativePathsResolver).resolveRelativePaths(0xc00047e6c0, {0x5ea29caa51c0, 0xc00035d590}, {0xc000723da0, 0xc})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:80 +0x22e
github.com/compose-spec/compose-go/v2/paths.(*relativePathsResolver).resolveRelativePaths(0xc00047e6c0, {0x5ea29caa51c0, 0xc00035d560}, {0xc000723c98, 0x8})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:80 +0x22e
github.com/compose-spec/compose-go/v2/paths.(*relativePathsResolver).resolveRelativePaths(0xc00047e6c0, {0x5ea29caa51c0, 0xc00035d380}, {0x0, 0x0})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:80 +0x22e
github.com/compose-spec/compose-go/v2/paths.ResolveRelativePaths(0xc00035d380, {0x5ea29c81ad28, 0x1}, {0xc0007137c0, 0x2, 0x2})
    github.com/compose-spec/compose-go/v2@v2.1.4/paths/resolve.go:50 +0x827
github.com/compose-spec/compose-go/v2/loader.getExtendsBaseFromFile({0x5ea29ce88128, 0xc00035c810}, {0xc000723bd8, 0x3}, {0xc000723c78, 0x3}, {0xc000781c80, 0x3e}, {0xc000723c50, 0x9}, ...)
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/extends.go:203 +0x985
github.com/compose-spec/compose-go/v2/loader.applyServiceExtends({0x5ea29ce88128, 0xc00035c810}, {0xc000723bd8, 0x3}, 0xc00035c9c0, 0xc00054fcb0, 0xc0005a0c20, {0xc0005a0900, 0x1, 0x1})
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/extends.go:101 +0x4da
github.com/compose-spec/compose-go/v2/loader.ApplyExtends({0x5ea29ce88128, 0xc00035c810}, 0xc00035c990, 0xc00054fcb0, 0xc0005a0c20, {0xc0005a0900, 0x1, 0x1})
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/extends.go:43 +0x174
github.com/compose-spec/compose-go/v2/loader.loadYamlFile.func1({0x5ea29caa51c0?, 0xc00035c990?}, {0xc0005a0900, 0x1, 0x1})
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/loader.go:438 +0x11d
github.com/compose-spec/compose-go/v2/loader.loadYamlFile({0x5ea29ce88160, 0xc0003eec30}, {{0xc000781c80, 0x3e}, {0xc000536200, 0x40, 0x200}, 0x0}, 0xc00054fcb0, {0xc000781c80, ...}, ...)
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/loader.go:490 +0x564
github.com/compose-spec/compose-go/v2/loader.loadYamlModel({0x5ea29ce88160, 0xc0003eec30}, {{0x0, 0x0}, {0xc000781c80, 0x2b}, {0xc00035c0f0, 0x1, 0x1}, 0xc00057dd70}, ...)
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/loader.go:364 +0x1a5
github.com/compose-spec/compose-go/v2/loader.load({0x5ea29ce88160, 0xc0003eec30}, {{0x0, 0x0}, {0xc000781c80, 0x2b}, {0xc00035c0f0, 0x1, 0x1}, 0xc00057dd70}, ...)
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/loader.go:512 +0x38b
github.com/compose-spec/compose-go/v2/loader.loadModelWithContext({0x5ea29ce88160, 0xc0003eec30}, 0xc00069eb40, 0xc00054fcb0)
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/loader.go:336 +0x105
github.com/compose-spec/compose-go/v2/loader.LoadWithContext({0x5ea29ce88160, 0xc0003eec30}, {{0x0, 0x0}, {0xc000781c80, 0x2b}, {0xc00035c0f0, 0x1, 0x1}, 0xc00057dd70}, ...)
    github.com/compose-spec/compose-go/v2@v2.1.4/loader/loader.go:312 +0xd8
github.com/compose-spec/compose-go/v2/cli.(*ProjectOptions).LoadProject(0xc00054fc20, {0x5ea29ce88160, 0xc0003eec30})
    github.com/compose-spec/compose-go/v2@v2.1.4/cli/options.go:445 +0x128
github.com/docker/compose/v2/cmd/compose.(*ProjectOptions).ToProject(0xc000315720, {0x5ea29ce88160, 0xc0003eec30}, {0x5ea29ce9f160, 0xc000420c80}, {0x5ea29e0fd3a0, 0x0, 0x0}, {0xc00057dce0, 0x6, ...})
    github.com/docker/compose/v2/cmd/compose/compose.go:318 +0x539
github.com/docker/compose/v2/cmd/compose.(*configOptions).ToProject(0xc0005a16d0, {0x5ea29ce88160, 0xc0003eec30}, {0x5ea29ce9f160, 0xc000420c80}, {0x5ea29e0fd3a0, 0x0, 0x0}, {0x0, 0x0, ...})
    github.com/docker/compose/v2/cmd/compose/config.go:62 +0x16a
github.com/docker/compose/v2/cmd/compose.runConfigInterpolate({0x5ea29ce88160, 0xc0003eec30}, {0x5ea29ce9f160, 0xc000420c80}, {0xc000315720, {0x5ea29c4a6c07, 0x4}, {0x0, 0x0}, 0x0, ...}, ...)
    github.com/docker/compose/v2/cmd/compose/config.go:181 +0x72
github.com/docker/compose/v2/cmd/compose.runConfig({0x5ea29ce88160?, 0xc0003eec30?}, {0x5ea29ce9f160, 0xc000420c80}, {0xc000315720, {0x5ea29c4a6c07, 0x4}, {0x0, 0x0}, 0x0, ...}, ...)
    github.com/docker/compose/v2/cmd/compose/config.go:159 +0xc5
github.com/docker/compose/v2/cmd/compose.configCommand.func2({0x5ea29ce88160?, 0xc0003eec30?}, {0x5ea29e0fd3a0?, 0xc0000061c0?, 0x5ea29c47933c?})
    github.com/docker/compose/v2/cmd/compose/config.go:126 +0x105
github.com/docker/compose/v2/cmd/compose.configCommand.Adapt.func4({0x5ea29ce88160?, 0xc0003eec30?}, 0x2?, {0x5ea29e0fd3a0?, 0x5ea29ce6d260?, 0x78b32a?})
    github.com/docker/compose/v2/cmd/compose/compose.go:125 +0x30
github.com/docker/compose/v2/cmd/compose.configCommand.Adapt.AdaptCmd.func7(0xc0001f6f08, {0x5ea29e0fd3a0, 0x0, 0x0})
    github.com/docker/compose/v2/cmd/compose/compose.go:101 +0x154
github.com/docker/cli/cli-plugins/plugin.RunPlugin.func1.1.2(0xc0001f6f08, {0x5ea29e0fd3a0, 0x0, 0x0})
    github.com/docker/cli@v27.0.3+incompatible/cli-plugins/plugin/plugin.go:64 +0x6c
github.com/docker/compose/v2/cmd/cmdtrace.Setup.wrapRunE.func2(0xc0001f6f08?, {0x5ea29e0fd3a0?, 0x0?, 0x0?})
    github.com/docker/compose/v2/cmd/cmdtrace/cmd_span.go:85 +0x63
github.com/spf13/cobra.(*Command).execute(0xc0001f6f08, {0xc00014baf0, 0x0, 0x0})
    github.com/spf13/cobra@v1.8.1/command.go:985 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc000020308)
    github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/docker/cli/cli-plugins/plugin.RunPlugin(0xc000420c80, 0xc000440908, {{0x5ea29c4a7737, 0x5}, {0x5ea29c4b1636, 0xb}, {0x5ea29c82072c, 0x6}, {0x0, 0x0}, ...})
    github.com/docker/cli@v27.0.3+incompatible/cli-plugins/plugin/plugin.go:79 +0x145
github.com/docker/cli/cli-plugins/plugin.Run(0x5ea29ce58b18, {{0x5ea29c4a7737, 0x5}, {0x5ea29c4b1636, 0xb}, {0x5ea29c82072c, 0x6}, {0x0, 0x0}, {0x0, ...}})
    github.com/docker/cli@v27.0.3+incompatible/cli-plugins/plugin/plugin.go:94 +0x165
main.pluginMain()
    github.com/docker/compose/v2/cmd/main.go:38 +0xa5
main.main()
    github.com/docker/compose/v2/cmd/main.go:98 +0x19c

Removing :ro (the readonly mount option):

d=/tmp/docker-compose-interpolation-test-extended-without-ro

set -e
mkdir "$d"
cd "$d"

cat > docker-compose.yml <<EOF
services:
  foo:
    extends: { file: foo.yml, service: foo }
EOF

cat > foo.yml <<EOF
services:
  foo:
    image: bash
    volumes:
      - \${FOO:-/dev/null}:/tmp/foo
EOF

docker compose config

Result:

invalid interpolation format for services.foo.volumes.[].source.
You may need to escape any $ with another $.
${FOO

And here's a successful interpolation if I don't extend and use a single yml:

d=/tmp/docker-compose-interpolation-test-not-extended-with-ro

set -e
mkdir "$d"
cd "$d"

cat > docker-compose.yml <<EOF
services:
  foo:
    image: bash
    volumes:
      - \${FOO:-/dev/null}:/tmp/foo:ro
EOF

docker compose config

Result:

name: docker-compose-interpolation-test-not-extended-with-ro
services:
  foo:
    image: bash
    networks:
      default: null
    volumes:
      - type: bind
        source: /dev/null
        target: /tmp/foo
        read_only: true
        bind:
          create_host_path: true
networks:
  default:
    name: docker-compose-interpolation-test-not-extended-with-ro_default

Compose Version

$ docker compose version
Docker Compose version 2.29.0
$ docker-compose version
Docker Compose version 2.29.0

Docker Environment

Client:
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.15.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.29.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 271
  Running: 1
  Paused: 0
  Stopped: 270
 Images: 284
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.9.6-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.7GiB
 ID: TVST:4T34:N6W7:FSIT:7HOI:IDG3:NUS4:A4AA:SUGG:WO3F:WZPJ:Y73Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Live Restore Enabled: false

Anything else?

No response

jolmg commented 1 month ago

As an additional check (since the linked commit mentions empty environment variables), changing docker compose config for FOO=/dev/zero docker compose config in each of the above reproduction scripts gives the same results. Same errors. The only difference is that, in the successful one, source: /dev/zero rather than source: /dev/null, of course.

idsulik commented 1 month ago

@jolmg hi! I think the issue with the yaml, you should put empty space after ":"

- ${FOO:-/dev/null}: /tmp/foo:to
                    ^ here
jolmg commented 1 month ago

@idsulik This is the short syntax for volumes.. If I added a space, it wouldn't be a yaml string; it'd a mapping, and I'd get this error:

validating /tmp/docker-compose-interpolation-test-t9H5/docker-compose.yml: services.foo.volumes.0 type is required

Because it's then expecting the different syntax with specific keys.

jolmg commented 1 month ago

@idsulik Thank you, though. Your comment does show that switching to long syntax is a good workaround for this bug.

d=/tmp/docker-compose-interpolation-test-long-syntax

set -e
mkdir "$d"
cd "$d"

cat > docker-compose.yml <<EOF
services:
  foo:
    extends: { file: other.yml, service: foo }
EOF

cat > other.yml <<EOF
services:
  foo:
    image: bash
    volumes:
      - type: bind
        source: \${FOO:-/dev/null}
        target: /tmp/foo
        read_only: true
EOF

docker compose config

Result:

name: docker-compose-interpolation-test-long-syntax
services:
  foo:
    image: bash
    networks:
      default: null
    volumes:
      - type: bind
        source: /dev/null
        target: /tmp/foo
        read_only: true
networks:
  default:
    name: docker-compose-interpolation-test-long-syntax_default