Error when trying to use CloudFormation intrinsic functions

MrChrisRodriguez commented 2 years ago

Description

Trying to use CloudFormation intrinsic functions like Fn::ImportValue in docker compose convert is failing. It prints the following error message along with a goroutine, then exits (albeit with a success instead of an error code):

panic: interface conversion: interface {} is map[string]interface {}, not string

The full output is shown below.

Steps to reproduce the issue:

Create a yaml file with the following:

x-aws-cluster:
  Fn::ImportValue:
    "exampleOutput-ClusterName"

Run COMPOSE_FILE=filename.yml docker compose convert

Describe the results you expected: I expected generated output cloudformation template to include the Fn::ImportValue intrinsic function.

Describe the results you received: Depending on how I run the command, I either get nothing, get the full string "exampleOutput-ClusterName", or get this error:

> COMPOSE_FILE=devops/compose.monorepo.ecs.example.yml docker compose convert
panic: interface conversion: interface {} is map[string]interface {}, not string

goroutine 1 [running]:
github.com/docker/compose-cli/ecs.(*ecsAPIService).parseClusterExtension(0xc0004681c0, 0x3809020, 0xc000468f00, 0xc000592000, 0xc00046faa0, 0x30, 0x5199a68, 0x30, 0xc0008a7ec0)
        github.com/docker/compose-cli/ecs/awsResources.go:151 +0x20f
github.com/docker/compose-cli/ecs.(*ecsAPIService).parse(0xc0004681c0, 0x3809020, 0xc000468f00, 0xc000592000, 0xc00046faa0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/docker/compose-cli/ecs/awsResources.go:126 +0xd8
github.com/docker/compose-cli/ecs.(*ecsAPIService).convert(0xc0004681c0, 0x3809020, 0xc000468f00, 0xc000592000, 0x0, 0x0, 0x2c28b80)
        github.com/docker/compose-cli/ecs/cloudformation.go:128 +0x1f0
github.com/docker/compose-cli/ecs.(*ecsAPIService).Convert(0xc0004681c0, 0x3809020, 0xc000468f00, 0xc000592000, 0x3406c95, 0x4, 0x0, 0x0, 0x10f398c, 0xc000544270, ...)
        github.com/docker/compose-cli/ecs/cloudformation.go:58 +0x165
github.com/docker/compose/v2/pkg/api.(*ServiceProxy).Convert(0xc0000a8000, 0x3809020, 0xc000468f00, 0xc000592000, 0x3406c95, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/docker/compose/v2@v2.2.0/pkg/api/proxy.go:227 +0x10a
github.com/docker/compose/v2/cmd/compose.runConvert(0x3809020, 0xc000468f00, 0x3829398, 0xc0000a8000, 0xc000053c80, 0x3406c95, 0x4, 0x0, 0x0, 0x0, ...)
        github.com/docker/compose/v2@v2.2.0/cmd/compose/convert.go:137 +0x1ac
github.com/docker/compose/v2/cmd/compose.convertCommand.func2(0x3809020, 0xc000468f00, 0x499b108, 0x0, 0x0, 0xc000000180, 0x1cd7925)
        github.com/docker/compose/v2@v2.2.0/cmd/compose/convert.go:93 +0x1db
github.com/docker/compose/v2/cmd/compose.Adapt.func1(0x3809020, 0xc000468f00, 0xc000565400, 0x499b108, 0x0, 0x0, 0x48, 0xc0003f5040)
        github.com/docker/compose/v2@v2.2.0/cmd/compose/compose.go:85 +0x57
github.com/docker/compose/v2/cmd/compose.AdaptCmd.func1(0xc000565400, 0x499b108, 0x0, 0x0, 0x0, 0x0)
        github.com/docker/compose/v2@v2.2.0/cmd/compose/compose.go:64 +0x13c
github.com/spf13/cobra.(*Command).execute(0xc000565400, 0xc00010e050, 0x0, 0x0, 0xc000565400, 0xc00010e050)
        github.com/spf13/cobra@v1.2.1/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc0008fd680, 0xc0004f3e10, 0x1, 0x1)
        github.com/spf13/cobra@v1.2.1/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.2.1/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.2.1/command.go:895
main.main()
        github.com/docker/compose-cli/cli/main.go:249 +0xaa7

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker-compose --version:

docker context use default
docker-compose --version --> docker-compose version 1.29.2, build 5becea4c
docker compose version --> Docker Compose version v2.4.1
--
docker context use tradesorg (ecs)
docker compose version --> Docker Compose version dev

Output of docker version:

Client:
 Cloud integration: v1.0.23
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:49:20 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.7.1 (77678)
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker context show:

docker context inspect tradesorg
[
    {
        "Name": "tradesorg",
        "Metadata": {
            "Type": "ecs"
        },
        "Endpoints": {
            "docker": {
                "SkipTLSVerify": false
            },
            "ecs": {
                "Profile": "tradesorg"
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "/Users/<user>/.docker/contexts/meta/<hash>",
            "TLSPath": "/Users/<user>/.docker/contexts/tls/<hash>"
        }
    }
]

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.4.1)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.773GiB
 Name: docker-desktop
 ID: MAKE:42FC:V2PQ:5S5W:R7AN:EVKX:QH4U:7V6L:4NKQ:57SO:GYN4:JD2N
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 60
  Goroutines: 58
  System Time: 2022-05-02T18:48:18.511375664Z
  EventsListeners: 5
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS ECS, Azure ACI, local, etc.):

JohnPreston commented 2 years ago

Not related to the ECS plugin, but I hope this might help with deploying your services to AWS ECS.

Maybe instead of looking at defining all that to simply import the cluster name, you could use the x-cluster.Lookup from that project. Not only does it allow to create new cluster or use an existing one (using Lookup), but it will also

detect the capacity providers available for the cluster
use the capacity providers set to compare with settings you'd set via x-ecs to avoid
detect cluster settings, i.e. ecs execute settings, to adapt permissions to be given to services in the cluster

ECS Compose-X will understand a mix of docker-compose and override files, CloudFormation syntax for new resources to create, can do cross-accounts discovery, and at the end of the day, "simply" renders CFN templates that you can use and where needed, tweak.

Note: links point to the nightly docs because the 0.18 version is set for release this week and will be mainstream then.

JohnPreston commented 2 years ago

Follow up @MrChrisRodriguez , v0.18.0 is officially out since yesterday, support all of that :)

joshuakcockrell commented 2 years ago

@MrChrisRodriguez Did you ever figure this out? In my case, it would be great to use x-aws-vpc: Fn::ImportValue: ... instead of manually looking up the VPC id in the AWS console.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

docker-archive / compose-cli

Error when trying to use CloudFormation intrinsic functions #2146