Using build_in_docker running docker in docker

[reikje]

Description

I am struggling using the build_in_docker option when running inside a VSCode devcontainer.

I arrived here because my Lambda needs a Python dependency called cryptography and I am hitting /lib64/libc.so.6: version 'GLIBC_2.28' not found when I run terraform apply on my Macbook.

My project is running inside a .devcontainer and I am mounting "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" into that container. When I then run terraform apply inside the container, it basically is running docker in docker utilising Docker Desktop on the host. Using the build_in_docker option will result in the following error:

21826ea7c95e: Pull complete
│ Digest: sha256:79efc767616c002acf83758a24da5fb02d3d079313321b0de304bbfec402b77d
│ Status: Downloaded newer image for public.ecr.aws/sam/build-python3.10:latest
│ docker: Error response from daemon: Mounts denied: 
│ The path /root/.ssh/known_hosts is not shared from the host and is not known to Docker.

Here is the mount causing the issue. I am really not a docker expert but I think instead of using /root (which is $HOME inside my devcontainer) I need to use the path actual available on the host. There seem to be no option to set the folder for the .ssh directory ...

• [ x ] ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: 6.0.1
• Terraform version: 1.6.1
• Provider version(s): 5.21.0

Reproduction Code [Required]

.devcontainer/devcontainer.json:

{
  "name": "Python 3",
  "build": {
    "dockerfile": "Dockerfile",
    "context": "..",
    "args": {
    }
  },
  "customizations": {
    "vscode": {
      "settings": {
        "editor.suggest.insertMode": "replace",
        "python.testing.pytestArgs": [
          "tests"
        ],
        "python.testing.unittestEnabled": false,
        "python.testing.pytestEnabled": true,
        "python.defaultInterpreterPath": "/var/folders/.venv/bin/python",
        "python.analysis.indexing": true,
        "python.analysis.packageIndexDepths": [
            { "name": "", "depth": 2, "includeAllSymbols": true }
        ]
      },

      "extensions": [
        "ms-python.python",
        "charliermarsh.ruff",
        "hashicorp.terraform",
        "njpwerner.autodocstring"
      ]
    }
  },
  "remoteUser": "root",
  "runArgs": ["--add-host=host.docker.internal:host-gateway"],
  "containerEnv": {
    "GITHUB_WORKSPACE": "${localWorkspaceFolder}"
  },
  "mounts": [
      "source=${env:HOME}${env:USERPROFILE}/.aws,target=/root/.aws,type=bind,readonly",
      "source=${env:HOME}${env:USERPROFILE}/.ssh,target=/root/.ssh,type=bind,readonly",
      "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind"
  ],
  "features": {
    "github-cli": "latest"
  },
  "workspaceMount": "source=${localWorkspaceFolder},target=/var/folders,type=bind",
  "workspaceFolder": "/var/folders"
}

lambda.tf:

module "lambda_function" {
  source              = "terraform-aws-modules/lambda/aws"
  version             = "~> 6.0"
  timeout             = 300
  source_path         = "../../src"
  function_name       = "whatever"
  handler             = "app.handler"
  runtime             = "python3.10"
  create_sam_metadata = true
  publish             = true
  build_in_docker     = true

  allowed_triggers = {
    AnyRule = {
      principal  = "events.amazonaws.com"
      source_arn = "arn:aws:events:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:rule/*"
    }
  }

  tags = local.tags
}

Steps to reproduce the behavior:

• create a VSCode devcontainer
• open the project in the devcontainer
• run terraform apply in a module that has the build_in_docker option set to true

Expected behavior

The build runs properly inside a docker (in docker) container.

Actual behavior

The build fails with:

...
│ 21826ea7c95e: Pull complete
│ Digest: sha256:79efc767616c002acf83758a24da5fb02d3d079313321b0de304bbfec402b77d
│ Status: Downloaded newer image for public.ecr.aws/sam/build-python3.10:latest
│ docker: Error response from daemon: Mounts denied: 
│ The path /root/.ssh/known_hosts is not shared from the host and is not known to Docker.

.. unless the host system has a /root/.ssh folder, which is not the case in MacOS.

[villekr]

I'm also looking at the similar case i.e. trying to use terraform-aws-lambda -module within dockerized build container.

I think mounting .ssh/known_hosts is necessary only if accessing packages in private repos via ssh. So basically it's possibly to use local version of terraform-aws-lambda-repository and modify package.py docker_run_command-function a bit. I commented out this part as a temp workaround. And of course one need to refer then to local path of terraform-aws-lambda directory.

    # docker_cmd.extend([
    #     # '-v', '{}/.ssh/id_rsa:/root/.ssh/id_rsa:z'.format(home),
    #     '-v', '{}/.ssh/known_hosts:/root/.ssh/known_hosts:z'.format(home),
    # ])

But then I'm facing another issue when install npm packages:

Error: local-exec provisioner error
│ 
│   with module.nodejs_lambda_layer.null_resource.archive[0],
│   on modules/terraform-aws-lambda/package.tf line 67, in resource "null_resource" "archive":
│   67:   provisioner "local-exec" {
│ 
│ Error running command
│ './builds/lambda_layer/9b33c113e714cbd7294f9da60e90d6c4ca1f8babd889b16ef17adc9f073a6fe6.plan.json':
│ exit status 1. Output: zip: creating
│ './builds/lambda_layer/9b33c113e714cbd7294f9da60e90d6c4ca1f8babd889b16ef17adc9f073a6fe6.zip'
│ archive
│ > docker images '--format={{.ID}}' ecs-test-pipeline-infra-nodejs18.x-build
│ Installing npm requirements: ./../nodejs-lambda/package.json
│ > mktemp -d terraform-aws-lambda-XXXXXXXX #
│ /tmp/terraform-aws-lambda-ljdocj4k
│ > cd /tmp/terraform-aws-lambda-ljdocj4k
│ > docker run --rm -w /var/task -v
│ /tmp/terraform-aws-lambda-ljdocj4k:/var/task:z --entrypoint '' 8a0a80350522
│ /bin/sh -c 'npm install && chown -R 0:0 .'
│ npm ERR! code ENOENT
│ npm ERR! syscall open
│ npm ERR! path /var/task/package.json
│ npm ERR! errno -2
│ npm ERR! enoent ENOENT: no such file or directory, open
│ '/var/task/package.json'
│ npm ERR! enoent This is related to npm not being able to find a file.
│ npm ERR! enoent

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

This issue was automatically closed because of stale in 10 days

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.