local-exec step fails when trying to run 'npm install' on Windows

james-mwakichako commented 1 year ago

Description

Trying to deploy a lambda on Windows and getting the following error:

module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): raise RuntimeError( module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): RuntimeError: Nodejs interpreter version equal to defined lambda runtime (nodejs16.x) should be available in system PATH I have verified that I have node and npm installed .

I have ran into the same error on GitBash and CMD. NOT seeing this issue on Mac.

Below are the system specs: Terraform : v1.3.7 Edition Windows 10 Enterprise Version 20H2 Installed on ‎09/‎01/‎2022 OS build 19042.2486 Experience Windows Feature Experience Pack 120.2212.4190.0

Versions

Module version [Required]: 4.7.1
Terraform version: v1.3.7
Provider version(s): v4.47.0
Python version 3.10

Reproduction Code

module "lambdaToDynamo_lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "4.7.1"

  function_name = "testing-lambda"
  handler       = "index.handler"
  runtime       = "nodejs18.x"

  source_path = "../lambda/src/lambdaToDynamo"

  memory_size = 128
  timeout     = 60

  cloudwatch_logs_retention_in_days = 7

  tags = local.common_tags
}

output "lambdaToDynamo_lambda_function_name" {
  value = module.lambdaToDynamo_lambda.lambda_function_name
}

output "lambdaToDynamo_lambda_role_arn" {
  value = module.lambdaToDynamo_lambda.lambda_role_arn
}

provider "aws" {
  region = var.region
}

terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.47.0"
    }
  }
}

locals {
  prefix        = "${var.projectPrefix}-${var.environment}-${var.serviceName}"
  prefix_short  = "${var.projectPrefix}-${var.environment}"
  region_prefix = "${var.projectPrefix}-${var.environment}-${var.region}-${var.serviceName}"
  global_prefix = "${var.projectPrefix}-${var.environment}-${local.region_short}-${var.serviceName}"

  common_tags = {
    Environment = var.environment
  }
}

locals {
  region_short = var.region == "us-east-1" ? "ue1" : "ue2"
}

variable "environment" {
  type    = string
  default = "test"
}

variable "projectPrefix" {
  type = string
  default = "TF-test"
}

variable "region" {
  type    = string
  default = "us-east-1"
}

variable "serviceName" {
  type = string
  default = "TestService"
}

Steps to reproduce the behavior:

terraform init
terraform plan
terraform apply

Expected behavior

Module should run successfully , run npm install against the package.json defined in the same directory as the lambda code and deploy the lambda to AWS.

Actual behavior

Full error message :

module.lambdaToDynamo_lambda.null_resource.archive[0]: Provisioning with 'local-exec'...
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): Executing: ["python.exe" ".terraform/modules/lambdaToDynamo_lambda/package.py" "build" "--timestamp" "1674855599800929000" "./.terraform/test-lambda-builds/835975921e6d253666fa437649970e48ec554ce92fc3951357c6233208daebe8.plan.json"]
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): zip: creating './.terraform/test-lambda-builds/835975921e6d253666fa437649970e48ec554ce92fc3951357c6233208daebe8.zip' archive
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): Installing npm requirements: ../../lambda/src/lambdaToDynamo\package.json
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): > mktemp -d terraform-aws-lambda-XXXXXXXX # 'C:\Users\XXXXXX\AppData\Local\Temp\terraform-aws-lambda-iqttatx9'
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): > cd 'C:\Users\XXXXXX\AppData\Local\Temp\terraform-aws-lambda-iqttatx9'
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): > npm install
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): zip: Error during zip archive creation
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): Traceback (most recent call last):
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1272, in install_npm_requirements
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     check_call(npm_command, env=subproc_env)
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 364, in check_call
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     retcode = call(*popenargs, **kwargs)
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 345, in call
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     with Popen(*popenargs, **kwargs) as p:
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 969, in __init__
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     self._execute_child(args, executable, preexec_fn, close_fds,
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1438, in _execute_child
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): FileNotFoundError: [WinError 2] The system cannot find the file specified

module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): The above exception was the direct cause of the following exception:

module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): Traceback (most recent call last):
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1522, in build_command
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     bpm.execute(build_plan, zs, query)
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 877, in execute
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     with install_npm_requirements(query, npm_requirements, tmp_dir) as rd:
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in __enter__
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     return next(self.gen)
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1274, in install_npm_requirements
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     raise RuntimeError(
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): RuntimeError: Nodejs interpreter version equal to defined lambda runtime (nodejs16.x) should be available in system PATH
╷
│ Error: local-exec provisioner error
│
│   with module.lambdaToDynamo_lambda.null_resource.archive[0],
│   on .terraform\modules\lambdaToDynamo_lambda\package.tf line 67, in resource "null_resource" "archive":
│   67:   provisioner "local-exec" {
│
│ Error running command
│ './.terraform/test-lambda-builds/835975921e6d253666fa437649970e48ec554ce92fc3951357c6233208daebe8.plan.json':
│ exit status 1. Output: zip: creating
│ './.terraform/test-lambda-builds/835975921e6d253666fa437649970e48ec554ce92fc3951357c6233208daebe8.zip'
│ archive
│ Installing npm requirements: ../../lambda/src/lambdaToDynamo\package.json
│ > mktemp -d terraform-aws-lambda-XXXXXXXX #
│ 'C:\Users\XXXXXX\AppData\Local\Temp\terraform-aws-lambda-iqttatx9'
│ > cd 'C:\Users\XXXXXX\AppData\Local\Temp\terraform-aws-lambda-iqttatx9'
│ > npm install
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1272, in install_npm_requirements
│     check_call(npm_command, env=subproc_env)
│   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 364, in check_call
│     retcode = call(*popenargs, **kwargs)
│   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 345, in call
│     with Popen(*popenargs, **kwargs) as p:
│   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 969, in __init__
│     self._execute_child(args, executable, preexec_fn, close_fds,
│   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1438, in _execute_child
│     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
│ FileNotFoundError: [WinError 2] The system cannot find the file specified
│
│ The above exception was the direct cause of the following exception:
│
│ Traceback (most recent call last):
│   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1522, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 877, in execute
│     with install_npm_requirements(query, npm_requirements, tmp_dir) as rd:
│   File "C:\Users\XXXXXX\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in __enter__
│     return next(self.gen)
│   File "C:\Users\XXXXXX\XXXXXX\XXXXXX\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1274, in install_npm_requirements
│     raise RuntimeError(
│ RuntimeError: Nodejs interpreter version equal to defined lambda runtime
│ (nodejs16.x) should be available in system PATH

Additional context

I have found two work arounds by updating package.py :

Set shell=True in check_call within package.py. This comes with it’s own security considerations
change "npm" "install" to "npm.cmd" "install" in the case the OS is Windows.

The lambdaToDynamo dir contains two files :

index.js
package.json

james-mwakichako commented 1 year ago

Additional Context:

I also went ahead and used commands option and got the following error in Windows .

module.lambdaToDynamo_lambda.local_file.archive_plan[0]: Creation complete after 0s [id=592971c7c21735af995af3dffbde97e352aae917]
module.lambdaToDynamo_lambda.null_resource.archive[0]: Creating...
module.lambdaToDynamo_lambda.null_resource.archive[0]: Provisioning with 'local-exec'...
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): Executing: ["python.exe" ".terraform/modules/lambdaToDynamo_lambda/package.py" "build" "--timestamp" "1675955070336843000" "./.terraform/test-app-ue1-lambda-builds/0b4f89a1c7e5af849c5d3a686943d420e47ef076ee13fd81af3279c678716fdf.plan.json"]
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): zip: creating './.terraform/test-app-ue1-lambda-builds/0b4f89a1c7e5af849c5d3a686943d420e47ef076ee13fd81af3279c678716fdf.zip' archive
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): zip: Error during zip archive creation
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): Traceback (most recent call last):
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXXX\test\sampleService\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1520, in build_command
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     bpm.execute(build_plan, zs, query)
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXXX\test\sampleService\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 891, in execute
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     p = subprocess.Popen(script, shell=True, cwd=path,
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 969, in __init__
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     self._execute_child(args, executable, preexec_fn, close_fds,
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):   File "C:\Users\XXXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1364, in _execute_child
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec):     assert not pass_fds, "pass_fds not supported on Windows."
module.lambdaToDynamo_lambda.null_resource.archive[0] (local-exec): AssertionError: pass_fds not supported on Windows.
╷
│ Error: local-exec provisioner error
│
│   with module.lambdaToDynamo_lambda.null_resource.archive[0],
│   on .terraform\modules\lambdaToDynamo_lambda\package.tf line 67, in resource "null_resource" "archive":
│   67:   provisioner "local-exec" {
│
│ Error running command
│ './.terraform/test-app-ue1-lambda-builds/0b4f89a1c7e5af849c5d3a686943d420e47ef076ee13fd81af3279c678716fdf.plan.json':
│ exit status 1. Output: zip: creating
│ './.terraform/test-app-ue1-lambda-builds/0b4f89a1c7e5af849c5d3a686943d420e47ef076ee13fd81af3279c678716fdf.zip'
│ archive
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "C:\Users\XXXXXXX\test\sampleService\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 1520, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "C:\Users\XXXXXXX\test\sampleService\infrastructure\regional\.terraform\modules\lambdaToDynamo_lambda\package.py", line 891, in execute
│     p = subprocess.Popen(script, shell=True, cwd=path,
│   File "C:\Users\XXXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 969, in __init__
│     self._execute_child(args, executable, preexec_fn, close_fds,
│   File "C:\Users\XXXXXXX\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1364, in _execute_child
│     assert not pass_fds, "pass_fds not supported on Windows."
│ AssertionError: pass_fds not supported on Windows.

Terraform code

module "lambdaToDynamo_lambda" {

  source  = "terraform-aws-modules/lambda/aws"
  version = "4.7.1"

  function_name = "${local.prefix}-lambdaToDynamo"
  description   = "Sample service lambda"
  handler       = "index.handler"
  runtime       = "nodejs18.x"

  source_path = [
    {
    path     = "../../lambda/src/lambdaToDynamo",
    commands = [
      "npm.cmd install",
      ":zip"
    ],
    patterns = [
      "node_modules/.+", # Include all node_modules
    ],
  }
  ]

  memory_size = var.memory_size
  timeout     = var.timeout_in_seconds

  cloudwatch_logs_retention_in_days = var.cw_log_retention_in_days

  environment_variables = {
    "SampleTable_Name" = data.aws_dynamodb_table.demoTable.id
  }

  tags = local.common_tags
}

locals {
  account_resource_prefix        = "${var.projectPrefix}-${var.accountProfile}"
  prefix                         = "${var.projectPrefix}-${var.app_name}-${var.environment}"
  region_short                   = var.region == "us-east-1" ? "ue1" : "ue2"
  region_prefix                  = "${var.projectPrefix}-${var.app_name}-${var.environment}-${local.region_short}"
  account_resource_region_prefix = "${var.projectPrefix}-${var.accountProfile}-${local.region_short}"
  lambda_role_arn                = "arn:aws:iam::${var.allowedAccountIds[0]}:role/${local.region_prefix}-lambdaRole"

  common_tags = {
    environment   = var.environment
  }
}

I tried running this on Mac and the lambda was deployed successfully .

james-mwakichako commented 1 year ago

More updates on this. We got around this on Windows by using WSL . Thanks to @jwicks

antonbabenko commented 1 year ago

Exactly, this is the solution all Windows users I know were using.

I am closing this issue.

james-mwakichako commented 1 year ago

@antonbabenko , could that be added in the documentation ?

antonbabenko commented 1 year ago

@james-mwakichako Sure, please open a PR with the updated README file. Highly appreciated!

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

terraform-aws-modules / terraform-aws-lambda