search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.52k stars 1.17k forks source link

Bug: Build error when using docker build secrets #7522

Open royassis opened 1 month ago

royassis commented 1 month ago

Description:

My goal is to build my images outside sam and use sam only for debugging using the AWS Toolkit in Pycharm. When running the AWS Toolkit debug configuration in Pycharm the sam build command is ran and fails.

Whenever I use a RUN --mount=type=secret in my Dockerfile I get the build error.

This is the error:

Error: unexpected key 'env' in 'env=PIP_INDEX_URL'

This is the part from the Dockerfile:

RUN --mount=type=secret,id=PIP_INDEX_URL,env=PIP_INDEX_URL \
      pip install -r requirements.codeartifact.txt

If I remove this part from the Dockerfile the error is solved.

Regardess of sam, I get the same error when I remove this line from my Dockerfile:

# syntax=docker/dockerfile:1

Observed result:

(base) PS C:\Users\roya\PycharmProjects\unigen_pipeline> sam.cmd build SendPipelineResultToCaller --use-container --debug
2024-09-28 19:13:21,921 | Config file location: C:\Users\roya\PycharmProjects\unigen_pipeline\samconfig.toml
2024-09-28 19:13:21,923 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at 'C:\Users\roya\PycharmProjects\unigen_pipeline\samconfig.toml'...
2024-09-28 19:13:21,925 | Configuration values successfully loaded.
2024-09-28 19:13:21,926 | Configuration values are: {'region': 'us-east-1', 'parameter_overrides': ['AwsAccessKeyId=[reducted]', 'AwsSecretAccessKey=[reducted]',
'AwsSessionToken=[reducted]']}
2024-09-28 19:13:21,932 | Using SAM Template at C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:21,949 | Using config file: samconfig.toml, config environment: default
2024-09-28 19:13:21,950 | Expand command line arguments to:
2024-09-28 19:13:21,950 | --template_file=C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml --use_container --resource_logical_id=SendPipelineResultToCaller --mount_with=READ --build_dir=.aws-sam\build --cache_dir=.aws-sam\cache
--parameter_overrides={'AwsAccessKeyId': '[reducted]', 'AwsSecretAccessKey': '[reducted]', 'AwsSessionToken':
'[reducted]'}
2024-09-28 19:13:22,015 | 'build' command is called
2024-09-28 19:13:22,016 | Starting Build inside a container
2024-09-28 19:13:22,022 | Collected default values for parameters: {'AwsAccessKeyId': '', 'AwsSecretAccessKey': '', 'AwsSessionToken': ''}
2024-09-28 19:13:22,037 | There is no customer defined id or cdk path defined for resource ReadSqs, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,038 | There is no customer defined id or cdk path defined for resource SendPipelineResultToCaller, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,038 | There is no customer defined id or cdk path defined for resource GetSampleLocations, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,039 | There is no customer defined id or cdk path defined for resource ValidateStateMachineInput, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,039 | There is no customer defined id or cdk path defined for resource SfLambda, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,040 | There is no customer defined id or cdk path defined for resource LinkDatasetToStudy, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,041 | There is no customer defined id or cdk path defined for resource AddCellCount, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,041 | 0 stacks found in the template
2024-09-28 19:13:22,042 | Collected default values for parameters: {'AwsAccessKeyId': '', 'AwsSecretAccessKey': '', 'AwsSessionToken': ''}
2024-09-28 19:13:22,052 | There is no customer defined id or cdk path defined for resource ReadSqs, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,053 | There is no customer defined id or cdk path defined for resource SendPipelineResultToCaller, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,053 | There is no customer defined id or cdk path defined for resource GetSampleLocations, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,054 | There is no customer defined id or cdk path defined for resource ValidateStateMachineInput, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,054 | There is no customer defined id or cdk path defined for resource SfLambda, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,055 | There is no customer defined id or cdk path defined for resource LinkDatasetToStudy, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,055 | There is no customer defined id or cdk path defined for resource AddCellCount, so we will use the resource logical id as the resource id
2024-09-28 19:13:22,056 | 7 resources found in the stack
2024-09-28 19:13:22,057 | Found Serverless function with name='ReadSqs' and ImageUri='readsqs'
2024-09-28 19:13:22,058 | --base-dir is not presented, adjusting uri ./functions/read_sqs relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,058 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,059 | Skip building pre-built function: ReadSqs
2024-09-28 19:13:22,060 | Found Serverless function with name='SendPipelineResultToCaller' and ImageUri='sendpipelineresulttocaller'
2024-09-28 19:13:22,060 | --base-dir is not presented, adjusting uri ./functions/send_pipeline_result_to_caller relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,061 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,061 | Skip building pre-built function: SendPipelineResultToCaller
2024-09-28 19:13:22,062 | Found Serverless function with name='GetSampleLocations' and ImageUri='getsamplelocations'
2024-09-28 19:13:22,062 | --base-dir is not presented, adjusting uri ./functions/get_sample_locations relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,063 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,063 | Skip building pre-built function: GetSampleLocations
2024-09-28 19:13:22,064 | Found Serverless function with name='ValidateStateMachineInput' and ImageUri='validatestatemachineinput'
2024-09-28 19:13:22,064 | --base-dir is not presented, adjusting uri ./functions/validate_state_machine_input relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,065 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,065 | Skip building pre-built function: ValidateStateMachineInput
2024-09-28 19:13:22,066 | Found Serverless function with name='SfLambda' and ImageUri='sflambda'
2024-09-28 19:13:22,066 | --base-dir is not presented, adjusting uri ./functions/sf_lambda relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,067 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,067 | Skip building pre-built function: SfLambda
2024-09-28 19:13:22,068 | Found Serverless function with name='LinkDatasetToStudy' and ImageUri='linkdatasettostudy'
2024-09-28 19:13:22,068 | --base-dir is not presented, adjusting uri ./functions/link_dataset_to_study relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,069 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,070 | Skip building pre-built function: LinkDatasetToStudy
2024-09-28 19:13:22,070 | Found Serverless function with name='AddCellCount' and ImageUri='addcellcount'
2024-09-28 19:13:22,071 | --base-dir is not presented, adjusting uri ./functions/add_cell_counts relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,071 | --base-dir is not presented, adjusting uri . relative to C:\Users\roya\PycharmProjects\unigen_pipeline\template.yaml
2024-09-28 19:13:22,072 | Skip building pre-built function: AddCellCount
2024-09-28 19:13:22,077 | 7 resources found in the stack
2024-09-28 19:13:22,078 | Found Serverless function with name='ReadSqs' and ImageUri='readsqs'
2024-09-28 19:13:22,079 | Skip building pre-built function: ReadSqs
2024-09-28 19:13:22,080 | Found Serverless function with name='SendPipelineResultToCaller' and ImageUri='sendpipelineresulttocaller'
2024-09-28 19:13:22,080 | Skip building pre-built function: SendPipelineResultToCaller
2024-09-28 19:13:22,081 | Found Serverless function with name='GetSampleLocations' and ImageUri='getsamplelocations'
2024-09-28 19:13:22,081 | Skip building pre-built function: GetSampleLocations
2024-09-28 19:13:22,082 | Found Serverless function with name='ValidateStateMachineInput' and ImageUri='validatestatemachineinput'
2024-09-28 19:13:22,082 | Skip building pre-built function: ValidateStateMachineInput
2024-09-28 19:13:22,083 | Found Serverless function with name='SfLambda' and ImageUri='sflambda'
2024-09-28 19:13:22,084 | Skip building pre-built function: SfLambda
2024-09-28 19:13:22,084 | Found Serverless function with name='LinkDatasetToStudy' and ImageUri='linkdatasettostudy'
2024-09-28 19:13:22,085 | Skip building pre-built function: LinkDatasetToStudy
2024-09-28 19:13:22,086 | Found Serverless function with name='AddCellCount' and ImageUri='addcellcount'
2024-09-28 19:13:22,086 | Skip building pre-built function: AddCellCount
2024-09-28 19:13:22,087 | Error occurred while trying to track an event: Event 'BuildFunctionRuntime' does not accept value 'None'.
2024-09-28 19:13:22,088 | Instantiating build definitions
2024-09-28 19:13:22,089 | No previous build graph found, generating new one
2024-09-28 19:13:22,089 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(None, C:\Users\roya\PycharmProjects\unigen_pipeline, Image, , 817b3ea2-2429-4835-adbb-a19b333d4211, {'SkipBuild': True, 'Dockerfile':
'Dockerfile', 'DockerContext': 'C:\\Users\\roya\\PycharmProjects\\unigen_pipeline\\functions\\send_pipeline_result_to_caller'}, {}, x86_64, []), Function: Function(function_id='SendPipelineResultToCaller', name='SendPipelineResultToCaller',
functionname='SendPipelineResultToCaller', runtime=None, memory=2048, timeout=30, handler=None, imageuri='sendpipelineresulttocaller', packagetype='Image', imageconfig=None, codeuri='C:\\Users\\roya\\PycharmProjects\\unigen_pipeline', environment={'Variables':
{'BUCKET_NAME': ''}}, rolearn=None, layers=[], events=None, metadata={'SkipBuild': True, 'Dockerfile': 'Dockerfile', 'DockerContext': 'C:\\Users\\roya\\PycharmProjects\\unigen_pipeline\\functions\\send_pipeline_result_to_caller', 'SamResourceId':
'SendPipelineResultToCaller'}, inlinecode=None, codesign_config_arn=None, architectures=None, function_url_config=None, function_build_info=<FunctionBuildInfo.SkipBuild: ('SkipBuild', 'A Function which is denoted with SkipBuild in metadata, non buildable')>,
stack_path='', runtime_management_config=None, logging_config=None))
2024-09-28 19:13:22,092 | Building codeuri: C:\Users\roya\PycharmProjects\unigen_pipeline runtime: None metadata: {'SkipBuild': True, 'Dockerfile': 'Dockerfile', 'DockerContext':
'C:\\Users\\roya\\PycharmProjects\\unigen_pipeline\\functions\\send_pipeline_result_to_caller'} architecture: x86_64 functions: SendPipelineResultToCaller
2024-09-28 19:13:22,093 | Building to following folder C:\Users\roya\PycharmProjects\unigen_pipeline\.aws-sam\build\SendPipelineResultToCaller
2024-09-28 19:13:22,093 | Building image for SendPipelineResultToCaller function
2024-09-28 19:13:22,099 | Setting DockerBuildArgs: {} for SendPipelineResultToCaller function
2024-09-28 19:13:22,274 | Failed building function SendPipelineResultToCaller

Build Failed
2024-09-28 19:13:22,284 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-09-28 19:13:22,534 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-09-28 19:13:22,534 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'daa61946-e7ce-47cb-8ccb-830d3fec1e60', 'installationId': '12857e84-3346-496c-987c-3c885abc8da7', 'sessionId': '97bd7ff3-f59d-43ca-a884-b31eeb59a267', 'executionEnvironment': 'CLI',
'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.115.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': 'us-east-1', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin':
'b434c6605bc3c7f9b872ba7661045d74b6075801499be400267f904460194dfb', 'projectName': 'cd7d3ce40c6d284416e2dd8eaf5ea424d5c71abac7e4ccc153d6f8a540b3df6f', 'initialCommit': '4aa3f951373a0e613daf16947fff86ebb08b9a47119e278a9f6b6c3b6a121e9e'}, 'duration': 335, 'exitReason':
'DockerBuildFailed', 'exitCode': 1}}]}
2024-09-28 19:13:22,535 | Unable to find Click Context for getting session_id.
2024-09-28 19:13:22,538 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'e45cf212-76cd-456e-b72a-03e53086cf3f', 'installationId': '12857e84-3346-496c-987c-3c885abc8da7', 'sessionId': '97bd7ff3-f59d-43ca-a884-b31eeb59a267', 'executionEnvironment': 'CLI', 'ci':
False, 'pyversion': '3.11.8', 'samcliVersion': '1.115.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '6b7b23808a894dd99e89f1b74e8946ba', 'time_stamp': '2024-09-28
16:13:21.921', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '66131a6249d54b50a6e9aeafde8b8a2b', 'time_stamp': '2024-09-28 16:13:21.949', 'exception_name': None}]}}}]}
2024-09-28 19:13:23,430 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2024-09-28 19:13:23,433 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: unexpected key 'env' in 'env=PIP_INDEX_URL'

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Windows 11
  2. sam --version: SAM CLI, version 1.115.0
  3. AWS region: us-east-1
{
  "version": "1.115.0",
  "system": {
    "python": "3.11.8",
    "os": "Windows-10-10.0.22631-SP0"
  },
  "additional_dependencies": {
    "docker_engine": "27.2.0",
    "aws_cdk": "Not available",
    "terraform": "1.8.1"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}
hawflau commented 1 month ago

Hi @royassis thanks for raising the issue. Looking at docker's doc, it seems you need to pass in the secrets. i.e. if you run docker build, you need to supply the --secret flag. However, sam build does not support --secret at the moment. Marking it as a feature request. I'll bring this up with team to prioritize it.

royassis commented 1 month ago

Thanks @hawflau

Actually I am also using the --build-context flag in some of my images.
Would be nice if this can also be added to the feature request, or even an option to pass a custom string to the docker build command.

JonZeolla commented 1 month ago

I work around this by buliding and distributing my image lambdas on my own, and then I use sam to deploy the IaC (and other things like generating events for testing, etc.)

royassis commented 1 month ago

@JonZeolla the issue is when I try to debug with the AWS Toolkit plugin it auto builds the image and then fails