search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.52k stars 1.17k forks source link

Bug: SAM does not write Layer build results to `build.toml` unless its table already exists #5585

Open davidjb opened 1 year ago

davidjb commented 1 year ago

Description:

Starting with an empty environment (e.g. no .aws-sam directory or no .aws-sam/build.toml file), attempting to build a layer by itself:

sam build --cached --use-container MyLayer

does not result in the layer's build results being written to .aws-sam/build.toml. In fact, this file does not even get created.

This means that subsequent calls to the same build command will not use the cache, despite cached files being created in .aws-sam/cache and that a problem such as #5507 cannot be worked around.

Edit: currently, a Layer build will only update its entry within .aws-sam/build.toml (e.g. SAM won't create this file just for building a Layer), essentially meaning that in order for a Layer to be cached, it must first at least have been attempted to be cached. A workaround for this situation is to force the creation of .aws-sam/build.toml, building in whatever way is necessary to generate the Layer's entry in this file (or otherwise manually creating the TOML would work too). In my case, it was quickest to just do:

sam build --cached MyFunction                 # this fails to build MyLayer because of issue #5507

# now, .aws-build/build.toml exists so we can carry on
sam build --cached --use-container MyLayer    # this caches, updating build.toml
sam build --cached MyFunction                 # this succeeds now

Steps to reproduce:

  1. Remove any .aws-sam directory or .aws-sam/build.toml file

  2. Define a Layer and Function such as this in template.yaml:

    MyLayer:
 Type: AWS::Serverless::LayerVersion
 Properties:
   ContentUri: layers/mylayer/
   CompatibleRuntimes:
     - provided.al2
 Metadata:
   BuildMethod: makefile

MyFunction:
Type: AWS::Serverless::Function
Metadata:
  BuildMethod: go1.x
Properties:
  CodeUri: functions/myfunction
  Handler: bootstrap
  Runtime: provided.al2
  Layers:
    - !Ref MyLayer
    build-MyLayer:
    mkdir "$(ARTIFACTS_DIR)/bin"
    echo 'dummy' > "$(ARTIFACTS_DIR)/bin/dumy"

  3. Build that layer with: sam build --cached --use-container MyLayer

  4. Observe that no .aws-sam/build.toml file is created.

  5. Try and re-run sam build command and layer will not have been cached.

Observed result:

$ sam build --debug --cached --use-container MyLayer
2023-07-21 04:44:17,571 | No config file found in this directory.
2023-07-21 04:44:17,574 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/home/circleci/project2/samconfig.toml'
2023-07-21 04:44:17,575 | Config file location: /home/circleci/project2/samconfig.toml
2023-07-21 04:44:17,576 | Config file '/home/circleci/project2/samconfig.toml' does not exist
2023-07-21 04:44:17,578 | Using SAM Template at /home/circleci/project2/template.yml
2023-07-21 04:44:17,593 | Using config file: samconfig.toml, config environment: default
2023-07-21 04:44:17,594 | Expand command line arguments to:
2023-07-21 04:44:17,594 | --template_file=/home/circleci/project2/template.yml --cached --use_container --resource_logical_id=MyLayer --mount_with=READ --build_dir=.aws-sam/build
--cache_dir=.aws-sam/cache
2023-07-21 04:44:17,633 | 'build' command is called
2023-07-21 04:44:17,634 | Starting Build use cache
2023-07-21 04:44:17,635 | Starting Build inside a container
2023-07-21 04:44:17,637 | No Parameters detected in the template
2023-07-21 04:44:17,653 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:17,654 | 0 stacks found in the template
2023-07-21 04:44:17,654 | No Parameters detected in the template
2023-07-21 04:44:17,668 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:17,669 | 1 resources found in the stack
2023-07-21 04:44:17,669 | --base-dir is not presented, adjusting uri layers/mylayer/ relative to /home/circleci/project2/template.yml
2023-07-21 04:44:17,672 | 1 resources found in the stack
2023-07-21 04:44:17,674 | Instantiating build definitions
2023-07-21 04:44:17,674 | No previous build graph found, generating new one
2023-07-21 04:44:17,675 | Unique Layer build definition found, adding as new (Layer Build Definition: LayerBuildDefinition(MyLayer, /home/circleci/project2/layers/mylayer, ,
ef3c3c4f-ca7b-4153-9bbd-6e33b801fbd7, makefile, ['provided.al2'], x86_64, {}), Layer: <samcli.lib.providers.provider.LayerVersion object at 0x7f4979f77fd0>)
2023-07-21 04:44:17,676 | Running cached build for runtime makefile for following resources (<bound method LayerBuildDefinition.get_resource_full_paths of
<samcli.lib.build.build_graph.LayerBuildDefinition object at 0x7f4979e08220>>)
2023-07-21 04:44:17,678 | Cache is invalid, running build and copying resources for following layers (MyLayer)
2023-07-21 04:44:17,679 | Building layer 'MyLayer'
2023-07-21 04:44:17,680 | For container layer build, first compatible runtime is chosen as build target for container.

Fetching public.ecr.aws/sam/build-provided.al2:latest-x86_64 Docker container image......
2023-07-21 04:44:17,788 | Mounting /home/circleci/project2/layers/mylayer as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Running workflow 'CustomMakeBuilder'
Running CustomMakeBuilder:CopySource
Copying source file (/tmp/samcli/source/Makefile) to destination (/tmp/samcli/scratch/Makefile)
CustomMakeBuilder:CopySource succeeded
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
executing Make: ['make', '--makefile', '/tmp/samcli/source/Makefile', 'build-MyLayer']
mkdir "/tmp/samcli/artifacts/bin"
echo 'dummy' > "/tmp/samcli/artifacts/bin/dumy"
CustomMakeBuilder:MakeBuild succeeded
2023-07-21 04:44:18,104 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
2023-07-21 04:44:18,105 | Build inside container was successful. Copying artifacts from container to host
2023-07-21 04:44:18,280 | Copying from container: /tmp/samcli/artifacts/. -> /home/circleci/project2/.aws-sam/build/MyLayer
2023-07-21 04:44:18,325 | Build inside container succeeded
2023-07-21 04:44:18,326 | Instantiating build definitions
2023-07-21 04:44:18,327 | No previous build graph found, generating new one
2023-07-21 04:44:18,328 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:18,329 | 1 resources found in the stack

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2023-07-21 04:44:18,330 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-21 04:44:18,332 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-21 04:44:18,332 | Unable to find Click Context for getting session_id.

$ ls .aws-sam/build.toml                                                                                               ls: .aws-sam/build.toml: No such file or directory

Expected result:

Layer to be cached and recorded, available for subsequent runs of sam build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.93.0",
  "system": {
    "python": "3.10.6",
    "os": "Linux-5.19.0-1022-aws-x86_64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.24",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}
mildaniel commented 1 year ago

Thanks for reporting. I was able to reproduce this and we welcome contributions if you'd like to help fix this issue!

moelasmar commented 1 year ago

@davidjb , the steps you mentioned as a workaround does not work on my side. My findings is SAM CLI does not write to build.toml file even if exist in case if SAM CLI was invoked to build specific resources:

https://github.com/aws/aws-sam-cli/blob/46c374f79c4a21869cd02423f91bd0eda1dbee79/samcli/lib/build/app_builder.py#L264

https://github.com/aws/aws-sam-cli/blob/d8418011244a6c21aa63093f8963713288801986/samcli/lib/build/build_graph.py#L320-L321

https://github.com/aws/aws-sam-cli/blob/d8418011244a6c21aa63093f8963713288801986/samcli/lib/build/build_graph.py#L442-L479

I will check with the team if this was done on purpose.

moelasmar commented 1 year ago

I checked this issue with the team, and we confirmed that SAM CLI does not create the build.toml file when ASM CLI build specific resources, because we do not create the build graph in this case, and so we will not be able to create the build.toml file.

The work around for this issue now is to first run sam build --cached which will create the build.toml file, and then the subsequent sam build --cached <<specific resource>> commands executions will check the cached build of this resource, and skipped it if it is already cached.

davidjb commented 1 year ago

@moelasmar So the issue is trying to reach a point where a sam build --cached can succeed in the first place, especially given other issues such as https://github.com/aws/aws-sam-cli/issues/5280. More specifically, a situation where certain resources need to be built in containers and others cannot currently be built in containers, meaning it's effectively impossible without hacks and workarounds for certain setups.

It may be possible to change configurations and hack things up until sam build yields an entry in build.toml, but I'm thinking that a more reliable workaround is to generate a fake resource entry into build.toml, such as:

[layer_build_definitions.00000000-0000-0000-0000-000000000000]
layer_name = "MyLayer"
codeuri = "/path/to/layers/mylayer"
build_method = "makefile"
compatible_runtimes = ["provided.al2"]
architecture = "arm64"
source_hash = "fake"
manifest_hash = ""
layer = "MyLayer"

Once this exists, then running sam build --cached <<specific resource>> updates source_hash correctly after that. Provided the UUID is any fake value, this appears to work.

Could a similar strategy be added into sam itself? For a LayerVersion in particular, there are no dependencies or build graphs, just that resource itself, so outputting the layer's config alone would be sufficient to solve this problem.