Reuse image for functions with same build metadata

billyshambrook commented 3 years ago

Description:

It looks like if you have 2 functions using PackageType: Image using the same Dockerfile, sam build does not correctly add ImageUri to all functions.

I have traced through the code that all functions are added to the same build_definition but the build results are not copied to all functions. There is an explicit if condition to only do this to ZIP, is there a reason for this or it's just a feature not supported yet?

https://github.com/aws/aws-sam-cli/blob/12efe15b682926b006ce2a846d7cc6c71f5f2d74/samcli/lib/build/build_strategy.py#L117

Steps to reproduce:

Example template:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Parameters:
  Tag:
    Type: String
    Default: latest
    Description: Docker tag to build and deploy.

Globals:
  Function:
    Timeout: 5

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageConfig:
        Command:
          - foo
    Metadata:
      DockerTag: !Ref Tag
      DockerContext: .

  BarFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageConfig:
        Command:
          - bar
    Metadata:
      DockerTag: !Ref Tag
      DockerContext: .

Observed result:

The "built" template only has ImageUri on the BarFunction:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  Tag:
    Type: String
    Default: latest
    Description: Docker tag to build and deploy.
Globals:
  Function:
    Timeout: 5
Resources:
  BarFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageConfig:
        Command:
        - bar
      ImageUri: barfunction:latest
    Metadata:
      DockerTag:
        Ref: Tag
      DockerContext: .
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageConfig:
        Command:
        - foo
    Metadata:
      DockerTag:
        Ref: Tag
      DockerContext: .

Expected result:

Expect to re-use the same image for multiple functions.

I can however see an issue because the image name includes the first function name.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: mac
sam --version: 1.13.2

billyshambrook commented 3 years ago

One workaround I have found is to use a different DockerTag for each function. This way sam thinks the build is different but docker will reuse the same image:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Parameters:
  Tag:
    Type: String
    Default: latest
    Description: Docker tag to build and deploy.

Globals:
  Function:
    Timeout: 5

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageConfig:
        Command:
          - foo
    Metadata:
      DockerTag: !Sub foo-${Tag}
      DockerContext: .

  BarFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageConfig:
        Command:
          - bar
    Metadata:
      DockerTag: !Sub bar-${Tag}
      DockerContext: .

jasonmk commented 3 years ago

Came here to report the same thing. The problem comes from here: https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/build/app_builder.py#L183

The built_artifacts only lists the first function because that's the only one that actually performed any build work. I'm still tracing through the code to figure out how that gets populated, but thank you for the workaround. That will at least get me past my current roadblock.

metaskills commented 3 years ago

I accidentally opened #2514 without realizing it duplicated this issue. Closing that one and joining the conversation here. I was using a symlink as a workaround in my post. But I can confirm that I can remove ImageUri and use a different DockerTag as @billyshambrook suggested and it works. THANK YOU!!!

jasonmk commented 3 years ago

FYI, I found another bug, though I believe this one is much deeper than SAM (or at least the part that runs locally). It appears that if you specify a Command in the template it will lose track of the EntryPoint. I found that I had to specify both Command and Entrypoint ('/lambda-entrypoint.sh') in order to be able to successfully override the command. If you run into issues about missing entry points then give that a shot.

jfuss commented 3 years ago

@jasonmk That is a know bug with how the CFN resources for Lambda (currently) work. The correct teams are aware internally already.

nitsujri commented 3 years ago

This solves the problem of getting the image-repository to populate for each function since now there's a Tag applied to each, but now this causes the same image to be rebuilt for each function.

The context loading for each function build causes it to take a while on larger containers even if the build itself is fast.

Is it possible to only build once?

billyshambrook commented 3 years ago

@nitsujri I ran into the exact same problem, as you say, even though docker eventually knows to use its cache, it still takes a while for it to load.

I'm not sure what contributes to this loading time but I'm assuming it's docker itself rather than aws sam cli. If that's the case, I haven't myself been able to find anything within docker to speed this up.

Taking a look at sam, this line seems to show it explicitly decides to not use the same caching logic as non image builds https://github.com/aws/aws-sam-cli/blob/686369638c9d23f2cf9d5821177bce716dd8893f/samcli/lib/build/build_strategy.py#L213

This makes sense as, even though the context is set for each function, not everything within that context may be copied into the image as that is controlled within the Dockerfile and/or dockerignore files.

However maybe at least caching based on the context is a step in the right direction - might still cover most user cases...

Am happy to contribute if there is an acceptable path forward to utilize sam cache build strategy for images!

nitsujri commented 3 years ago

@billyshambrook yeah the time I'm complaining about is docker build copying the context. THe time is consumed for each function in the current workaround, in my case 3x.

For now, since SAM must build all functions individually, I'm avoiding using sam build and sam package and returned to docker build w/ ecr commands.

echo "== Docker Build..."
docker build --tag $IMAGE_URI \
  --file ./docker/Dockerfile.production $LAMBY_BUILD_CACHE_DIR

echo "== ECR Login..."
aws ecr get-login-password --profile $AWS_PROFILE | docker login --username AWS --password-stdin $IMAGE_REPOSITORY

echo "== ECR Push..."
docker push $IMAGE_URI

echo "== SAM deploy..."
sam deploy \
  --profile "${AWS_PROFILE}" \
  --template-file ./template.yaml \
  --stack-name "lamby-discovery-${RAILS_ENV}" \
  --image-repository "$IMAGE_REPOSITORY" \
  --capabilities "CAPABILITY_IAM" \
  --parameter-overrides \
    RailsEnv="${RAILS_ENV}" \
    RailsImageUri=$IMAGE_URI

This completely sidesteps the problem in a rather ugly way.

This does mean I lost the template.yaml packaging ability which is nice for fancy CF template things, but my project is a pathfinder (heavily modified lamby) so it's quite simple and doesn't need it. If I do need it, I could use aws cloudformation package and get similar functionality without the docker steps (I haven't tried it yet).

Going back to the issue at hand. It would probably have to be similar to:

In sam build, if the Build Context && Dockerfile are the same, don't rebuild just reuse previous build, (or tag if different tag).

I can agree though that feels quite brittle and the number of corner cases could be quite high. Perhaps manual control of the ImageUri for each function is the only real way.

DarrinProtag commented 3 years ago

I'm hitting this too, in testing out SAM. I'm concerned, because I expect to have dozens of lambdas that reuse the same image (same .net code base). Waiting while uploading what amounts to the same image dozens of times to different ecr repos for every build doesn't sound great. Perhaps the concerns about automatic detection of the image being identical can be addressed by allowing us to pass a variable or flag on 2nd, 3rd function resources' ImageURI:. Or maybe that's already a possibility, if the format of the auto-created image name were constant/predictable (automatic repo names currently contain guids or hashes), and things were guaranteed to happen in order.