Need an independent layer of Sharp supporting HEIC

zoellner / sharp-heic-lambda-layer

Lambda Layer providing sharp with HEIC support

MIT License

69 stars 71 forks source link

Need an independent layer of Sharp supporting HEIC #2

Closed abidzafar closed 2 years ago

abidzafar commented 2 years ago

Thanks for this super helpful repo. But I have a few problems getting that sharp layer to support HEIC format.

I created an AWS lambda layer for sharp to support HEIC format via following steps:

Created this folder on AWS cloud9 https://github.com/aws-samples/aws-lambda-layers-aws-sam-examples/tree/master/sharp-layer.
Changed directory: cd ./sharp-layer
Installed sharp dependency: npm install
Created nodejs directory: mkdir -p ./layer/nodejs
Moved the node modules to that newly created directory: mv ./node_modules ./layer/nodejs
Deployed that directory having nodejs as a layer: sam deploy --guided
Added that layer to lambda function using ARN.

Still getting Runtime.UnhandledPromiseRejection: Error: Input buffer contains unsupported image format error on triggering that lambda function.

I tried your repository and followed the readme but this gets install the new function. What I want is to create a layer that I can embed in my function. So can you please guide me on how to achieve an independent layer of sharp that can support HEIC?

abidzafar commented 2 years ago

@zoellner - Would appreciate if you can help me on this.

zoellner commented 2 years ago

I highly recommend to use this repo by setting up a CodeBuild project. The necessary buildspec is included. I am using a CloudFormation stack with a CodeBuild Project that looks like this:

  SharpHEICCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: sharp-heic-lambda-layer
      Description: Sharp Lambda Layer with HEIC Support
      ServiceRole: !Ref CodeBuildRole
      BadgeEnabled: false
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Auth:
          Type: OAUTH
        Type: GITHUB
        Location: https://github.com/zoellner/sharp-heic-lambda-layer.git
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_MEDIUM
        Image: aws/codebuild/standard:5.0
        ImagePullCredentialsType: CODEBUILD
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: SAM_BUCKET
            Type: PLAINTEXT
            Value: !Ref DeploymentBucket
          - Name: STACK_NAME
            Type: PLAINTEXT
            Value: bcb-sharp-heic-lambda-layer
      Cache:
        Type: NO_CACHE
      Triggers:
        FilterGroups:
          - - Type: EVENT
              Pattern: PUSH
              ExcludeMatchedPattern: false
            - Type: HEAD_REF
              Pattern: '^refs/tags/.*'
              ExcludeMatchedPattern: false
        Webhook: true
      EncryptionKey: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/s3

The two things you need to setup in addition are the DeploymentBucket referenced in there and an IAM role for CodeBuild to use. It needs at the very least write access to the DeploymentBucket and a few lambda permissions to publish the layer.

That said, you can deploy the layer from a local machine as well following the readme:

npm run build
SAM_BUCKET=your-s3-bucket npm run deploy

This does create a new lambda layer that you can then reference in your existing functions. The only place this repo creates new functions should be if you deploy the example from the example folder as well but that's not needed if you want to use the layer in your own function.

abidzafar commented 2 years ago

@zoellner - I tried using the aws/codebuild/standard:5.0 and when tried to build artifact via command ./codebuild_build.sh -i aws/codebuild/standard:5.0 -a ~/environment/artifacts -s ~/environment/sharp-heic-lambda-layer got the following error.

Note: artifacts.zip is still generated by above command.

zoellner commented 2 years ago

From what I can tell from this you did not link the codebuild project to the github repo but maybe provided the code manually in some way? The corresponding output from my CodeBuild is like this:

[Container] 2021/10/07 19:17:48 Phase context status code:  Message:
--
106 | [Container] 2021/10/07 19:17:48 Entering phase BUILD
107 | [Container] 2021/10/07 19:17:48 Running command npm run build
108 |  
109 | > sharp-heic-lambda-layer@2.0.2 build /codebuild/output/src550635503/src/github.com/zoellner/sharp-heic-lambda-layer
110 | > sam build --use-container
111 |  
112 | Starting Build inside a container
113 | Building layer 'SharpHEICLayer'
114 | First compatible runtime has been chosen as build runtime
115 |  
116 | Fetching public.ecr.aws/sam/build-nodejs14.x:latest-x86_64 Docker container image..............................................................................................................................................................................................................................................................................................................................................................................................
117 | Mounting /codebuild/output/src550635503/src/github.com/zoellner/sharp-heic-lambda-layer/layer as /tmp/samcli/source:ro,delegated inside runtime container
118 |  
119 | Build Succeeded
120 |  
121 | Built Artifacts  : .aws-sam/build
122 | Built Template   : .aws-sam/build/template.yaml
123 |  
124 | Commands you can use next
125 | =========================
126 | [*] Invoke Function: sam local invoke
127 | [*] Deploy: sam deploy --guided
128 |  
129 | Running CustomMakeBuilder:CopySource
130 | Running CustomMakeBuilder:MakeBuild
131 | Current Artifacts Directory : /tmp/samcli/artifacts
132 |  
133 | [Container] 2021/10/07 19:26:17 Phase complete: BUILD State: SUCCEEDED
134 | [Container] 2021/10/07 19:26:17 Phase context status code:  Message:
135 | [Container] 2021/10/07 19:26:17 Entering phase POST_BUILD
136 | [Container] 2021/10/07 19:26:17 Running command bash -c "if [ /"$CODEBUILD_BUILD_SUCCEEDING/" == /"0/" ]; then exit 1; fi"
137 |  
138 | [Container] 2021/10/07 19:26:17 Running command npm run deploy

abidzafar commented 2 years ago

@zoellner - I followed the following steps:

Cloned these images https://github.com/aws/aws-codebuild-docker-images
And switched the directory to ubuntu/standard/5.0 using cd aws-codebuild-docker-images/ubuntu/standard/5.0
Then I build the codebuild using command docker build -t aws/codebuild/standard:5.0 .
It took some time and when it completed I pulled the codebuild using docker pull amazon/aws-codebuild-local:latest --disable-content-false=false
Then I downloaded the script for codebuild https://raw.githubusercontent.com/aws/aws-codebuild-docker-images/master/local_builds/codebuild_build.sh
And using the script executed it with sharp-heic-lambda-layer repository code as ./codebuild_build.sh -i aws/codebuild/standard:5.0 -a ~/environment/artifacts -s ~/environment/sharp-heic-lambda-layer

On this command I got the above error. Also, I'm doing this all on AWS Cloud9 IDE environment.

zoellner commented 2 years ago

Not sure how you came up with these steps. All you need to do is to deploy the above Cloudformation snippet to setup a CodeBuild project. Might have to create a connection to Github first (it's been a while since I connected my AWS accounts, might not be needed for the public repo).

No need to clone images or pull any docker images manually. Just let CodeBuild do its job and run everything for you.

abidzafar commented 2 years ago

It worked, I was missing permissions that was causing problems in creating stack. I used the code manually by pulling the repository locally and uploading to AWS IDE.

s3 ksm arn:aws:cloudformation sqs lambda

Thanks for all your help @zoellner ! 😊

agcty commented 1 year ago

I don't entirely understand if you're supposed to create one big template.yaml that defines all resources for a project, eg. lambda function, sharp layer, s3 bucket etc, or if you you should separate them into their own stacks? Do you have any recommendation @zoellner would be awesome to hear an expert's opinion :)

zoellner commented 1 year ago

I would recommend to use the above CF template snippet to setup the build project (together with a DeploymentBucket in case you don't have one already for other build things, and a CodeBuildRole) That will create the lambda layer that you can then use elsewhere.

I would have all your application related resources (lambda function, etc) in a separate CF stack.

The thinking behind this is that the build project etc rarely changes and you might use it across different applications, so having it integrated with one application would mean you need to do things differently when you start a second application.

agcty commented 1 year ago

This is what worked for me now:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  Codebuild heic layer
Resources:
  DeploymentBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: photosai-user-images
  BuilderIAMRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: photosai-codebuild-role
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "codebuild.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: "codebuild-policy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: "Allow"
                Action:
                  - "s3:PutObject"
                Resource: !Sub "arn:aws:s3:::${DeploymentBucket}/*"
              - Effect: "Allow"
                Action:
                  - "lambda:AddLayerVersionPermission"
                  - "lambda:PublishLayerVersion"
                Resource: "*"

  SharpHEICCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: sharp-heic-lambda-layer
      Description: Sharp Lambda Layer with HEIC Support
      ServiceRole: !Ref BuilderIAMRole
      BadgeEnabled: false
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Auth:
          Type: OAUTH
        Type: GITHUB
        Location: https://github.com/zoellner/sharp-heic-lambda-layer.git
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_MEDIUM
        Image: aws/codebuild/standard:5.0
        ImagePullCredentialsType: CODEBUILD
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: SAM_BUCKET
            Type: PLAINTEXT
            Value: !Ref DeploymentBucket
          - Name: STACK_NAME
            Type: PLAINTEXT
            Value: bcb-sharp-heic-lambda-layer
      Cache:
        Type: NO_CACHE
      # Triggers:
      #   FilterGroups:
      #     - - Type: EVENT
      #         Pattern: PUSH
      #         ExcludeMatchedPattern: false
      #       - Type: HEAD_REF
      #         Pattern: "^refs/tags/.*"
      #         ExcludeMatchedPattern: false
      #   Webhook: true
      EncryptionKey: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/s3

Note that I had to comment out the Triggers section. This is because codebuild will try to connect to @zoellner's github repo and create a webhook that will re-trigger building but it obviously doesn't have permission to do so.

agcty commented 1 year ago

Ok scrap my previous comment, what I did is fork the repo so the trigger still works. This is what works for me now:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  Codebuild heic layer
Resources:
  DeploymentBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: photosai-codebuild-deployments
  BuilderIAMRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: photosai-codebuild-role
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "codebuild.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: "codebuild-policy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: "Allow"
                Action:
                  - "s3:PutObject"
                  - "s3:PutObjectAcl"
                  - "s3:GetObject"
                Resource: !Sub "arn:aws:s3:::${DeploymentBucket}/*"
              - Effect: "Allow"
                Action:
                  - "lambda:AddLayerVersionPermission"
                  - "lambda:PublishLayerVersion"
                  - "lambda:GetLayerVersion"
                  - "lambda:PublishLayerVersion"
                  - "lambda:DeleteLayerVersion"
                  - "lambda:ListLayerVersions"
                  - "lambda:ListLayers"
                  - "lambda:AddLayerVersionPermission"
                  - "lambda:RemoveLayerVersionPermission"
                Resource: "*"
              - Effect: "Allow"
                Action:
                  - "cloudformation:CreateChangeSet"
                  - "cloudformation:DescribeStacks"
                  - "cloudformation:DescribeStackEvents"
                  - "cloudformation:DescribeChangeSet"
                  - "cloudformation:ExecuteChangeSet"
                Resource: "*"

  SharpHEICCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: sharp-heic-lambda-layer
      Description: Sharp Lambda Layer with HEIC Support
      ServiceRole: !Ref BuilderIAMRole
      BadgeEnabled: false
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Auth:
          Type: OAUTH
        Type: GITHUB
        Location: https://github.com/agcty/sharp-heic-lambda-layer.git
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_MEDIUM
        Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0
        ImagePullCredentialsType: CODEBUILD
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: SAM_BUCKET
            Type: PLAINTEXT
            Value: !Ref DeploymentBucket
          - Name: STACK_NAME
            Type: PLAINTEXT
            Value: bcb-sharp-heic-lambda-layer
      Cache:
        Type: NO_CACHE
      Triggers:
        FilterGroups:
          - - Type: EVENT
              Pattern: PUSH
              ExcludeMatchedPattern: false
            - Type: HEAD_REF
              Pattern: "^refs/tags/.*"
              ExcludeMatchedPattern: false
        Webhook: true
      EncryptionKey: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/s3

deathemperor commented 1 year ago

Ok scrap my previous comment, what I did is fork the repo so the trigger still works. This is what works for me now:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  Codebuild heic layer
Resources:
  DeploymentBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: photosai-codebuild-deployments
  BuilderIAMRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: photosai-codebuild-role
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "codebuild.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: "codebuild-policy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: "Allow"
                Action:
                  - "s3:PutObject"
                  - "s3:PutObjectAcl"
                  - "s3:GetObject"
                Resource: !Sub "arn:aws:s3:::${DeploymentBucket}/*"
              - Effect: "Allow"
                Action:
                  - "lambda:AddLayerVersionPermission"
                  - "lambda:PublishLayerVersion"
                  - "lambda:GetLayerVersion"
                  - "lambda:PublishLayerVersion"
                  - "lambda:DeleteLayerVersion"
                  - "lambda:ListLayerVersions"
                  - "lambda:ListLayers"
                  - "lambda:AddLayerVersionPermission"
                  - "lambda:RemoveLayerVersionPermission"
                Resource: "*"
              - Effect: "Allow"
                Action:
                  - "cloudformation:CreateChangeSet"
                  - "cloudformation:DescribeStacks"
                  - "cloudformation:DescribeStackEvents"
                  - "cloudformation:DescribeChangeSet"
                  - "cloudformation:ExecuteChangeSet"
                Resource: "*"

  SharpHEICCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: sharp-heic-lambda-layer
      Description: Sharp Lambda Layer with HEIC Support
      ServiceRole: !Ref BuilderIAMRole
      BadgeEnabled: false
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Auth:
          Type: OAUTH
        Type: GITHUB
        Location: https://github.com/agcty/sharp-heic-lambda-layer.git
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_MEDIUM
        Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0
        ImagePullCredentialsType: CODEBUILD
        PrivilegedMode: true
        EnvironmentVariables:
          - Name: SAM_BUCKET
            Type: PLAINTEXT
            Value: !Ref DeploymentBucket
          - Name: STACK_NAME
            Type: PLAINTEXT
            Value: bcb-sharp-heic-lambda-layer
      Cache:
        Type: NO_CACHE
      Triggers:
        FilterGroups:
          - - Type: EVENT
              Pattern: PUSH
              ExcludeMatchedPattern: false
            - Type: HEAD_REF
              Pattern: "^refs/tags/.*"
              ExcludeMatchedPattern: false
        Webhook: true
      EncryptionKey: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/s3

this helped me in building with CodeBuild, however there's still some permission problem when deploying the artifact to lambda layer with command npm run deploy

Error: Failed to create/update the stack: sharp-heic-lambda-layer, An error occurred (AccessDenied) when calling the DescribeStacks operation: User: arn:aws:sts::*:assumed-role/codebuild-heic-service-role/AWSCodeBuild-f74d2566-* is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:ap-southeast-1:*:stack/sharp-heic-lambda-layer/* because no identity-based policy allows the cloudformation:DescribeStacks action

The artifact is created however, I manually create the layer from that file.

zoellner commented 1 year ago

@deathemperor are you trying to do a cross account deploy or something? The error you posted would be unexpected given the role which has an explicit Allow statement for that action

deathemperor commented 1 year ago

@deathemperor are you trying to do a cross account deploy or something? The error you posted would be unexpected given the role which has an explicit Allow statement for that action

No I was doing same account.