agutoli / serverless-layers

Serverless.js plugin implementing AWS Lambda Layers, reducing lambda size, warm-up, and deployment time.
MIT License
229 stars 54 forks source link

Python Layers using other layers packages #153

Closed ayush-mehta closed 7 months ago

ayush-mehta commented 8 months ago

Using serverless-layers@2.7 Getting the same issue mentioned in #114 @agutoli you mentioned that you made some changes in version 2.7 so it should work, but I am facing the same issue, here is my serverless.yml file


  serverless-layers:
    - numpy-pandas:
        layersDeploymentBucket: clara-lambda-layers
        dependenciesPath: layers/numpy-pandas/requirements.txt
    - pdf2image-deskew:
        layersDeploymentBucket: clara-lambda-layers
        functions:
          - PytesseractFetchOCR
        dependenciesPath: layers/pdfPreprocessing/requirements.txt
    - pdfplumber:
        layersDeploymentBucket: clara-lambda-layers
        functions:
          - PdfPlumberFetchOCR
        dependenciesPath: layers/pdfplumber/requirements.txt
    - pytesseract:
        layersDeploymentBucket: clara-lambda-layers
        functions:
          - PytesseractFetchOCR
        dependenciesPath: layers/pytesseract/requirements.txt```
agutoli commented 8 months ago

@ayush-mehta You are using OpenCV, Pandas, NumPy, and other large libraries, which are probably exceeding the Lambda limit/quota. https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

ayush-mehta commented 8 months ago

I agree that the libraries are large that is why I am breaking them into multiple layers, I have checked individually that each layer's requirements are less then 250MB individually. But the thing that is happening here is that the plugin is appending the requirements of the second layer into the first one. I tried it out and checked in s3 bucket where the zip files are stored, it created the first layer perfectly, but when installing the second layer, it again installed the first layer's requirement into the same folder, so I believe the issue first described in #114 is recurring here. Correct me if I am wrong anywhere @agutoli

agutoli commented 7 months ago

@ayush-mehta sorry for late reply (really busy with my job). I've published a bugfix:

serverless-layers@2.8.3