search

serverless / serverless-python-requirements

⚡️🐍📦 Serverless plugin to bundle Python packages
MIT License
1.11k stars 290 forks source link

Layers: Requirements being processed different than in non-layer mode #485

Open ffedee7 opened 4 years ago

ffedee7 commented 4 years ago

Bug Report

Description

What did you do?

I had a deployment that worked fine. Then I added a layer for requirements.

What happened?

I tried to deploy it and the Layer was larger than 250mb

  An error occurred: PythonRequirementsLambdaLayer - Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 17cd4a66-de6a-4c4a-b6f5-2d9f9e79440d).

What should've happened?

It should work as before since the requirements haven't changed.

What's the content of your serverless.yml file?

service: service

provider:
  name: aws
  runtime: python3.6
  versionFunctions: false
  region: us-east-1
  timeout: 30

functions:
    function:
        handler: handlers.function.run
        layers:
          - {Ref: PythonRequirementsLambdaLayer}
        events:
          - http:
              method: get
              path: function
plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    layer: true
    dockerizePip: non-linux
    slim: true
    zip: true
    useDownloadCache: true
    dockerRunCmdExtraArgs:
      ['-e', 'CFLAGS=-std=c11', '-v', '${env:PWD}:/src']
    invalidateCaches: true
AndrewFarley commented 4 years ago

To clarify @ffedee7 ... you're saying that before you split your requirements off into a layer this stack worked perfectly and didn't exceed the max size?

Do us a favor and include your requriements.txt file, and include your previous serverless.yml that worked, in addition to showing us the file sizes it generates (before and after) (serverless package && ls -la .serverless).

AndrewFarley commented 4 years ago

I'm guessing real quick, but you probably need to use the exclude feature. If you look at the zip file(s) it generates, you should extract them and look inside. More than likely there's extra stuff in there that's pushing over your limit. Use the exclude feature in the serverless framework to exclude certain files. But, just a thought. Need more info from you.

ffedee7 commented 4 years ago

@AndrewFarley exactly. The serverless.yml that I posted is shrunk version of the original because it's huge (we use the exclude feature). Basically what changed is layer: true to false and the layer on the function.

The requirement is:

./backend
./backend/uberc_code
authlib==0.12.1
aws-requests-auth==0.4.2
base58==2.0.0
beautifulsoup4==4.8.2
cached-property==1.5.1
cachetools==4.0.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
cryptography==2.8
cython==0.29.14
dynet==2.1
elasticsearch-dsl==6.4.0
elasticsearch==6.4.0
google-api-python-client==1.7.11
google-auth-httplib2==0.0.3
google-auth==1.10.2
httplib2==0.16.0
idna==2.8
joblib==0.13.2
langcodes==1.4.1
marisa-trie==0.7.5
maxminddb-geolite2==2018.703
maxminddb==1.5.2
msgpack==0.6.2
nagisa==0.2.5
numpy==1.16.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycparser==2.19
pynamodb==4.2.0
pyyaml==5.3
regex==2018.2.21
requests==2.22.0; python_version >= '3.0'
rsa==4.0
scikit-learn==0.21.1
scipy==1.4.1
sentry-sdk==0.8.0
soupsieve==1.9.5
stripe==2.42.0
tenacity==6.0.0
uritemplate==3.0.1
urllib3==1.25.8
wordfreq==2.2.1

Before:

total 345736
drwxr-xr-x  43 fgrunwald  staff       1376 Mar  3 23:41 .
drwxr-xr-x  36 fgrunwald  staff       1152 Mar  3 23:40 ..
-rw-r--r--   1 fgrunwald  staff       1238 Mar  3 23:41 cloudformation-template-API-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       6828 Mar  3 23:41 cloudformation-template-Admin-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11838 Mar  3 23:41 cloudformation-template-CancelUnderscoresubscription-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9778 Mar  3 23:41 cloudformation-template-CancelUnderscoresubscriptionUnderscorewebhook-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11590 Mar  3 23:41 cloudformation-template-Checkout-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9129 Mar  3 23:41 cloudformation-template-DeleteUnderscoreuser-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       8453 Mar  3 23:41 cloudformation-template-GetUnderscoreallUnderscorenewUnderscorelostUnderscorebacklinks-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12020 Mar  3 23:41 cloudformation-template-GetUnderscorebacklinksUnderscorerefdomainsUnderscorehistory-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       4213 Mar  3 23:41 cloudformation-template-GetUnderscorenewUnderscorelostUnderscorebacklinks-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11215 Mar  3 23:41 cloudformation-template-GetUnderscoreplans-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11272 Mar  3 23:41 cloudformation-template-GetUnderscorethumbnail-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11738 Mar  3 23:41 cloudformation-template-GetUnderscoreuser-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11666 Mar  3 23:41 cloudformation-template-GoogleUnderscoreauth-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11411 Mar  3 23:41 cloudformation-template-GoogleUnderscoreoauth2Underscorecallback-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11724 Mar  3 23:41 cloudformation-template-GoogleUnderscorerevoke-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11631 Mar  3 23:41 cloudformation-template-Onboarding-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11253 Mar  3 23:41 cloudformation-template-PageUnderscoreshares-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       1301 Mar  3 23:41 cloudformation-template-Policies-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       4522 Mar  3 23:41 cloudformation-template-ProcessUnderscoreaudit-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9288 Mar  3 23:41 cloudformation-template-ProjectUnderscoreallUnderscoreprojects-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11962 Mar  3 23:41 cloudformation-template-ProjectUnderscoreavailableUnderscoredomains-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11978 Mar  3 23:41 cloudformation-template-ProjectUnderscoreavailableUnderscorekeywords-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9419 Mar  3 23:41 cloudformation-template-ProjectUnderscorecreateUnderscoreuserUnderscoreproject-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12177 Mar  3 23:41 cloudformation-template-ProjectUnderscoregetUnderscorehistoricUnderscorebacklinks-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12307 Mar  3 23:41 cloudformation-template-ProjectUnderscoregetUnderscorehistoricUnderscoresearchUnderscoretraffic-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12228 Mar  3 23:41 cloudformation-template-ProjectUnderscoregetUnderscorekeywordUnderscorepositionUnderscoreinfo-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12058 Mar  3 23:41 cloudformation-template-ProjectUnderscoreprocessUnderscorerankUnderscoretracking-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       6950 Mar  3 23:41 cloudformation-template-ProjectUnderscoreupdater-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11687 Mar  3 23:41 cloudformation-template-ProjectUnderscorewithUnderscoreidUnderscorehandlers-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11564 Mar  3 23:41 cloudformation-template-Signup-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11922 Mar  3 23:41 cloudformation-template-UpdateUnderscorepaymentUnderscoremethod-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11704 Mar  3 23:41 cloudformation-template-UpdateUnderscoreplan-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9792 Mar  3 23:41 cloudformation-template-UpdatedUnderscoresubscriptionUnderscorewebhook-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11152 Mar  3 23:41 cloudformation-template-Webproxy-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       1653 Mar  3 23:38 cloudformation-template-create-stack.json
-rw-r--r--   1 fgrunwald  staff     313835 Mar  3 23:41 cloudformation-template-update-stack.json
-rw-r--r--   1 fgrunwald  staff    6707046 Mar  3 23:41 custom-resources.zip
-rw-r--r--   1 fgrunwald  staff  164685346 Mar  3 23:40 new-ubersuggest.zip
lrwxr-xr-x   1 fgrunwald  staff        134 Mar  3 23:38 requirements -> /Users/fgrunwald/Library/Caches/serverless-python-requirements/81eaec5b81ba22dfb0af9aa43bcc875a9840857157326ec20aac1aa03d0436eb_slspyc
-rw-r--r--   1 fgrunwald  staff        846 Mar  3 23:38 requirements.txt
-rw-r--r--   1 fgrunwald  staff     457003 Mar  3 23:41 serverless-state.json

After:

total 674120
drwxr-xr-x  44 fgrunwald  staff       1408 Mar  3 23:50 .
drwxr-xr-x  36 fgrunwald  staff       1152 Mar  3 23:47 ..
-rw-r--r--   1 fgrunwald  staff       1238 Mar  3 23:50 cloudformation-template-API-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       7021 Mar  3 23:50 cloudformation-template-Admin-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12031 Mar  3 23:50 cloudformation-template-CancelUnderscoresubscription-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9971 Mar  3 23:50 cloudformation-template-CancelUnderscoresubscriptionUnderscorewebhook-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11783 Mar  3 23:50 cloudformation-template-Checkout-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9322 Mar  3 23:50 cloudformation-template-DeleteUnderscoreuser-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       8646 Mar  3 23:50 cloudformation-template-GetUnderscoreallUnderscorenewUnderscorelostUnderscorebacklinks-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12213 Mar  3 23:50 cloudformation-template-GetUnderscorebacklinksUnderscorerefdomainsUnderscorehistory-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       4213 Mar  3 23:50 cloudformation-template-GetUnderscorenewUnderscorelostUnderscorebacklinks-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11408 Mar  3 23:50 cloudformation-template-GetUnderscoreplans-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11465 Mar  3 23:50 cloudformation-template-GetUnderscorethumbnail-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11931 Mar  3 23:50 cloudformation-template-GetUnderscoreuser-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11859 Mar  3 23:50 cloudformation-template-GoogleUnderscoreauth-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11604 Mar  3 23:50 cloudformation-template-GoogleUnderscoreoauth2Underscorecallback-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11917 Mar  3 23:50 cloudformation-template-GoogleUnderscorerevoke-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11824 Mar  3 23:50 cloudformation-template-Onboarding-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11446 Mar  3 23:50 cloudformation-template-PageUnderscoreshares-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       1301 Mar  3 23:50 cloudformation-template-Policies-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       4522 Mar  3 23:50 cloudformation-template-ProcessUnderscoreaudit-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9481 Mar  3 23:50 cloudformation-template-ProjectUnderscoreallUnderscoreprojects-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12155 Mar  3 23:50 cloudformation-template-ProjectUnderscoreavailableUnderscoredomains-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12171 Mar  3 23:50 cloudformation-template-ProjectUnderscoreavailableUnderscorekeywords-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9612 Mar  3 23:50 cloudformation-template-ProjectUnderscorecreateUnderscoreuserUnderscoreproject-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12370 Mar  3 23:50 cloudformation-template-ProjectUnderscoregetUnderscorehistoricUnderscorebacklinks-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12500 Mar  3 23:50 cloudformation-template-ProjectUnderscoregetUnderscorehistoricUnderscoresearchUnderscoretraffic-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12421 Mar  3 23:50 cloudformation-template-ProjectUnderscoregetUnderscorekeywordUnderscorepositionUnderscoreinfo-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12251 Mar  3 23:50 cloudformation-template-ProjectUnderscoreprocessUnderscorerankUnderscoretracking-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       7143 Mar  3 23:50 cloudformation-template-ProjectUnderscoreupdater-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11880 Mar  3 23:50 cloudformation-template-ProjectUnderscorewithUnderscoreidUnderscorehandlers-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11757 Mar  3 23:50 cloudformation-template-Signup-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      12115 Mar  3 23:50 cloudformation-template-UpdateUnderscorepaymentUnderscoremethod-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11897 Mar  3 23:50 cloudformation-template-UpdateUnderscoreplan-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       9985 Mar  3 23:50 cloudformation-template-UpdatedUnderscoresubscriptionUnderscorewebhook-nested-stack.json
-rw-r--r--   1 fgrunwald  staff      11345 Mar  3 23:50 cloudformation-template-Webproxy-nested-stack.json
-rw-r--r--   1 fgrunwald  staff       1653 Mar  3 23:45 cloudformation-template-create-stack.json
-rw-r--r--   1 fgrunwald  staff     321826 Mar  3 23:50 cloudformation-template-update-stack.json
-rw-r--r--   1 fgrunwald  staff    6707044 Mar  3 23:50 custom-resources.zip
-rw-r--r--   1 fgrunwald  staff  164690706 Mar  3 23:47 new-ubersuggest.zip
-rw-r--r--   1 fgrunwald  staff  154794485 Mar  3 23:49 pythonRequirements.zip
lrwxr-xr-x   1 fgrunwald  staff        134 Mar  3 23:45 requirements -> /Users/fgrunwald/Library/Caches/serverless-python-requirements/81eaec5b81ba22dfb0af9aa43bcc875a9840857157326ec20aac1aa03d0436eb_slspyc
-rw-r--r--   1 fgrunwald  staff        846 Mar  3 23:45 requirements.txt
-rw-r--r--   1 fgrunwald  staff     472525 Mar  3 23:50 serverless-state.json

Inside the requirements folder (they should be the same in both cases, since they don't change)

4.0K    ./apiclient
4.0K    ./uberc
 12K    ./base58
 12K    ./estimate_clicks
 16K    ./aws_requests_auth
 32K    ./uritemplate
 36K    ./pyximport
 40K    ./maxminddb
 40K    ./tests
 44K    ./cachetools
 48K    ./.libs_cffi_backend
 80K    ./jmespath
 80K    ./tenacity
 96K    ./ubs_models
116K    ./bin
120K    ./rsa
136K    ./soupsieve
180K    ./ubs_clients
196K    ./elasticsearch_dsl
204K    ./handlers
204K    ./msgpack
208K    ./googleapiclient
212K    ./requests
220K    ./ubs_commons
244K    ./httplib2
248K    ./yaml
268K    ./idna
272K    ./s3transfer
272K    ./sentry_sdk
276K    ./google
288K    ./certifi
320K    ./pynamodb
396K    ./cffi
408K    ./urllib3
420K    ./pyasn1
428K    ./elasticsearch
432K    ./bs4
444K    ./chardet
468K    ./dateutil
488K    ./pkg_resources
576K    ./pycparser
740K    ./stripe
872K    ./pyasn1_modules
1.0M    ./authlib
1.0M    ./boto3
1.1M    ./joblib
1.4M    ./setuptools
2.2M    ./docutils
4.3M    ./cryptography
7.0M    ./.libs_dynet
7.0M    ./Cython
7.2M    ./langcodes
 19M    ./sklearn
 35M    ./wordfreq
 39M    ./ubs_assets
 42M    ./botocore
 42M    ./numpy
 45M    ./nagisa
 54M    ./_maxminddb_geolite2
 78M    ./scipy
399M    .
399M    total
ffedee7 commented 4 years ago

@AndrewFarley my guess is that zip: true feature isn't working with Layers

AndrewFarley commented 4 years ago

That’s a possibility. I’ll confirm later this evening and let you know. If this is the case a quick workaround for this could be to split up your requirements into two layers.

ffedee7 commented 4 years ago

@AndrewFarley That wouldn't work because the sum of Layers will still be > 250MB. Thanks!

ffedee7 commented 4 years ago

@AndrewFarley do you have news about this??

AndrewFarley commented 4 years ago

@ffedee7

Sorry I haven't taken the time to look into it... but I did just now... it looks like this is a totally valid edge case that we're not handling. Currently when you enable layers it disables zipping the requirements.

See: https://github.com/UnitedIncome/serverless-python-requirements/blob/master/lib/inject.js#L90

So, just having a quick think, I'm not sure layers could use compression just because of how they work. We would need to expand the unzip_requirements code to support any number of zipped layer zip files which currently it can only be a single zip file. To support that, those zip file names would need to differ from layer to layer which currently its a static named file.

I'd be happy to review PR's if you (or anyone) want to dig into making this possible. It seems like it should be possible, just not how this plugin works currently. If you want a "simple" version of support for your use-case you can omit the idea of having multiple zipped requirements and just modify the code I linked to above to support zipping those requirements first. It might just be a matter of copying code from HERE into an if statement HERE

If you want to test/play with that @ffedee7 or anyone else please do.

tomharvey commented 4 years ago

I just ran into the same edge case here.

While looking at how to zip the requirements in the layer deployment is it worth a warning or error if the yml has both zip and layer options in use?