search

serverless / serverless-python-requirements

⚡️🐍📦 Serverless plugin to bundle Python packages
MIT License
1.11k stars 290 forks source link

`exclude` does not work on dependencies #138

Open hinnerk opened 6 years ago

hinnerk commented 6 years ago

Files that are installed via requirements.txt can't be excluded. For example this doesn't work:

package:
  exclude:
    - __pycache__/**
    - pandas/tests/**
dschep commented 6 years ago

That's on purpose so that you don't have to enumerate the packages if you go with an exclude all, then include some approach that I often use:

package:
  exclude:
    - '**/*'
  include:
    - handler.py

I think the best approach to what you're asking for would be to implement a plugin specific exclude option that would make your example look like:

custom:
  pythonRequirements:
    exclude:
      - __pycache__/**
      - pandas/tests/**

or just a toggle to turn off the plugin's code that adds to the includes:

custom:
  pythonRequirements:
    modifyPackageInclude: false
dschep commented 6 years ago

I don't really have the time to implement either feature, but would be happy to accept a PR for either approach.

hinnerk commented 6 years ago

We've more or less solved that for us with the serverless-scriptable-plugin:

plugins:
  - serverless-scriptable-plugin

custom:
  scriptHooks:
    before:package:createDeploymentArtifacts:
      - "rm -rf .serverless/requirements/pandas/tests || echo ERR: UNABLE TO REMOVE pandas/tests"
      - "rm -rf .serverless/requirements/numpy/core/tests || echo ERR: UNABLE TO REMOVE numpy/core/tests"
dschep commented 6 years ago

Nice workaround using a great plugin!

ghost commented 5 years ago

I'm trying to remove some language models from spacy to reduce the overall size, but when the above is used in combination with serverless-python-requirements this doesn't seem to work.

What I wrote

scriptHooks:
    before:package:createDeploymentArtifacts:
      - "mv .serverless/requirements/spacy/lang .serverless/requirements/spacy/lang_old"
      - "mkdir .serverless/requirements/spacy/lang"
      - "mv .serverless/requirements/spacy/lang_old/en .serverless/requirements/spacy/lang/"
      - "mv .serverless/requirements/spacy/lang_old/xx .serverless/requirements/spacy/lang/"
      - "cp .serverless/requirements/spacy/lang_old/*.py .serverless/requirements/spacy/lang/"
      - "rm -rf .serverless/requirements/spacy/lang_old"

Output:

Serverless: Adding Python requirements helper...
Serverless: Generated requirements from /home/user/Projects/test-spacy/requirements.txt in /home/user/Projects/test-spacy/.serverless/requirements.txt...
Serverless: Installing requirements from /home/user/Projects/test-spacy/.serverless/requirements/requirements.txt ...
Serverless: Running ...
Serverless: Zipping required Python packages...
Running command: mv .serverless/requirements/spacy/lang .serverless/requirements/spacy/lang_old
Running command: mkdir .serverless/requirements/spacy/lang
Running command: mv .serverless/requirements/spacy/lang_old/en .serverless/requirements/spacy/lang/
Running command: mv .serverless/requirements/spacy/lang_old/xx .serverless/requirements/spacy/lang/
Running command: cp .serverless/requirements/spacy/lang_old/*.py .serverless/requirements/spacy/lang/
Running command: rm -rf .serverless/requirements/spacy/lang_old
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Removing Python requirements helper...
Serverless: Injecting required Python packages to package...

Interesting that the .serverless/requirements folder which is used for creating the .requirements.zip has been modified appropriately. Any guidelines would be most helpful!

Sam152 commented 3 years ago

Hey @kalinkirev, I found that I had to remove files from the built zip file to get this working. I wrote a blog post with an example here: https://dev.to/sam152/crude-python-tree-shaking-for-squeezing-into-aws-lambda-package-size-limits-357a