chalice deployment package missing libraries

ereboschi commented 6 years ago

When my Bitbucket pipeline runs chalice deploy --stage dev, the deployment works just fine. When I run chalice deploy --stage dev from my command line, the package deployed to AWS is missing most libraries... so I get errors like Unable to import module 'app': No module named 'phpserialize' when the Lambda is executed.

I've verified that the deployment packages differ by running deployments, downloading the deployment packages from AWS (using the Lambda console), and diff'ing the contents of the .zip files.

Clearly there's some difference between my personal environment and the Bitbucket pipeline environment that causes a difference in how the deployment .zip is built, but I can't figure it out. Here's what I've tried:

Changing the commands to/from chalice deploy && chalice deploy --stage dev. No affect.
Rebuilding my vendor dependencies: In the past I've been able to 'fix' deployments missing modules by manually rebuilding the wheels in the vendor directory, but this hasn't worked in the past few days: pip download x && pip wheel x-y.y.y.tar.gz && rm *.gz
python/pip versions: Bitbucket was using python 3.6.3 & pip 9.0.1 and my personal machine was python 3.6.5 & pip 9.0.3. I changed my versions 3.6.3/9.0.1 and my deployment packages were still missing libraries. I changed the Bitbucket pipeline to 3.6.5/10.0.1 and the deployment was still perfect.
I've also reviewed issues: #106, #155, #189

Anyone else seen this problem? It's plaguing my office here and some of us have turned to pushing all changes to Bitbucket because we can't trust a local chalice deploy to work.

Bitbucket OS: Linux 44faaabc-f42c-4803-993e-f72fa3365e1e 4.14.48-coreos-r2 #1 SMP Thu Jun 14 08:23:03 UTC 2018 x86_64 GNU/Linux Laptop OS: Darwin Macbook.localdomain 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64

Here's my requirements.txt file:

chalice==1.3.0
phpserialize==1.3
yoyo-migrations==5.1.5

Here're the contents of my vendor directory:

MarkupSafe-1.0-cp36-cp36m-macosx_10_13_x86_64.whl   idna-2.7-py2.py3-none-any.whl
PyYAML-3.13-cp36-cp36m-macosx_10_13_x86_64.whl      iniherit-0.3.9-py3-none-any.whl
argh-0.26.2-py2.py3-none-any.whl            jmespath-0.9.3-py2.py3-none-any.whl
argparse-1.4.0-py2.py3-none-any.whl         pathtools-0.1.2-py3-none-any.whl
asn1crypto-0.24.0-py2.py3-none-any.whl          phpserialize-1.3-py3-none-any.whl
bidict-0.17.2-py3-none-any.whl              pycparser-2.18-py2.py3-none-any.whl
boto3-1.7.42-py2.py3-none-any.whl           python_dateutil-2.7.3-py2.py3-none-any.whl
botocore-1.10.42-py2.py3-none-any.whl           pytz-2018.4-py2.py3-none-any.whl
cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl        s3transfer-0.1.13-py2.py3-none-any.whl
ciso8601-2.0.1.tar.gz                   schemapi-0.3.0-py3-none-any.whl
credstash-1.14.0-py3-none-any.whl           simplejson-3.15.0-cp36-cp36m-macosx_10_13_x86_64.whl
cryptography-2.0.3-cp36-cp36m-macosx_10_6_intel.whl six-1.11.0-py2.py3-none-any.whl
dateutils-0.6.6-py3-none-any.whl            typing-3.5.3.0-py3-none-any.whl
docutils-0.14-py3-none-any.whl              watchdog-0.8.3-cp36-cp36m-macosx_10_13_x86_64.whl
expiringdict-1.1.4-py3-none-any.whl

Here's a sanitized copy of my Bitbucket pipeline file:

image: python:3.6.5

pipelines:
  branches:
    develop:
      - step:
          script:
            - export AWS_ACCESS_KEY_ID=$DEV_AWS_ACCESS_KEY_ID
            - export AWS_SECRET_ACCESS_KEY=$DEV_AWS_SECRET_ACCESS_KEY
            - pip install -r requirements.txt
            - pip install pytest
            - python -m pytest tests/
            - chalice deploy --stage dev
          deployment: staging

Here's a diff of the deployment package (Bitbucket deployed vs. manually deployed):

Only in notifications-poller-dev-Bitbucket: MarkupSafe-1.0.dist-info
Only in notifications-poller-dev-Bitbucket: PyYAML-3.13.dist-info
Only in notifications-poller-dev-Bitbucket: _yaml.cpython-36m-x86_64-linux-gnu.so
Only in notifications-poller-dev-Bitbucket: bidict
Only in notifications-poller-dev-Bitbucket: bidict-0.17.2.dist-info
Only in notifications-poller-dev-Bitbucket: ciso8601-2.0.1.dist-info
Only in notifications-poller-dev-Bitbucket: ciso8601.cpython-36m-x86_64-linux-gnu.so
Only in notifications-poller-dev-Bitbucket: dateutils
Only in notifications-poller-dev-Bitbucket: dateutils-0.6.6.dist-info
Only in notifications-poller-dev-Bitbucket: iniherit
Only in notifications-poller-dev-Bitbucket: iniherit-0.3.9.dist-info
Only in notifications-poller-dev-Bitbucket: markupsafe
Only in notifications-poller-dev-Bitbucket: pathtools
Only in notifications-poller-dev-Bitbucket: pathtools-0.1.2.dist-info
Only in notifications-poller-dev-Bitbucket: phpserialize-1.3.dist-info
Only in notifications-poller-dev-Bitbucket: phpserialize.py
Only in notifications-poller-dev-Bitbucket: pycparser
Only in notifications-poller-dev-Bitbucket: pycparser-2.18.dist-info
Only in notifications-poller-dev-Bitbucket: schemapi
Only in notifications-poller-dev-Bitbucket: schemapi-0.3.0.dist-info
Only in notifications-poller-dev-Bitbucket: watchdog
Only in notifications-poller-dev-Bitbucket: watchdog-0.8.3.dist-info
Only in notifications-poller-dev-Bitbucket: yaml

stealthycoin commented 6 years ago

Clarifying point do you get that error when trying to deploy or when the app is running in lambda? If you are getting it locally that just means you don't have that dependency installed locally which is required.

If you are getting the error during the lambda runtime then thats more of an issue:

First thing that stands out is to remove chalice==1.3.0 from your requirements.txt. That's for runtime dependencies only.

When I make a bundle with the chalice==1.30 removed it looks correct:

$ chalice package out
Creating deployment package.
$ unzip -l out/deployment.zip
Archive:  out/deployment.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
    18400  07-24-18 14:52   phpserialize.py
    30888  07-24-18 14:52   six.py
      648  07-24-18 14:52   iniherit/__init__.py
     5353  07-24-18 14:52   iniherit/cli.py
     4920  07-24-18 14:52   iniherit/interpolation.py
     2684  07-24-18 14:52   iniherit/mixin.py
     9823  07-24-18 14:52   iniherit/parser.py
    19028  07-24-18 14:52   iniherit/test.py
      991  07-24-18 14:52   iniherit/test_cli.py
       48  07-24-18 14:52   iniherit-0.3.9.dist-info/entry_points.txt
     8605  07-24-18 14:52   iniherit-0.3.9.dist-info/METADATA
     1026  07-24-18 14:52   iniherit-0.3.9.dist-info/RECORD
        9  07-24-18 14:52   iniherit-0.3.9.dist-info/top_level.txt
       92  07-24-18 14:52   iniherit-0.3.9.dist-info/WHEEL
        1  07-24-18 14:52   iniherit-0.3.9.dist-info/zip-safe
     7546  07-24-18 14:52   phpserialize-1.3.dist-info/METADATA
      388  07-24-18 14:52   phpserialize-1.3.dist-info/RECORD
       13  07-24-18 14:52   phpserialize-1.3.dist-info/top_level.txt
       92  07-24-18 14:52   phpserialize-1.3.dist-info/WHEEL
     1098  07-24-18 14:52   six-1.11.0.dist-info/DESCRIPTION.rst
     1607  07-24-18 14:52   six-1.11.0.dist-info/METADATA
      703  07-24-18 14:52   six-1.11.0.dist-info/metadata.json
      540  07-24-18 14:52   six-1.11.0.dist-info/RECORD
        4  07-24-18 14:52   six-1.11.0.dist-info/top_level.txt
      110  07-24-18 14:52   six-1.11.0.dist-info/WHEEL
      484  07-24-18 14:52   text_unidecode/__init__.py
   311077  07-24-18 14:52   text_unidecode/data.bin
     1108  07-24-18 14:52   text_unidecode-1.2.dist-info/DESCRIPTION.rst
     6317  07-24-18 14:52   text_unidecode-1.2.dist-info/LICENSE.txt
     2206  07-24-18 14:52   text_unidecode-1.2.dist-info/METADATA
     1197  07-24-18 14:52   text_unidecode-1.2.dist-info/metadata.json
      789  07-24-18 14:52   text_unidecode-1.2.dist-info/RECORD
       15  07-24-18 14:52   text_unidecode-1.2.dist-info/top_level.txt
      110  07-24-18 14:52   text_unidecode-1.2.dist-info/WHEEL
     1020  07-24-18 14:52   yoyo/__init__.py
    18406  07-24-18 14:52   yoyo/backends.py
     1748  07-24-18 14:52   yoyo/compat.py
     2307  07-24-18 14:52   yoyo/config.py
     4011  07-24-18 14:52   yoyo/connections.py
      981  07-24-18 14:52   yoyo/exceptions.py
    18909  07-24-18 14:52   yoyo/migrations.py
     3449  07-24-18 14:52   yoyo/utils.py
      575  07-24-18 14:52   yoyo/scripts/__init__.py
    10116  07-24-18 14:52   yoyo/scripts/main.py
    10878  07-24-18 14:52   yoyo/scripts/migrate.py
     6915  07-24-18 14:52   yoyo/scripts/newmigration.py
     1195  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/DESCRIPTION.rst
       87  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/entry_points.txt
     2328  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/METADATA
     1413  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/metadata.json
     1567  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/RECORD
        5  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/top_level.txt
      110  07-24-18 14:52   yoyo_migrations-5.1.5.dist-info/WHEEL
    37579  07-17-18 15:04   chalice/app.py
      461  03-02-18 14:17   chalice/__init__.py
      732  07-24-18 14:50   app.py
 --------                   -------
   562712                   56 files

Even if I do leave in the chalice==1.3.0 in the requirements file I get a broken deployment but I see that it includes phpserialize:

$ unzip -l badout/deployment.zip | head
Archive:  badout/deployment.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
    18400  07-24-18 14:57   phpserialize.py
    30888  07-24-18 14:57   six.py
    71396  07-24-18 14:57   typing.py
      495  07-24-18 14:57   argh/__init__.py
    17555  07-24-18 14:57   argh/assembling.py
     2834  07-24-18 14:57   argh/compat.py
     2943  07-24-18 14:57   argh/completion.py

I'm not sure I understand the point of the vendor directory there. Its just copying those into the top level and those are still whl files so they will not be importable. Without unzipping those into the vendor directory they won't be usable. You also have a few that are binary whls built for the wrong platform (for example PyYAML-3.13-cp36-cp36m-macosx_10_13_x86_64.whl), which wont be usable even if they were importable in lambda.

So I would suggest not having a vendor directory at all, it doesn't appear to need one with those dependencies from my testing.

Edit: the diff at the bottom does look suspicious. If none of the suggestions help I would add debug logs to the packaging system so we can get a better look at what is going on.

ereboschi commented 6 years ago

Apologies, I monitored this ticket for almost two weeks and stopped watching it on Tuesday. (I rarely check the email address associated with my profile.) To answer your questions:

The error message ("Unable to import app...") is not a local error message. It happens in AWS is at Lambda runtime.
I don't doubt that the deployment package that you built seems correct and complete, but this is not the case for me or my co-workers when we run chalice deploy locally. We are all forced to push our changes to Bitbucket and let chalice deploy run from the Bitbucket pipeline in order to get a properly functioning Lambda deployment.
As for you suggestion that we not use the vendor directory at all, my coworkers and I have all verified that the standard chalice deployment usually does not package properly and we are using the vendor directory as a work around in order to ensure that required packages are included in the .zip deployed to AWS. When a chalice build is run locally, many packages don't get placed into the .zip file (hence our use of the vendor directory as a work-around).

I suspect that there is a chalice bug related to our environments (and included some OS info hoping that someone could verify). I've not found anything in debug output that has shed light on our issues, but I can include output next week if you think it'll help.

Thank you

stealthycoin commented 6 years ago

@ereboschi I don't think the debug output will be useful without me going in and adding some debug logging to the packager. If we can't get this sorted out I will do that so we can figure out what is going on.

Rather than give me debug logs can you get me the listing of one of your deployment zipfiles? Specifically

Delete the vendor directory
Remove chalice from the requirements.txt
Run chalice package out
Show me the output of unzip -l out/deployment.zip.

I'm curious exactly what is getting through.

ereboschi commented 6 years ago

Hi, I've attached my updated "requirements.txt" file and the output for the steps listed above: output.txt requirements.txt

(Note: I've had to redact the names of our company dependencies and pypi server, but we haven't had any packaging problems with them.)

I really appreciate you looking into this for us.

stealthycoin commented 6 years ago

Ok so it looks to me like there maybe multiple issues that we are conflating as one. What I notice right off the bat is that it appears like deps that require a compiler and do not have wheel files available on pypi are failing which is expected and can be fixed with vendoring as you mentioned. I was confused by the earlier vendor directory contents so lets try adding some things selectively back to it. The main thing that confuses me is phpserialize being listed as not bundled since that is pure python and should work no matter what machine tries to package it.

So lets separate these:

watchdog==0.8.3
iniherit==0.3.9
markupsafe==1.0
pycparser==2.18
ciso8601==2.0.1
pathtools==0.1.2
bidict==0.17.2
phpserialize==1.3
pyyaml==3.13
schemapi==0.3.0
dateutils==0.6.6

Into two categories.

I would expect the following to fail on OS X without a vendor directory entry:

watchdog==0.8.3 (no wheel and binary)
pyyaml==3.13 (no wheel and binary)
ciso8601==2.0.1 (not in pypi)

Things I would expect to work:

iniherit==0.3.9
markupsafe==1.0
pycparser==2.18
pathtools==0.1.2
bidict==0.17.2
phpserialize==1.3
schemapi==0.3.0
dateutils==0.6.6

So lets try adding vendor entries for just the ones that I expect to fail for missing binary. Since the build system you mention works that means it is producing compatible wheel files with the lambda runtime. Can you get your hands on the wheel files it produces with the names:

PyYAML-3.13-cp36-cp36m-linux_x86_64.whl watchdog-0.8.3-cp36-cp36m-linux_x86_64.whl

And unzip them into the vendor directory?

The only other thing I can think of is that on your dev machines pip may be hitting your local cache and somehow doing the wrong thing. If you can test out the above suggestion I will get debug statements added to the packager so we can get more useful info.

stealthycoin commented 6 years ago

@ereboschi I pushed up a branch here which has debug logs when --debug is supplied for the chalice package ... command https://github.com/aws/chalice/tree/package-debug. That will have to do for now till I can get something more production-ready code reviewed and released.

Here is what it does for me in a similar case:

$ chalice --debug package out
2018-08-01 10:00:49,092 packager [DEBUG] calling pip download -r /private/tmp/repro/requirements.txt --dest /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu
2018-08-01 10:00:54,314 packager [DEBUG] Full dependency closure: {chalice==1.6.0(wheel), jmespath==0.9.3(wheel), pip==10.0.1(wheel), enum-compat==0.0.2(sdist), botocore==1.10.67(wheel), click==6.7(wheel), attrs==17.4.0(wheel), python-dateutil==2.7.3(wheel), phpserialize==1.3(sdist), text-unidecode==1.2(wheel), typing==3.6.4(wheel), docutils==0.14(wheel), yoyo-migrations==5.1.5(wheel), iniherit==0.3.9(sdist), six==1.11.0(wheel)}
2018-08-01 10:00:54,315 packager [DEBUG] compatible: {chalice==1.6.0(wheel), pip==10.0.1(wheel), botocore==1.10.67(wheel), click==6.7(wheel), attrs==17.4.0(wheel), python-dateutil==2.7.3(wheel), text-unidecode==1.2(wheel), typing==3.6.4(wheel), jmespath==0.9.3(wheel), docutils==0.14(wheel), yoyo-migrations==5.1.5(wheel), six==1.11.0(wheel)}
2018-08-01 10:00:54,315 packager [DEBUG] incompatible: {iniherit==0.3.9(sdist), phpserialize==1.3(sdist), enum-compat==0.0.2(sdist)}
2018-08-01 10:00:54,315 packager [DEBUG] Downloading missing wheels: {iniherit==0.3.9(sdist), phpserialize==1.3(sdist), enum-compat==0.0.2(sdist)}
2018-08-01 10:00:54,316 packager [DEBUG] calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu iniherit==0.3.9
2018-08-01 10:00:54,949 packager [DEBUG] calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu phpserialize==1.3
2018-08-01 10:00:55,569 packager [DEBUG] calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu enum-compat==0.0.2
2018-08-01 10:00:56,168 packager [DEBUG] compatible: {chalice==1.6.0(wheel), jmespath==0.9.3(wheel), pip==10.0.1(wheel), botocore==1.10.67(wheel), click==6.7(wheel), attrs==17.4.0(wheel), python-dateutil==2.7.3(wheel), text-unidecode==1.2(wheel), typing==3.6.4(wheel), docutils==0.14(wheel), yoyo-migrations==5.1.5(wheel), six==1.11.0(wheel)}
2018-08-01 10:00:56,168 packager [DEBUG] Build missing wheels from sdists (C compiling True): {enum-compat==0.0.2(sdist), phpserialize==1.3(sdist), iniherit==0.3.9(sdist)}
2018-08-01 10:00:56,168 packager [DEBUG] calling pip wheel --no-deps --wheel-dir /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu/enum-compat-0.0.2.tar.gz
2018-08-01 10:00:57,647 packager [DEBUG] calling pip wheel --no-deps --wheel-dir /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu/phpserialize-1.3.tar.gz
2018-08-01 10:00:59,137 packager [DEBUG] calling pip wheel --no-deps --wheel-dir /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu /var/folders/sv/p_5qj57d7y312j1vcxbncz097fs8py/T/tmphfxztpgu/iniherit-0.3.9.tar.gz
2018-08-01 10:01:00,632 packager [DEBUG] compatible: {chalice==1.6.0(wheel), jmespath==0.9.3(wheel), pip==10.0.1(wheel), enum-compat==0.0.2(wheel), botocore==1.10.67(wheel), click==6.7(wheel), attrs==17.4.0(wheel), python-dateutil==2.7.3(wheel), phpserialize==1.3(wheel), text-unidecode==1.2(wheel), typing==3.6.4(wheel), docutils==0.14(wheel), yoyo-migrations==5.1.5(wheel), iniherit==0.3.9(wheel), six==1.11.0(wheel)}
2018-08-01 10:01:00,632 packager [DEBUG] Build missing wheels from sdists (C compiling False): set()
2018-08-01 10:01:00,633 packager [DEBUG] Final compatible: {chalice==1.6.0(wheel), jmespath==0.9.3(wheel), pip==10.0.1(wheel), enum-compat==0.0.2(wheel), botocore==1.10.67(wheel), click==6.7(wheel), attrs==17.4.0(wheel), python-dateutil==2.7.3(wheel), phpserialize==1.3(wheel), text-unidecode==1.2(wheel), typing==3.6.4(wheel), docutils==0.14(wheel), yoyo-migrations==5.1.5(wheel), iniherit==0.3.9(wheel), six==1.11.0(wheel)}
2018-08-01 10:01:00,633 packager [DEBUG] Final incompatible: set()
2018-08-01 10:01:00,633 packager [DEBUG] Final missing wheels: set()

This shows the pip commands its executing and then how it is sorting the resultant packages by lambda compatibility. I can't show an exact repro case since I do not have access to some of the packages you do. But looking at this you can see that phpserialize at least is not valid immediately since it cannot download a wheel for it. Its not until after the 2018-08-01 10:00:56,168 packager [DEBUG] Build missing wheels from sdists (C compiling True): {enum-compat==0.0.2(sdist), phpserialize==1.3(sdist), iniherit==0.3.9(sdist)} step that it gets moved to the compatible list. So possibly something in your setup is preventing it from being built?

ereboschi commented 6 years ago

@stealthycoin, I started to implement your build suggestions. Unfortunately, I've had trouble properly installing the debug build of chalice that you made, into my virtual environment.

Given the amount of time we've spent on trying to debug these build issues, I think that our group has to continue using chalice as we have been (by pushing changes to Bitbucket and having its pipeline deploy for us)... and continue working on replacing the chalice framework with plain AWS tooling (AWS CLI w/ CloudFormation) and custom scripting (which sadly, is exactly what chalice is for).

I'm so sorry that I wasn't able to provide a debug build log using the chalice version that you built for me. I can't tell you how thankful I am that you've really gone the extra mile (or two) to help me understand chalice and try to identify our bugs. I know that chalice developers aren't paid support staff under contract, but you've conducted yourselves no differently.

Thank you

stealthycoin commented 6 years ago

@ereboschi sorry you coudn't get it to work.

For future reference this should do it:

(tmpenv) $ git clone git@github.com:aws/chalice.git
(tmpenv) $ cd chalice
(tmpenv) $ git checkout package-debug
(tmpenv) $ pip install -e .

brainstorm commented 6 years ago

I might take/reopen this issue then since it's biting me right now :)

Please @stealthycoin, would you mind taking a look at this?: https://github.com/HumanCellAtlas/data-store/issues/1740

I did clone chalice within the hca repo as they do, but with your package-debug branch. The error seems to be the same, without further debugging info:

2018-11-21 06:23:49         94 HASH.TIMESTAMP
eSTART RequestId: HASH Version: $LATEST
Unable to import module 'app': No module named 'inflection'

END RequestId: HASH
REPORT RequestId: HASH  Duration: 0.62 ms       Billed Duration: 100 ms         Memory Size: 1536 MB    Max Memory Used: 42 MB

Happy to follow some indications on how to debug this. Meanwhile, this is my environment where I'm running this:

(hca-venv-py36) ubuntu@ip-172-31-21-87:~/data-store/chalice$ chalice --version
chalice 1.6.0, python 3.7.0, linux 4.18.0-1002-aws
(hca-venv-py36) ubuntu@ip-172-31-21-87:~/data-store/chalice$ python --version
Python 3.6.7

Shall I use python 3.7 instead since AWS seems to support it nowadays anyway? Could that be the root of the problem?

/cc @reisingerf

jamesls commented 5 years ago

Just following up here, are you still running into issues? The package debug logs PR and support for python3.7 are now in master and will go out in the next chalice release, so I'm curious if the chalice master branch works for you. Looking at the linked repo, inflection is a pure python module so there shouldn't be an issues on our end being able to package that dependency. If I create a new chalice app with just inflection in the requirements.txt chalice package foo correctly packages the dependency.

ereboschi commented 5 years ago

Sorry for the delay, I don't check this email address often. I'd say, yes, my team is still having issues with Chalice failing to package certain dependencies... just last week, a team member was getting errors in the CloudWatch logs on Lambda startup that dependencies were missing. We work around the issue by manually adding packages to the vendor directory.

I couldn't say which version he's using, precisely, but I could check if you like.

ereboschi commented 5 years ago

Update: Yes, @jamesls, last week we had an issue with Chalice 1.6.1 not packaging ciso8601. It also wouldn't vendor, so we manually built a version to vendor.

That said, ciso8601 does contain code from another language, so working with it can be tricky anyway.

duaneking commented 5 years ago

I'm currently running into an issue wher Chalice wont deploy local code correctly and is actively skipping the packaging of code we depend on that is work we did ourselves that is a local module that is not installed via pip because its just a local library we created and consume; could this be related?

eliyaba commented 3 years ago

I'm running into this issue as well. Packages numpy==1.21.0 and pandas==1.2.4 are missing from generated zip.

2021-07-05 19:47:39,921 chalice.deploy.packager [DEBUG] Final incompatible: {pandas==1.2.4(wheel), numpy==1.21.0(wheel)} 2021-07-05 19:47:39,921 chalice.deploy.packager [DEBUG] Final missing wheels: {pandas==1.2.4(wheel), numpy==1.21.0(wheel)}

Putting them into vendor folder consumes all the lambda memory. Did anyone find a way to fix this?

duaneking commented 3 years ago

No this was never resolved; The company I was with decided to stop using this library due to it not being maintained enough. Nobody I know uses it anymore, and my perception of it is that you should not use it in production code because its not maintained or functional enough to trust.

refeed commented 1 year ago

I have this problem too, where I have a module written locally inside my project directory, it worked with chalice local but when I deployed it, it couldn't find the local module. When I try to unpack the deployment zip, chalice just omit the local module, I don't know why is this

aws / chalice

chalice deployment package missing libraries #896