support for gathering local pip packages on deploy.

[ghost]

I have a local python package installed via pip on my machine. chalice deploy does not work after doing a pip freeze > requirements.txt. Reading the documentation, the chalicelibs/ and vendor/ folders are my only options. The problem comes from building multiple chalice projects. I want a way to share code between projects without having to manually drag and drop everything into chalicelib/ or vendor/ for each deploy.

chalice local works without issue after configuring my package locally via pip.

My req.txt that doesnt work.

pymongo
-e git+https://github.com/company/somesharedlib.git@d0fcd1e0cb0b43ac4cbf1d587cd7e62814245549#egg=somesharedlib

from chalice import Chalice 
from chalicelib.models import SomeModel
# I would like to do this instead
from somesharedlib.models import SomeModel

app = Chalice(app_name='temp-svc')

@app.route('/') 
def index():
  log = PresenceStatusLog('test')
  return {'hello': 'world'}

It would be great to have a simple feature that allows for say pointing the chalicelib/ directory at some other directory on your local machine at the very least so people could share code between projects. Currently writing a bash script to copy the contents of that directory each time i deploy.

[jlujan-invitae]

The chalice packager appears to not handle source packages (or git) correctly. The packager attempts to download existing 'manylinux' wheels. This solves many of the problems encountered when packaging your chalice app on macOS, for instance. However, the code involved does not properly handle intentional source packages defined in the requirements.txt, i.e. git or file paths. My understanding is that the packager:

1. Downloads each package using pip download chalice/deploy/packager.py#L568
2. Inspects the file name to determine if the package is suitable for "manylinux", i.e. is pure python. chalice/deploy/packager.py#L582
3. If the package is "incompatible" or a source distribution, the packager attempts to download a suitable binary distribution using pip download --only-binary=:all: --no-deps --platform 'manylinux2014_x86_64' --implementation 'cp' --abi cp38 chalice/deploy/packager.py#L593
4. If no binary distribution was found for "incompatible" packages, try to build it from the source distribution. chalice/deploy/packager.py#L620

The problem occurs in step 3 as the packager lumps both source distributions and incompatible packages together chalice/deploy/packager.py#L593. At this point in the process it has no knowledge of the intent to use a source package. This causes problems if you need to use a specific fork or revision of the package as it will inspect the sdist package info using setuptoolschalice/deploy/packager.py#L807 and only use the the name and version found to download a binary distribution, overriding the intentional use of a source checkout. I have currently run into this issue attempting to test a PR on a GH fork of a public package.

I think the appropriate solution would be to differentiate source-only distributions from packages intentionally specified as source distributions in the requirements.txt which is not possible with the current packager.

@rtorres280, in your specific case with a private package checked out from source, it should work assuming you have an appropriate setup .py in your repository. Assuming you do not have a PyPI repository hosting your custom package, it will fail to find a binary distribution and attempt to build it. It uses pip wheel --no-deps <package> to build the wheel. If you run it against your package you should get a<package>-<version>-py3-none-any.whl file as the output.

[Brianthered]

@jlujan-invitae 's description of the packager is also discussed in https://github.com/aws/chalice/issues/1516. Cross linking for informational purposes.