search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.51k stars 1.17k forks source link

git dependencies in pip don't install on Windows #1144

Open johnc44 opened 5 years ago

johnc44 commented 5 years ago

Description

I'm not able to use git/ssh based dependencies using PIP and SAM.

Using a git dependency in my requirements.txt file like this: git+ssh://git@bitbucket.org/myaccount/example.git

When running sam build from a Windows command prompt, it gives an error like thi: AssertionError: name 'c:\users\johnc\appdata\local\temp\pip-req-build-puwxrmxy\.git' doesn't start with prefix 'c:\users\johnc\appdata\local\temp\pip-req-build-puwxrmxy\'

Steps to reproduce

From a Windows 10 command prompt:

  1. sam init -r python3.6 -n git-dependencies
  2. Edit requirements.txt and add in a git dependency: git+ssh://git@bitbucket.org/myaccount/somethingorother.git
  3. sam build

Observed result

c:\temp\git-dependencies>sam build --debug 2019-04-23 13:51:15 Using SAM Template at c:\temp\git-dependencies\template.yaml 2019-04-23 13:51:15 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-04-23 13:51:15 Changing event name from before-call.apigateway to before-call.api-gateway 2019-04-23 13:51:15 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-04-23 13:51:15 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-04-23 13:51:15 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-04-23 13:51:15 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-04-23 13:51:15 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-04-23 13:51:15 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-04-23 13:51:15 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-04-23 13:51:15 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-04-23 13:51:15 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-04-23 13:51:15 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-04-23 13:51:15 Changing event name from before-call.apigateway to before-call.api-gateway 2019-04-23 13:51:15 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-04-23 13:51:15 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-04-23 13:51:15 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-04-23 13:51:15 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-04-23 13:51:15 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-04-23 13:51:15 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-04-23 13:51:15 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-04-23 13:51:15 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-04-23 13:51:15 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-04-23 13:51:15 'build' command is called 2019-04-23 13:51:15 No Parameters detected in the template 2019-04-23 13:51:15 2 resources found in the template 2019-04-23 13:51:15 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/' 2019-04-23 13:51:15 Building resource 'HelloWorldFunction' 2019-04-23 13:51:15 Loading workflow module 'aws_lambda_builders.workflows' 2019-04-23 13:51:15 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' 2019-04-23 13:51:15 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' 2019-04-23 13:51:15 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 2019-04-23 13:51:15 Running workflow 'PythonPipBuilder' 2019-04-23 13:51:15 Running PythonPipBuilder:ResolveDependencies 2019-04-23 13:51:15 calling pip download -r C:\temp\git-dependencies\hello_world\requirements.txt --dest C:\Users\johnc\AppData\Local\Temp\tmpjm5d5o79 2019-04-23 13:51:17 PythonPipBuilder:ResolveDependencies failed Traceback (most recent call last): File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute self.scratch_dir File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies requirements_path, artifacts_dir_path, scratch_dir_path) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 209, in build_site_packages scratch_directory, requirements_filepath) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 234, in _download_dependencies requirements_filename, directory) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 330, in _download_all_dependencies self._pip.download_all_dependencies(requirements_filename, directory) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 632, in download_all_dependencies raise PackageDownloadError(error) aws_lambda_builders.workflows.python_pip.packager.PackageDownloadError: Exception: Traceback (most recent call last): File "C:\Program Files\Python36\lib\site-packages\pip_internal\cli\base_command.py", line 179, in main status = self.run(options, args) File "C:\Program Files\Python36\lib\site-packages\pip_internal\commands\download.py", line 164, in run resolver.resolve(requirement_set) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 131, in resolve self._resolve_one(requirement_set, req) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 294, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 242, in _get_abstract_dist_for self.require_hashes File "C:\Program Files\Python36\lib\site-packages\pip_internal\operations\prepare.py", line 353, in prepare_linked_requirement req.archive(self.download_dir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 873, in archive rootdir=dir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 831, in _get_archive_name name = self._clean_zip_name(path, rootdir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 822, in _clean_zip_name "name %r doesn't start with prefix %r" % (name, prefix) AssertionError: name 'c:\users\johnc\appdata\local\temp\pip-req-build-unr3otwk\.git' doesn't start with prefix 'c:\users\johnc\appdata\local\temp\pip-req-build-unr3otwk\'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 248, in run action.execute() File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 42, in execute raise ActionFailedError(str(ex)) aws_lambda_builders.actions.ActionFailedError: Exception: Traceback (most recent call last): File "C:\Program Files\Python36\lib\site-packages\pip_internal\cli\base_command.py", line 179, in main status = self.run(options, args) File "C:\Program Files\Python36\lib\site-packages\pip_internal\commands\download.py", line 164, in run resolver.resolve(requirement_set) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 131, in resolve self._resolve_one(requirement_set, req) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 294, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 242, in _get_abstract_dist_for self.require_hashes File "C:\Program Files\Python36\lib\site-packages\pip_internal\operations\prepare.py", line 353, in prepare_linked_requirement req.archive(self.download_dir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 873, in archive rootdir=dir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 831, in _get_archive_name name = self._clean_zip_name(path, rootdir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 822, in _clean_zip_name "name %r doesn't start with prefix %r" % (name, prefix) AssertionError: name 'c:\users\johnc\appdata\local\temp\pip-req-build-unr3otwk\.git' doesn't start with prefix 'c:\users\johnc\appdata\local\temp\pip-req-build-unr3otwk\'

Build Failed Error: PythonPipBuilder:ResolveDependencies - Exception: Traceback (most recent call last): File "C:\Program Files\Python36\lib\site-packages\pip_internal\cli\base_command.py", line 179, in main status = self.run(options, args) File "C:\Program Files\Python36\lib\site-packages\pip_internal\commands\download.py", line 164, in run resolver.resolve(requirement_set) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 131, in resolve self._resolve_one(requirement_set, req) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 294, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "C:\Program Files\Python36\lib\site-packages\pip_internal\resolve.py", line 242, in _get_abstract_dist_for self.require_hashes File "C:\Program Files\Python36\lib\site-packages\pip_internal\operations\prepare.py", line 353, in prepare_linked_requirement req.archive(self.download_dir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 873, in archive rootdir=dir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 831, in _get_archive_name name = self._clean_zip_name(path, rootdir) File "C:\Program Files\Python36\lib\site-packages\pip_internal\req\req_install.py", line 822, in _clean_zip_name "name %r doesn't start with prefix %r" % (name, prefix) AssertionError: name 'c:\users\johnc\appdata\local\temp\pip-req-build-unr3otwk\.git' doesn't start with prefix 'c:\users\johnc\appdata\local\temp\pip-req-build-unr3otwk\'

Expected result

Should have installed dependency from git.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Windows 10
  2. sam --version:0.15.0
  3. python version: 3.6.7
  4. pip version: 19.0.3

If I run from Ubuntu WSL then it installs OK. (But not if I use "--use-container", but then that never works for me from WSL anyway).

It also doesn't work from Windows if I add "--use-container" in which case I get:

Error: PythonPipBuilder:ResolveDependencies - Command "git clone -q ssh://git@bitbucket.org/myaccount/sample.git /tmp/pip-req-build-1n5kwnr1" failed with error code 128 in None

Is doing this sort of thing supported with "--use-container"? (If not, I can maybe raise it as a feature request)

From Windows, installing using pip directly works, i.e. like this: pip install -r src\requirements.txt -t .aws-sam\tests

So I know that the requirements file is good, and that Windows, git and pip are playing nicely together.

sriram-mv commented 5 years ago

this is definitely frustrating and is something we need to look into. But as a workaround i would suggest building a wheel of your git dependency locally, and sticking it in your requirements.txt.

johnc44 commented 5 years ago

Thanks, @TheSriram for the reply. Sorry it took me so long to respond, but I needed to find time to play about with your suggestion.

Am I right in thinking that you are proposing:

I tried this. My setup is something like:

mylambda
    packages/samplelibrary-1.0-py3-none-any.whl
    src/
        requirements.txt (containing one line: "packages/samplelibrary-1.0-py3-none-any.whl")
        main.py
    template.yml

This does work OK if I use "sam build". However, it does not work if I use "sam build --use-container". I also tried moving the packages into "src" but that also doesn't work with --use-container because the relative path is based on where I run the command from, not relative to requirements.txt. (Not that I particularly want to do it that way anyway).

At the moment, I don't need --use-container but I don't want to start building things that won't work with it because we might find at some point that we do start using a dependency that needs to be built that way.

johnc44 commented 5 years ago

Digging around in the error some more, it does look like this is a pip problem.

pip download -r requirements.txt --dest temp - this throws the same error as I described above.

However, pip install -r requirements.txt --dest temp - this works fine.

I'll be honest, I've never used pip download so not sure why one should work and not the other.

(Found pip issue: https://github.com/pypa/pip/pull/6436)

This should fix half of the issue, but won't address the problem of it not working with "--use-container".

jdonboch commented 5 years ago

Having same issue. Would be interested if there was a work around. We do use the --use-container flag