search

aws / aws-lambda-builders

Python library to compile, build & package AWS Lambda functions for several runtimes & framework
Apache License 2.0
334 stars 139 forks source link

Bug: sam build throws error with python packages installed with git+https url #500

Closed mbtx2 closed 1 year ago

mbtx2 commented 1 year ago

Description:

When using a url to install a python package via a git+https url, sam build fails.

For instance, if you have this in your requirements.txt file:

a_package @ git+https://a_git_api_key@github.com/a_repo_name/a_repo_name.git@main#egg=a_package[an_extra]&subdirectory=a_subdir

sam build fails, although pip is able to install this requirements.txt file without error, so this is a fully-supported pip install string. Have tried with @main, @a_tag, etc., doesn't work.

Error:

Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: a_package

Steps to reproduce:

In a Lambda project, put a git+https package url in the requirements.txt file. run sam build. You will get the error:

Running PythonPipBuilder:ResolveDependencies

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: a_package

Observed result:

2023-05-07 16:03:27,826 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "aws_lambda_builders/workflow.py", line 369, in run
  File "aws_lambda_builders/workflows/python_pip/actions.py", line 54, in execute
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 256, in build_site_packages
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 280, in _download_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 363, in _download_all_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 363, in <setcomp>
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 512, in __init__
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 554, in _calculate_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 625, in get_package_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 608, in _get_pkg_info_filepath
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: a_package

Build Failed

Expected result:

Should be able to install git+https package without error. If pip can install it, sam should not throw errors.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.80.0",
  "system": {
    "python": "3.7.10",
    "os": "Linux-5.15.0-70-generic-x86_64-with-debian-bookworm-sid"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.18",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  }
}

One note... These git+https package references do work with lambda container images, because they do work in dockerfiles/docker builds. This only affects the standard Lambda build process.

hnnasit commented 1 year ago

Hi @mbtx2, thanks for opening the issue. Moving this issue to lambda builders repository. We will investigate it further.

nsteffens commented 1 year ago

This is probably the same issue as #253 - Maybe you could have a look into this? Unfortunately, it's been open for almost 2 years and it hasn't been fixed, yet 😞

mndeveci commented 1 year ago

@nsteffens do you have a real example in requirements that I can use to re-produce this? I've tried with requests library but I was able to build it.

requests @ git+ssh://git@github.com/psf/requests.git@v2.30.0

In the meantime, I will resolve this issue to continue discussion in #253 since this is duplicate as you mentioned.