search

aws / aws-lambda-builders

Python library to compile, build & package AWS Lambda functions for several runtimes & framework
Apache License 2.0
338 stars 139 forks source link

Unable to build Python lambda with package dependency in a private Git repository #253

Closed Berciq closed 10 months ago

Berciq commented 3 years ago

Description:

I am trying to use sam build to build a Python 3.7 lambda. My requirements.txt file contains the following dependency:

git+ssh://git@github.com/Berciq/commons.git@v0.0.4

So, as you can see, I am trying to include a package hosted in a private Git in my requirements.txt file as per the documentation. The package installs fine when using pip install -r requirements.txt but sam build throws the following error:

Build Failed
Error: PythonPipBuilder:ResolveDependencies - list index out of range

If I remove that dependency from my requirments.txt, sam build runs successfully.

Steps to reproduce the issue:

  1. Use sam init to create any Python-based AWS SAM application (e.g. hello-world)
  2. Add a private Git repo dependency to the requirements.txt file
  3. Run sam build

Observed result: The build will fail with a PythonPipBuilder:ResolveDependencies error.

Expected result: The build should succeed, given that running pip install -r requirements.txt succeeds.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc) I'm running Ubuntu 20, AWS SAM CLI v.1.26.

But I was also able to reproduce the problem by installing and importing aws-lambda-builders as a module and running the PythonPipWorkflow, hence I'm raising the issue here.

This appears to be related in some way to Issue #229 but at the same time it's not exactly the same issue, as mine has to do specifically with dependencies hosted in a private repository.

sriram-mv commented 3 years ago

I tested this on a mac by supplying build of aws-sam-cli directly from github and that seemed to work.

I'm on python version 3.8.10, AWS SAM CLI version: 1.27.2

My requirements.txt looks like below

git+https://github.com/aws/aws-sam-cli

Could you try upgrading and also additional traceback information as well?

prenx4x commented 3 years ago

I tried and it was working for me for private repo and python 3.7. my requirements.txt file looked like 

requests
-e git+git@github.com:prenx4x/sampy38-jfs.git#egg=my-git-pack

Can you upgrade to latest sam-cli and try again and then provide additional info so we can try to repro it.

Berciq commented 3 years ago

It looks as though PR #265 should address my issue. I will wait for to confirm once those changes carry over to the newest version of SAM CLI.

I am still getting the error with Python 3.7 and SAM CLI version 1.29.0. Here is what my requirements.txt looks like:

requests
git+ssh://git@github.com-p/Berciq/sample-pkg.git@v0.0.4

And here is the output from sam build --debug:

2021-08-30 11:45:01,869 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-08-30 11:45:01,870 | Using config file: samconfig.toml, config environment: default
2021-08-30 11:45:01,870 | Expand command line arguments to:
2021-08-30 11:45:01,870 | --template_file=/home/bartek/sandbox/sample-sam-app/template.yaml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-08-30 11:45:01,901 | 'build' command is called
2021-08-30 11:45:01,906 | No Parameters detected in the template
2021-08-30 11:45:01,925 | 2 stacks found in the template
2021-08-30 11:45:01,925 | No Parameters detected in the template
2021-08-30 11:45:01,942 | 2 resources found in the stack 
2021-08-30 11:45:01,942 | No Parameters detected in the template
2021-08-30 11:45:01,959 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-08-30 11:45:01,959 | --base-dir is not presented, adjusting uri hello_world/ relative to /home/bartek/sandbox/sample-sam-app/template.yaml
2021-08-30 11:45:01,959 | No Parameters detected in the template
2021-08-30 11:45:01,977 | Instantiating build definitions
2021-08-30 11:45:01,977 | No previous build graph found, generating new one
2021-08-30 11:45:01,977 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(python3.7, /home/bartek/sandbox/sample-sam-app/hello_world, Zip, , f5cc0c06-8e35-4ddf-90c9-e24b594ef076, {}, {}, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.7', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/home/bartek/sandbox/sample-sam-app/hello_world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, stack_path=''))
2021-08-30 11:45:01,978 | Building codeuri: /home/bartek/sandbox/sample-sam-app/hello_world runtime: python3.7 metadata: {} functions: ['HelloWorldFunction']
2021-08-30 11:45:01,978 | Building to following folder /home/bartek/sandbox/sample-sam-app/.aws-sam/build/HelloWorldFunction
2021-08-30 11:45:01,979 | Loading workflow module 'aws_lambda_builders.workflows'
2021-08-30 11:45:01,981 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-08-30 11:45:01,982 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-08-30 11:45:01,982 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-08-30 11:45:01,983 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-08-30 11:45:01,984 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-08-30 11:45:01,984 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-08-30 11:45:01,985 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-08-30 11:45:01,986 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-08-30 11:45:01,987 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-08-30 11:45:01,987 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-08-30 11:45:01,999 | Running workflow 'PythonPipBuilder'
2021-08-30 11:45:01,999 | Running PythonPipBuilder:ResolveDependencies
2021-08-30 11:45:02,017 | calling pip download -r /home/bartek/sandbox/sample-sam-app/hello_world/requirements.txt --dest /tmp/tmpev9tvprz --exists-action i
2021-08-30 11:45:05,288 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "aws_lambda_builders/workflow.py", line 269, in run
  File "aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 145, in build_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 223, in build_site_packages
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 247, in _download_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 330, in _download_all_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 330, in <setcomp>
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 468, in __init__
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 510, in _calculate_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 570, in get_package_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 553, in _generate_egg_info
IndexError: list index out of range

Build Failed
2021-08-30 11:45:05,290 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7a7b2f7a-5fd4-4f80-a17e-ecf6f7869f0c', 'installationId': '8b2c648e-3b81-425e-8ddd-9194c667999e', 'sessionId': '42352b2a-fb70-49f6-804e-e63eb87a5956', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.29.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 3419, 'exitReason': 'WorkflowUnknownError', 'exitCode': 1}}]}
2021-08-30 11:45:06,218 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - list index out of range

I should add that running the following works correctly:

pip3.7 install -r /home/bartek/sandbox/sample-sam-app/hello_world/requirements.txt -t /tmp/pip-test
bartek-hitachi commented 2 years ago

Contrary to what I wrote earlier, I'm still facing this issue (or a related one) with SAM CLI version 1.34.1. The error message changed. Please see attached output of sam build --debug below.

I'm on Ubuntu 20.04.3 LTS.

Here is what my requirements.txt looks like:

requests
git+ssh://git@github.com-p/Berciq/sample-pkg.git@v0.0.4

And here is the output from sam build --debug:

2021-11-17 18:42:36,429 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-11-17 18:42:36,429 | Using config file: samconfig.toml, config environment: default
2021-11-17 18:42:36,429 | Expand command line arguments to:
2021-11-17 18:42:36,429 | --template_file=/home/bartek/sandbox/sample-sam-app/template.yaml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-11-17 18:42:36,516 | 'build' command is called
2021-11-17 18:42:36,520 | No Parameters detected in the template
2021-11-17 18:42:36,538 | 2 stacks found in the template
2021-11-17 18:42:36,539 | No Parameters detected in the template
2021-11-17 18:42:36,556 | 2 resources found in the stack 
2021-11-17 18:42:36,556 | No Parameters detected in the template
2021-11-17 18:42:36,573 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-11-17 18:42:36,573 | --base-dir is not presented, adjusting uri hello_world/ relative to /home/bartek/sandbox/sample-sam-app/template.yaml
2021-11-17 18:42:36,573 | No Parameters detected in the template
2021-11-17 18:42:36,591 | Instantiating build definitions
2021-11-17 18:42:36,595 | Same function build definition found, adding function (Previous: BuildDefinition(python3.7, /home/bartek/sandbox/sample-sam-app/hello_world, Zip, , f5cc0c06-8e35-4ddf-90c9-e24b594ef076, {}, {}, x86_64, []), Current: BuildDefinition(python3.7, /home/bartek/sandbox/sample-sam-app/hello_world, Zip, , 96314c54-ef15-494e-a22a-0fa2ab73ab93, {}, {}, x86_64, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.7', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/home/bartek/sandbox/sample-sam-app/hello_world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, architectures=None, stack_path=''))
2021-11-17 18:42:36,596 | Building codeuri: /home/bartek/sandbox/sample-sam-app/hello_world runtime: python3.7 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction']
2021-11-17 18:42:36,596 | Building to following folder /home/bartek/sandbox/sample-sam-app/.aws-sam/build/HelloWorldFunction
2021-11-17 18:42:36,597 | Loading workflow module 'aws_lambda_builders.workflows'
2021-11-17 18:42:36,599 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-11-17 18:42:36,600 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-11-17 18:42:36,600 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-11-17 18:42:36,601 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-11-17 18:42:36,602 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-11-17 18:42:36,603 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-11-17 18:42:36,604 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-11-17 18:42:36,605 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-11-17 18:42:36,605 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-11-17 18:42:36,606 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-11-17 18:42:36,617 | Running workflow 'PythonPipBuilder'
2021-11-17 18:42:36,617 | Running PythonPipBuilder:ResolveDependencies
2021-11-17 18:42:36,634 | calling pip download -r /home/bartek/sandbox/sample-sam-app/hello_world/requirements.txt --dest /tmp/tmpdpnjclo8 --exists-action i
2021-11-17 18:42:40,910 | Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for help.\n\nError: no such option: -c\n"
2021-11-17 18:42:40,910 | Using fallback location for PKG-INFO file in package directory: /tmp/tmpe8l5xmx9/sample-pkg
2021-11-17 18:42:40,911 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "aws_lambda_builders/workflow.py", line 291, in run
  File "aws_lambda_builders/workflows/python_pip/actions.py", line 54, in execute
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 258, in build_site_packages
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 282, in _download_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in _download_all_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in <setcomp>
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 519, in __init__
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 561, in _calculate_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 632, in get_package_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 615, in _get_pkg_info_filepath
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: sample-pkg

Build Failed
2021-11-17 18:42:40,912 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'ba585dc5-963d-40ed-b31c-d03ac9cc812d', 'installationId': '8b2c648e-3b81-425e-8ddd-9194c667999e', 'sessionId': '639e019f-22c6-4df5-b747-2799e8ce1cc1', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.34.1', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 4482, 'exitReason': 'WorkflowUnknownError', 'exitCode': 1}}]}
2021-11-17 18:42:41,826 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: sample-pkg
Berciq commented 2 years ago

To add to the previous comment, I'm able to successfully build this package inside Docker using the following command:

docker run --rm -v ~/.ssh/known_hosts:/etc/ssh/ssh_known_hosts -v $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent -v $(pwd):/var/task "amazon/aws-sam-cli-build-image-python3.7" sam build

The SSH related params in the above command are necessary for Docker to use my local system's SSH keys to fetch the sample-pkg package from my private repo. But the key thing to note is that sam build executes successfully within that Docker container.

This leads me to think that the problem is in some way operating-system related or configuration related. I'm on Ubuntu 20.04.3 LTS.

Here is another thing I noticed. Please note how one of the debug messages says:

Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for help.\n\nError: no such option: -c\n"

This error message is a result of executing this command.

The strange thing is that this command should be executing within the context of the Python interpreter, like so: 

/usr/bin/python -c [other params]

But instead, it appears to be executing within the context of sam build as indicated by the log message:

Usage: sam [OPTIONS] COMMAND [ARGS]... 
Try 'sam -h' for help.

Error: no such option: -c

So it appears that sys.executable points to the SAM CLI executable on my system instead of the Python interpreter. Which is strange, because it doesn't under normal circumstances. Here is the output from a simple test script:

~$ python -c "import sys; print(sys.executable)"
/usr/bin/python

This is how far I've gotten so far troubleshooting this. Maybe someone can pick up where I left off and identify the root cause.

shumer commented 2 years ago

Is there any ideas how to solve this issue? Because I'm getting the same error during the build process.

Yop-La commented 2 years ago

Upgrading sam cli fixes the issue on my side.

javivdm commented 2 years ago

I am getting the same error: Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: with version SAM CLI, version 1.37.0 (latest) during the build process when specifying a dependency from a public github repo.

josefinb commented 2 years ago

Also getting this error, with version 1.40.1, when specifying a private git repo in requirements.txt. I am using python 3.8.

Berciq commented 2 years ago

To follow up on my last comment, I'm quite certain that the issue I'm experiencing on my system is caused by Python's sys.executable returning the path of the SAM CLI executable as opposed to the path of the Python interpreter when executing sam build. The stacktrace from sam build --debug reveals that the issue happens on the following lines:

cmd = [sys.executable, "-c", script, "--no-user-cfg", "egg_info", "--egg-base", "egg-info"]
p = subprocess.Popen(
    cmd, cwd=package_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=self._osutils.original_environ()
)
_, stderr = p.communicate()
info_contents = self._osutils.get_directory_contents(egg_info_dir)
if p.returncode != 0:
    LOG.debug("Non zero rc (%s) from the setup.py egg_info command: %s", p.returncode, stderr)

Here is why I believe this is the case:

  1. My output from sam build --debug includes the following log message:

Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for help.\n\nError: no such option: -c\n"

This, along with the stack trace included in the output, makes it apparent that the above call to subprocess.Popen() is trying to execute the command against the SAM CLI.

  1. The Python docs indicate that sys.executable returns:

A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.

I would expect for the Python interpreter path to be undefined within a compiled binary such as the SAM CLI executable. What is odd, is that the sys.executable is not returning an empty string or None, as per the documentation, which would result in a different and perhaps clearer error message. However, it actually makes sense that sys.executable is returning the path to the compiled binary itself.

  1. After following the instructions here to clone the aws-sam-cli repo and set up a local development environment, I was able to successfully build my stack using samdev build, thereby confirming that the command works as expected when not in a compiled binary.

I think that's an accurate diagnosis of the issue. How to solve it? I'm not sure at this point. It seems that sys.executable may need to be substituted by a different approach for reliably obtaining the Python interpreter path.

svenemtell commented 2 years ago

Any news on this? We get the Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package:... and the workaround setting up a local development environment (in no 3 above) seems a little cumbersome.

Tomperez98 commented 1 year ago

Any updates on this?

dunaden001 commented 1 year ago

Just chiming in here in case this is useful for others that find this thread while troubleshooting.

The way I have temporarily resolved this issue for my own situation (with a bunch of less than ideal caveats) is to list the local file locations of the private repositories on the machine I'm using for deployment in the requirements.txt file for the project.

This makes the project no longer portable (though I may end up running everything in a container that first pulls the latest versions of the repos and then references them from there so that I can still deploy from CI/CD scripts) and means I need to manually list all the private sub-dependencies in the correct resolution order. But it at least allowed me to successfully build a project at all with a system that relies heavily on independently maintained private repos.

codrin-lanterne commented 1 year ago

Also facing this issue with 1.66.0

mikolajmalecki commented 1 year ago

Issue in my case was resolved by using (compiled on host machine) development version samdev instead of sam -- instructions for getting samdev working: https://github.com/aws/aws-sam-cli/blob/develop/DEVELOPMENT_GUIDE.md

moelasmar commented 1 year ago

I am able to reproduce this issue. We will work on a solution for this issue.

moelasmar commented 1 year ago

if you installed AWS SAM CLI using pip, you will not face this issue. But if you used any of the installers provided here, you can easily reproduce this issue.

I think the fix is to change this line https://github.com/aws/aws-lambda-builders/blob/8305159c383f286937ff8eb851fc2ba92faa6e29/aws_lambda_builders/workflows/python_pip/packager.py#L589, and replace sys.executable with the python binaries as defined here https://github.com/aws/aws-lambda-builders/blob/8305159c383f286937ff8eb851fc2ba92faa6e29/aws_lambda_builders/workflow.py#L330-L339

nsteffens commented 1 year ago

Any idea when this gets merged? We have this issue happening to us both locally and within our pipelines. It's really frustrating to use SAM with this.

5t33 commented 1 year ago

Hey yall this bug is rather annoying. A work around yall might be able to use is a git submodule. Alternatively, since we are on AWS, you could use AWS CodeArtifact as a package repo. Best of luck.

nsteffens commented 1 year ago

Hey @mndeveci! Thanks for coming back to me in #500:

If I recall this correctly, I've tried using it with a package hosted in a private repo with sub-dependencies to another repo. But I guess that shouldn't be the core issue, as sam build (without --use-container) would use my local credentials.

Another difference I've seen is that there is https used instead of ssh, maybe this is related, too?

mndeveci commented 1 year ago

Hey all,

We've applied the solution that @moelasmar mentioned here https://github.com/aws/aws-lambda-builders/issues/253#issuecomment-1486617634

Can you try to download ZIP file (one for MacOS x86 and one for Linux x86) in the following page (which contains the fix) and try to run sam-beta build (executable is named as sam-beta)?

https://github.com/aws/aws-sam-cli/actions/runs/5459308213

I've tested with a requirements.txt file which points to a private GH repo. It fails with sam build but it succeeds with the version above.

MajorTal commented 1 year ago

Hey - thanks for the patch. Not working for me (Linux). I'm using pyenv with virtualenvs, which might explain:

2023-07-18 14:48:25,497 | Non zero rc (127) from the setup.py egg_info command: b"pyenv: python3.10: command not found\n\nThe `python3.10' command exists in  
these Python versions:\n  3.10.5\n  3.10.5/envs/agi\n  3.10.5/envs/bgpt\n  3.10.5/envs/demucs\n  3.10.5/envs/spleeter\n  3.10.5/envs/whisper\n  3.10.10\n     
3.10.10/envs/aiquest\n  3.10.10/envs/autogpt\n  3.10.10/envs/cdk.venv\n  3.10.10/envs/sports_motivator\n  3.10.10/envs/synthlifer\n  3.10.10/envs/zoomer\n    
agi\n  aiquest\n  autogpt\n  bgpt\n  cdk.venv\n  demucs\n  spleeter\n  sports_motivator\n  synthlifer\n  whisper\n  zoomer\n\nNote: See 'pyenv help global'   
for tips on allowing both\n      python2 and python3 to be found.\n"

and eventually: Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: solara

whereas with the stable sam I'm getting:

2023-07-18 14:51:38,529 | Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for help.\n\nError: No such option: -c\n" and eventually Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: solara

mndeveci commented 1 year ago

Hey @MajorTal ,

I've tried the patch on a linux machine where I pointed a private git repository in my requirements.txt file, and it worked fine for me. I've tried with regular SAM CLI and it failed. Can you elaborate how you tested the patched SAM CLI version?

I am linking the ZIP files into this dummy release (https://github.com/mndeveci/aws-sam-cli/releases/tag/lb-fix) which contains sam-beta builds and that contains the fix that was suggested above. Can you try with these ZIP files once again? (These are only build for x86 architecture for Linux and MacOS, and executable is named as sam-beta).

Just to provide the source of these ZIP files, this is the PR which generated them: https://github.com/aws/aws-sam-cli/pull/5666 And this is the Github Action definition which generates these artifacts: https://github.com/aws/aws-sam-cli/blob/develop/.github/workflows/validate_pyinstaller.yml

Thanks!

lucashuy commented 1 year ago

Hi @MajorTal just checking in, have you given the release/steps that @mndeveci posted in the above reply a test yet? Want to confirm that this is still indeed an issue.

mndeveci commented 1 year ago

Noting this PR here for future reference: https://github.com/aws/aws-lambda-builders/pull/519

iainelder commented 1 year ago

if you installed AWS SAM CLI using pip, you will not face this issue.

I face the issue and I installed with pip via pipx.

$ pipx list | grep aws-sam-cli
   package aws-sam-cli 1.98.0, installed using Python 3.8.10

When requirements.txt has a requirement like this:

aws_sso_lib @ git+https://github.com/iainelder/aws-sso-util.git@skip-suspended#subdirectory=lib

Or like this:

-e git+https://github.com/iainelder/aws-sso-util.git@skip-suspended#egg=aws_sso_lib&subdirectory=lib

sam build --use-container gives an error like this:

Error: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: aws_sso_lib

This happens for public Git repositories too. You can follow those links to see what I'm trying to install from GitHub.

iainelder commented 1 year ago

In my case sam build expects a setup.py file to coexist with my pyproject.toml file before it will process the dependency.

I used the dephell utility to convert my pyproject.toml file to a setup.py file like this.

pipx install dephell
env --chdir=lib -- dephell deps convert --from=pyproject.toml --to=setup.py

With setup.py, it works. sam build gives this debug output. No error after the first pip download command.

Running workflow 'PythonPipBuilder'
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i

Without setup.py, it fails. sam build gives this debug output that shows an error.

Running workflow 'PythonPipBuilder'
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Non zero rc (1) from the setup.py egg_info command: b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/var/lang/lib/python3.8/tokenize.py", line 392, in open\n    buffer = _builtin_open(filename, \'rb\')\nFileNotFoundError: [Errno 2] No such file or directory: \'/tmp/tmphnl2j1yp/aws_sso_lib/setup.py\'\n'
Using fallback location for PKG-INFO file in package directory: /tmp/tmphnl2j1yp/aws_sso_lib
PythonPipBuilder:ResolveDependencies raised unhandled exception
wesleyjin commented 1 year ago

adding a setup.py to my private packages fixed the issue for me. I was using just setup.cfg and pyproject.toml - thinking it wasn't needed b/c of PEP 517/518

I just needed to add the following to a setup.py to the project root (requires setup.cfg):

from setuptools import setup

if __name__ == "__main__":
    setup()
iainelder commented 1 year ago

Glad to hear it, @wesleyjin . Can you share or point to a representative example of setup.cfg?

I have never used setup.cfg in my own projects. Until now I always used just pyproject.toml.

The setup.cfg-only projects doc you linked has a caveat: "if you can rely on your project always being built by a PEP 517/PEP 518 compatible frontend".

I don't know the details of the PEPs, but is it fair to say that sam build isn't yet compatible with them?

wesleyjin commented 1 year ago

@iainelder

here's a setup.cfg file that works for me with the setup.py above. There's other metadata fields that are optional you might want to include like author & description. (replaced private info with {{}} placeholders)

all the source code is located in "src/{{name}}" from the root of my repository where setup.cfg is located.

# setup.cfg

[metadata]
name = {{name}}
version = {{version}}
classifiers =
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3.8
    Programming Language :: Python :: 3.9
    Programming Language :: Python :: 3.10

[options]
package_dir = 
    =src
packages = find:
python_requires = >=3.8
install_requires =
    {{your versioned pip requirements here e.g.:}}
    {{gql[aiohttp, websockets, requests]>=3.0,<4}}

[options.packages.find]
where = src
yasntrk commented 11 months ago

Same error in Python 3.8.4

sidhujus commented 11 months ago

Hi @yasntrk, could you provide more information on your error? If its the one above to do with setup.py files thats because currently sam build doesn't support pyproject.toml files and requires a setup.py instead.

If its to do with the original issue could you try running the commands again but using one of the versions of sam linked here https://github.com/sidhujus/aws-sam-cli/actions/runs/7011049269

The version of sam there includes the fix that was previously mentioned in this thread and did fix the error for me. Just make sure to run the command as sam-beta build

yasntrk commented 11 months ago

Hi @yasntrk, could you provide more information on your error? If its the one above to do with setup.py files thats because currently sam build doesn't support pyproject.toml files and requires a setup.py instead.

If its to do with the original issue could you try running the commands again but using one of the versions of sam linked here https://github.com/sidhujus/aws-sam-cli/actions/runs/7011049269

The version of sam there includes the fix that was previously mentioned in this thread and did fix the error for me. Just make sure to run the command as sam-beta build

Hi, Thanks for the reply!

I solved the issue. I was using conda environment while building Lambda. I think somehow aws-sam and git or conda are having an incompatibility. When I used pyenv instead of conda I had no problems with the same version.

github-actions[bot] commented 10 months ago

Patch is released in v1.44.0. If you are AWS SAM CLI user, please wait for next AWS SAM CLI release. Closing

chrisntb commented 8 months ago

I am a SAM user, using verison 'SAM CLI, version 1.109.0' which was released after the message above:

"Patch is released in v1.44.0. If you are AWS SAM CLI user, please wait for next AWS SAM CLI release. Closing".

I get this error locally on Debian Linux using pyenv and Python 3.12. The error also happens in a GitHub action using Python 3.12 'actions/setup-python@v5'.

I tried adding setup.py/setup.cfg to the library repository but no luck.

Is it supported to use private Python libraries with SAM?

chrisntb commented 8 months ago

My problem was 'No module named 'setuptools'' in the GitHub action:

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

Solved with 'pip install --upgrade pip setuptools' just before running 'sam build --debug'