Can not install custom python dependency from custom git url

mrtj commented 9 months ago

Description:

I am trying to install a python dependency from a custom git repo, with a specific commit and a subdirectory specified.

Steps to reproduce:

I have the following line in my requirements.txt:

langchain @ git+https://github.com/mrtj/langchain.git@7b52eff#subdirectory=libs/langchain

Observed result:

sam build fails with the error PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package

Expected result:

sam build works.

Note: pip install -r requirements.txt do correctly install the required version of the package.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.98.0",
  "system": {
    "python": "3.8.13",
    "os": "macOS-12.5-x86_64-i386-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.23",
    "aws_cdk": "Not available",
    "terraform": "1.3.7"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

Debug output:

2023-10-10 18:26:23,075 |  Running PythonPipBuilder:ResolveDependencies                                                                    
2023-10-10 18:26:23,104 | calling pip download -r /Users/janos/Projects/teamsystem_personalizer/personalizer/requirements.txt --dest       
/var/folders/6r/wc15p6m13nl_nl_n_xfqpc5c0000gp/T/tmpvma9ri9e --exists-action i                                                             
2023-10-10 18:26:39,016 | Non zero rc (2) from the setup.py egg_info command: b"Usage: sam [OPTIONS] COMMAND [ARGS]...\nTry 'sam -h' for   
help.\n\nError: No such option: -c\n"                                                                                                      
2023-10-10 18:26:39,019 | Using fallback location for PKG-INFO file in package directory:                                                  
/var/folders/6r/wc15p6m13nl_nl_n_xfqpc5c0000gp/T/tmpve6hox12/langchain                                                                     
2023-10-10 18:26:39,146 | PythonPipBuilder:ResolveDependencies raised unhandled exception                                                  
Traceback (most recent call last):                                                                                                         
  File "aws_lambda_builders/workflow.py", line 371, in run                                                                                 
  File "aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute                                                          
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 158, in build_dependencies                                             
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 259, in build_site_packages                                            
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 283, in _download_dependencies                                         
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in _download_all_dependencies                                     
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 366, in <setcomp>                                                      
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 515, in __init__                                                       
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 557, in _calculate_name_and_version                                    
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 698, in get_package_name_and_version                                   
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 610, in _get_pkg_info_filepath                                         
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: langchain          
2023-10-10 18:26:39,154 | Exception raised during the execution

lucashuy commented 9 months ago

Hi, thanks for raising this issue. This is a result of how the package resolution logic works in Lambda Builders. Essentially, if the pip workflow finds a package that is not built already (the langchain fork that is provided), it'll try to resolve some basic info from that package by looking at the setup.py file. Since this file is non-existent, it fails like you see here.

While we look for a proper way forward, would it be possible to build this package locally and reference it inside of the requirements.txt file as a local dependency with file:// instead of pointing to the git repository? This could be done as a step prior to running sam build, or as part of a Makefile build workflow.

mrtj commented 9 months ago

Hello, thank you for the workaround. Indeed compiling my version of langchain into a wheel and passing it as a local file pointer worked.

I am sure that you're aware of the fact that setup.py is deprecated since years and nowadays most projects use pyproject.toml as package configuration structure. So looking for the existence of setup.py might be error prone.

Also, in langchain package the pyproject.toml is not placed in the root of the repo but instead in a subfolder pointed by the #subdirectory URL tag as specified in the pip documentation. I do not know if this inferred with the bug somehow.

lucashuy commented 8 months ago

Thanks for confirming that building the project and then using the wheel file worked. As for moving away from setup.py, this would have to be something that would likely be taken in as a feature request since this is something that we don't currently check/handle.

Using the url fragments as you've done in the requirements file works fine however (though granted the project would probably have to have a setup.py file as discovered)

dtcaciuc-ggs commented 1 week ago

I would also like to see sam build recognizing pyproject.toml files. We tried publishing wheels through build artifacts, however can't download them because the code is in a private repo and they cannot be accessed through SSH. We had to resort to hand-rolling PKG-INFO which is another thing to keep up to date and duplicates the info that pyproject.toml already provides.

aws / aws-lambda-builders