aws / aws-toolkit-jetbrains

AWS Toolkit for JetBrains - a plugin for interacting with AWS from JetBrains IDEs
https://plugins.jetbrains.com/plugin/11349-aws-toolkit
Apache License 2.0
747 stars 214 forks source link

Error: PythonPipBuilder:ResolveDependencies - list index out of range #1169

Closed OscarVanL closed 5 years ago

OscarVanL commented 5 years ago

Describe the bug

When using PyCharm to create a new AWS Lambda to run remotely or running it locally, I get a list index out of range error when PythonPipBuilder tries to resolve the dependencies.

To reproduce

  1. Run [Local] configuration of serverless application with the AWS Toolkit

Expected behavior

Requirements should be successfully parsed from the requirements.txt file inside the project's folder.

Screenshots

image

Your Environment

Additional context

C:\Users------.PyCharm2019.2\system\log\idea.log:

2019-08-19 10:28:57,790 [3420678]   INFO - .services.lambda.LambdaBuilder - "C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" build --template C:\Users\-------\PycharmProjects\AutomatedSageMakerTraining\template.yaml --build-dir C:\Users\-------\PycharmProjects\AutomatedSageMakerTraining\.aws-sam\build AutomatedSageMakerGeneration

2019-08-19 10:28:57,826 [3420714]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:28:59,253 [3422141]   INFO - .services.lambda.LambdaBuilder - 2019-08-19 10:28:59 Building resource 'AutomatedSageMakerGeneration'

2019-08-19 10:28:59,430 [3422318]   INFO - .services.lambda.LambdaBuilder - 2019-08-19 10:28:59 Running PythonPipBuilder:ResolveDependencies

2019-08-19 10:29:08,723 [3431611]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:29:10,863 [3433751]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:30:03,970 [3486858]   INFO - .services.lambda.LambdaBuilder - 

2019-08-19 10:30:03,970 [3486858]   INFO - .services.lambda.LambdaBuilder - Build Failed

2019-08-19 10:30:04,848 [3487736]   INFO - .services.lambda.LambdaBuilder - Error: PythonPipBuilder:ResolveDependencies - list index out of range

2019-08-19 10:30:04,913 [3487801]   WARN - xecution.local.SamInvokeRunner - Failed to create Lambda package 
java.util.concurrent.CompletionException: java.lang.IllegalStateException: SAM build command failed
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
    at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
    at software.aws.toolkits.jetbrains.services.lambda.LambdaBuilderUtils$buildAndReport$$inlined$runSamBuildInBackground$1.run(LambdaBuilderUtils.kt:104)
    at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:894)
    at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:169)
    at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:591)
    at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:537)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:59)
    at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:156)
    at com.intellij.openapi.progress.impl.CoreProgressManager$4.run(CoreProgressManager.java:408)
    at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:294)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: SAM build command failed
    at software.aws.toolkits.jetbrains.services.lambda.LambdaBuilder$buildLambdaFromTemplate$2$3.processTerminated(LambdaBuilder.kt:155)
    at jdk.internal.reflect.GeneratedMethodAccessor495.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.intellij.execution.process.ProcessHandler$2.invoke(ProcessHandler.java:214)
    at com.sun.proxy.$Proxy26.processTerminated(Unknown Source)
    at com.intellij.execution.process.ProcessHandler.lambda$notifyTerminated$2(ProcessHandler.java:172)
    at com.intellij.execution.process.ProcessHandler$TasksRunner.execute(ProcessHandler.java:247)
    at com.intellij.execution.process.ProcessHandler.notifyTerminated(ProcessHandler.java:155)
    at com.intellij.execution.process.ProcessHandler.notifyProcessTerminated(ProcessHandler.java:151)
    at com.intellij.execution.process.KillableProcessHandler.notifyProcessTerminated(KillableProcessHandler.java:124)
    at com.intellij.execution.process.BaseProcessHandler.onOSProcessTerminated(BaseProcessHandler.java:61)
    at com.intellij.execution.process.OSProcessHandler.onOSProcessTerminated(OSProcessHandler.java:168)
    at com.intellij.execution.process.BaseOSProcessHandler$1.lambda$startNotified$0(BaseOSProcessHandler.java:99)
    at com.intellij.execution.process.ProcessWaitFor.lambda$null$0(ProcessWaitFor.java:61)
    at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:208)
    at com.intellij.execution.process.ProcessWaitFor.lambda$new$1(ProcessWaitFor.java:43)
    ... 5 more
2019-08-19 10:30:04,915 [3487803]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:30:04,946 [3487834]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:30:16,920 [3499808]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:30:18,835 [3501723]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:32:16,548 [3619436]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:32:17,618 [3620506]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:32:18,842 [3621730]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:32:22,754 [3625642]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:35:24,968 [3807856]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:36:10,513 [3853401]   INFO - ces.lambda.sam.SamVersionCache - Evaluating C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd 
2019-08-19 10:36:52,210 [3895098]   INFO - j.ide.plugins.RepositoryHelper - using cached plugin list (updated at 19/08/2019 10:08 am) 
2019-08-19 10:37:35,387 [3938275]   INFO - ide.actions.ShowFilePathAction - 
Exit code 1 

Contents of requirements.txt:

sagemaker
boto3
pandas
s3fs
numpy
pathlib

Contents of template.yaml:

Transform: 'AWS::Serverless-2016-10-31'
Resources:
  AutomatedSageMakerGeneration:
    # This resource creates a Lambda function.
    Type: "AWS::Serverless::Function"
    Properties:
      Runtime: python3.6
      FunctionName: AutomatedSageMakerTraining
      Description: 'A service ran daily generating a new SageMaker model from last days data and making a days predictions.'
      MemorySize: 128
      Timeout: 300
      Role: 'XXXXXXXXXXXXXXXXXXXXXXX'
      # This is the Lambda function's handler.
      Handler: sagemaker_train_task.lambda_handler
      # The location of the Lambda function code.
      CodeUri: .
OscarVanL commented 5 years ago

I decided to run the same SAM CLI command the plugin was calling, along with the --debug parameter.

This gave the following logs:

"C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" build --template C:\Users\-------\PycharmProjects\AutomatedSageMakerTraining\template.yaml --build-dir C:\Users\-------\PycharmProjects\AutomatedSageMakerTraining\.aws-sam\build AutomatedSageMakerGeneration --debug
2019-08-19 11:00:48 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-08-19 11:00:48 Changing event name from before-call.apigateway to before-call.api-gateway
2019-08-19 11:00:48 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-08-19 11:00:48 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-19 11:00:48 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-08-19 11:00:48 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-08-19 11:00:48 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-19 11:00:48 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-19 11:00:48 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-08-19 11:00:48 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-08-19 11:00:48 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-08-19 11:00:48 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-08-19 11:00:48 Changing event name from before-call.apigateway to before-call.api-gateway
2019-08-19 11:00:48 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-08-19 11:00:48 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-19 11:00:48 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-08-19 11:00:48 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-08-19 11:00:48 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-19 11:00:48 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-19 11:00:48 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-08-19 11:00:48 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-08-19 11:00:48 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-08-19 11:00:48 Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2019-08-19 11:00:48 'build' command is called
2019-08-19 11:00:48 No Parameters detected in the template
2019-08-19 11:00:48 1 resources found in the template
2019-08-19 11:00:48 Found Serverless function with name='AutomatedSageMakerGeneration' and CodeUri='.'
2019-08-19 11:00:48 Building resource 'AutomatedSageMakerGeneration'
2019-08-19 11:00:48 Loading workflow module 'aws_lambda_builders.workflows'
2019-08-19 11:00:48 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-08-19 11:00:48 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-08-19 11:00:48 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-08-19 11:00:48 Running workflow 'PythonPipBuilder'
2019-08-19 11:00:48 Running PythonPipBuilder:ResolveDependencies
2019-08-19 11:00:48 calling pip download -r C:\Users\-------\PycharmProjects\AutomatedSageMakerTraining\requirements.txt --dest C:\Users\-------\AppData\Local\Temp\tmpibkvse64
2019-08-19 11:01:48 PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 248, in run
    action.execute()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute
    self.scratch_dir
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 209, in build_site_packages
    scratch_directory, requirements_filepath)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 234, in _download_dependencies
    requirements_filename, directory)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 332, in _download_all_dependencies
    in self._osutils.get_directory_contents(directory)}
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 331, in <setcomp>
    deps = {Package(directory, filename) for filename
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 437, in __init__
    self._name, self._version = self._calculate_name_and_version()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 480, in _calculate_name_and_version
    sdist_path)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 540, in get_package_name_and_version
    pkg_info_filepath = self._generate_egg_info(package_dir)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 523, in _generate_egg_info
    egg_info_dir, info_contents[0], 'PKG-INFO')
IndexError: list index out of range

Build Failed
2019-08-19 11:01:48 Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 60435, 'exitReason': 'UserException', 'exitCode': 1, 'requestId': '8a427a74-46dc-4dbc-921c-4767c8b2877e', 'installationId': '0c5d4463-668a-4706-ae40-afd0704d66bd', 'sessionId': 'b9067c32-9256-428e-9c78-7510f80d5635', 'executionEnvironment': 'CLI', 'pyversion': '3.6.7', 'samcliVersion': '0.19.0'}}]}
2019-08-19 11:01:48 Starting new HTTPS connection (1): aws-serverless-tools-telemetry.us-west-2.amazonaws.com:443
2019-08-19 11:01:49 HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - list index out of range

It seems this is a bug with aws-lambda-builders, not the JetBrains AWS Toolkit. To be specific, it is the same bug as in this issue.

I will close this issue as it is not related to aws-toolkit-jetbrains, but the issue with aws-lambda-builders is not fixed.