SAM direct-invoke stalls at "Found workflow" for python app

ckoppelman commented 1 week ago

Problem

I have three Debian machines with very similar builds. On only one of them, the Debug in the VSCode extension stalls out at the below line and times out. This does not happen on that same machine via the command line.

Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'

Steps to reproduce the issue

samconfig.toml:

version = 0.1

[default]
[default.global.parameters]
stack_name = "my-app-test"

[default.build.parameters]
cached = true
parallel = true

[default.validate.parameters]
lint = true

[default.deploy.parameters]
confirm_changeset = true
resolve_s3 = true
s3_prefix = "my-app"
region = "us-east-1"
profile = "dev"
image_repositories = []
capabilities = []

[default.package.parameters]
resolve_s3 = false

[default.sync.parameters]
watch = true

[default.local_start_api.parameters]
warm_containers = "EAGER"

[default.local_start_lambda.parameters]
warm_containers = "EAGER"
debug_port = 4711

[dev.deploy.parameters]
stack_name = "my-app"
resolve_s3 = true
s3_prefix = "my-app"
region = "us-east-1"
profile = "dev"
image_repositories = []

template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  my-app Load

Globals:
  Function:
    Timeout: 60
    MemorySize: 128

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: path.to.handler
      Runtime: python3.11
      Architectures:
        - x86_64
      Environment:
        Variables:
          MY_VAR: var_value

Outputs:
  MyFunction:
    Description: "My Function ARN"
    Value: !GetAtt MyFunction.Arn

launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "aws-sam",
            "request": "direct-invoke",
            "name": "My lambda",
            "invokeTarget": {
                "target": "code",
                "projectRoot": "${workspaceFolder}/path/to/root",
                "lambdaHandler": "path.to.handler"
            },
            "lambda": {
                "runtime": "python3.11",
                "payload": {
                    "json": {}
                },
                "environmentVariables": {
                    "MY_VAR": "var_value"
                }
            },
            "sam": {
                "buildArguments": [
                    "--debug"
                ],
                "buildDir": "/tmp/aws-toolkit-vscode/directory/",
            }
        }
    ]
}

Log output

2024-06-26 17:10:31.316 [warning] SAM debug: missing AWS credentials (Toolkit is not connected)
2024-06-26 17:10:34.177 [info] Preparing to debug locally: Lambda "path.to.handler"
2024-06-26 17:10:34.177 [info] Building SAM application...
2024-06-26 17:10:34.925 [info] SAM CLI not configured, using SAM found at: '/usr/local/bin/sam'
2024-06-26 17:10:34.926 [info] Command: (not started) [/usr/local/bin/sam build --build-dir /tmp/aws-toolkit-vscode/directory/output --template /tmp/aws-toolkit-vscode/directory/app___vsctk___template.yaml --base-dir /var/www/directory/path/to/root --manifest /tmp/aws-toolkit-vscode/directory/debug-requirements.txt --debug]
2024-06-26 17:10:35.968 [info] 2024-06-26 17:10:35,967 | No config file found in this directory.
2024-06-26 17:10:35.969 [info] 2024-06-26 17:10:35,967 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/tmp/aws-toolkit-vscode/directory/samconfig.toml'
2024-06-26 17:10:35,968 | Config file location: /tmp/aws-toolkit-vscode/directory/samconfig.toml
2024-06-26 17:10:35,968 | Config file '/tmp/aws-toolkit-vscode/directory/samconfig.toml' does not exist
2024-06-26 17:10:35.992 [info] 2024-06-26 17:10:35,991 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/tmp/aws-toolkit-vscode/directory/samconfig.toml'
2024-06-26 17:10:35.993 [info] 2024-06-26 17:10:35,991 | Using config file: samconfig.toml, config environment: default
2024-06-26 17:10:35,991 | Expand command line arguments to:
2024-06-26 17:10:35,991 | --template_file=/tmp/aws-toolkit-vscode/directory/app___vsctk___template.yaml --build_dir=/tmp/aws-toolkit-vscode/directory/output --base_dir=/var/www/directory/path/to/root --manifest=/tmp/aws-toolkit-vscode/directory/debug-requirements.txt --mount_with=READ --cache_dir=.aws-sam/cache
2024-06-26 17:10:36.043 [info] 2024-06-26 17:10:36,042 | 'build' command is called
2024-06-26 17:10:36.046 [info] 2024-06-26 17:10:36,044 | No Parameters detected in the template
2024-06-26 17:10:36.129 [info] 2024-06-26 17:10:36,128 | There is no customer defined id or cdk path defined for resource applicationname, so we will use the resource logical id as the resource id
2024-06-26 17:10:36.130 [info] 2024-06-26 17:10:36,128 | 0 stacks found in the template
2024-06-26 17:10:36,129 | No Parameters detected in the template
2024-06-26 17:10:36.153 [info] 2024-06-26 17:10:36,151 | There is no customer defined id or cdk path defined for resource applicationname, so we will use the resource logical id as the resource id
2024-06-26 17:10:36.154 [info] 2024-06-26 17:10:36,152 | 1 resources found in the stack
2024-06-26 17:10:36.154 [info] 2024-06-26 17:10:36,153 | Found Serverless function with name='applicationname' and CodeUri='/var/www/directory/path/to/root'
2024-06-26 17:10:36.157 [info] 2024-06-26 17:10:36,157 | 1 resources found in the stack
2024-06-26 17:10:36.158 [info] 2024-06-26 17:10:36,157 | Found Serverless function with name='applicationname' and CodeUri='/var/www/directory/path/to/root'
2024-06-26 17:10:36.158 [info] 2024-06-26 17:10:36,158 | Instantiating build definitions
2024-06-26 17:10:36.163 [info] 2024-06-26 17:10:36,162 | Same function build definition found, adding function (Previous: BuildDefinition(python3.11, /var/www/directory/path/to/root, Zip, , 75d8ada6-6786-4ab1-8ec8-67dbfd8ea874, {}, {}, x86_64, []), Current: BuildDefinition(python3.11, /var/www/directory/path/to/root, Zip, , 5ad1dece-1b0c-413b-8b9d-decb95743669, {}, {}, x86_64, []), Function: Function(function_id='applicationname', name='applicationname', functionname='applicationname', runtime='python3.11', memory=None, timeout=None, handler='path.to.handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/var/www/directory/path/to/root', environment={'Variables': {'MY_VAR': 'var_value'}}, rolearn=None, layers=[], events=None, metadata={'SamResourceId': 'applicationname'}, inlinecode=None, codesign_config_arn=None, architectures=None, function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))
2024-06-26 17:10:36.164 [info] 2024-06-26 17:10:36,164 | Building codeuri: /var/www/directory/path/to/root runtime: python3.11 metadata: {} architecture: x86_64 functions: applicationname
2024-06-26 17:10:36.165 [info] 2024-06-26 17:10:36,164 | Building to following folder /tmp/aws-toolkit-vscode/directory/output/applicationname
2024-06-26 17:10:36.166 [info] 2024-06-26 17:10:36,165 | Loading workflow module 'aws_lambda_builders.workflows'
2024-06-26 17:10:36.167 [info] 2024-06-26 17:10:36,166 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2024-06-26 17:10:36.168 [info] 2024-06-26 17:10:36,167 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2024-06-26 17:10:36.169 [info] 2024-06-26 17:10:36,169 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2024-06-26 17:10:36.171 [info] 2024-06-26 17:10:36,171 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2024-06-26 17:10:36.174 [info] 2024-06-26 17:10:36,173 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2024-06-26 17:10:36.176 [info] 2024-06-26 17:10:36,176 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2024-06-26 17:10:36.181 [info] 2024-06-26 17:10:36,180 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
2024-06-26 17:10:36.185 [info] 2024-06-26 17:10:36,184 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2024-06-26 17:10:36.186 [info] 2024-06-26 17:10:36,185 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2024-06-26 17:10:36.188 [info] 2024-06-26 17:10:36,187 | Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
2024-06-26 17:10:36.189 [info] 2024-06-26 17:10:36,188 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2024-06-26 17:13:36.191 [error] SamLaunchRequestError: "sam build" failed. Timed out waiting for build. [BuildFailure]

And then it times out.

Expected behavior

Things continue onward as they do in the CLI, with

Running workflow 'PythonPipBuilder'

System details (run `AWS: About` and/or `Amazon Q: About`)

OS: Debian, but run over SSH. The same client works on one host and fails on another
- Client: macOS 13.6.7 (22G720).
- Host: Linux 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08) x86_64 GNU/Linux
Visual Studio Code version: Version: 1.90.2
AWS Toolkit version:
- OS: Linux x64 4.19.0-16-amd64
- Visual Studio Code extension host: 1.90.2
- AWS Toolkit: 3.11.0
- node: 20.11.1
sam version: SAM CLI, version 1.118.0
Amazon Q version: n/a

justinmk3 commented 1 week ago

The aws.samcli.lambdaTimeout setting controls how long AWS Toolkit waits for the "sam build" command. Does increasing this setting resolve your issue?

ckoppelman commented 1 week ago

No, I tried extending it quite a bit, and the other machines run that part almost instantly.

justinmk3 commented 1 week ago

And on this machine, running sam cli in your terminal doesn't have the same behavior?

This timeout indicates that sam isn't sending output to AWS Toolkit. If you check the sam process in your OS process tree, is it active (i.e. non-zero CPU usage)?

ckoppelman commented 1 week ago

Correct. The CLI works fine. What could be wrong with SAM here?

ckoppelman commented 1 day ago

I was able to resolve this by reinstalling sam

aws / aws-toolkit-vscode