search

aws-observability / aws-otel-lambda

AWS Distro for OpenTelemetry - AWS Lambda
https://aws-otel.github.io/
Apache License 2.0
143 stars 52 forks source link

ADOT Lambda Layer local testing #931

Open roger-zhangg opened 1 month ago

roger-zhangg commented 1 month ago

Is your feature request related to a problem? Please describe. AWS ADOT Lambda Python Sample app stuck till timeout in sam local invoke. Although the code in handler executed successfully, the ADOT Lambda Layer seems blocking the local invoke from exiting normally. This example works as expected when deployed to Lambda.

Describe the solution you'd like When Local testing, Function with ADOT Lambda layer should exit normally after execution

Describe alternatives you've considered Remove the ADOT layer when local testing, but this requires repeated manual work

Additional context

Steps to reproduce:

git clone git@github.com:aws-observability/aws-otel-lambda.git
cd aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam

sam build -u
sam local invoke --debug

Observed result:

Elysia % sam local invoke --region us-west-2 --debug                                                      
2024-05-28 18:38:58,817 | Using config file: samconfig.toml, config environment: default                                                            
2024-05-28 18:38:58,817 | Expand command line arguments to:                                                                                         
2024-05-28 18:38:58,818 | --template_file=/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/template.yaml  --no_event --layer_cache_basedir=/home/ruojiazh/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1                 
2024-05-28 18:38:58,819 | local invoke command is called                                                                                            
2024-05-28 18:38:58,824 | No Parameters detected in the template                                                                                    
2024-05-28 18:38:58,862 | There is no customer defined id or cdk path defined for resource api, so we will use the resource logical id as the       
resource id                                                                                                                                         
2024-05-28 18:38:58,863 | Sam customer defined id is more priority than other IDs. Customer defined id for resource function is function            
2024-05-28 18:38:58,864 | 0 stacks found in the template                                                                                            
2024-05-28 18:38:58,864 | No Parameters detected in the template                                                                                    
2024-05-28 18:38:58,880 | There is no customer defined id or cdk path defined for resource api, so we will use the resource logical id as the       
resource id                                                                                                                                         
2024-05-28 18:38:58,880 | Sam customer defined id is more priority than other IDs. Customer defined id for resource function is function            
2024-05-28 18:38:58,881 | 2 resources found in the stack                                                                                            
2024-05-28 18:38:58,882 | Found Serverless function with name='function' and CodeUri='function'                                                     
2024-05-28 18:38:58,882 | --base-dir is not presented, adjusting uri function relative to                                                           
/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/template.yaml                                            
2024-05-28 18:38:58,887 | Found one Lambda function with name 'function'                                                                            
2024-05-28 18:38:58,888 | Invoking lambda_function.lambda_handler (python3.9)                                                                       
2024-05-28 18:38:58,888 | Loading AWS credentials from session with profile 'None'                                                                  
2024-05-28 18:38:59,227 | Resolving code path. Cwd=/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build,      
CodeUri=/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function                                         
2024-05-28 18:38:59,229 | Resolved absolute path to code is                                                                                         
/local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function                                                 
2024-05-28 18:38:59,229 | Code /local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function is not a zip/jar 
file                                                                                                                                                
2024-05-28 18:38:59,230 | arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-24-0:1 is already cached. Skipping download       
2024-05-28 18:38:59,636 | Local image is up-to-date                                                                                                 
2024-05-28 18:38:59,640 | Checking free port on 127.0.0.1:8100                                                                                      
2024-05-28 18:38:59,642 | Using local image: samcli/lambda-python:3.9-x86_64-3260d442615df22e4be53571e.                                             

2024-05-28 18:38:59,643 | Mounting /local/home/ruojiazh/aws-otel-lambda/sample-apps/python-aws-sdk-aiohttp-sam/.aws-sam/build/function as           
/var/task:ro,delegated, inside runtime container                                                                                                    
2024-05-28 18:39:00,014 | Starting a timer for 15 seconds for function 'function'                                                                   
START RequestId: 18e3b183-d240-40f3-bc3c-2d660f6dea39 Version: $LATEST
{"level":"info","ts":1716921540.029972,"msg":"Launching OpenTelemetry Lambda extension","version":"v0.39.0"}
{"level":"info","ts":1716921540.0309246,"logger":"telemetryAPI.Listener","msg":"Listening for requests","address":":53612"}
{"level":"info","ts":1716921540.0309997,"logger":"telemetryAPI.Client","msg":"Subscribing","baseURL":"http://127.0.0.1:9001/2022-07-01/telemetry"}
{"level":"error","ts":1716921540.0312636,"logger":"telemetryAPI.Client","msg":"Subscription failed. Logs API is not supported! Is this extension running in a local sandbox?","status_code":202}
{"level":"info","ts":1716921540.0312836,"logger":"telemetryAPI.Client","msg":"Subscription success","response":"{\"errorMessage\":\"Telemetry API is not supported\",\"errorType\":\"Telemetry.NotSupported\"}\n"}
{"level":"info","ts":1716921540.0329914,"caller":"service@v0.98.0/telemetry.go:55","msg":"Setting up own telemetry..."}
{"level":"info","ts":1716921540.0331674,"caller":"service@v0.98.0/telemetry.go:97","msg":"Serving metrics","address":"localhost:8888","level":"Basic"}
{"level":"info","ts":1716921540.034368,"caller":"exporter@v0.98.0/exporter.go:275","msg":"Deprecated component. Will be removed in future releases.","kind":"exporter","data_type":"metrics","name":"logging"}
{"level":"info","ts":1716921540.0594833,"caller":"service@v0.98.0/service.go:143","msg":"Starting aws-otel-lambda...","Version":"v0.39.0","NumCPU":16}
{"level":"info","ts":1716921540.0595214,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"info","ts":1716921540.0596437,"caller":"otlpreceiver@v0.98.0/otlp.go:102","msg":"Starting GRPC server","kind":"receiver","name":"otlp","data_type":"traces","endpoint":"localhost:4317"}
{"level":"info","ts":1716921540.0598555,"caller":"otlpreceiver@v0.98.0/otlp.go:152","msg":"Starting HTTP server","kind":"receiver","name":"otlp","data_type":"traces","endpoint":"localhost:4318"}
{"level":"info","ts":1716921540.0599985,"caller":"service@v0.98.0/service.go:169","msg":"Everything is ready. Begin running and processing data."}
{"level":"warn","ts":1716921540.0600317,"caller":"localhostgate/featuregate.go:63","msg":"The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.","feature gate ID":"component.UseLocalHostAsDefaultHost"}

MyBucket1
MyBucket2

-2024-05-28 18:39:15,017 | Function 'function' timed out after 15 seconds                                                                            
-28 May 2024 18:39:15,028 [ERROR] (rapid) Invoke failed error=errResetReceived InvokeID=f3183651-cc6b-42fc-a050-cb60bc82b647
2024-05-28 18:39:15,736 | Container ID not defined, unable to fetch container state                                                                 
2024-05-28 18:39:15,738 | Container was not created. Skipping deletion                                                                              
2024-05-28 18:39:15,738 | Cleaning all decompressed code dirs                                                                                       
2024-05-28 18:39:15,739 | No response from invoke container for function                                                                            
2024-05-28 18:39:15,739 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                
2024-05-28 18:39:15,747 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                
2024-05-28 18:39:15,748 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '064e6001-623a-48cc-bee3-ed8f62126453', 'installationId':    
'f5068bb9-add9-4617-8f83-fdf7f78ec852', 'sessionId': '48d4dc59-f4e7-4716-b67b-fe983cedd9a1', 'executionEnvironment': 'CLI', 'ci': False,            
'pyversion': '3.11.3', 'samcliVersion': '1.108.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName':    
'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName':                                            
'fcd52188567211bf1788ec5003f8a07ca1de3afc4605c8c2887fd8cb0d96f172', 'initialCommit': None}, 'duration': 16922, 'exitReason': 'success', 'exitCode': 
0}}]}                                                                                                                                               
2024-05-28 18:39:15,748 | Unable to find Click Context for getting session_id.                                                                      
2024-05-28 18:39:15,752 | Sending Telemetry: {'metrics': [{'events': {'requestId': '66e243b6-0509-46c7-b171-f72bb048b391', 'installationId':        
'f5068bb9-add9-4617-8f83-fdf7f78ec852', 'sessionId': '48d4dc59-f4e7-4716-b67b-fe983cedd9a1', 'executionEnvironment': 'CLI', 'ci': False,            
'pyversion': '3.11.3', 'samcliVersion': '1.108.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name':        
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '5a1d73ed92374980819d53bb918cc9a3', 'time_stamp': '2024-05-28 18:38:58.790',         
'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '9fcd57b9cd85422aafc966b6d902cafd',          
'time_stamp': '2024-05-28 18:38:58.817', 'exception_name': None}]}}}]}                                                                              
2024-05-28 18:39:15,786 | Telemetry response: 200                                                                                                   
2024-05-28 18:39:15,803 | Telemetry response: 200

Env


us-west-2

sam --info
{
  "version": "1.108.0",
  "system": {
    "python": "3.11.3",
    "os": "Linux-5.10.216-182.855.amzn2int.x86_64-x86_64-with-glibc2.26"
  },
  "additional_dependencies": {
    "docker_engine": "25.0.3",
    "aws_cdk": "2.134.0 (build 265d769)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}```
roger-zhangg commented 1 week ago

Hey Team, has anyone checked out this yet?