EventBridge: Function gets stuck at PutEvetsAsync() during runtime until it timeouts

smoothstill commented 9 months ago

Describe the bug

We have a function which is supposed to send an event to event bridge event bus:

PutEventsResponse putEventsResponse = await _busClient.PutEventsAsync(new PutEventsRequest
{
      Entries = new List<PutEventsRequestEntry> { entry }
});

However, during runtime the function gets stuck at method PutEventsAsync(), and it remains stuck until the function timeouts (even if I set maximum timeout of 900 seconds to the function).

The problem is that I have zero clue on why this happens because there are no exceptions or any errors in the log files to tell me what is the issue. I only know that no code is executed after PutEventAsync(). So my question is, is there a way to increase the verbosity of the event bridge client so that I know why it gets stuck here?

Here are the logs:

INIT_START Runtime Version: dotnet:6.v25    Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:REDACTED
Amazon Information: 0 : UserCrypto is not supported.  This may be due to use of a non-Windows operating system or Windows Nano Server, or the current user account may not have its profile loaded. Unable to load shared library 'Crypt32.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libCrypt32.dll: cannot open shared object file: No such file or directory
Amazon Information: 0 : Credentials found using environment variables.
Amazon Information: 0 : The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
Amazon Information: 1 : The environment variable AWS_MAX_ATTEMPTS was not set with a value.
Amazon Information: 2 : The environment variable AWS_RETRY_MODE was not set with a value.
Amazon Information: 3 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
Amazon Information: 4 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
Amazon Information: 5 : The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
Amazon Information: 6 : The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
Amazon Information: 0 : Unable to find a profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 0 : Region found using environment variable.
Amazon Information: 0 : Resolved DefaultConfigurationMode for RegionEndpoint [eu-west-1] to [Legacy].
START RequestId: fa49d7f2-9d2d-5e14-bf76-89d21e86a8ae Version: $LATEST
2024-01-16T13:15:39.279Z    fa49d7f2-9d2d-5e14-bf76-89d21e86a8ae    info    Sending event to event bus: 'REDACTED'
Amazon Information: 1 : Credentials found using environment variables.
Amazon Information: 2 : Credentials found using environment variables.
END RequestId: fa49d7f2-9d2d-5e14-bf76-89d21e86a8ae
REPORT RequestId: fa49d7f2-9d2d-5e14-bf76-89d21e86a8ae  Duration: 30035.89 ms   Billed Duration: 30000 ms   Memory Size: 2048 MB    Max Memory Used: 168 MB Init Duration: 373.56 ms    
XRAY TraceId: REDACTED  SegmentId: REDACTED Sampled: true

Interestingly, sending events seem to work well in our staging environment, just not in sandbox environment, even though there should not be any differences that would effect sending events. It is really difficult to understand why this happens when there are no information to tell me what is wrong with the event bridge client, so if anyone has any clues on why this happens, feel free to share them.

Expected Behavior

I would expect to receive an exception or error/warning message from event bridge client when the runtime gets stuck on PutEventsAsync() method.

Current Behavior

While executing the function, it appears to get stuck during runtime to PutEventsAsync() without giving any visible reason why this happens.

Reproduction Steps

Uknown

Possible Solution

PutEventsAsync throws an exception
Increased verbosity of PutEventsAsync
Better documentation

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.EventBridge 3.7.5.27

Targeted .NET Platform

.Net 6

Operating System and version

x86-64 Linux (The aws/codebuild/standard:7.0 build image)

ashishdhingra commented 9 months ago

@smoothstill Good morning. Could you please share your execution environment? In case of Lambda execution environment, using custom logging also suffers from the same issue due to reasons mentioned in https://github.com/aws/aws-logging-dotnet?tab=readme-ov-file#aws-lambda. In your case, you are sending PutEventsAsync request using async/await pattern and by the time the result of PutEventsAsync is received, Lambda event is already processed and the execution thread is frozen.

Could you try the following:

While you notice that your PutEventsAsync call is frozen, could you send another request to the Lambda function in a separate thread and examine if the previous request is now processed?
Could you modify your code to reference Task object of PutEventsAsync call and instead of awaiting it, use Task.WaitAll(taskObject);

Thanks, Ashish

github-actions[bot] commented 9 months ago

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

aws / aws-sdk-net