bzhou-sw
commented
3 years ago If I set S3 client's enableTcpKeepAlive to false, then no crashing regardless iteration interval. So this seems some issue inside SDK?
Closed bzhou-sw closed 3 years ago
bzhou-sw
commented
3 years ago If I set S3 client's enableTcpKeepAlive to false, then no crashing regardless iteration interval. So this seems some issue inside SDK?
KaibaLopez
commented
3 years ago Hi @bzhou-sw , that's odd, I would expect the opposite to happen with tcpkeepalive.... can you enable the logs for both true and false and post them here? (Remember to censor private info first though).
bzhou-sw
commented
3 years ago @KaibaLopez SDK didn't log anything (even I set log level to TRACE) after first iteration is done, it just silently crashed at the first request of second iteration. Or maybe there was something but not flushed to file? Is there any way to let SDK instantly flush to log file for every message (we are using SDK default logging system)? And what level of log you want?
bzhou-sw
commented
3 years ago The code logic is:
int main(int argc, char *argv[]) {
Aws::SDKOptions options;
Aws::Utils::Logging::InitializeAWSLogging(
Aws::MakeShared<Aws::Utils::Logging::DefaultLogSystem>(
"flowlogs-test", convert_to_aws_log_level(log_level),
input_config.cfg_log_file_prefix));
Aws::InitAPI(options);
// We are not ec2 instance, so disable EC2MetadataClient
setenv("AWS_EC2_METADATA_DISABLED", "true", 1);
// Client config
Aws::Client::ClientConfiguration config;
if (input_config.cfg_disable_keepalive) {
config.enableTcpKeepAlive = 0;
}
config.region = input_config.cfg_region;
config.useDualStack = 1;
config.proxyHost = input_config.cfg_proxy;
config.proxyPort = atoi(input_config.cfg_proxy_port);
Aws::Auth::AWSCredentials credentials;
credentials.SetAWSAccessKeyId(input_config.cfg_accecc_key_id);
credentials.SetAWSSecretKey(input_config.cfg_secret_access_key);
bool useVirtualAddressing = true;
if (input_config.cfg_aws_force_path_style_access &&
strcmp(input_config.cfg_aws_force_path_style_access, "true") == 0) {
AWS_LOG_INFO(__func__, "Setting useVirtualAddressing to false");
useVirtualAddressing = false;
}
Aws::S3::S3Client s3Client(
credentials, config,
Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,
useVirtualAddressing);
for (every 5 minutes) {
Aws::S3::Model::ListObjectsV2Request requestV2;
requestV2.WithBucket(input_config.cfg_bucket_name);
requestV2.WithPrefix(someprefix);
requestV2.SetDelimiter("/");
AWS_LOG_INFO(__func__, "Calling s3Client.ListObjectsV2 ...\n");
auto acctOutcome = s3Client.ListObjectsV2(requestV2);
if (acctOutcome.IsSuccess()) {
AWS_LOG_INFO(__func__, "s3Client.ListObjectsV2 IsSuccess\n");
} else {
AWS_LOG_ERROR(__func__, "s3Client.ListObjectsV2 failed\n");
}
}
Aws::ShutdownAPI(options);
Aws::Utils::Logging::ShutdownAWSLogging();
// Exit only something is wrong
return -1;
}Here is logs.zip (of keepalive_enabled.log and keepalive_disabled.log): logs.zip
KaibaLopez
commented
3 years ago Hi @bzhou-sw , Yea I'm able to reproduce this, we'll take a closer look into this, but yea this all looks weird to me, keepalive should not even be related to this, it should only affect delays when the request has been made and not in between requests.
For now I 'd say just disable it since that works for you.
bzhou-sw
commented
3 years ago @KaibaLopez Thanks for looking into this.
ebinans
commented
3 years ago This is most likely caused by SIGPIPE.
SDK uses following option to make it compatible with multi-thread applications:
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);https://curl.se/libcurl/c/CURLOPT_NOSIGNAL.html
That in turn means that you need to handle the signal yourself e.g.:
signal(SIGPIPE, SIG_IGN);@ebinans Thanks for your comments. Ignore SIGPIPE in our application solved the issue!
wps132230
commented
3 years ago Or you may try this in your options when initializing your application:
options.httpOptions.installSigPipeHandler = true;@wps132230 Thanks!
github-actions[bot]
commented
3 years ago Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
adamcalhoon
commented
2 years ago @KaibaLopez -- I am running into this issue with version 1.9.176 of the SDK library. In particular it happens occasionally when calling GetObject.
I first tried the options.httpOptions.installSigPipeHandler = true; suggestion which did not solve the problem. Next I tried the enableTcpKeepAlive = false in the S3 Client configuration which does seem to have worked.
Should I open a new issue? Are there any other things I should try?
jmklix
commented
2 years ago Yes, please open a new issue and include all relevant information/logs
shrkamat
commented
2 years ago @adamcalhoon is the issue resolved for you. I am facing similar error, but with Openssl 1.0 version. With Openssl 1.1 version it works fine.
Confirm by changing [ ] to [x] below:
Platform/OS/Hardware/Device VM with Linux Debian 4.19.0-8-amd64 SDK release 1.8.63
Describe the question I have an application calling S3 client ListObjectsV2 () periodically and call S3 client GetObject () if any objects in certain time frame. If the interval is 1 minute, then it works fine. If interval is 2 minutes or larger, then first iteration works for both list and get objects. But second iteration crashes. See back trace below.
We use default aws config except:
region,proxyHost,proxyPort, and AWS credentials.Do you have any idea why this happens? Any advice or suggestion are highly appreciated.
Logs/output link information;
gdb crash backtrace:
REMEMBER TO SANITIZE YOUR PERSONAL INFO