In the current implementation, sys_info.c::aws_system_info_processor_count() is used to determine the parallelization factor for thread pools. On POSIX systems, the API uses sysconf(_SC_NPROCESSORS_ONLN).
On large systems, such as the u-12tb1.112xlarge instance type, the SDK initiates thread pools with up to 448 threads. This high thread count can potentially lead to S3 operations returning "503 Slow Down" errors due to the high load on a bucket.
This commit introduces a fix to this issue by allowing users to limit the number of threads through a new environment variable, AWS_COMMON_MAX_PROCS. This enhancement provides users with greater control over the parallelization factor, particularly on large systems.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
In the current implementation, sys_info.c::aws_system_info_processor_count() is used to determine the parallelization factor for thread pools. On POSIX systems, the API uses sysconf(_SC_NPROCESSORS_ONLN).
On large systems, such as the u-12tb1.112xlarge instance type, the SDK initiates thread pools with up to 448 threads. This high thread count can potentially lead to S3 operations returning "503 Slow Down" errors due to the high load on a bucket.
This commit introduces a fix to this issue by allowing users to limit the number of threads through a new environment variable, AWS_COMMON_MAX_PROCS. This enhancement provides users with greater control over the parallelization factor, particularly on large systems.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.