search

intel-cloud / cosbench

a benchmark tool for cloud object storage service
Other
573 stars 242 forks source link

S3: Need configurable max_connections for AmazonS3Client #265

Closed zoran-rajic closed 9 years ago

zoran-rajic commented 9 years ago

If using COSBench to test the S3 with more than 50 buckets (ie. 75 or more), the (Unix/Linux) Driver system will deplete all the system's sockets, and the tests will fail.

The issue is that the AmazonS3Client uses a connection pool set to 50 connections max (see DEFAULT_MAX_CONNECTIONS in https://github.com/aws/aws-sdk-java/blob/cf21041bc0129512cd30431fc4a960f9e73e6cbc/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L39), so if for example 75 buckets will be used sequentially in a COSBench test - 1) the usage of the connection pool will be void (no cached connections will actually be reused), and 2) the AmazonS3Client will keep closing old, and adding new connections into the connection pool.

With a fast S3 server, this may deplete all sockets on the Driver's system within minutes (ie. expect over 50'000 sockets stuck in a TIME_WAIT state when you run 'netstat -ant'), and the AmazonS3Client will start throwing following exceptions as it can no longer connect to the S3 server:

[...] Caused by: java.net.ConnectException: Cannot assign requested address at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:123) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) ... 25 more

Expected behavior:

Workarounds (avoids caching host-routes in the AmazonS3Client connection pool):

ywang19 commented 9 years ago

Hi Zoran,

Thansk for the fix on max connection limit, the fix looks fine to me, I will merge it soon.

-yaguang

hrchu commented 9 years ago

I encounter the same problem. Thanks Zoran report this, so I can understand what's happening.

There is another approach to solve the problem: adjust TCP stack config. Here is my setting in /etc/sysctl.conf:

net.ipv4.tcp_syncookies=1
net.ipv4.tcp_tw_reuse =1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=5
ywang19 commented 9 years ago

Thanks to share the new method.

From: hrchu [mailto:notifications@github.com] Sent: Friday, October 16, 2015 4:28 PM To: intel-cloud/cosbench Cc: Wang, Yaguang Subject: Re: [cosbench] S3: Need configurable max_connections for AmazonS3Client (#265)

I encounter the same problem. Thanks Zoran report this, so I can understand what's happening.

There is another approach to solve the problem: adjust TCP stack config. Here is my setting in /etc/sysctl.conf:

net.ipv4.tcp_syncookies=1

net.ipv4.tcp_tw_reuse =1

net.ipv4.tcp_tw_recycle=1

net.ipv4.tcp_fin_timeout=5

— Reply to this email directly or view it on GitHubhttps://github.com/intel-cloud/cosbench/issues/265#issuecomment-148651805.