Terminating large number of instances will fail with "Connection reset"

[GoogleCodeExporter]

What steps will reproduce the problem?
This is, unfortunately, an intermittent problem.  Calling 
Jec2.terminateInstances() with a very 
long list of instance ID's (in the hundreds) will sometimes fail with a socket 
exception and the 
message "Connection reset".

What is the expected output? What do you see instead?
The instances should terminate successfully.

What version of the product are you using? On what operating system?
Typica v1.5 on Fedora Core 8 64-bit, Java 1.6.0_16 64-bit.

Please provide any additional information below.
This *might* be because terminateInstances() uses GET instead of POST.  I found 
a similar issue 
in the "boto" Python library (see 
http://code.google.com/p/boto/issues/detail?id=133) which 
was resolved by switching to POST.  Furthermore, the problem does not seem to 
happen in 
Elasticfox, which also uses POST.

I can provide a patch, but first I would need to know which version of 
HttpClient we are going 
with :-).

Original issue reported on code.google.com by msol...@gmail.com on 25 Nov 2009 at 6:32

[GoogleCodeExporter]

As it turns out, this is not that hard to reproduce.  Here is an example:

    final int COUNT = 340;

    String[] ids = new String[COUNT];

    for (int i = 0; i < COUNT; i++)
    {
      ids[i] = "i-100" + i;
    }

    Jec2 jec2 = new Jec2("replace with a real access key", "replace with a real secret key", true, "us-east-
1.ec2.amazonaws.com");
    jec2.setMaxRetries(0);
    jec2.terminateInstances(ids);

This consistently produces the following exception:

Exception in thread "main" com.xerox.amazonws.ec2.EC2Exception: Number of 
retries exceeded : 
TerminateInstances
    at com.xerox.amazonws.ec2.Jec2.makeRequestInt(Jec2.java:1838)
    at com.xerox.amazonws.ec2.Jec2.terminateInstances(Jec2.java:625)
    at com.xerox.amazonws.ec2.Jec2.terminateInstances(Jec2.java:606)
Caused by: org.apache.commons.httpclient.HttpException: Number of retries 
exceeded : TerminateInstances
    at com.xerox.amazonws.common.AWSQueryConnection.makeRequest(AWSQueryConnection.java:373)
    at com.xerox.amazonws.ec2.Jec2.makeRequestInt(Jec2.java:1830)
    ... 3 more
Caused by: com.xerox.amazonws.common.AWSException: Connection reset
    at com.xerox.amazonws.common.AWSQueryConnection.makeRequest(AWSQueryConnection.java:344)
    ... 4 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:168)
    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
    at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
    at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
    at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
    at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
    at 
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionA
dapter.readLine(Mult
iThreadedHttpConnectionManager.java:1413)
    at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
    at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
    at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
    at com.xerox.amazonws.common.AWSQueryConnection.makeRequest(AWSQueryConnection.java:340)
    ... 4 more

Original comment by msol...@gmail.com on 25 Nov 2009 at 7:10

[GoogleCodeExporter]

I can confirm that after making the following changes, everything works.  I 
tried up to 1,000 instance ID's and 
got the correct error ("Invalid id") instead of the connection reset:

1) Using POST instead of GET
2) Using a request entity (i.e. request body) instead of a query string
3) Removing the initial "?" from the request (since it's no longer a query 
string)

Note that according to the "boto" issue, this same change might be required to 
support the runInstances() 
method with very large user data.

Original comment by msol...@gmail.com on 25 Nov 2009 at 5:05

[GoogleCodeExporter]

Thanks, I'm going to try changing both runInstances and terminateInstances

Original comment by dkavan...@gmail.com on 23 Mar 2010 at 11:13

• Changed state: Started

[GoogleCodeExporter]

This issue was closed by revision r317.

Original comment by dkavan...@gmail.com on 24 Mar 2010 at 12:48

• Changed state: Fixed

[GoogleCodeExporter]

This has been fixed in SVN r317

Original comment by dkavan...@gmail.com on 24 Mar 2010 at 12:48

[GoogleCodeExporter]

Thanks!

Original comment by msol...@gmail.com on 24 Mar 2010 at 8:12