content() methods suitable for large content

[GoogleCodeExporter]

Some REST APIs, such as those for object storage clouds from Amazon, OpenStack, 
and Google, take very large objects as content, but do not accept multi-part 
form data. The natural way in rest-assured to test these is using the content() 
methods, something like this:

given().
    content(Files.readAllBytes(new File(filename).toPath())).
when().
    put(filename).
then().
    statusCode(is(201));

There are currently two forms of the content() method, one that takes an array 
of Bytes and one that takes an Object. This means, of course, that to form a 
PUT request the test case must read the entire contents of the file into memory 
as bytes and pass the whole thing in. This is fine for small files, but these 
object storage clouds often accept content of up to 5 GB in a single request. 
So it's not practical to read one or more of these files into memory on most 
test client machines.

A better approach for files of this size is for RA to accept an InputStream and 
only read enough from the stream to efficiently write the data to the wire, so 
the only memory overhead is some reasonable cache. This is a request to add an 
enhancement to rest-assured in the form of new content() methods.

I think content(InputStream is) and content(File f) would cover every case. In 
fact, the File version is technically redundant because it's easy for the test 
designer to turn a File into an InputStream. But the File version is more 
convenient since most test cases that try to PUT 5 billion bytes are going to 
store those in a file anyhow.

Original issue reported on code.google.com by todd...@gmail.com on 20 Mar 2015 at 7:13

[GoogleCodeExporter]

Original comment by johan.ha...@gmail.com on 21 Mar 2015 at 5:39

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Original comment by johan.ha...@gmail.com on 21 Mar 2015 at 5:40

• Changed state: Started

[GoogleCodeExporter]

I've added support for this now I think. Please try it out by depending on 
2.4.1-SNAPSHOT after having added the following repo:

<repositories>
        <repository>
            <id>sonatype</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
            <snapshots />
        </repository>
</repositories>

Original comment by johan.ha...@gmail.com on 21 Mar 2015 at 5:48

[GoogleCodeExporter]

I haven't tested it with large files/inputstreams so I'm still not sure if it 
works

Original comment by johan.ha...@gmail.com on 21 Mar 2015 at 5:49

[GoogleCodeExporter]

I've tried the code from the 2.4.1-SNAPSHOT but it didn't work using the code 
listed below:

     given().spec(adminBasicAuth).content(new FileInputStream(new File(testFile))).when().put("/" + test_container + "/" +testFile);

and

     given().spec(adminBasicAuth).content(new File(testFile)).when().put("/" + test_container + "/" +testFile);

I tried several test files with sizes of 157 bytes and 4GB.  Would you please 
confirm that the form of these commands is correct?

Original comment by siwarsha...@gmail.com on 23 Mar 2015 at 8:40

[GoogleCodeExporter]

Thanks for trying it out. It actually works for small files in my tests. Please 
have a look at 
https://github.com/jayway/rest-assured/blob/master/examples/rest-assured-itest-j
ava/src/test/java/com/jayway/restassured/itest/java/NonMultiPartUploadITest.java
. The actual services that handles the "file" request is a simple Scalatra 
resource that is defined like this (same for post and put):

post("/file") {
    val content: String = IOUtils.toString(request.getInputStream)
    content
}

Original comment by johan.ha...@gmail.com on 24 Mar 2015 at 10:44

[GoogleCodeExporter]

Also, what error do you get?

Original comment by johan.ha...@gmail.com on 24 Mar 2015 at 10:44

[GoogleCodeExporter]

That works for small files, now I need to get it working with the large files.  
Running into heap space error when running Eclipse project when uploading 1G 
file.  Once I get that resolved I'll try 4G and 6G files.

Original comment by siwarsha...@gmail.com on 24 Mar 2015 at 2:26

[GoogleCodeExporter]

Hmm I wonder why that is. It shouldn't need to copy everything into memory.

Original comment by johan.ha...@gmail.com on 24 Mar 2015 at 4:09

[GoogleCodeExporter]

I found the source of the heap space error, it was in my code.  After fixing it 
I ran tests with 1Gb and 4Gb files -- no problems, it worked as advertised.  
Thanks for your assistance, Johan.

Original comment by siwarsha...@gmail.com on 24 Mar 2015 at 7:50

[GoogleCodeExporter]

Great! Thanks for helping out and testing it.

Original comment by johan.ha...@gmail.com on 24 Mar 2015 at 7:52

• Changed state: Fixed

[GoogleCodeExporter]

We've started using com.jayway.restassured:rest-assured:2.4.1 and this feature 
no longer appears to work.

Original comment by siwarsha...@gmail.com on 26 May 2015 at 8:19

[GoogleCodeExporter]

Hmm but it worked in the snapshot version?

Original comment by johan.ha...@gmail.com on 27 May 2015 at 6:16

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Yes, the call hangs and we don't see a file transfer.  Here's a code sample:

        given().spec(adminAStorageAccessBasicAuth).
            contentType(ContentType.BINARY).
            content(new FileInputStream(file)).
        when().
            put(containerName + "/" + testFile4G).
        then().
            statusCode(201);

Original comment by siwarsha...@gmail.com on 27 May 2015 at 3:25

[GoogleCodeExporter]

Seems to work with small files, need to understand difference between a 25K and 
4G file upload.

Original comment by siwarsha...@gmail.com on 27 May 2015 at 3:50

[GoogleCodeExporter]

I ran a series of tests with files ranging from 25K to 819M.  The transfer rate 
in our system was around 1MB/sec during the tests so a 4G file would take over 
an hour to complete.  No idea where there's a hangup, looking into it.

Original comment by siwarsha...@gmail.com on 27 May 2015 at 4:44

[GoogleCodeExporter]

Thanks. I don't think I've changed anything in the released version form the 
snapshot.

Original comment by johan.ha...@gmail.com on 27 May 2015 at 6:15