How to upload a simple File from InputStream ?

[emmanuel-D]

I am pretty new with this lib and I am trying to upload simple images or videos to my S3. I am using REST-API and I am already able to get the InputStream from my HttpServletRequest. So how, do I upload my InputStream to the S3 using the StreamTransferManager ?

Here is a code snippet:

1) Get the InputStream from the request InputStream inputStream = item.openStream();

2) Upload it to S3 final StreamTransferManager manager = new StreamTransferManager(bucketName, key, s3client);

3) What comes next ??

[alexmojaki]

Your title mentions a File - is there a file saved to disk, or is it all still in memory?

[emmanuel-D]

I updated my Post

[emmanuel-D]

No there is no file saved to disk. I want to upload the retrieved InputStream in the HttpServletRequest directly to S3

[alexmojaki]

I believe your best option is https://stackoverflow.com/a/39440936/2482744

Otherwise Google "java copy inputstream to outputstream"

[emmanuel-D]

The problem is that I don't know how to upload those Stream to the S3. I have followed your example on the JavaDoc, but I don't see the place where you use the InputStream.

// Setting up
        int numStreams = 2;
        final StreamTransferManager manager = new StreamTransferManager(bucketName, key, s3client)
                .numStreams(numStreams)
                .numUploadThreads(2)
                .queueCapacity(2)
                .partSize(10);
        final List<MultiPartOutputStream> streams = manager.getMultiPartOutputStreams();

        ExecutorService pool = Executors.newFixedThreadPool(numStreams);
        for (int i = 0; i < numStreams; i++) {
            final int streamIndex = i;
            pool.submit(() -> {
                try {
                    MultiPartOutputStream outputStream = streams.get(streamIndex);
                    for (int lineNum = 0; lineNum < 1000000; lineNum++) {
                        String line = generateData(streamIndex, lineNum);

                        // Writing data and potentially sending off a part
                        outputStream.write(line.getBytes());
                    }

                    // The stream must be closed once all the data has been written
                    outputStream.close();
                } catch (Exception e) {

                    // Aborts all uploads
                    manager.abort(e);
                }
            });
        }
        pool.shutdown();
        pool.awaitTermination(5, TimeUnit.SECONDS);

        // Finishing off
        manager.complete();

[emmanuel-D]

And what does the method generateData() do exactly do?

[alexmojaki]

What all users need to do is write to the output streams. In this example the data to write is generated by a made up algorithm. In your case it comes from an input stream, but that doesn't apply to everyone.

I suggest you use .numStreams(1) and then item.openStream().transferTo(streams.get(0)).

[emmanuel-D]

Sorry I am not understanding that syntax item.openStream().transferTo(streams.get(0)) 😅 And the method .transfertTo does not exist

[emmanuel-D]

Can you simply help me by telling how exactly I am suppose to upload the InputStream to the S3 by updating my example on the top please ?

[alexmojaki]

What version of Java are you using?

[emmanuel-D]

I am using Java 1.8

[alexmojaki]

That's quite an old version, is there a reason you're not using a newer version?

[emmanuel-D]

Because the project is old and certains libraries are also Java 1.8 dependant

[alexmojaki]

Then I suggest looking at the other answers in https://stackoverflow.com/questions/43157/easy-way-to-write-contents-of-a-java-inputstream-to-an-outputstream

item.openStream() is your InputStream, manager.getMultiPartOutputStreams().get(0) is the OutputStream you need to write to.

[emmanuel-D]

All right. But do I really need the following code. I mean, I am not understanding what is going on here:

ExecutorService pool = Executors.newFixedThreadPool(numStreams);
        for (int i = 0; i < numStreams; i++) {
            final int streamIndex = i;
            pool.submit(() -> {
                try {
                    MultiPartOutputStream outputStream = streams.get(streamIndex);
                    for (int lineNum = 0; lineNum < 1000000; lineNum++) {
                        String line = generateData(streamIndex, lineNum);

                        // Writing data and potentially sending off a part
                        outputStream.write(line.getBytes());
                    }

                    // The stream must be closed once all the data has been written
                    outputStream.close();
                } catch (Exception e) {

                    // Aborts all uploads
                    manager.abort(e);
                }
            });
        }
        pool.shutdown();
        pool.awaitTermination(5, TimeUnit.SECONDS);

[alexmojaki]

You don't need any of that because you will be using one stream and one thread. Just write to the stream, close the stream, and complete the manager.

[emmanuel-D]

I am getting the following error exception after refactoring the code:

java.lang.IllegalArgumentException: The given part size (1048576) is less than 5 MB.

How can I upload file that are less than this size ?

[alexmojaki]

That's not about your file size, that's for specifying .partSize(1). Don't do that.

[emmanuel-D]

The execution is being blocked and the lines after manager.complete(); are never reached. What I am doing wrong here ?

// Setting up
            int numStreams = 1;
            final StreamTransferManager manager = new StreamTransferManager(bucketName, fileName, s3client)
                    .numStreams(numStreams)
                    .numUploadThreads(1)
                    .queueCapacity(2)
                    ;

            IOUtils.copy(inputStream, manager.getMultiPartOutputStreams().get(0));
            final List<MultiPartOutputStream> streams = manager.getMultiPartOutputStreams();

            // Finishing off
            manager.complete();

            // The bellow code is never call ????
            log.info(String.format("Erfolgreich Uploaded Media to with name: %s", fileName));
            mediaUriService.save(mediaUri);

[alexmojaki]

Close the stream.