S3 MultipleFileDownload fails joining parts and gives filelock exception

Hi all,

I've uploaded a 3.4GB folder to an AWS Bucket and am trying to write a java application to download the content using transfer acceletration on that bucket and MultipleFileDownload ;

I've declared a

private MultipleFileDownload download;

and

private TransferManager tx;

then I start the download process with the following line:

download = tx.downloadDirectory(bucketName, DCPname, path);

the download seems working fine, the folder gets created, the single chunks get downloaded, and the joining process goes on, but after the total folder size has been downloaded, joining the last chunks of the larger file in the folder fails and I have this error:

com.amazonaws.services.s3.transfer.exception.FileLockException: java.io.FileNotFoundException: C:\Users\Alessandro\Desktop\SILENCE_TLR-1_F_IT-XX_51_2K_O1_20161206_AM\SILENCE_TLR-1_F_IT-XX_51_2K_O1_20161206_AM_01.mxf (Impossibile accedere al file. Il file è utilizzato da un altro processo) at com.amazonaws.services.s3.internal.FileLocks.lock(FileLocks.java:76) at com.amazonaws.services.s3.internal.ServiceUtils.appendFile(ServiceUtils.java:451) at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:46) at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:29) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: C:\Users\Alessandro\Desktop\SILENCE_TLR-1_F_IT-XX_51_2K_O1_20161206_AM\SILENCE_TLR-1_F_IT-XX_51_2K_O1_20161206_AM_01.mxf (Impossibile accedere al file. Il file è utilizzato da un altro processo) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) at com.amazonaws.services.s3.internal.FileLocks.lock(FileLocks.java:70)

If I try 10 times, 1 out of 10 it succeeds, but most times it fails.

If the folder contains just 1 file it works perfectly with the same code(without changing a line). It only fails with multilple files = downloading directories with several 3GB files in. Can anyone help to unterstand why it downloads every chunk but stops while joning and Leaves some chunks out? I'm using sdk 1.11.73 .jar in Windows 10

Thanks for reporting this. Looks like it's because a file cannot be opened by multiple processes on a Windows system. We will get a fix out.

My pleasure! If a set of file to test would be useful here's a link to the folder I have used to test (I uploaded it zipped here https://www.sendspace.com/pro/dl/4hk80y), you could upload it in a bucket and test yourself. There are files of several sizes inside (up to 3.3GB). I've also seen a minor issue is the ProgressListener sometimes fails to trigger the ProgressChange event when the listener is added to a multiplefile download as folderdownload.

`com.amazonaws.event.ProgressListenerChain progressChanged AVVERTENZA: Couldn't update progress listener com.amazonaws.AbortedException: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:797) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:706) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518) at com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient.doInvoke(AmazonIdentityManagementClient.java:6882) at com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient.invoke(AmazonIdentityManagementClient.java:6858) at com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient.detachUserPolicy(AmazonIdentityManagementClient.java:2749) at S3Sample$ButtonListener1$1.progressChanged(S3Sample.java:439) at com.amazonaws.event.ProgressListenerChain.progressChanged(ProgressListenerChain.java:98) at com.amazonaws.services.s3.transfer.MultipleFileTransferProgressUpdatingListener.progressChanged(MultipleFileTransferProgressUpdatingListener.java:41) at com.amazonaws.event.ProgressListenerChain.progressChanged(ProgressListenerChain.java:98) at com.amazonaws.event.SDKProgressPublisher.quietlyCallListener(SDKProgressPublisher.java:86) at com.amazonaws.event.SDKProgressPublisher.deliverEvent(SDKProgressPublisher.java:68) at com.amazonaws.event.SDKProgressPublisher.publishProgress(SDKProgressPublisher.java:58) at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.fireProgressEvent(AbstractTransfer.java:253) at com.amazonaws.services.s3.transfer.internal.DownloadImpl.setState(DownloadImpl.java:168) at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:50) at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:29) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.amazonaws.http.timers.client.SdkInterruptedException at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:841) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:827) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1006) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704) ... 21 more

Just a quick update:

the 1.11.74 has been released and I can see the following charges:

1) No more filelock exceptions in the IDE (Netbeans) while running the program but the issue still stands 2) The difference with the 1.11.73 is that joining doesn't start anymore before all of the parts have been downloaded, 3.3GB file size stays on 0 while the parts get downloaded; see picture below:

state1

once all of the parts have been downloaded and folder size is the right one, joining auto starts but it gets stuck at a random point: see picture below:

state2

The execution is stuck at this point.

If I close the swing frame window and manually stop execution some part files get dropped =discarded=deleted but the main file size remains the same.

picture below shows the final state:

state 3

In the very end rebuilding of a large file fails under win 10. I've not added any listener to the process to have as less objects as possible involved.

I have used the 1.11.75 and assume the issue hasn't been fixed in such a release, behaviour remains unchanged- From 1.11.74 on the filelock exception doesn't get thrown anymore on netbeans, but the problem the last file doesn't get joined still stands. Also if I either put a listener or insert a download.isDone() to test when it's completed, it never reaches that point as joining process is stuck; and even using a listener the listener can't see the end of process

@studioam Are you using the default threadpool in TransferManger?

You said you are not seeing the filelock exception anymore. Now what error are you seeing? Or is the application just struck without any progress?

@varunnvs92

Before replying to your question I would tell I've installed the 1.11.76 but nothing has changed even using it. I forgot to add in my previous replies I have tested the built jar executable file on more than a machine to make sure the issue wasn't connected to a single computer with its Windows 10 x64 installation.

I'm not sure what do you mean by "default threadpool" , I can't see any option in TransferManager for Threadpool choice.

Reply to your 2nd question: I can't see any exception after version 1.11.74 but "the Read Timeout" so I've circumvented it with the following lines (read timeout increased to 60 secs):

`private final int connectionTimeout = 50 1000; private final int readTimeout = 60 1000;

ClientConfiguration config = new ClientConfiguration(); config.setConnectionTimeout(connectionTimeout); config.setSocketTimeout(readTimeout); config.setRetryPolicy(PredefinedRetryPolicies.getDefaultRetryPolicyWithCustomMaxRetries(5));`

but the application gets struck without any progress as explained above. Anyhow just for being more precise, I declared it all as follows:

` AmazonS3 s3 = new AmazonS3Client(credentials, config); -----setting region here--- s3.setS3ClientOptions(S3ClientOptions.builder().setAccelerateModeEnabled(true).build());

TransferManager tx = new TransferManager(s3);

MultipleFileDownload download = tx.downloadDirectory(bucketName, DCPname, path);

----where DCPname is keyprefix and path the path--- ----since I had troubles with listener I removed the listener and added the following line, I don't think it should change the final result---

while (download.isDone()==false) jProgressBar1.setValue((int)download.getProgress().getPercentTransferred());

tx.shutdownNow();

Please lmk if you see anything wrong with it and if I have to specify a different threadpool in any way, thanks-

Tested with 1.11.77, nothing changes. No exception and process struck. Please don't forget the download of each part works fine. The Whole folder gets downloaded and the size seems matching, it's just the joining process to get struck at some point

The fix is not released yet. I am still investigating on why this is happening. I was not able to reproduce this on my side which is delaying progress.

I will let you know when the bug is identified and fixed.

As per you previous message, your configuration seems good. FYI, TransferManager by default uses a threadpool with 10 threads. You can also provide your own pool using this constructor. Using the default pool should be good enough.

This is the code where parts are combined. I am not sure why the application is being struck. Can you please give us a thread dump so that we can see where the thread(s) are being struck?

You said you tried the application on different machines. Did you try on both windows and Linux machines?

here you have the dump I've taken when the program is being struck.

Thread_Dump.txt

I'm trying to test it on Linux too, but my aim is to have it working on Windows 10 as the 99.99% of the machines where it will run are Windows machines. Thanks in advance

From the thread dump, all transfer manager threads are waiting as all the scheduled tasks are completed. This means that the CompleteMultipartDownload task (which combines part files) should have been completed. But you said that the joining of parts didn't even start which is weird.

As you can see here, the CompleteMultipartDownload task is submitted along with DownloadPartCallable tasks. So if all parts are downloaded, the joining should have started.

Is it possible for you to keep a breakpoint in CompleteMultipartDownload#call method and debug the application. If its not possible for you to debug, I can add log statements in the CompleteMultipartDownload class and give you a custom SDK jar. This might help us know if the application is reaching this class or not.

Hi, thanks for your reply, the real behaviour is the joining of parts starts but never completes; but please note the process starts. Please look at the above pictures I posted some posts ago. if you could add the log statement it would be great , I would have it in the custom jar file. Please lmk, thanks

This is the custom S3client jar with extra log statements in DownloadCallable and CompleteMultipartDownload classes. Use this and provide us the new logs. Lets see if we can find something new.

FYI, I have tested the downloadDirectory operation with the same configuration and files you gave in this thread. I tested on both windows (win 10, JDK 8) and Mac (OS X Elcapitan, JDK8) and I was not able to reproduce the issue. This is the generated log output when I ran the test in Mac.

Few more questions: 1) Which JDK version are you using? 2) Any luck with testing the application in Linux. 3) You said the process hangs in the middle of joining parts. But the thread dump you gave doesn't indicate that. All the transfer manager threads in the thread dump were just waiting for tasks and none seems to be locked/blocked by another resource. You can try some java profiler and capture thread dumps at multiple points during download and share it.

Hi, first let me thank you for the time you dedicated to this. I've made some progresses.

Here there are the answers to your questions:

1)I was using the 8.111 and now moved to 8.112 2)not yet 3)After having moved to 8.112 I had the tool working sometimes. I made 6 tests and had it working 4 times out of 6. Then I started thinking the issue is not totally connected to the library but maybe the SDK was involved.

Ive taken some dumps again, both while downloading and joining. I wasn't able to use your jar, or better I've added to the jar files but I can see any log anywhere, where it's supposed to save them? Thanks again Dumps.zip

Update: after haveing moved to JDK 8.112 the tool works fine on a Windows 7 X64 machine, it never stops or slowers, I've made 10 tests on that machine and 10 out of 10 are succesful. So it's useless to test on a Linux one IMHO. On the Windows 10 machine where I had it working in the 1st tests it is completely struck now with no progress (not slowered but struck now), tested also on another Windows 10 pro x64 machine and same issue.

Its interesting that the application works in windows 7 but getting randomly struck in windows10. I have to do some research about it.

In the meantime, can you try the application with SDK version 1.11.63 on Windows 10 machine. The logic for copying data between files has changed a little in 1.11.64. I want to make sure that the code change is not the reason for this issue.

I will get back to you if I have something new.

Hi, I've got some good news: I've found the source of the problem. I've tried SDK version 1.11.63 and the issue still stands. Then I've tried on a 3rd machine with Windows 10 and had 10 out of 10 success. So I went back to my installation and performed further tests.

The problem is I'm saving on desktop and the system disk is the following:

https://www.kingston.com/it/ssd/system-builder/sm2280s3g2

I was saving on desktop on C:...... where Windows 10 OS is installed. I've tried then to save the file on storage disk , D: and everything goes fine . Here I had 10 out of 10 tests succeful. It's unbelievable that very fast disk drive (https://www.kingston.com/it/ssd/system-builder/sm2280s3g2) has this issue, it's very fast on writing but very slow to join..... and the result is completely random. Also tested on an external drive (regular USB ard disk drive) and 100% success. So it seems the result is related to the physiscal Disk hardware and how it's managed by Windows 10.

now I've got an additiona issue not present in previous versions and it's not related to joining and transfermanager but to Progress Listener:

MultipleFileDownload download = tx.downloadDirectory(bucketName, DCPname, path); download.addProgressListener(progressListener);

and this is the previously declared listener:

`ProgressListener progressListener = new ProgressListener() { @Override public void progressChanged(ProgressEvent progressEvent) {

        if (download == null) return;

         jProgressBar1.setValue((int)download.getProgress().getPercentTransferred());    

        if (download.isDone()){

             tx.shutdownNow();
            AmazonIdentityManagementClient cliente = new AmazonIdentityManagementClient();

            DetachUserPolicyRequest cancellazione = new DetachUserPolicyRequest();

           cliente.detachUserPolicy(cancellazione.withUserName(account).withPolicyArn("arn:aws:iam::579503845502:policy/policygen-201612221527"));

// more actions performed here.....

The aim is the policy gets removed once download is complete, so when download.isDone() the transfermanager is being shut down and the policy detached (then I send an email confirmation with javaxmail and more actions) The problem is the " more actions performed" is never reached since I have an exception when this line is hit:

'cliente.detachUserPolicy(cancellazione.withUserName(account).withPolicyArn("arn:aws:iam::579503845502:policy/policygen-201612221527"));

and this is the exception:

gen 16, 2017 12:44:26 PM com.amazonaws.event.ProgressListenerChain progressChanged AVVERTENZA: Couldn't update progress listener com.amazonaws.AbortedException: at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:797) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:706) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518) at com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient.doInvoke(AmazonIdentityManagementClient.java:7367) at com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient.invoke(AmazonIdentityManagementClient.java:7343) at com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient.detachUserPolicy(AmazonIdentityManagementClient.java:2946) at NewJFrame$7.progressChanged(NewJFrame.java:455) at com.amazonaws.services.s3.model.LegacyS3ProgressListener.progressChanged(LegacyS3ProgressListener.java:47) at com.amazonaws.event.ProgressListenerChain.progressChanged(ProgressListenerChain.java:98) at com.amazonaws.services.s3.transfer.MultipleFileTransferProgressUpdatingListener.progressChanged(MultipleFileTransferProgressUpdatingListener.java:41) at com.amazonaws.event.ProgressListenerChain.progressChanged(ProgressListenerChain.java:98) at com.amazonaws.event.SDKProgressPublisher.quietlyCallListener(SDKProgressPublisher.java:86) at com.amazonaws.event.SDKProgressPublisher.deliverEvent(SDKProgressPublisher.java:68) at com.amazonaws.event.SDKProgressPublisher.publishProgress(SDKProgressPublisher.java:58) at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.fireProgressEvent(AbstractTransfer.java:253) at com.amazonaws.services.s3.transfer.internal.DownloadImpl.setState(DownloadImpl.java:168) at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:50) at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:29) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.amazonaws.http.timers.client.SdkInterruptedException at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:841) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:827) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1006) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704) ... 22 more what does it mean? Thanks in advance.

Good to hear that you found a temporary fix for the locking issue. Its interesting that the device hardware is affecting the joining of parts. These kind of cases are rare and we can't find a permanent fix as its hard to reproduce them. Thanks for experimenting in different ways and finding the root cause.

For the new issue, I guess the code where you run the AmazonIdentityManagementClient operations are in a background thread. Am I right? If so, I suspect that the main process (JVM) is exiting which is interrupting and closing all other threads. This might be causing the AbortedException. https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/http/AmazonHttpClient.java#L705-L709

Is it possible for you to share the application code which is run the main thread? Even partial codes at the end of application should suffice. Thanks.

My pleasure to contribute! I've tested with an external drive too and it works fine; I agree it's a rare case and it would have been hard to think to such a source of the problem at a first glance.

About the new issue I have I thin you're right, this is the application code, please lmk if I failed with using the listener or if there's a smarter way to update the progress bar and detect the end of main process to start performing final actions, thanks in advance:

`public class NewJFrame extends javax.swing.JFrame {

private File TxtFile;
private static NewJFrame frame;
private  AWSCredentials credentials = null;
private String credenziali;
private File path = null;
private final int connectionTimeout = 50 * 1000;
private final int readTimeout = 55 * 1000;
private String account ="";
private AmazonS3 s3;
private TransferManager tx;
private int completedfiles =0; 
private int nfiles = 1;
private String DCPname;
private MultipleFileDownload download;
private int percentCompl=0;

    //invio email
private void sendFromGMail(String from, String pass, String[] to, String subject, String body) throws MessagingException {
    Properties props = System.getProperties();
    String host = "smtp.gmail.com";
    props.put("mail.smtp.starttls.enable", "true");
    props.put("mail.smtp.host", host);
    props.put("mail.smtp.user", from);
    props.put("mail.smtp.password", pass);
    props.put("mail.smtp.port", "587");
    props.put("mail.smtp.auth", "true");

    Session session = Session.getDefaultInstance(props);
    MimeMessage message = new MimeMessage(session);

    try {
        message.setFrom(new InternetAddress(from));
        InternetAddress[] toAddress = new InternetAddress[to.length];

        // To get the array of addresses
        for( int i = 0; i < to.length; i++ ) {
            toAddress[i] = new InternetAddress(to[i]);
        }

        for (InternetAddress toAddres : toAddress) {
            message.addRecipient(Message.RecipientType.TO, toAddres);
        }

        message.setSubject(subject);
        message.setText(body);
        Transport transport = session.getTransport("smtp");
        transport.connect(host, from, pass);
        transport.sendMessage(message, message.getAllRecipients());
        transport.close();
    }
    catch (AddressException ae) {
        ae.printStackTrace();
    }
    catch (MessagingException me) {
        me.printStackTrace();
    }
}

/**
 * Creates new form NewJFrame
 */
public NewJFrame() {
    initComponents();
}

/**
 * This method is called from within the constructor to initialize the form.
 * WARNING: Do NOT modify this code. The content of this method is always
 * regenerated by the Form Editor.
 */
@SuppressWarnings("unchecked")
// <editor-fold defaultstate="collapsed" desc="Generated Code">                          
private void initComponents() {

    jButton1 = new javax.swing.JButton();
    jLabel1 = new javax.swing.JLabel();
    jButton2 = new javax.swing.JButton();
    jLabel2 = new javax.swing.JLabel();
    jButton3 = new javax.swing.JButton();
    jLabel3 = new javax.swing.JLabel();
    jButton4 = new javax.swing.JButton();
    jProgressBar1 = new javax.swing.JProgressBar();

    setDefaultCloseOperation(javax.swing.WindowConstants.EXIT_ON_CLOSE);

    jButton1.setText("Chiave 1");
    jButton1.addMouseListener(new java.awt.event.MouseAdapter() {
        public void mouseClicked(java.awt.event.MouseEvent evt) {
            jButton1MouseClicked(evt);
        }
    });
    jButton1.addActionListener(new java.awt.event.ActionListener() {
        public void actionPerformed(java.awt.event.ActionEvent evt) {
            jButton1ActionPerformed(evt);
        }
    });

    jButton2.setText("Chiave 2");
    jButton2.addMouseListener(new java.awt.event.MouseAdapter() {
        public void mouseClicked(java.awt.event.MouseEvent evt) {
            jButton2MouseClicked(evt);
        }
    });
    jButton2.addActionListener(new java.awt.event.ActionListener() {
        public void actionPerformed(java.awt.event.ActionEvent evt) {
            jButton2ActionPerformed(evt);
        }
    });

    jButton3.setText("Download To");
    jButton3.addMouseListener(new java.awt.event.MouseAdapter() {
        public void mouseClicked(java.awt.event.MouseEvent evt) {
            jButton3MouseClicked(evt);
        }
    });

    jButton4.setText("Start Download");
    jButton4.addMouseListener(new java.awt.event.MouseAdapter() {
        public void mouseClicked(java.awt.event.MouseEvent evt) {
            jButton4MouseClicked(evt);
        }
    });

    javax.swing.GroupLayout layout = new javax.swing.GroupLayout(getContentPane());
    getContentPane().setLayout(layout);
    layout.setHorizontalGroup(
        layout.createParallelGroup(javax.swing.GroupLayout.Alignment.LEADING)
        .addGroup(layout.createSequentialGroup()
            .addContainerGap()
            .addGroup(layout.createParallelGroup(javax.swing.GroupLayout.Alignment.LEADING)
                .addGroup(layout.createSequentialGroup()
                    .addComponent(jButton3, javax.swing.GroupLayout.PREFERRED_SIZE, 106, javax.swing.GroupLayout.PREFERRED_SIZE)
                    .addPreferredGap(javax.swing.LayoutStyle.ComponentPlacement.RELATED)
                    .addComponent(jLabel3, javax.swing.GroupLayout.PREFERRED_SIZE, 569, javax.swing.GroupLayout.PREFERRED_SIZE))
                .addGroup(layout.createSequentialGroup()
                    .addComponent(jButton1)
                    .addPreferredGap(javax.swing.LayoutStyle.ComponentPlacement.RELATED)
                    .addComponent(jLabel1, javax.swing.GroupLayout.PREFERRED_SIZE, 273, javax.swing.GroupLayout.PREFERRED_SIZE)
                    .addGap(18, 18, 18)
                    .addComponent(jButton2)
                    .addPreferredGap(javax.swing.LayoutStyle.ComponentPlacement.UNRELATED)
                    .addComponent(jLabel2, javax.swing.GroupLayout.PREFERRED_SIZE, 353, javax.swing.GroupLayout.PREFERRED_SIZE))
                .addGroup(layout.createSequentialGroup()
                    .addComponent(jButton4)
                    .addPreferredGap(javax.swing.LayoutStyle.ComponentPlacement.RELATED)
                    .addComponent(jProgressBar1, javax.swing.GroupLayout.PREFERRED_SIZE, 236, javax.swing.GroupLayout.PREFERRED_SIZE)))
            .addContainerGap(232, Short.MAX_VALUE))
    );
    layout.setVerticalGroup(
        layout.createParallelGroup(javax.swing.GroupLayout.Alignment.LEADING)
        .addGroup(layout.createSequentialGroup()
            .addContainerGap()
            .addGroup(layout.createParallelGroup(javax.swing.GroupLayout.Alignment.LEADING, false)
                .addComponent(jButton2, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE)
                .addComponent(jButton1, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE)
                .addComponent(jLabel1, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE)
                .addComponent(jLabel2, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE))
            .addGap(18, 18, 18)
            .addGroup(layout.createParallelGroup(javax.swing.GroupLayout.Alignment.LEADING, false)
                .addComponent(jButton3, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE)
                .addComponent(jLabel3, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE))
            .addGap(18, 18, 18)
            .addGroup(layout.createParallelGroup(javax.swing.GroupLayout.Alignment.LEADING, false)
                .addComponent(jButton4, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE)
                .addComponent(jProgressBar1, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE))
            .addContainerGap(javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE))
    );

    pack();
}// </editor-fold>                        

private void jButton1MouseClicked(java.awt.event.MouseEvent evt) {                                      
    // TODO add your handling code here:
    //Selezione Chiave 1
    JFileChooser fileChooser = new JFileChooser();
        //fileChooser1.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
        int showOpenDialog = fileChooser.showOpenDialog(frame);
        if (showOpenDialog != JFileChooser.APPROVE_OPTION) return;
        if (fileChooser.getSelectedFile().getName().equals("")) jLabel1.setText("Nessun file selezionato! Selezionare Percorso");
        else {
            jLabel1.setText(fileChooser.getSelectedFile().getName());         
            TxtFile = new File(fileChooser.getSelectedFile().getPath());
        }
}                                     

private void jButton1ActionPerformed(java.awt.event.ActionEvent evt) {                                         
    // TODO add your handling code here:
}                                        

private void jButton2ActionPerformed(java.awt.event.ActionEvent evt) {                                         
    // TODO add your handling code here:
}                                        

private void jButton2MouseClicked(java.awt.event.MouseEvent evt) {                                      
    // TODO add your handling code here:
    //Selezione Chiave 2
    JFileChooser fileChooser = new JFileChooser();
        //fileChooser1.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
        int showOpenDialog = fileChooser.showOpenDialog(frame);
        if (showOpenDialog != JFileChooser.APPROVE_OPTION) return;
        if (fileChooser.getSelectedFile().getName().equals("")) jLabel2.setText("Nessun file selezionato! Selezionare Percorso");
        else {
            jLabel2.setText(fileChooser.getSelectedFile().getName());         
            credenziali = fileChooser.getSelectedFile().getPath();
        }
}                                     

private void jButton3MouseClicked(java.awt.event.MouseEvent evt) {                                      
    // TODO add your handling code here:
    //Selezione Cartella Destinazione copia DCP
    JFileChooser fileChooser = new JFileChooser();
        fileChooser.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
        int showOpenDialog = fileChooser.showOpenDialog(frame);
        if (showOpenDialog != JFileChooser.APPROVE_OPTION) return;
        if (fileChooser.getSelectedFile().getName().equals("")) jLabel3.setText("Nessun percorso selezionato! Selezionare Percorso");
        else {
            jLabel3.setText(fileChooser.getSelectedFile().getName());
            path = fileChooser.getSelectedFile();
        }
}                                     

private void jButton4MouseClicked(java.awt.event.MouseEvent evt) {                                      
    // TODO add your handling code here:
    //Azione Main
    //Controllo Percorso Salvataggio
    if (jLabel3.getText().equals("")){
        JOptionPane.showMessageDialog(frame, "Nessun Percorso di Salvataggio selezionato! Selezionare Percorso" );                  
    }

    String bucketName = "";

    //Controllo file Chiave 1
    if(TxtFile==null)try {
            throw new Exception();
        } catch (Exception e) {
            // TODO Auto-generated catch block              
            JOptionPane.showMessageDialog(frame,
                    "File Chiave 1 Non Presente! Selezionare File Chiave 1" + e.getMessage(),
                    "Selezionare File Chiave 1", JOptionPane.ERROR_MESSAGE);
            e.printStackTrace();
        }
    //Controllo Credenziali
    if(credenziali==null)try {
            throw new Exception();
        } catch (Exception e) {
            // TODO Auto-generated catch block              
            JOptionPane.showMessageDialog(frame,
                    "File Chiave 2 Non Presente! Selezionare File Chiave 2" + e.getMessage(),
                    "Selezionare File Chiave 2", JOptionPane.ERROR_MESSAGE);
            e.printStackTrace();
        }

    BufferedReader br=null;

        try {
            br = new BufferedReader(new FileReader(TxtFile));
        } catch (FileNotFoundException e2) {
            // TODO Auto-generated catch block
            JOptionPane.showMessageDialog(frame,
                    "File Chiave 1 Non Presente! Selezionare File Chiave 1" + e2.getMessage(),
                    "Selezionare File Chiave 1", JOptionPane.ERROR_MESSAGE);
            e2.printStackTrace();
        }
                String line="";

        try {
            line = br.readLine();
        } catch (IOException e2) {
            // TODO Auto-generated catch block
            JOptionPane.showMessageDialog(frame,
                    "Nessuna identità Selezionata" + e2.getMessage(),
                    "Inserire Identità", JOptionPane.ERROR_MESSAGE);
            e2.printStackTrace();
        }

    ///Login
account=line;

try {
       credentials = new ProfileCredentialsProvider(credenziali,"test-user").getCredentials();
    } catch (Exception e) {
        // TODO Auto-generated catch block
    JOptionPane.showMessageDialog(frame,
                        "File Chiave 2 Errato! Caricare un File Chiave 2 Valido" + e.getMessage(),
                        "Caricare un File Chiave 2 Valido", JOptionPane.ERROR_MESSAGE);
                e.printStackTrace();            
        }

//Configurazione Client
ClientConfiguration config = new ClientConfiguration(); 
config.setConnectionTimeout(connectionTimeout);
config.setSocketTimeout(readTimeout); 
    //config.setRetryPolicy(PredefinedRetryPolicies.getDefaultRetryPolicyWithCustomMaxRetries(5));

//creazione Connessione s3
    s3 = new AmazonS3Client(credentials,config);
    //s3 = new AmazonS3Client(credentials);
Region usWest2 = Region.getRegion(Regions.EU_WEST_1);
s3.setRegion(usWest2);
s3.setS3ClientOptions(S3ClientOptions.builder().setAccelerateModeEnabled(true).build());

//Creazione e Configurazione Transfer Manager 
    //TransferManagerConfiguration configuraz= new TransferManagerConfiguration();
    //configuraz.setMultipartCopyThreshold(16*1024*1024);
    //configuraz.setMultipartCopyPartSize(100*1024*1024);        
tx = new TransferManager(s3); 
    //tx.setConfiguration(configuraz);

    //lettura bucket

        try {
                    line = br.readLine();
        } catch (IOException e2) {
            // TODO Auto-generated catch block
            JOptionPane.showMessageDialog(frame,
                    "File Chiave 1 Errato! Caricare un File Chiave 1 Valido" + e2.getMessage(),
                    "Caricare un File Chiave 1 Valido", JOptionPane.ERROR_MESSAGE);
            e2.printStackTrace();
        }

    try{
            for (com.amazonaws.services.s3.model.Bucket bucket : s3.listBuckets())  if (bucket.getName().equals(line)) bucketName = bucket.getName() ;
        }catch (Exception e){
            JOptionPane.showMessageDialog(frame,
                    "Chiavi Scadute! Per effettuare un nuovo Download richiedere nuove chiavi    " + e.getMessage(),
                    "Per effettuare un nuovo Download richiedere nuove chiavi", JOptionPane.ERROR_MESSAGE);
            e.printStackTrace();
        }

    if (bucketName.equals(""))
        try {

                            throw new Exception();
             } catch (Exception e) {
                // TODO Auto-generated catch block

                JOptionPane.showMessageDialog(frame,
                        "File Chiave 1 Errato! Caricare un File Chiave 1 Valido" + e.getMessage(),
                        "Caricare un File Chiave 1 Valido", JOptionPane.ERROR_MESSAGE);
                e.printStackTrace();
            }        

    //Numero degli oggetti nel bucket
    nfiles = s3.listObjects(new ListObjectsRequest().withBucketName(bucketName)).getObjectSummaries().size();

    //lettura nome DCP
    try {
    DCPname = br.readLine();
    } catch (IOException e1) {
        // TODO Auto-generated catch block
        JOptionPane.showMessageDialog(frame,
            "File Chiave 1 Errato! Caricare un File Chiave 1 Valido" + e1.getMessage(),
                        "Caricare un File Chiave 1 Valido", JOptionPane.ERROR_MESSAGE);

            e1.printStackTrace();
        }
    if (DCPname.equals("")) try {
              throw new Exception();
            } catch (Exception e) {
            // TODO Auto-generated catch block

            JOptionPane.showMessageDialog(frame,
            "File Chiave 1 Errato! Caricare un File Chiave 1 Valido" + e.getMessage(),
                        "Caricare un File Chiave 1 Valido", JOptionPane.ERROR_MESSAGE);
                e.printStackTrace();
            } 

    //Aggiunta progress Listener

    ProgressListener progressListener = new ProgressListener() {
        @Override
        public void progressChanged(ProgressEvent progressEvent) {

        if (download == null) return;

                      jProgressBar1.setValue((int)download.getProgress().getPercentTransferred());    

        if (progressEvent.getEventType()== ProgressEventType.TRANSFER_COMPLETED_EVENT) {

               if (download.isDone()){

            tx.shutdownNow();
            AmazonIdentityManagementClient cliente = new AmazonIdentityManagementClient();
                           DetachUserPolicyRequest cancellazione = new DetachUserPolicyRequest();

            cliente.detachUserPolicy(cancellazione.withUserName(account).withPolicyArn("arn:aws:iam::579503845502:policy/policygen-201612221527"));

            }

           // else percentCompl = (int)(completedfiles*(100/nfiles));   

        }
        if (progressEvent.getEventType() == ProgressEventType.TRANSFER_FAILED_EVENT){
            try {
                AmazonClientException e = download.waitForException();
                JOptionPane.showMessageDialog(frame,
                        "Unable to Download file from Amazon S3: " + e.getMessage(),
                        "Error Downloading File", JOptionPane.ERROR_MESSAGE);
            } catch (InterruptedException e) {}
        }
        }
    }; 

    download = tx.downloadDirectory(bucketName, DCPname, path);
    download.addProgressListener(progressListener);

}                                     

/**
 * @param args the command line arguments
 */
public static void main(String args[]) {
    /* Set the Nimbus look and feel */
    //<editor-fold defaultstate="collapsed" desc=" Look and feel setting code (optional) ">
    /* If Nimbus (introduced in Java SE 6) is not available, stay with the default look and feel.
     * For details see http://download.oracle.com/javase/tutorial/uiswing/lookandfeel/plaf.html 
     */
    try {
        for (javax.swing.UIManager.LookAndFeelInfo info : javax.swing.UIManager.getInstalledLookAndFeels()) {
            if ("Nimbus".equals(info.getName())) {
                javax.swing.UIManager.setLookAndFeel(info.getClassName());
                break;
            }
        }
    } catch (ClassNotFoundException ex) {
        java.util.logging.Logger.getLogger(NewJFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
    } catch (InstantiationException ex) {
        java.util.logging.Logger.getLogger(NewJFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
    } catch (IllegalAccessException ex) {
        java.util.logging.Logger.getLogger(NewJFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
    } catch (javax.swing.UnsupportedLookAndFeelException ex) {
        java.util.logging.Logger.getLogger(NewJFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
    }
    //</editor-fold>

    /* Create and display the form */
    java.awt.EventQueue.invokeLater(new Runnable() {
        public void run() {
            frame = new NewJFrame();
            frame.setVisible(true);
        }
    });
}

// Variables declaration - do not modify                     
private javax.swing.JButton jButton1;
private javax.swing.JButton jButton2;
private javax.swing.JButton jButton3;
private javax.swing.JButton jButton4;
private javax.swing.JLabel jLabel1;
private javax.swing.JLabel jLabel2;
private javax.swing.JLabel jLabel3;
private javax.swing.JProgressBar jProgressBar1;
// End of variables declaration

Sorry for the delay in responding.

I think JVM is exiting after calling tx.shutdownNow() which means there are no other non-daemon threads running. I am not that familiar with swing framework. As you know, JVM will exit if all non-daemons threads are finished. If all JFrame and Jbutton listeners are daemon threads, then transfer manager shutdown is closing all non-daemons threads started by Java SDK which is causing JVM to exit.

I think you should wait for download to complete using waitForCompletion method, then shutdown the transfer manager and do the IAM actions.

So after download.addProgressListener(progressListener) line, add:

download.waitForCompletion();
 tx.shutdownNow();
 AmazonIdentityManagementClient cliente = new AmazonIdentityManagementClient();
....

Remember waitForCompletion is blocking call. As this download is happening in a listener, it should block your application.

Btw for these Java/Threading generic questions, you will get better help from stack overflow community.

Hi, thanks for your reply. Unfortunately stack overflow community requires 1500 reputation to start a new thread. I'd close this thread as it's solved about the issue explained in the title and open a new one about the progressbar, listener and blocking calls. Thanks

Did you try the fix from my last correspondence?

Unfortunately stack overflow community requires 1500 reputation to start a new thread.

I don't think we need any reputation to ask a new question on stack overflow.

yes teh fix works

download.waitForCompletion();

but the progress bar won't update anymore if I use it.

aws / aws-sdk-java

S3 MultipleFileDownload fails joining parts and gives filelock exception #963