s-u / REngine

General Java interface to R supporting multiple back-ends such as JRI and Rserve
Other
67 stars 47 forks source link

Multiple connections using multithreading #17

Closed hlms closed 7 years ago

hlms commented 7 years ago

Need to download few files from RServe.

I have written the following code which creates several threads, and downloads the files in multithreading. So far so good. It seems to be working fine on both - unix and windows.

But is it okay to use multithreading with RConnection and openFile() method?

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import org.rosuda.REngine.REXPMismatchException;
import org.rosuda.REngine.Rserve.RConnection;
import org.rosuda.REngine.Rserve.RFileInputStream;
import org.rosuda.REngine.Rserve.RserveException;

public class TestConcurrentDownloads2Windows {

    public static void main(String[] args) throws RserveException {
        try {
            Thread t1 = new Thread(new Downloader2("E:/test data/113MB.xlsx"));
            t1.start();

            Thread t2 = new Thread(new Downloader2("E:/test data/75MB.xlsx"));
            t2.start();

            Thread t3 = new Thread(new Downloader2("E:/test data/37MB.xlsx"));
            t3.start();

            Thread t4 = new Thread(new Downloader2("E:/test data/52MB.xlsx"));
            t4.start();

            t1.join();
            t2.join();
            t3.join();
            t4.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

class Downloader2 implements Runnable {

    private String filepath;
    private RFileInputStream rfis;
    private FileOutputStream fos = null;

    public Downloader2(String filepath) {
        this.filepath = filepath;
    }

    @Override
    public void run() {
        RConnection rconnection = null;
        try {
            // windows
            String username = "username";
            String password = "password";
            String url = "url";
            int port = 6311;

            System.out.println("Starting ... " + filepath);
            rconnection = new RConnection(url, port);
            rconnection.login(username, password);

            String osType = rconnection.eval(".Platform$OS.type").asString();
            System.out.println("osType: " + osType);

            rfis = rconnection.openFile(filepath);
            Path p = Paths.get(filepath);
            String fileNameToCreate = p.getFileName().toString();
            File f = new File(fileNameToCreate);
            fos = new FileOutputStream(f);
            byte[] arr = new byte[8192];
            int len = 0;
            while ((len = rfis.read(arr)) > 0) {
                fos.write(arr, 0, len);
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (RserveException e) {
            e.printStackTrace();
        } catch (REXPMismatchException e) {
            e.printStackTrace();
        } finally {
            try {
                if (rfis != null) {
                    rfis.close();
                }
            } catch (IOException e) {
            }
            try {
                if (fos != null) {
                    fos.close();
                }
            } catch (IOException e) {
            }
            if (rconnection != null) {
                rconnection.close();
            }
        }
        System.out.println("Downloaded: " + filepath);
    }
}
s-u commented 7 years ago

Yes, it's fine on unix since unix supports parallel connections. Windows doesn't so the threads will be running in parallel but they will transfer sequentially - i.e. only the first thread will be connected and the other threads will get a connection one at a time only after the first thread is done. So you can use the same code, but on Windows it won't actually be able to do anything in parallel due to limitations in the Windows OS.

hlms commented 7 years ago

Awesome! Thanks for the quick response! :)