Setting Up Airavata for File Transfer - Java

[ushaheu]

I would like to know if there is a feature to define multiple transfer paths in database. In my use case I have over 6000 different paths I want to pick files from and move to different locations. Is there a way to dynamically define my paths

[DImuthuUpe]

@ushaheu Do you want to move data within your local workstation?

[ushaheu]

@DImuthuUpe I want to move data from my server where I generate reports to an s3 bucket

[smarru]

@ushaheu I see you will need to push data to S3, what will be the source protocol? A server with SCP/SSH access or other cloud storages?

[ushaheu]

Thanks for your response @smarru my source protocol will be SCP. Would you be kind to also share a sample java snippet or javadoc to guide me use the mft with Java?

[smarru]

@ushaheu in general, MFT seems to be a good fit for what you want to achieve. It can translate the protocol from SCP to S3, allowing you to plug in different credentials. I will let @DImuthuUpe respond on the java client. To explore options, will a Python client also work, or do you prefer the client to be in Java?

[ushaheu]

Thanks alot for your response @smarru. My application is a java application and I have mastery of Java not Python hence my preference for Java. I will be glad to get the response from @DImuthuUpe on how to integrate airavata mft with Java/Spring boot. Thanks for the great work

[DImuthuUpe]

@ushaheu seems like you are planning to integrate MFT into your application. The obvious way is to have MFT running as a separate process and connect to it through its gRPC API. It does not need spring to be running in your application. Here are the steps you need to follow

1. Start MFT by following the steps https://github.com/apache/airavata-mft#download-and-install
2. Build MFT Project using maven on your laptop (You need to do this as you plan to programmatically access MFT)
3. Import this MFT client module into your maven project https://github.com/apache/airavata-mft/blob/master/api/client/pom.xml
4. Use the MFTClient https://github.com/apache/airavata-mft/blob/master/api/client/src/main/java/org/apache/airavata/mft/api/client/MFTApiClient.java to talk to MFT

Update us once you managed to successfully complete the above steps. I will send a client usage sample code quickly

[ushaheu]

Thank you very much for the detailed information @DImuthuUpe. I will go through these steps and come back to you for the client code. Regards

[ushaheu]

Thanks once again @DImuthuUpe and @smarru I have been able to run the steps 1 to 4. I attached screenshots. I will be waiting for your guidance on the client

[DImuthuUpe]

@ushaheu This is great. I encountered a couple of bugs in SCP transport and updated the binary version with the fix. Can you please update mft-cli and reinstall mft? Here are the commands

pip3.10 install --upgrade airavata-mft-cli mft update

Above commands will remove your existing mft setup and reinstall the latest version. To verify if you have it running properly, please send the output of

mft log

I am still working on the java snippet and I'll send you it as soon as it is available

[ushaheu]

Thanks alot @DImuthuUpe for your support and kind words. I have run the commands above. Kindly find attached the logs.

[DImuthuUpe]

@ushaheu Great. Looks like MFT running fine now. You might have to get a pull of the MFT repository and rebuild through maven before moving into next step. After you have MFT built,

1. Run mft storage add to add both SCP and S3 endpoints. This is a one-time step and you can reuse the storage you have added in multiple transfers.
2. Run mft storage list to list the storages you have already registered. Notice that there is a storage id for each storage.

Here is the java code to submit transfers for the storage you have added. You need to replace storage ids with the ids you got

` package org.apache.airavata.mft.api.client;

import org.apache.airavata.mft.api.service.*;
import org.apache.airavata.mft.resource.stubs.storage.common.SecretForStorage;
import org.apache.airavata.mft.resource.stubs.storage.common.SecretForStorageGetRequest;

public class Test {
    public static void main(String args[]) throws InterruptedException {

        String s3StorageId = "07e4865f-7b35-4c8d-a977-81c5ba0f6ff4";
        String scpStorageId = "5344d7f6-1ab6-458b-8e15-47108d7b5ead";
        String scpSourcePath = "/home/exouser/a.txt";
        String s3DestinationPath = "a.txt";

        MFTApiClient client = MFTApiClient.MFTApiClientBuilder.newBuilder()
                .withResourceServicePort(7003)
                .withSecretServicePort(7003)
                .withTransferServicePort(7003).build();

        SecretForStorage scpSecret = client.getStorageServiceClient()
                .common()
                .getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(scpStorageId).build());

        SecretForStorage s3Secret = client.getStorageServiceClient()
                .common()
                .getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(s3StorageId).build());

        EndpointPaths endpoints = EndpointPaths.newBuilder()
                .setSourcePath(scpSourcePath)
                .setDestinationPath(s3DestinationPath).build();

        TransferApiResponse transferResponse = client.getTransferClient().submitTransfer(TransferApiRequest.newBuilder()
                .setSourceSecretId(scpSecret.getSecretId())
                .setSourceStorageId(scpStorageId).setDestinationSecretId(s3Secret.getSecretId())
                .setDestinationStorageId(s3StorageId).addEndpointPaths(endpoints).build()); // You can add many endpointpaths into one request 

        String transferId = transferResponse.getTransferId();
        System.out.println("Transfer Id: " + transferId);

        // Monitoring transfer progress. You can write your own logic to do this
        for (int i = 0; i < 10; i ++) {
            TransferStateSummaryResponse transferStateSummary = client.getTransferClient()
                    .getTransferStateSummary(
                            TransferStateApiRequest.newBuilder().setTransferId(transferId).build());
            System.out.println("Transfer state: " + transferStateSummary.getState());
            Thread.sleep(1000);
        }
    }
}

`

[ushaheu]

Thanks alot for this support @DImuthuUpe I will implement this and provide update as soon as am done. I have two quick question, is there a possibility that mft storage add and mft storage list will be exposed as apis? Then can I transfer to s3 bucket from my local drive?

Also, I am trying to add a storage for SCP, its prompting me for a path to my key. How can I by pass it if I dont have a key. I noticed that when I try to add the s3 storage, it does not prompt me for access key and secret key then it throws an error

[DImuthuUpe]

Yes they are. Please refer to sample APIs like below.

s3Storage = client.getStorageServiceClient().s3().createS3Storage()
s3Secret = client.getSecretServiceClient().s3().createS3Secret()

[ushaheu]

I appreciate you @DImuthuUpe. I will try it out and provide an update as soon as I get it right. Thanks alot

[ushaheu]

Hi @DImuthuUpe thanks once again for your support. I have written a small application for a local transfer on my laptop from one directory to another. I first of all created the storage entry for local as commented in my code. Then I used the sample you shared. However, the file was not transferred as expected.

public class APIManager {
    public static void main(String[] args) throws InterruptedException {
        MFTApiClient mftApiClient = MFTApiClient.MFTApiClientBuilder.newBuilder()
                .withResourceServicePort(7003)
                .withSecretServicePort(7003)
                .withTransferServicePort(7003).build();
//mftApiClient.getStorageServiceClient().local().createLocalStorage(LocalStorageCreateRequest.newBuilder().setStorageId(UUID.randomUUID().toString()).setName("Mac Local").build());
        String s3StorageId = "b6336695-7127-4c84-9ea4-c6a157032f32";
        String scpStorageId = "50292023-f4ff-496f-8382-e5c20632244e";
        String localSourcePath = "/Users/ushahembaukange/Downloads/visa-clearing-2021-09-03-1731c3c2-febe-43ae-b832-f91d043fc173.ecf";
        String localDestinationPath = "/Users/ushahembaukange/Documents/filetransfer/visa-clearing-2021-09-03-1731c3c2-febe-43ae-b832-f91d043fc173.ecf";
        SecretForStorage scpSecret = mftApiClient.getStorageServiceClient()
                .common()
                .getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(scpStorageId).build());

        SecretForStorage s3Secret = mftApiClient.getStorageServiceClient()
                .common()
                .getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(s3StorageId).build());

        EndpointPaths endpoints = EndpointPaths.newBuilder()
                .setSourcePath(localSourcePath)
                .setDestinationPath(localDestinationPath).build();
        TransferApiResponse transferResponse = mftApiClient.getTransferClient().submitTransfer(TransferApiRequest.newBuilder()
                .setSourceSecretId(scpSecret.getSecretId())
                .setSourceStorageId(scpStorageId).setDestinationSecretId(s3Secret.getSecretId())
                .setDestinationStorageId(s3StorageId).addEndpointPaths(endpoints).build()); // You can add many endpointpaths into one request
        String transferId = transferResponse.getTransferId();
        System.out.println("Transfer Id: " + transferId);

        // Monitoring transfer progress. You can write your own logic to do this
        for (int i = 0; i < 10; i ++) {
            TransferStateSummaryResponse transferStateSummary = mftApiClient.getTransferClient()
                    .getTransferStateSummary(
                            TransferStateApiRequest.newBuilder().setTransferId(transferId).build());
            System.out.println("Transfer state: " + transferStateSummary.getPercentage());
            Thread.sleep(1000);
        }
    }
}

Kindly find the logs from mft log

[DImuthuUpe]

Can you do mft stop mft init

and retry?

[ushaheu]

Thank you very much @DImuthuUpe . I was able to transfer now. Is there any reason we had to do that? Is it a command I will need to run often.

[DImuthuUpe]

No it is not required. We need to investigate why the agent was down. Can you send the full log file available in ~/.mft/Standalone-Service-0.01/logs/airavata.log ?