The diagram below shows the top level transfer architecture of Blaze to move data from a S3 bucket to any cloud resource endpoint
Blaze is a product with the integration of several software tools to build end-to-end data transfer workflows which includes data preparation, transfers and post-processing. To accomplish that, Blaze uses following software
Deployment of the platform includes installation of the Data Transfer Layer mentioned in the above diagram and registering source and destination storages in the Data Transfer Layer
To have a minimal transfer layer, there should be 2 EC2 VMs (one for MFT Master and one for MFT Agent) created inside us-west-2 region. These VMs should have following features
cd scripts
python3 -m venv ENV
source ENV/bin/activate
pip install -r requirements.txt
ansible-playbook -i inventories/example install-master.yml
Install the transfer agent into the agent node
ansible-playbook -i inventories/example install-agents.yml
To make sure that everything installed properly, go to http://
If you are configuring a NASA S3 endpoint, you need to install and setup the transfer orchestration workflow. First, you need to have an account created at https://urs.earthdata.nasa.gov and installation script will ask for those credentials. This is required to generate access credentials for NASA S3 endpoint
ansible-playbook -i inventories/example configure-endpoints.yml
While executing above ansible script, You will be asked to configure the destination storage endpoint in the MFT-Master node and get the storage_id. Here is an example of configuring a storage endpoint
Example storage creation command for AWS S3 cloud storage in us-west-2 region
java -jar /home/ubuntu/mft_deployment/mft-client.jar s3 remote add --name=<Name for the Storage> --bucket=<Bucket Name> --key=<key> --secret=<secret> --endpoint=https://s3.us-west-2.amazonaws.com --region=us-west-2
Example storage creation command for IBM cloud storage in us-east region
java -jar /home/ubuntu/mft_deployment/mft-client.jar s3 remote add --name=<Name for the Storage> --bucket=<Bucket Name> --key=<key> --secret=<secret> --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --region=us-east-smart
Then go back to Airflow dashboard, and you should be able to see a workflow named "transfer-workflow". Enable it and click the trigger workflow button in the top right corner. It will start transferring sample dataset mentioned in the scripts/roles/catalog/files/sample-data.csv
Once you are able to see the data being transferred to destination storage, you can pause the transfer any time by clicking the toggle button present in the top left corner in the airflow transfer-workflow DAG page.