Script to push files to apollo production server

[mpoelchau]

The following steps need to be put into a workflow/script to move files to the apollo-node1 server. (We can't pull from the apollo-stage server, so we can't re-use add_trackList.py completely).

• back up trackList.json file on apollo-node1 server
• create subdirectory for files on apollo-node1 server (/app/data/[containing directory]/[gggsss]/[assemblyname]/scaffold/analyses/[species]-[genus]-RNA-Seq_[date]_v1.0)
• move .bam, .bai, .bigwig, .bed files from apollo-stage-node1 server to apollo-node1 server

[HsiuKangHuang]

Before running add_tracklist.py on apollo-stage server, check the following things are in bashrc or bash_profile file.

1. /app/data/WebApollo_deployment/apollo_master/jbrowse/bin/ and /app/data/shared_tools/perl5/lib/perl5 are in your PATH.

2. PATH="/app/data/shared_tools/perl5/bin${PATH:+:${PATH}}"; export PATH;
   PERL5LIB="/app/data/shared_tools/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
   PERL_LOCAL_LIB_ROOT="/app/data/shared_tools/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
   PERL_MB_OPT="--install_base \"/app/data/shared_tools/perl5\""; export PERL_MB_OPT;
   PERL_MM_OPT="INSTALL_BASE=/app/data/shared_tools/perl5"; export PERL_MM_OPT;

These five lines are below export PATH

[HsiuKangHuang]

Using Globus is another way to transfer files from Ceres. (https://3.basecamp.com/3625179/buckets/5538276/documents/1060605516)

Globus provides a faster way for transferring files between two servers or from local to server. By installing globus-cli python package, it can be used via command line interface.(https://docs.globus.org/cli/) This page provides some examples of using CLI for Globus.(https://docs.globus.org/cli/examples/)

There are some questions before using Globus.

1. To transfer files from Ceres to apollo server, we need the endpoint name or UUID of both two servers. (The endpoint name and uuid of Ceres is Ceres DTN 1/d46403e2-38d7-11e8-b985-0ac6873fc732) We need the name and uuid of apollo servers.

2. In our current pipelines, we use scp and rsync to transfer files. It takes about 5 minutes for transferring process. How much processing time can be shorten by using globus?