Add article on large file transfers and `sciencedmz` partition

[wwarriner]

What would you like to see added?

Issue for info on large file transfers and using sciencedmz partition.

[wwarriner]

Draft case study:

How do I transfer large quantities of data from a non-Globus source to Cheaha using a specialized S3 interface? (Specifically, Seven Bridges/Velsera command line interface, https://www.sevenbridges.com/, https://sb-biodatacatalyst.readme.io/docs/upload-and-download-files#download)

Typically, for large data transfers, we highly recommend using Globus. In this particular case, Globus was not an option. The only option was to use the specialized sb download command to transfer data from an external S3 store to our LTS S3.

Initially we had the researcher run their jobs unrestricted. We received notification from UAB Central IT informing us of network congestion. We contacted the researcher involved and discussed their workflow with an eye to limiting network bandwidth usage.

Their starting workflow was to transfer 100 files in each of 109 job scripts. This meant that about 100 files were being downloaded simultaneously. Cheaha routes external S3 transfers through a 10 Gbps connection with the UAB Campus Core Network and the transfers were saturating that connection.

After some discussion of options, we settled on using an "array" job. Using an array job would allow us to more carefully tune the number of files downloaded at the same time. With minimal rework to the script, we set up two files per task, with about 5450 tasks. The array job was initially limited to four simultaneous tasks, meaning four files. This helped reduce the transfer rate to about 5 Gbps. This was a temporary workaround to allow the researcher to continue without impacting campus operations.

John-Paul and Mike followed up by creating a special sciencedmz partition using two of the nodes from our primary partition. The nodes in that partition had their internet connection routed through the Science DMZ, which has a 40 Gbps connection to the internet.

The researcher's job was stopped, moved to the sciencedmz partition and restarted where it had left off. Experimenting with different numbers of simultaneous tasks revealed the remote server had a bandwidth limit of 10 Gbps.

All factors considered we were able to more than double the throughput from about 70 files per hour to 180 files per hour, getting the remaining half of the files in about 30 hours, saving well over a day's wait. We were able to transfer to a connection where we could obtain a full 10 Gbps. We learned how to set up a restricted-access partition so we can easily set up high-throughput data transfers in the future.

[wwarriner]

To get list of nodes use sinfo show partition and find sciencedmz.

We can quasi-automate this using https://github.com/wwarriner/slurm_status_tools

[mdefende]

Can also add globus-cli as an extra option. The CLI can be run on any node, but the transfer should still be localized to the DTNs since everything would be run through Globus. This would solve cases where a user wanted to perform compute and transfer within the same job since:

1. They wouldn't need to be added to the sciencedmz partition group
2. We don't want people running compute on sciencedmz, just transfers. Without globus-cli, they would need to run the analysis in an initial job on some other partition and then separately submit a transfer job on sciencedmz manually after the initial job completed or know how to submit a job that depends on a previous job completing in Slurm.

[mdefende]

globus-cli docs: https://docs.globus.org/cli/