I remembered I was meant to provide some information/suggestions about Swift temporary URLs. A benefit of this is that users can directly download (or upload) files from ECHO (1) without having the credentials for accessing ECHO, and (2) without the data having to flow through another box.
You first of all need to create a key which can then be used for creating the temporary URLs. Once you have created this key you don’t need the original credentials anymore. The temporary URLs can be generated by a few lines of Python (an example of which is in the link above).
For downloading output files, your REST API could generate a temporary URL and redirect the user to this.
To create the key, a simplest way is to install the Swift client somewhere (pip install python-swiftclient) and run:
swift -A https://s3.echo.stfc.ac.uk/auth/1.0 -U <username> -K <password> post -m "Temp-URL-Key:<key>"
Here you need to specify your Swift credentials and the key you want to use. I used uuidgen to generate the key.
I remembered I was meant to provide some information/suggestions about Swift temporary URLs. A benefit of this is that users can directly download (or upload) files from ECHO (1) without having the credentials for accessing ECHO, and (2) without the data having to flow through another box.
Swift temporary URLs are described here:
http://docs.ceph.com/docs/jewel/radosgw/swift/tempurl/
An example of what they look like for ECHO (note that I've modified it to be invalid however):
https://s3.echo.stfc.ac.uk/swift/v1/prominence-jobs/alahiff/jintrac-20190130.tar?temp_url_sig=27d3ba7c37877b64d050ac70a1096c42181012a9&temp_url_expires=1553272013
You first of all need to create a key which can then be used for creating the temporary URLs. Once you have created this key you don’t need the original credentials anymore. The temporary URLs can be generated by a few lines of Python (an example of which is in the link above).
For downloading output files, your REST API could generate a temporary URL and redirect the user to this.
To create the key, a simplest way is to install the Swift client somewhere (
pip install python-swiftclient) and run:Here you need to specify your Swift credentials and the key you want to use. I used uuidgen to generate the key.