[X] This issue does not match the other templates (i.e., it is not a bug report, documentation request, feature request, or a question.)
Issue description
Using zstash with Globus is now much more cumbersome than it used to be; to the point that it may become questionable whether integrating Globus actually adds value to zstash.
Here is the workflow I have to use:
(1) Login to Globus web interface and activate end points
(2) Delete existing globus cfg file
rm ~/.globus-native-apps.cfg
(3) Start interactive zstash test transfer, something like
This one should complete without issue or any prompt.
(5) Start long transfer (in a screen session to keep it running)
zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/${EXP} . 2>&1 | tee zstash/zstash_create_${stamp}.log
Note that this transfer is limited to 48 hours due to Globus token expiration. Given the relatively slow transfer speeds between chrysalis and NERSC HPSS (~100 MB/s), 48 hours is often insufficient for a large simulation.
(6) Using Globus web interface, manually transfer zstash files that were not transferred due to token expiration.
(7) Repeat steps (2) to (4) above. Restart zstash archiving that stopped:
zstash update -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/${EXP} 2>&1 | tee zstash/zstash_create_${stamp}.log
I should note that in the not so distant past, only steps (1) and (5) were needed. It is also possible that I'm doing something wrong, in which case I'd like to know what.
golaz
commented
3 months ago
Request criteria
Issue description
Using zstash with Globus is now much more cumbersome than it used to be; to the point that it may become questionable whether integrating Globus actually adds value to zstash.
Here is the workflow I have to use:
(1) Login to Globus web interface and activate end points
(2) Delete existing globus cfg file
rm ~/.globus-native-apps.cfg(3) Start interactive zstash test transfer, something like
zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/tmp tmpDuring the the interactive test run, zstash will ask to copy and paste authorization code twice. zstash runs but files are not transferred.
(4) Start a second interactive zstash transfer
zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/tmp tmpThis one should complete without issue or any prompt.
(5) Start long transfer (in a
screensession to keep it running)zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/${EXP} . 2>&1 | tee zstash/zstash_create_${stamp}.logNote that this transfer is limited to 48 hours due to Globus token expiration. Given the relatively slow transfer speeds between chrysalis and NERSC HPSS (~100 MB/s), 48 hours is often insufficient for a large simulation.
(6) Using Globus web interface, manually transfer zstash files that were not transferred due to token expiration.
(7) Repeat steps (2) to (4) above. Restart zstash archiving that stopped:
zstash update -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/${EXP} 2>&1 | tee zstash/zstash_create_${stamp}.logI should note that in the not so distant past, only steps (1) and (5) were needed. It is also possible that I'm doing something wrong, in which case I'd like to know what.