Open DavidHuber-NOAA opened 1 year ago
The coding work for this issue is complete. I am waiting for the rstprod group to be added to Niagara before updating the branch, completing testing, and submitting a PR.
@arunchawla-NOAA Can we submit a request to get rstprod on Niagara? It would help with the lack of direct HPSS access from Orion (and Hercules). Thanks!
This will remain blocked until rstprod is added to Niagara.
Curious about the status of this as we evaluate whether we can use Orion/Hercules for future GFSv17 prototypes/retros. Was there ever a request made to get rstprod on Niagara?
Tagging @JacobCarley-NOAA on this as acting EIB chief.
@KateFriedman-NOAA Do you know if Arun ever made the request for rstprod on Niagara?
Do you know if Arun ever made the request for rstprod on Niagara?
To my knowledge he never submitted the request. @JacobCarley-NOAA is this something you could submit?
@KateFriedman-NOAA @DavidHuber-NOAA @JessicaMeixner-NOAA @JacobCarley-NOAA
Good news everyone: Rstprod has now been added to Niagara.
Hooray! Thanks for the update @CatherineThomas-NOAA !
I am working with Georgy Fekete to establish a workflow for these transfers (RDHPCS ticket number 2024061454000228).
Description
As part of the archive (earc, arch) jobs, an option to push data to Niagara from Globus Endpoint machines would allow users to stage data for archive to HPSS. This feature would require that
LOCAL_ARCH="YES"
be set. This option would be useful for Orion, S4, and eventually Hercules.Acceptance Criteria (Definition of Done)
Data is pushed to HPSS from Orion and verified against the original .tar files created on Orion.
(Optional): Suggest A Solution
The globus transfers could be made part of the *arc j-jobs and added after each successful creation of a tar file.
At the end of each *arc script, an HPSS script would be generated and globus transferred to Niagara containing a list of all .tar files that were sent. A crontab entry could then be placed on Niagara to look for and execute these scripts and report any errors via email.
@erinjones2 developed scripts that worked on Orion for the feature/ops-orion branch. I plan on adapting these to the purpose.