search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
74 stars 164 forks source link

Enable transfers of archived data to Niagara from non-HPSS-connected systems #1357

Open DavidHuber-NOAA opened 1 year ago

DavidHuber-NOAA commented 1 year ago

Description

As part of the archive (earc, arch) jobs, an option to push data to Niagara from Globus Endpoint machines would allow users to stage data for archive to HPSS. This feature would require that LOCAL_ARCH="YES" be set. This option would be useful for Orion, S4, and eventually Hercules.

Acceptance Criteria (Definition of Done)

Data is pushed to HPSS from Orion and verified against the original .tar files created on Orion.

(Optional): Suggest A Solution

The globus transfers could be made part of the *arc j-jobs and added after each successful creation of a tar file.

At the end of each *arc script, an HPSS script would be generated and globus transferred to Niagara containing a list of all .tar files that were sent. A crontab entry could then be placed on Niagara to look for and execute these scripts and report any errors via email.

@erinjones2 developed scripts that worked on Orion for the feature/ops-orion branch. I plan on adapting these to the purpose.

DavidHuber-NOAA commented 1 year ago

The coding work for this issue is complete. I am waiting for the rstprod group to be added to Niagara before updating the branch, completing testing, and submitting a PR.

KateFriedman-NOAA commented 1 year ago

@arunchawla-NOAA Can we submit a request to get rstprod on Niagara? It would help with the lack of direct HPSS access from Orion (and Hercules). Thanks!

DavidHuber-NOAA commented 10 months ago

This will remain blocked until rstprod is added to Niagara.

CatherineThomas-NOAA commented 5 months ago

Curious about the status of this as we evaluate whether we can use Orion/Hercules for future GFSv17 prototypes/retros. Was there ever a request made to get rstprod on Niagara?

CatherineThomas-NOAA commented 5 months ago

Tagging @JacobCarley-NOAA on this as acting EIB chief.

@KateFriedman-NOAA Do you know if Arun ever made the request for rstprod on Niagara?

KateFriedman-NOAA commented 5 months ago

Do you know if Arun ever made the request for rstprod on Niagara?

To my knowledge he never submitted the request. @JacobCarley-NOAA is this something you could submit?

CatherineThomas-NOAA commented 4 months ago

@KateFriedman-NOAA @DavidHuber-NOAA @JessicaMeixner-NOAA @JacobCarley-NOAA

Good news everyone: Rstprod has now been added to Niagara.

KateFriedman-NOAA commented 4 months ago

Hooray! Thanks for the update @CatherineThomas-NOAA !

DavidHuber-NOAA commented 1 week ago

I am working with Georgy Fekete to establish a workflow for these transfers (RDHPCS ticket number 2024061454000228).