search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
75 stars 168 forks source link

Staging the gfs-obs for Marine DA #2706

Closed apchoiCMD closed 2 months ago

apchoiCMD commented 4 months ago

Target fix directory

NEW

Type of change

Any workflow changes needed?

No change needed

Related issues

https://github.com/NOAA-EMC/GDASApp/issues/1120

Pickup location

/work/noaa/da/marineda/gfs-marine/data/obs OR /scratch1/NCEPDEV/global/glopara/data/experimental_obs

Size

0

WalterKolczynski-NOAA commented 4 months ago 
ls: cannot access '/work/noaa/da/marineda/gfs-marine/data/obs': Permission denied
apchoiCMD commented 4 months ago

@WalterKolczynski-NOAA Sorry for your inconvenience- I've changed the permissions so you will be able to access the folders/files

WalterKolczynski-NOAA commented 4 months ago

Looks like these need to go in the obs dump rather than a fix directory. Will coordinate with @KateFriedman-NOAA on how to do this.

guillaumevernieres commented 4 months ago

Looks like these need to go in the obs dump rather than a fix directory. Will coordinate with @KateFriedman-NOAA on how to do this.

@WalterKolczynski-NOAA , these are still experimental obs. They currently are here for the gdas: /scratch1/NCEPDEV/global/data/experimental_obs/ We just need to add the gfs obs at the same level.

apchoiCMD commented 4 months ago

Updated obs location with Hera as well, please use/copy the gfs obs:

/scratch1/NCEPDEV/global/glopara/data/experimental_obs
apchoiCMD commented 4 months ago

@KateFriedman-NOAA I just wanted to touch base to see if there are any updates on this issue. Always thanks!

KateFriedman-NOAA commented 4 months ago

@apchoiCMD Apologies, haven't been able to get to this yet. I've just taken a look at the obs you staged on Orion. I have some questions...would you have time for a short Meet tomorrow between 11am-noon or 1-2pm? Thanks!

apchoiCMD commented 4 months ago

@KateFriedman-NOAA 11am-12pm should work for me, thanks for your offer-

apchoiCMD commented 4 months ago

Either you can use them at /scratch1/NCEPDEV/da/common

KateFriedman-NOAA commented 4 months ago

Just spoke offline with @apchoiCMD and discussed the contents of the experimental obs. Given the holiday, I have planned to pull the obs into the experimental_obs folder on Hera next Monday and when @WalterKolczynski-NOAA returns from leave he will sync them to Orion. There is no need currently for the obs on any other supported platforms.

Will update this issue once obs are in place on Hera.

KateFriedman-NOAA commented 4 months ago

@apchoiCMD I have begun copying the experimental marine obs into the glopara space on Hera: /scratch1/NCEPDEV/global/glopara/data/experimental_obs

I have several streams of rsyncs happening in parallel. It will likely take most of today, possibly longer to copy them all in. Once in, I'll ask that you take a look to make sure they look good on Hera. I'll let you know when the rsyncs complete.

When do you need them on Orion/Hercules?

apchoiCMD commented 4 months ago

@apchoiCMD I have begun copying the experimental marine obs into the glopara space on Hera: /scratch1/NCEPDEV/global/glopara/data/experimental_obs

I have several streams of rsyncs happening in parallel. It will likely take most of today, possibly longer to copy them all in. Once in, I'll ask that you take a look to make sure they look good on Hera. I'll let you know when the rsyncs complete.

When do you need them on Orion/Hercules?

@KateFriedman-NOAA No worries, we will test the cycle once obs are ready on Orion- Appreciate your update!

JessicaMeixner-NOAA commented 3 months ago

There is no need currently for the obs on any other supported platforms.

@KateFriedman-NOAA (FYI: @apchoiCMD @guillaumevernieres): @CatherineThomas-NOAA and I were just talking. We need these obs on WCOSS2 as well. Is that possible?

KateFriedman-NOAA commented 3 months ago

@apchoiCMD The marine obs are now in place on Hera, see /scratch1/NCEPDEV/global/glopara/data/experimental_obs and let me know if I'm missing any dates. @WalterKolczynski-NOAA will sync them to Orion/Hercules when he's back next week.

@JessicaMeixner-NOAA Yeah, I can sync them to WCOSS2 as well. I have made a note to start that sync tomorrow.

JessicaMeixner-NOAA commented 3 months ago

Thank you @KateFriedman-NOAA !

apchoiCMD commented 3 months ago

More thanks from me @KateFriedman-NOAA

KateFriedman-NOAA commented 3 months ago

@JessicaMeixner-NOAA @apchoiCMD The experimental marine obs are now also on both WCOSS2s. I placed them in a separate folder within the global dump archive (GDA) space since the GDA has it's own separate quota on WCOSS2 and plenty of space right now. I made a symlink from the data folder to the GDA folder so it's similar to the placement on the other platforms. Example:

WCOSS2 (PRIMARYSYS-D) data> pwd
/lfs/h2/emc/global/noscrub/emc.global/data
WCOSS2 (PRIMARYSYS-D) data> ll | grep experimental_obs
lrwxrwxrwx  1 emc.global global      46 Jul 15 13:08 experimental_obs -> /lfs/h2/emc/dump/noscrub/dump/experimental_obs

Once @WalterKolczynski-NOAA is able to copy these experimental obs to Orion we can close this issue.

WalterKolczynski-NOAA commented 2 months ago

Sync'd to MSU.