NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)

https://global-workflow.readthedocs.io/en/latest

GNU Lesser General Public License v3.0

70 stars 162 forks source link

Add links to the ocean insitu obs processing tools #2644

Closed guillaumevernieres closed 3 weeks ago

guillaumevernieres commented 1 month ago

Description

Add links to the marine bufr to ioda converters for the marine insitu observations.

fixes https://github.com/NOAA-EMC/GDASApp/issues/1106
waiting for https://github.com/NOAA-EMC/GDASApp/pull/1135

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Tested on Hera

Checklist

[ ] Any dependent changes have been merged and published
[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] My changes generate no new warnings
[ ] New and existing tests pass with my changes
[ ] I have made corresponding changes to the documentation if necessary

guillaumevernieres commented 1 month ago

looks good to me.

Thanks for the quick review @aerorahul , I'll un-draft and update the gdas.cd # once the companion gdasapp PR is merged .

guillaumevernieres commented 1 month ago

@WalterKolczynski-NOAA , @aerorahul , I finished testing and just updated gdas.cd to the head of develop.

emcbot commented 1 month ago


CI Update on Wcoss2 at 06/03/24 02:00:16 AM
============================================
Cloning and Building global-workflow PR: 2644
with PID: 79025 on host: clogin01

emcbot commented 1 month ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Mon Jun  3 02:09:50 UTC 2024 on clogin01
---------------------------------------------------
Build: Completed at 06/03/24 02:21:47 AM
Case setup: Completed for experiment C48_ATM_547c5de0
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_547c5de0
Case setup: Skipped for experiment C48_S2SWA_gefs_547c5de0
Case setup: Completed for experiment C48_S2SW_547c5de0
Case setup: Completed for experiment C96_atm3DVar_extended_547c5de0
Case setup: Skipped for experiment C96_atm3DVar_547c5de0
Case setup: Skipped for experiment C96_atmaerosnowDA_547c5de0
Case setup: Completed for experiment C96C48_hybatmDA_547c5de0
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_547c5de0

emcbot commented 1 month ago

Experiment C48_ATM_547c5de0 SUCCESS on Wcoss2 at 06/03/24 03:33:08 AM

emcbot commented 1 month ago

Experiment C48_S2SW_547c5de0 SUCCESS on Wcoss2 at 06/03/24 03:45:11 AM

emcbot commented 4 weeks ago

Experiment C96C48_hybatmDA_547c5de0 SUCCESS on Wcoss2 at 06/03/24 06:21:22 AM

emcbot commented 4 weeks ago

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2644

emcbot commented 4 weeks ago

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2644

emcbot commented 4 weeks ago

Experiment C96_atm3DVar_extended_547c5de0 FAIL on Wcoss2 at 06/03/24 09:00:26 AM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2644/RUNTESTS/COMROOT/C96_atm3DVar_extended_547c5de0/logs/2021122118/gfsatmos_prod_f261-f267.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2644/RUNTESTS/COMROOT/C96_atm3DVar_extended_547c5de0/logs/2021122118/gfsfcst.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 4 weeks ago

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2644

WalterKolczynski-NOAA commented 4 weeks ago

Looks like WCOSS failed because stmp filled.

emcbot commented 4 weeks ago


CI Update on Wcoss2 at 06/04/24 02:32:54 AM
============================================
Cloning and Building global-workflow PR: 2644
with PID: 182343 on host: clogin01

emcbot commented 4 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Tue Jun  4 02:41:01 UTC 2024 on clogin01
---------------------------------------------------
Build: Completed at 06/04/24 02:52:57 AM
Case setup: Completed for experiment C48_ATM_547c5de0
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_547c5de0
Case setup: Skipped for experiment C48_S2SWA_gefs_547c5de0
Case setup: Completed for experiment C48_S2SW_547c5de0
Case setup: Completed for experiment C96_atm3DVar_extended_547c5de0
Case setup: Skipped for experiment C96_atm3DVar_547c5de0
Case setup: Skipped for experiment C96_atmaerosnowDA_547c5de0
Case setup: Completed for experiment C96C48_hybatmDA_547c5de0
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_547c5de0

emcbot commented 4 weeks ago

Experiment C48_ATM_547c5de0 SUCCESS on Wcoss2 at 06/04/24 04:12:09 AM

emcbot commented 4 weeks ago

Experiment C48_S2SW_547c5de0 SUCCESS on Wcoss2 at 06/04/24 04:15:14 AM

emcbot commented 4 weeks ago

Experiment C96C48_hybatmDA_547c5de0 SUCCESS on Wcoss2 at 06/04/24 05:21:18 AM

emcbot commented 4 weeks ago

Experiment C96_atm3DVar_extended_547c5de0 FAIL on Wcoss2 at 06/04/24 08:36:16 AM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2644/RUNTESTS/COMROOT/C96_atm3DVar_extended_547c5de0/logs/2021122118/gfspostsnd.log

Follow link here to view the contents of the above file(s): (link)

WalterKolczynski-NOAA commented 4 weeks ago

Restart files didn't get scrubbed properly from last attempt, so forecast started partway through. Will run through again once #2620 gets merged.

WalterKolczynski-NOAA commented 3 weeks ago

@guillaumevernieres please merge in develop with Russ's enabling of UFSDA on WCOSS and resolve the version conflict, then we'll rerun.

guillaumevernieres commented 3 weeks ago

@guillaumevernieres please merge in develop with Russ's enabling of UFSDA on WCOSS and resolve the version conflict, then we'll rerun.

@guillaumevernieres please merge in develop with Russ's enabling of UFSDA on WCOSS and resolve the version conflict, then we'll rerun.

Do we really need to rerun the ci @WalterKolczynski-NOAA ? The failure had nothing to do with the changes in this PR.

WalterKolczynski-NOAA commented 3 weeks ago

I was just going to rerun it on WCOSS

emcbot commented 3 weeks ago


CI Update on Wcoss2 at 06/05/24 08:56:51 PM
============================================
Cloning and Building global-workflow PR: 2644
with PID: 13330 on host: clogin05

emcbot commented 3 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Wed Jun  5 21:05:07 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/05/24 09:50:02 PM
Case setup: Completed for experiment C48_ATM_e6fe8740
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_e6fe8740
Case setup: Skipped for experiment C48_S2SWA_gefs_e6fe8740
Case setup: Completed for experiment C48_S2SW_e6fe8740
Case setup: Completed for experiment C96_atm3DVar_extended_e6fe8740
Case setup: Skipped for experiment C96_atm3DVar_e6fe8740
Case setup: Skipped for experiment C96_atmaerosnowDA_e6fe8740
Case setup: Completed for experiment C96C48_hybatmDA_e6fe8740
Case setup: Completed for experiment C96C48_ufs_hybatmDA_e6fe8740

emcbot commented 3 weeks ago

Experiment C48_S2SW_e6fe8740 FAIL on Wcoss2 at 06/05/24 10:36:19 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2644/RUNTESTS/COMROOT/C48_S2SW_e6fe8740/logs/2021032312/gfswaveinit.log

Follow link here to view the contents of the above file(s): (link)

WalterKolczynski-NOAA commented 3 weeks ago

stmp still/again full on WCOSS2. This PR doesn't have any changes that should impact only WCOSS, so I'm just going to merge without the test passing.

RussTreadon-NOAA commented 3 weeks ago

FYI: The files linked by this PR to ush/ should be added to .gitignore.

After executing sorc/link_workflow.sh the bufr2ioda_insitu_profile_${platform}.py files are flagged as untracked files when we execute git status

Hera(hfe08):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/pr2641$ git status
On branch feature/update_gdasapp
Your branch is up to date with 'origin/feature/update_gdasapp'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        ush/bufr2ioda_insitu_profile_argo.py
        ush/bufr2ioda_insitu_profile_bathy.py
        ush/bufr2ioda_insitu_profile_glider.py
        ush/bufr2ioda_insitu_profile_marinemammal.py
        ush/bufr2ioda_insitu_profile_tesac.py
        ush/bufr2ioda_insitu_profile_xbtctd.py
        ush/bufr2ioda_insitu_surface_altkob.py
        ush/bufr2ioda_insitu_surface_trkob.py

no changes added to commit (use "git add" and/or "git commit -a")

aerorahul commented 3 weeks ago

Thanks @RussTreadon-NOAA for identifying the linked files that are untracked.

RussTreadon-NOAA commented 3 weeks ago

@aerorahul , I can add

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        ush/bufr2ioda_insitu_profile_argo.py
        ush/bufr2ioda_insitu_profile_bathy.py
        ush/bufr2ioda_insitu_profile_glider.py
        ush/bufr2ioda_insitu_profile_marinemammal.py
        ush/bufr2ioda_insitu_profile_tesac.py
        ush/bufr2ioda_insitu_profile_xbtctd.py
        ush/bufr2ioda_insitu_surface_altkob.py
        ush/bufr2ioda_insitu_surface_trkob.py

to .gitignore in PR #2641. Is this acceptable or would you prefer a separate issue and PR?