AndrewEichmann-NOAA
commented
2 months ago @guillaumevernieres Apologies if we covered this already, but how do you replicate the PYTHONPATH problem?
Closed guillaumevernieres closed 2 months ago
AndrewEichmann-NOAA
commented
2 months ago @guillaumevernieres Apologies if we covered this already, but how do you replicate the PYTHONPATH problem?
JohnSteffen-NOAA
commented
2 months ago @AndrewEichmann-NOAA The issue is in the /jobs/rocoto/prepoceanobs.sh script. Below is a diff of the fix versus the original file. I hope this helps.
17,19c17,18 < pyiodaPATH="${HOMEgfs}/sorc/gdas.cd/build/lib/python$(detect_py_ver)/" < PYTHONPATH="${pyiodaPATH}:${PYTHONPATH}" < export PYTHONPATH
PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush"
guillaumevernieres
commented
2 months ago Profiling stats from a run done in ~June:
0: OOPS_STATS ------------------------------------------------------------------------------------------------------------------
0: OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 480 MPI tasks) -----------------------------------
0: OOPS_STATS ------------------------------------------------------------------------------------------------------------------
0: OOPS_STATS Name : min (ms) max (ms) avg (ms) % total imbal (%)
0: OOPS_STATS oops::Covariance::SABER::inverseMultiply : 39089.10 39118.49 39108.56 9.14 0.08
0: OOPS_STATS oops::Covariance::SABER::multiply : 318555.12 320508.12 319208.12 74.59 0.61
0: OOPS_STATS oops::Geometry::Geometry : 2175.04 2229.15 2202.86 0.51 2.46
0: OOPS_STATS oops::GeometryData::setGlobalTree : 2359.65 2625.32 2498.37 0.58 10.63
0: OOPS_STATS oops::GetValues::GetValues : 522.64 934.61 623.87 0.15 66.03
0: OOPS_STATS oops::GetValues::fillGeoVaLsTL : 839.27 1993.21 1340.86 0.31 86.06
0: OOPS_STATS oops::GetValues::finalizeAD : 137.30 1737.96 617.54 0.14 259.20
0: OOPS_STATS oops::GetValues::processAD : 1764.53 3689.34 2544.26 0.59 75.65
0: OOPS_STATS oops::GetValues::processTL : 2680.58 3699.18 3243.10 0.76 31.41
0: OOPS_STATS oops::Increment::Increment : 5349.88 6016.86 5518.57 1.29 12.09
0: OOPS_STATS oops::Increment::axpy : 657.32 700.57 672.96 0.16 6.43
0: OOPS_STATS oops::Increment::dot_product_with : 624.08 788.95 719.35 0.17 22.92
0: OOPS_STATS oops::Increment::fromFieldSet : 1139.07 1290.27 1187.17 0.28 12.74
0: OOPS_STATS oops::Increment::operator+= : 571.70 647.03 603.61 0.14 12.48
0: OOPS_STATS oops::Increment::operator= : 2531.70 3147.49 2791.42 0.65 22.06
0: OOPS_STATS oops::Increment::read : 2396.09 2421.44 2402.44 0.56 1.06
0: OOPS_STATS oops::Increment::toFieldSet : 2602.39 6067.22 4234.71 0.99 81.82
0: OOPS_STATS oops::Increment::write : 10341.35 10604.79 10388.49 2.43 2.54
0: OOPS_STATS oops::LinearObsOper::adt_rads_all::simulateObsTL : 261.54 651.17 479.06 0.11 81.33
0: OOPS_STATS oops::LinearVariableChange::changeVarAD : 5383.53 7051.18 6397.39 1.49 26.07
0: OOPS_STATS oops::LinearVariableChange::changeVarTL : 4489.01 5022.80 4649.39 1.09 11.48
0: OOPS_STATS oops::Model::step : 5893.19 5938.91 5921.36 1.38 0.77
0: OOPS_STATS oops::ObsSpace::ObsSpace : 2184.46 2291.20 2239.69 0.52 4.77
0: OOPS_STATS oops::ObsSpace::save : 2131.04 2240.83 2151.79 0.50 5.10
0: OOPS_STATS oops::ObsVector::dot_product : 880.91 2022.12 1730.50 0.40 65.95
0: OOPS_STATS oops::State::State : 3223.66 3279.44 3253.54 0.76 1.71
0: OOPS_STATS oops::State::read : 5886.17 5923.47 5911.78 1.38 0.63
0: OOPS_STATS oops::State::write : 12839.37 13136.83 12941.26 3.02 2.30
0: OOPS_STATS oops::UnstructuredInterpolator::apply : 1706.06 2109.63 1879.18 0.44 21.48
0: OOPS_STATS oops::UnstructuredInterpolator::applyAD : 1101.73 1354.02 1213.13 0.28 20.80
0: OOPS_STATS oops::VariableChange::changeVar : 363.96 597.46 493.61 0.12 47.31
0: OOPS_STATS oops::mpi::broadcast : 1479.06 1690.45 1558.76 0.36 13.56
0: OOPS_STATS util::Timers::Total : 427934.68 427986.61 427952.39 100.00 0.01
0: OOPS_STATS util::Timers::measured : 423683.67 424970.24 424467.41 99.19 0.30
0: OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 480 MPI tasks) -----------------------------------
0:
0: OOPS_STATS Run end - Runtime: 431.38 sec, Memory: total: 557.89 Gb, per task: min = 1.11 Gb, max = 2.39 Gb
0: Run: Finishing oops::Variational<SOCA, UFO and IODA observations> with status = 0 Current profiling stats:
0: OOPS_STATS ------------------------------------------------------------------------------------------------------------------
0: OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 800 MPI tasks) -----------------------------------
0: OOPS_STATS ------------------------------------------------------------------------------------------------------------------
0: OOPS_STATS Name : min (ms) max (ms) avg (ms) % total imbal (%)
0: OOPS_STATS oops::Covariance::SABER::inverseMultiply : 33942.63 33962.76 33951.64 8.77 0.06
0: OOPS_STATS oops::Covariance::SABER::multiply : 264555.83 265312.47 265018.90 68.47 0.29
0: OOPS_STATS oops::Geometry::Geometry : 4057.45 4340.30 4146.50 1.07 6.82
0: OOPS_STATS oops::Geometry::~Geometry : 766.74 921.28 857.29 0.22 18.03
0: OOPS_STATS oops::GeometryData::setGlobalTree : 9490.34 11204.47 10293.48 2.66 16.65
0: OOPS_STATS oops::GetValues::GetValues : 343.66 861.01 606.20 0.16 85.34
0: OOPS_STATS oops::GetValues::fillGeoVaLsTL : 860.45 3527.18 1596.59 0.41 167.03
0: OOPS_STATS oops::GetValues::finalizeAD : 81.55 2907.96 821.58 0.21 344.02
0: OOPS_STATS oops::GetValues::process : 371.89 468.54 399.88 0.10 24.17
0: OOPS_STATS oops::GetValues::processAD : 2511.47 3372.11 2743.77 0.71 31.37
0: OOPS_STATS oops::GetValues::processTL : 3079.93 4190.25 3459.69 0.89 32.09
0: OOPS_STATS oops::Increment::Increment : 8762.98 10540.19 9570.84 2.47 18.57
0: OOPS_STATS oops::Increment::axpy : 1385.07 1774.20 1482.51 0.38 26.25
0: OOPS_STATS oops::Increment::diff : 436.48 498.87 462.05 0.12 13.50
0: OOPS_STATS oops::Increment::dot_product_with : 830.86 1389.40 1232.65 0.32 45.31
0: OOPS_STATS oops::Increment::fromFieldSet : 540.98 734.41 591.54 0.15 32.70
0: OOPS_STATS oops::Increment::operator= : 10905.20 12450.11 11705.23 3.02 13.20
0: OOPS_STATS oops::Increment::read : 1971.61 2163.65 2087.15 0.54 9.20
0: OOPS_STATS oops::Increment::toFieldSet : 994.47 1253.32 1094.25 0.28 23.66
0: OOPS_STATS oops::Increment::write : 7728.28 7997.03 7737.61 2.00 3.47
0: OOPS_STATS oops::LinearVariableChange::changeVarAD : 12752.55 15365.65 13806.07 3.57 18.93
0: OOPS_STATS oops::LinearVariableChange::changeVarTL : 12411.24 13035.42 12794.49 3.31 4.88
0: OOPS_STATS oops::Model::step : 3046.85 3273.22 3239.17 0.84 6.99
0: OOPS_STATS oops::ObsFilter::Perform Action::preProcess : 227.97 1263.81 765.32 0.20 135.35
0: OOPS_STATS oops::ObsFilter::preProcess : 229.28 1265.21 766.85 0.20 135.09
0: OOPS_STATS oops::ObsFilter::~ObsFilter : 331.33 779.50 568.45 0.15 78.84
0: OOPS_STATS oops::ObsSpace::ObsSpace : 1151.00 1281.51 1224.88 0.32 10.65
0: OOPS_STATS oops::ObsSpace::save : 1793.36 1870.46 1812.89 0.47 4.25
0: OOPS_STATS oops::ObsVector::dot_product : 1180.66 3569.78 3161.18 0.82 75.58
0: OOPS_STATS oops::State::State : 2242.17 2665.12 2404.72 0.62 17.59
0: OOPS_STATS oops::State::read : 3046.81 3273.16 3239.13 0.84 6.99
0: OOPS_STATS oops::State::write : 9370.17 9631.28 9441.63 2.44 2.77
0: OOPS_STATS oops::UnstructuredInterpolator::apply : 2928.83 3962.74 3250.56 0.84 31.81
0: OOPS_STATS oops::UnstructuredInterpolator::applyAD : 1870.10 2647.06 2057.14 0.53 37.77
0: OOPS_STATS oops::VariableChange::changeVar : 876.20 1048.42 974.35 0.25 17.68
0: OOPS_STATS oops::mpi::broadcast : 883.52 1175.28 955.26 0.25 30.54
0: OOPS_STATS util::Timers::Total : 386985.58 387191.19 387036.71 100.00 0.05
0: OOPS_STATS util::Timers::measured : 378808.84 382775.36 380793.78 98.39 1.04
0: OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 800 MPI tasks) -----------------------------------
0:
0: OOPS_STATS Run end - Runtime: 390.33 sec, Memory: total: 2015.25 Gb, per task: min = 2.44 Gb, max = 3.56 Gb
0: Run: Finishing oops::Variational<SOCA, UFO and IODA observations> with status = 0
[x] Running out of memory The 3DVAR went from using ~0.5TB in June to more than 2-2.5TB as of now (07/30/2024)
[x] Missing python path to pyioda Add/export PYTHONPATH in this script:
similarly to what is done for the atm obs.
[x] Looks like I was wrong about {{ FIXgfs }} I know, shocker!!! in the gfsv17 yaml, the
SOCA_INPUT_FIX_DIRkey needs to be replaced:[x] soca MOM_input for the 1/4 deg is outdated 1 fatal error and a few warnings We need to remove
DEFAULT_2018_ANSWERS(error) andETA_TOLERANCE_AUX(warning)[x] No error trapping in the checkpoint task One of the job within that task has been failing but the error was not reported by the script exgdas_global_marine_analysis_chkpt.sh#L70
[x] soca to cice variable change is sometimes failing when using 20+ pe's profiling for the 0.25deg res:
we're currently using 40pe's, which fails most of the time and according to the profiling figure above, it is not optimal.
Switch to 16 after stress testing that layout.
[x] ice background fields are all zeros at 0.25deg res. Not quite sure what happened, the low-res seems to be ok.