Current tasks

[ilianagenkova]

• [x] satingest RN - provide only Short names and Dest dirs (SN/SS/IG)
• [x] Day before implementation - verify Short Names for OPS, and if same as Short Names for I&T (IG)
• [x] new product names in fix/ files; keep old names as commented (and date of change/implementation date) (SN/SS/IG)
• [x] Set meeting - PDA, OSPO/PALs, NCO SPA & DF, satingest - present I&T challenges (IG)
• [x] CBU failover stats; see ingest_history - how many files we lost ; add a flag (where?) (SN/SS)
• [x] Ask PDA and OSPO/PALs if OPS can have old and new products (like ASCAT winds) (IG)
• [x] CIMSS winds - Jeff A. will confirm, pass to Li Bi , satwhr(SS)
• [x] CONUS - for RRFS in a satwhr (SN)
• [x] introduce PALs to DA team, connect product provider and user, comm test data (IG)
• [x] GOES SkyCover rewrite - netcdf to ascii , tested with UW test data(SS)
• [x] satingest.v5.0.0 (SN/SS/IG)
• [ ] MTG IR John Derber (SN/SS/IG)
• [x] bufr-tranjb for saildrone (SS)
• [x] WIGOSIDs - future (IG)
• [x] tell PALs to provide WMO compliant test data (no missing local tables) (IG)
• satingest spreadsheet - share with PAL (to populate) and DA (SN/SS/IG)
• AF_* -> AF_Iband (Ravan) in next satingest , Short names: PDA OPS with Short Names of AF_Iband_VIIRS_EDR and AF_Iband_VIIRS_ASCII_List(SS)
• two google docs to be updated - (SN) https://docs.google.com/spreadsheets/d/1MVNwbnkhziNrqkwgObtWj-_aobJ9c9jaOIU44_d9u9I/edit#gid=0 https://docs.google.com/spreadsheets/d/1seZke7x3HDnVk7svyg3KsY3noIi5gt1BR6uEjGUXWVc/edit
• [ ] Resolve the snow run and the date issue - NDATE, N* ?? (SS)
• [x] When will CrIS be updated? - on 15 Dec 2022
• [x] Subs for Metop-B/C ASCAT (11 Nov- 11 Feb 2023) (IG, SN)
• [x] G18- AMVs global, rap, cdas (see Trigs.KEEP jcdasREL as example) (SN)
• [x] G18-CSR global (SN)
• [x] test PDA I&T accounts (IG, SS)

• [x] add the CONUS winds fix file (and relevant code) to GiitHUb, to start making these tanks operationally in early 2023

  **!!! Below are the high priority tasks for the Dec14-15 upgrades !!!**

• [x] test Ingest the AFIband* files in aerosol.af_viirs : AF-Iband_v%_npp%.txt (SS)
• [x] prepare satingest with retired product filenames removed - oct 31 2022 (SS)
• [x] test satingest with I&T products from updated BUFR toolkit - upgrade on 14 Dec - https://docs.google.com/document/d/10kWm3S1Q2Gne23QpkReE1KkZl4YYtN_X5_pVSovJo1c/edit
  if jobs get to be too many, pass some to Steve (SN)
• [x] keep the ASCAT winds ingesting and dumping, but don't put in Dec release until Emily OKs it (SN)
• [x] ingest GBBEP products - the most right column will tell you which filenames need to be added to aerosol.burning_emissions : https://docs.google.com/spreadsheets/d/11cKJk-CGnbwo2w6NbjUj0-RG81YzsgTykbPfhv5bR20/edit#gid=0 The subs were just made for these yesterday (SS) GBBEPx is now ingested by Data Flow , since 20230221 (see emails with Nate Janechek)
• [x] make sure subscriptions for ATMS and CRIS files (I&T) are ACTIVE- short names: ATMS_BUFR_TEST , Full: ATMS_v1r0_npp_sxxxxxxxxxxxxxxx_exxxxxxxxxxxxxxx_cxxxxxxxxxxxxxxx.bufr CrIS_C0431_BUFRTEST, Full: NUCAPS-C0431v3r0_x_sxxxxxxxxxxxxxxx_exxxxxxxxxxxxxxx_cxxxxxxxxxxxxxxx.bufr (only the 431, not the 2211) (SN)
• [x] resume ingest and dump of ATMS and CRIS files: ATMS name are unchanged, CrIS new names are: NUCAPS-C0431_v3r0_x_sxxxxxxxxxxxxxxx_exxxxxxxxxxxxxxx_cxxxxxxxxxxxxxxx.bufr (SN)

DECEMBER Sudhir:

• [x] Resume G-18 RAP AMVs dumping
• [x] Make I&T subs for GMI
• [x] Make I&T subs for ATMS NOAA-21
• [x] Make new task hrsatwind and hrsatwind.conus5*, hrsawnd.ciimss families - pass to Steve for v4.5
• [x] Keep OMPS-NP ingest, may be ingest from emcrzdm
• [x] Help Steve with dumping for v4.5

Steve:

• [x] Work with Sudhir on hrsatwind
• [x] Make feature/satingest.v4.5 branch
• [x] Add hrsatwind to v4.5
• [x] Add AF-Iband to v4.5
• [x] Add new GBBEPx names (3 files only) to v4.5 GBBEPx_BC.emissionsv003????????.nc (will be replaced by GBBEPx-BC-emissions_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}.nc$ ) GBBEPx_OC.emissionsv003????????.nc (GBBEPx-OC-emissions_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}.nc$ ) GBBEPx_SO2.emissionsv003????????.nc (GBBEPx-SO2-emissions_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}.nc$ )
• [x] Add new GMI name to v4.5 OLD: 1C-R.GPM.GMI.XCAL\d{4}-\w{1}.\d{8}-S\d{6}-E\d{6}.\w{4}.bufr New | Short: NCCF_GPM_GMI_L1C-R_BUFR | Full: GMI-1CR_v7r0_gpm_s\d{15}_e\d{15}_c\d{15}.bufr
• [x] May be add OMPS-NP to v4.5 (if nesdis is ready) - I saved the fix file with new nams: /lfs/h2/emc/obsproc/noscrub/iliana.genkova/GIT/satingest.v4.4.0/fix/npp_omps.ompsnp_v8.KEEP
• [x] *Keep snowday COPY_FORWARD_GRIB_16USAF,* but turn off PDA pulls for MESH16 in snowore
• [x] Check with Bert if he wants it - test poes_sst group with uncommented npp and n20 filenames, and remove metop-a (row 18) in https://github.com/NOAA-EMC/satingest/blob/develop/fix/poes_sst.navy_avhrr . Add to satingest.v4.5.0
• [x] backfill CIMSS winds for Li

[SteveStegall-NOAA]

How can we tell which tasks are mine vs SN?

[ilianagenkova]

The goal was to make an exhaustive list of all satingest tasks first, confirm priorities with EMC, and then divvy up. Some tasks are already assigned, so I'll add initials. If you can think of any satingest tasks not included in the list we put together with SN, please add them.

[ilianagenkova]

Satingest v4.4.1 (wrap up)

• G18 CSR,ASR,AMVs - Jan 4, 2023
• SkyCover, Cloud Products (if we get to test them) - Jan 4, 2023
  • GVF-WKL-GLBv3r0* test and add to aerosol.viirs_weekly_gvf - ASAP

Satingest v4.5.0

• [x] add RN (follow the v4.4.0 section as an example)

• [ ] - rename /TRIGGERS to /dev "git mv /TRIGGERS /dev"

• [x] - make README.md build/install instructions clearer

• [ ] - try making "cd ush && ./build.sh " work in addition to "./ush/build.sh"

• [x] - GMI new names

• [x] - AF-Iband new names

• [x] - CIMSS AMVs (in new task)

• [x] - CONUS AMVs (in new task)

• [x] - keep GBBEPx with v003

• [x] - X Keep snowday COPY_FORWARD_GRIB_16USAF, but turn off PDA pulls for MESH16 in snowore

• [x] - Clean up HIRS files (see HotPlate)

• [x] - Correct 005081's FXY value (remember to check against Decoders bufr tables too)

• [x] - update isdb*.html file

• [x] - OMPS NP (if ready)

[ilianagenkova]

SN:

• [x] Autosnow ingest test with new names - /lfs/h2/emc/obsproc/noscrub/sudhir.nadiga/GITHUBDIR/rel_sat.v4.5.0
• [x] Prepare feature/satingest.v4.6.0
• [x] Make and maintain (or add to a release) a satingest task for use with NRT crons, to pull data from PDA I&T for test purposes, where longer than 5 days: New task was created COPYINGEST, for now we have /fix/copyingest.smap (?), if one needs another data type ingested, make new copyingest.DATANAME, and change : filenames, dest dir, tank name, if ftp is the source, add eval MACHINE= /lfs/h2/emc/obsproc/noscrub/sudhir.nadiga/GITHUBDIR/f_sat451.COPYINGEST
• [x] Continue SMAP pulls for Youlong
• [x] what crons are NRT while on leave???
• [x] keep N-21 ATMS ingest (from OPS) and dump (global)
• [x] start N-21 CrIS ingest from NDE/NUCAPS I&T (only 431) and dump (global ), add filenames for n21, so all 3 instruments in one tank ; exglobal_dump.sh (crisfs), switch CrIS to OPS

SS:

• [x] RN for v4.6.0
• [x] oversee the completion of bufr_tranjb saildrone fix and bufr-tranjb.v*** to go with satingest.v4.6.0
• [x] Maintain a CrIS Cluster ingest and dump for Jim Jung (on hold)
• [x] Continue work on updating documents
• [x] start a cron to generate CONUS and CIMSS dumps, maintain 1 month data on /noscrub , delete weekly non-satwnd / hrsatwind dumps to free space

[ilianagenkova]

7 April 2023 SN

• [x] Dump CrIs and ATMS from OPS - global (& populate gaps after prod switches) Optimize PBS resources and frequency of job submission, share with Steve, save it (in the comments or this issue)
• [x] Is COPYINGEST task in v4.6? if not, add to v4.7, Write in the comment section "how to use it"
• [ ] Activate subs for all operationally ingested data, for comprehensive testing of releases
• [x] PS4 - add lines to wrapper and in /TRIGGERS/JISATINGEST DEBUG_LEVEL=4
• [ ] LOGs will be split by date (down to day), remove 3+ day old logs
• [ ] Receipt time - RCYR RCMN RC - identify which Santingest tanks have RC, which codes used for the b tanks, which time to use for RC, how to implement
• [ ] Add WIGOS IDs mnemonics to the bufr tables

SS

• [x] Dump CrIs and ATMS from OPS -RAP (& populate gaps after prod switches)
• [x] VIIRS winds
• [x] Start feature/satingest.v4.6_clean - turn off discontinued ingests (snowore.gribcopy_usaf_nh/sh - for MESH16, BURNING _EMISSIONS in aerosol - for GBBEPx), remove unused fixed files, test all tasks
• [x] Add bufr-tranjb saildrone to feature/satingest.v4.6_clean - test
• [x] PS4 - add lines to wrapper and in /TRIGGERS/JISATINGEST DEBUG_LEVEL=4
• [x] Retest all tasks v4.6 starting on 10 April - use logs to figure out the receipt time solution
• [x] Continue updating satingest inventory document
• [x] 1 May - start dump satwhr
• [ ] Receipt time - RCYR RCMN RC - identify which Santingest tanks have RC, which codes used for the b tanks, which time to use for RC, how to implement

IG

• [x] Ask PDA to reduce I&T to 24 hours, to increase the pulls, cut 48 to 24, when data moved I&T to OPS could they mirror the I&T sub to OPS, ask Mun for a sub list I&T and OPS

• [x] feature/satingest.v4.6_clean will become release/satingest.v4.7.0

• [ ] open DF ticket to move over GMASI to DF (not done yet! Oct 23, 2023)

TO DO before handing CDF to SPA:

• "diff -r" against the operationally used satingest, pick any NCO edits, make sure list of diff files == RN changed files
• test all satingest tasks
• use Sudhir's /lfs/h2/emc/obsproc/noscrub/sudhir.nadiga/comparetanks.sh to help verify output
• ADD the Implementation date to calendar (IG,SS,SN)
• keep an eye on naming conventions coming from NESDIS
• open new Dest Dir if new obs type comes (in I&T and OPS0

for v4.7

• [x] REMOVE: EXECUTE=copy_to_target (replace with something that makes more sense for that Family)

echo $EXECUTE [ $DEBUGSCRIPTS = ON -o $DEBUGSCRIPTS = YES ] && set -x if [ $toterr -eq 0 ] ; then if [ $EXECUTE = nullexec ] ; then bufrerror=0 elif [ $EXECUTE = ingest_copytotarget.sh ] ; then ... (to match the fix file)

[ilianagenkova]

October 2023

IG:

• [x] Discuss satingest v5.0.0 with SPAs and EIB
• [x] Discuss NCCF data pulls approach with SPA and DF
• [x] CDF for b-t v1.0.5 after Decoders test it
• [x] Test new NCCF credentials

SS:

• [x] Follow with Decoders that b-t v1.0.5 performs as intended
• [x] Continue work on adding receipt time in tanks
• [x] Begin work on UW Sky cover name change (when Short Names are provided)
• [x] Test new NCCF credentials

SN:

• [x] Ingest/dump CrIS (j01 and n21) from PDA I & T (with cloud and cluster inof)
• [x] Test satingest v5.0.0 Tier2 and Tier3 proposals
• [ ] Implement v5.0.0
• [ ] IMS snow data - we currently pull&copy&rename the imssnowgrb files. Add to this step the NIC.IMS??*.asc files (lower priority, this might be actually part of obsproc code)
• [x] Test new NCCF credentials

[ilianagenkova]

July 2024

SN

• [ ] AWS test with Metop AMVs, CrIS rads, snow data, hardwired path first

• [ ] sat v5.0 - single family per single task (??? 4.7.6)

• [ ] Satingest tanks receipt time - message time

• [ ] Keep SMAP, GBBEPx, RAVE backup

• [ ] Make subs for GMGSI McIDAS and Netcdf data - I&T

• [ ] release/satingest v4.7.5 + bufr-tranjv v1.1 (both still use bufr.11/7) clean unused fix files, may be 3x CRIS, may be 3x VIIRS SST, HM8 cloudy landley, HEAP IASI new version v3r1->v3r2 (test data on I&T); -> test ingest I&T data

• [ ]

• [ ] Help iliana with IMS snow data - we currently pull&copy&rename the imssnowgrb files. Add to this step the NIC.IMS??*.asc files (lower priority, this might be actually part of obsproc code)

SS

• [ ] Satingest tanks receipt time - message time
• [ ] Metop-SG IASI-NG ingest of EUMETSAT data
• [ ] Keep SMAP, GBBEPx, RAVE backup