search

NOAA-EMC / GDASApp

Global Data Assimilation System Application
GNU Lesser General Public License v2.1
15 stars 31 forks source link

Stage marine DA files and data to common directory #1175

Closed JohnSteffen-NOAA closed 4 weeks ago

JohnSteffen-NOAA commented 4 months ago

Stage marine DA related data files for verification etc. from Shastri's local directories to a common place:

/scratch2/NCEPDEV/marine/Shastri.Paturi/ /scratch2/NCEPDEV/stmp1/Shastri.Paturi/forLydiaSulagna/validation/ /scratch1/NCEPDEV/stmp4/Shastri.Paturi/

/scratch1/NCEPDEV/da/Shastri.Paturi

Clean up unnecessary data files as well.

guillaumevernieres commented 4 months ago

this needs to be rm'ed

/scratch2/NCEPDEV/marineda/validation

Duplicated ...

apchoiCMD commented 4 months ago

Is it also, correct? since da/common has it as well,

/scratch2/NCEPDEV/stmp1/Shastri.Paturi/forLydiaSulagna/validation/
guillaumevernieres commented 4 months ago

That's 3 places with the exact same stuff ... We might want to start digging into the fs and look for "Shastri" and "validation".

apchoiCMD commented 4 months ago

Group Name : stmp1

Group Name : marine

apchoiCMD commented 4 months ago

@guillaumevernieres @AndrewEichmann-NOAA @JohnSteffen-NOAA I just get an approvals getting permission from Daryl, Vijay and Avichal on Shastri's folder so, I kindly ask you to look in the his directory and let me know what should be kept/moved to common folder based on what I wrote. Thanks!

AndrewEichmann-NOAA commented 4 months ago

The only thing I remember touching was /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew, and that for samples of the DMPDIR for testing preoceanobs

apchoiCMD commented 4 months ago

The only thing I remember touching was /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew, and that for samples of the DMPDIR for testing preoceanobs

@AndrewEichmann-NOAA I also remember stmp4 also is need to take care of- , all in his folder are duplicated to common

apchoiCMD commented 3 months ago

Final Arrangement

Group Name : stmp1 /scratch2/NCEPDEV/stmp1/Shastri.Paturi/forKatherine => duplicated validation files for ostia2mom6 : RM'ED

Group Name : stmp4 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/ => RM'ED

Group Name : marine /scratch2/NCEPDEV/marine/Shastri.Paturi/COAPS => NEEDED TO LOOK AT -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/DATM_INPUT => Permission Denied -> gefs data RM'ED /scratch2/NCEPDEV/marine/Shastri.Paturi/ForJim => Permission Denied -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/GIOMAS => NEEDED TO LOOK AT -> moved to common /scratch2/NCEPDEV/marine/Shastri.Paturi/GOMb0.08 => Permission Denied -> RM'ED /scratch2/NCEPDEV/marine/Shastri.Paturi/Latency_table => Latency related but Permission Denied -> RM'ED /scratch2/NCEPDEV/marine/Shastri.Paturi/NG-GODAS => NEEDED TO LOOK AT -> RM'ED /scratch2/NCEPDEV/marine/Shastri.Paturi/OISST => Permission Denied -> moved to common /scratch2/NCEPDEV/marine/Shastri.Paturi/OceanObsQC => NEEDED TO LOOK AT -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/PIOMAS => NEEDED TO LOOK AT -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/QC_task => NEEDED TO LOOK AT -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/WCOSS1 => Permission Denied -> RM'ED /scratch2/NCEPDEV/marine/Shastri.Paturi/frommars => Permission Denied -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/ice.L2.emc => Some ice data files : NEEDED TO LOOK AT -> moved to common /scratch2/NCEPDEV/marine/Shastri.Paturi/ice.cryosat_GDR.esa => CS_OFFL_SIR_GDR_2 from 20100715 to 20200829 : NEEDED TO LOOK AT -> RM'ED /scratch2/NCEPDEV/marine/Shastri.Paturi/icec.nsidc => SeaIce related data files : NEEDED TO LOOK AT -> moved to common /scratch2/NCEPDEV/marine/Shastri.Paturi/insitu.fnmoc => FNMOC data files : NEEDED TO LOOK AT -> moved to common /scratch2/NCEPDEV/marine/Shastri.Paturi/ioda_observations => Seems some old files there : NEEDED TO LOOK AT -> moved to common /scratch2/NCEPDEV/marine/Shastri.Paturi/ncoda_code => Permission Denied -> copied /scratch2/NCEPDEV/marine/Shastri.Paturi/rtofs_prod => RTOFSDA.zip : NEEDED TO LOOK AT -> moved to common

apchoiCMD commented 3 months ago

Another big folder is /scratch1/NCEPDEV/da/Shastri.Paturi/ -> 3.6 TB

guillaumevernieres commented 3 months ago

I assume this was closed unintentionally? Do we need to keep this open @apchoiCMD ?

apchoiCMD commented 3 months ago

I would like to discuss it during tomorrow's meeting since I have not touched one of Shastri's folder where is at /scratch1/NCEPDEV/da/Shastri.Paturi/ , I am sure that we need to remove it sooner or later-

apchoiCMD commented 4 weeks ago

Those below are removed

/scratch1/NCEPDEV/da/Shastri.Paturi/ /scratch2/NCEPDEV/marine/Shastri.Paturi/