search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
9 stars 4 forks source link

add missing value to ob value and error #1481

Closed BenjaminRuston closed 6 months ago

BenjaminRuston commented 7 months ago

Description

properly use the IODA missing value for the observation and error missing values

Issue(s) addressed

Resolves #1480

Checklist

fabiolrdiniz commented 7 months ago

@BenjaminRuston, let's remove the -d option too.

fabiolrdiniz commented 7 months ago

@BenjaminRuston, just pushed that change.

BenjaminRuston commented 7 months ago

@YoulongXia-NOAA and @ClaraDraper-NOAA do you use the smap9km_ssm2ioda.py I just use the file in this PR should we remove the other one?

YoulongXia-NOAA commented 7 months ago

@BenjaminRuston, for 9km smap data, @ClaraDraper-NOAA group is using for testing. For the other SMAP ioda converter, it will use for NRT smap data when EMC move soil moisture DA to ops work. I also developed the ioda converts for other soil moisture data. Which converter do you want to keep and which ones want to remove, you need to talk with @ClaraDraper-NOAA and EMC land DA/DA feds to make the decision.

ClaraDraper-NOAA commented 7 months ago

@BenjaminRuston, for 9km smap data, @ClaraDraper-NOAA group is using for testing. For the other SMAP ioda converter, it will use for NRT smap data when EMC move soil moisture DA to ops work. I also developed the ioda converts for other soil moisture data. Which converter do you want to keep and which ones want to remove, you need to talk with @ClaraDraper-NOAA and EMC land DA/DA feds to make the decision.

@YoulongXia-NOAA I was going to tell Ben to ask you! (or EMC). My expectation is that we should be supporting all of the converters that have been added (as in, I'd be surprised if anything has changed and we don't need them all anymore).

YoulongXia-NOAA commented 7 months ago

@ClaraDraper-NOAA, @BenjaminRuston, now land has the converters below: afwa_snod2ioda.py ims_scf2ioda.py smap9km_ssm2ioda.py, ascat_ssm2ioda.py, imsfv3_scf2ioda.py, smap_ssm2ioda.py, ghcn_snod2ioda.py owp_snow_obs.py smos_ssm2ioda.py.

imsfv3_scf2ioda.py and ghcn_snod2ioda.py are used for snow DA, owp_snow_obs.py developed by NCAR people. smap9km_ssm2ioda.py is used by Clara, smap_ssm2ioda.py will be used by EMC for NRT smap soil moisture.

ascat_ssm2ioda.py and smos_ssm2ioda.py I developed. I am not sure if they will used. If Clara think that they are not useful, they can be removed. afwa_snod2ioda.py and ms_scf2ioda.py use pygrib to read grib file I developed originally and no one is planned to use. You can remove these two converters as well. In total you can remove these four files. What do you think?

BenjaminRuston commented 6 months ago

appreciate the comments...

would you like me to make corresponding changes to the 9km at this point let's keep both

regarding the new converters we should open a new issue and get those in and tested

YoulongXia-NOAA commented 6 months ago

Okay, please do so. Thank you.

ClaraDraper-NOAA commented 6 months ago

@ClaraDraper-NOAA, @BenjaminRuston, now land has the converters below: afwa_snod2ioda.py ims_scf2ioda.py smap9km_ssm2ioda.py, ascat_ssm2ioda.py, imsfv3_scf2ioda.py, smap_ssm2ioda.py, ghcn_snod2ioda.py owp_snow_obs.py smos_ssm2ioda.py.

imsfv3_scf2ioda.py and ghcn_snod2ioda.py are used for snow DA, owp_snow_obs.py developed by NCAR people. smap9km_ssm2ioda.py is used by Clara, smap_ssm2ioda.py will be used by EMC for NRT smap soil moisture.

ascat_ssm2ioda.py and smos_ssm2ioda.py I developed. I am not sure if they will used. If Clara think that they are not useful, they can be removed. afwa_snod2ioda.py and ms_scf2ioda.py use pygrib to read grib file I developed originally and no one is planned to use. You can remove these two converters as well. In total you can remove these four files. What do you think?

We should keep the ASCAT and SMOS converters. @BenjaminRuston - can you please make the same changes to those, if it's relevant?

BenjaminRuston commented 6 months ago

@ClaraDraper-NOAA this PR is just dealing with the SMAP converter and was wondering if we needed both the: smap_ssm2ioda.py and the smap9km_ssm2ioda.py

so for now made parallel changes. Will not touch the ASCAT and SMOS

but think this is ready and please let me know if this is OK... of note we decided to remove the shorthand switches as they could be misinterpreted too easily.

so to run with the masking which is typical the -m option is removed leaving only the --maskMissing similarly the -d (which often mean date) but in this case is assumed depth of measurement now can only be triggered using the full name --assumedSoilDepth

so please review any scripts you may have in place as these may need to be updated