Bugfix 2867 point2grid qc flag main v11.1

hsoh-u commented 2 months ago

Expected Differences

The meaning of ADP QC values were changed (it was 3 for high, 2 for medium, and 1 for low). The baseline algorithm and the enterprise algorithm produce different QC values for high, medium, and low. MET reads QC values and meanings from the variable attribute and apply them to -qc options (where 0 is high, 1 is medium, and 2 is low).

QC values for Baseline algorithm:
- high: 3 (raw value: 12 or 48)
- medium: 1 (raw value: 4 or 16)
- low: 0
QC values for Enterprise algorithm:
- high: 0
- medium: 1 (raw value: 4 or 16)
- low: 2 (raw value: 8 or 32)

The -qc options at the unittests were changed to -qc 0,1,2.

[x] Do these changes introduce new tools, command line arguments, or configuration file options? [No]
If yes, please describe:
[x] Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
If yes, please describe:

Pull Request Testing

[x] Describe testing already performed for these changes:

New GOES16 data with Enterprise algorithm:

/d1/personal/hsoh/git/pull_request/MET_bugfix_2867_point2grid_qc_flag_main_v11.1/bin/point2grid  /d1/projects/MET/MET_test_data/unit_test/model_data/goes_16/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/projects/MET/MET_test_data/unit_test/model_data/goes_16/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1

==>

DEBUG 4: set_adp_gc_values()  high_confidence = 0, medium_confidence = 1, low_confidence = 2

DEBUG 4: regrid_goes_variable() -> Count: actual: 6, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 62127, by absent: 590193, total: 652320
DEBUG 4:    Range:  data: [-0.05 - 4.99997]  QC: [0 - 2]
DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=4 medium=87, low=62050, no_retrieval=10, adjusted=47142

Note: if only high quality is given with-qc 0, all data will be filtered out.

Meaning of the debug message DEBUG 4: ADP QC: high=4 medium=87, low=62050, no_retrieval=10, adjusted=47142

number of high confidence: 4
number of medium confidence=87
number of low confidence=62050
number of no_retrieval_qf=10
number of adjusted confidences=47142: confidences were reduced by combining with data QC flags

Old GOES16 data with Baseline algorithm:

/d1/personal/hsoh/git/pull_request/MET_bugfix_2867_point2grid_qc_flag_main_v11.1/bin/point2grid  /d1/projects/MET/MET_test_data/unit_test/model_data/goes_16/OR_ABI-L2-AODC-M6_G16_s20192662141196_e20192662143569_c20192662145547.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/projects/MET/MET_test_data/unit_test/model_data/goes_16/OR_ABI-L2-ADPC-M6_G16_s20192662141196_e20192662143569_c20192662144526.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1

==>

DEBUG 4: set_adp_gc_values()  high_confidence = 3, medium_confidence = 1, low_confidence = 0

DEBUG 4: regrid_goes_variable() -> Count: actual: 121, missing: 1937596, non_missing: 1473666
DEBUG 4:    Filtered: by QC: 0, by adp QC: 86130, by absent: 1387116, total: 1473246
DEBUG 4:    Range:  data: [-0.05 - 4.99997]  QC: [0 - 2]
DEBUG 4:    AOD QC: high=222 medium=1938, low=84390, no_retrieval=0
DEBUG 4:    ADP QC: high=634 medium=9, low=85907, no_retrieval=0, adjusted=84770

[x] Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

More AOD files with Enterprise algorithm are at seneca:/d1/personal/hsoh/data/MET-2853/20240419

[x] Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]
[x] Do these changes include sufficient testing updates? [Yes] One unit test was added and -qc options were changed.
[x] Will this PR result in changes to the MET test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:

New output for Enterprise algorithm: point2grid/point2grid_GOES_16_ADP_Enterprise_high.nc

Three existing output will be different because -qc 1,2,3 was changed to -qc 0,1,2:

point2grid/point2grid_GOES_16_ADP.nc
point2grid_GOES_16_AOD_TO_G212_grid_map.nc
point2grid_GOES_16_AOD_TO_G212.nc
[x] Will this PR result in changes to existing METplus Use Cases? [Yes]
If yes, create a new Update Truth METplus issue to describe them.

The meaning of QC values were changed. So the output will be different if -qc option is given.

[ ] Do these changes introduce new SonarQube findings? [Yes or No]
If yes, please describe:
[ ] Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

[ ] Review the source issue metadata (required labels, projects, and milestone).
[ ] Complete the PR definition above.
[ ] Ensure the PR title matches the feature or bugfix branch name.
[ ] Define the PR metadata, as permissions allow. Select: Reviewer(s) and Development issue Select: Milestone as the version that will include these changes Select: Coordinated METplus-X.Y Support project for bugfix releases or MET-X.Y.Z Development project for official releases
[ ] After submitting the PR, select the :gear: icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
[ ] After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
[ ] Close the linked issue and delete your feature or bugfix branch from GitHub.

jprestop commented 2 months ago

@hsoh-u I see that you pointed me to the Enterprise AOD files, but I am wondering if I should also be testing the Baseline files? If so, could you please point me to those as well?

jprestop commented 2 months ago

@hsoh-u Actually, I think I may have found them from the section where you listed what testing you did.

/d1/projects/MET/MET_test_data/unit_test/model_data/goes_16/

Apologies for not catching that before I sent the last message.

jprestop commented 2 months ago

@hsoh-u Can you please confirm that the failed check for differences is expected?

hsoh-u commented 2 months ago

@hsoh-u Can you please confirm that the failed check for differences is expected?

One output is added and three files are expected to be different because the logic to identify high, medium, and low confidences is changed.

point2grid/point2grid_GOES_16_ADP.nc
point2grid_GOES_16_AOD_TO_G212_grid_map.nc
point2grid_GOES_16_AOD_TO_G212.nc

hsoh-u commented 2 months ago

@hsoh-u Actually, I think I may have found them from the section where you listed what testing you did.

/d1/projects/MET/MET_test_data/unit_test/model_data/goes_16/

Apologies for not catching that before I sent the last message.

More data at seneca:/d1/personal/hsoh/data/MET-2853/20240419

jprestop commented 2 months ago

Thanks @hsoh-u Can you please confirm that the failed Check for Differences is expected?

hsoh-u commented 2 months ago

Thanks @hsoh-u Can you please confirm that the failed Check for Differences is expected?

Yes, three output files from GOES-16 inputs are different because of different logic for high, medium, and low confidences.

dtcenter / MET