Bugfix 2884 point2grid no qc

hsoh-u commented 1 month ago

Expected Differences

Removed duplicated StringArray flag_meanings Don't check QC flags without -qc option

[x] Do these changes introduce new tools, command line arguments, or configuration file options? [No]
If yes, please describe:
[x] Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
If yes, please describe:

Pull Request Testing

[x] Describe testing already performed for these changes:

./point2grid /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1,2

DEBUG 4: regrid_goes_variable() -> Count: actual: 997, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 10, by absent: 590193, total: 590203
DEBUG 4:    Range:  data: [-0.05 - 4.99997]  QC: [0 - 2]
DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=0 (4), medium=24 (87), low=62117 (62050), no_retrieval=10
DEBUG 4:    adjusted: high to medium=0, high to low=4, medium to low=63, total=67

./point2grid /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4

DEBUG 4: regrid_goes_variable() -> Count: actual: 999, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 0, by absent: 590193, total: 590193
DEBUG 4:    Range:  data: [-0.05 - 4.99997]  QC: [1e+11 - -1e+11]

[x] Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Same commends with above, but with different data files. Expected DEBUG 4: Filtered: by QC: 0, by adp QC: 0 without -qc option. The testing is checking the differences between -qc 0,1,2 and without -qc options. There will be no differences if no_retrieval=0 for both AOD and ADP qc flags. Above case have 10 no_retrieval at ADP QC flags. So the filtering results are different between -qc 0,1,2 and without -qc options.

[x] Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]
[x] Do these changes include sufficient testing updates? [No]
[x] Will this PR result in changes to the MET test suite? [No]
If yes, describe the new output and/or changes to the existing output:
[x] Will this PR result in changes to existing METplus Use Cases? [No]
If yes, create a new Update Truth METplus issue to describe them.
[x] Do these changes introduce new SonarQube findings? [No]
If yes, please describe:
[x] Please complete this pull request review by [5/14/2024].

Pull Request Checklist

See the METplus Workflow for details.

[ ] Review the source issue metadata (required labels, projects, and milestone).
[ ] Complete the PR definition above.
[ ] Ensure the PR title matches the feature or bugfix branch name.
[ ] Define the PR metadata, as permissions allow. Select: Reviewer(s) and Development issue Select: Milestone as the version that will include these changes Select: Coordinated METplus-X.Y Support project for bugfix releases or MET-X.Y.Z Development project for official releases
[ ] After submitting the PR, select the :gear: icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
[ ] After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
[ ] Close the linked issue and delete your feature or bugfix branch from GitHub.

jprestop commented 1 month ago

@hsoh-u What testing, if any, would you like the reviewer to perform? Could you please fill out this section of the template?

Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Also, could you please fill in the date in the template for this section?

Please complete this pull request review by [Fill in date].

hsoh-u commented 1 month ago

@hsoh-u What testing, if any, would you like the reviewer to perform? Could you please fill out this section of the template?

Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Also, could you please fill in the date in the template for this section?

Please complete this pull request review by [Fill in date].

Filled [Fill in date] and added comment

jprestop commented 1 month ago

Thank you @hsoh-u. Can you please point me to the version of point2grid that you are testing with? I see in your command ./point2grid and looked for /d1/personal/hsoh/git/pull_request/bugfix_2884_point2grid_no_qc, but that directory does not exist on seneca.

hsoh-u commented 1 month ago

I built it at /d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc

jprestop commented 1 month ago

Thank you, @hsoh-u. Please excuse my ignorance in testing this. When I test with the same case that you test, I get the same output whether or not I use -qc 0, -qc 0,1, or -qc 0,1,2:

/d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc/bin/point2grid  /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1,2

/d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc/bin/point2grid  /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1

/d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc/bin/point2grid  /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0

Output:

DEBUG 4: regrid_goes_variable() -> Count: actual: 0, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 62151, by absent: 590193, total: 652344
DEBUG 4:    Range:  data: [-0.05 - 4.99997]  QC: [0 - 2]
DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=0 (4), medium=24 (87), low=62117 (62050), no_retrieval=10
DEBUG 4:    adjusted: high to medium=0, high to low=4, medium to low=63, total=67

Is that the expected behavior?

hsoh-u commented 1 month ago

Yes for the same log messages regardless -qc x because they were counted before applying QC options.

DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=0 (4), medium=24 (87), low=62117 (62050), no_retrieval=10
DEBUG 4:    adjusted: high to medium=0, high to low=4, medium to low=63, total=67

/d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc/bin/point2grid /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4

DEBUG 4: regrid_goes_variable() -> Count: actual: 999, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 0, by absent: 590193, total: 590193
DEBUG 4:    Range:  data: [-0.05 - 4.99997]  QC: [1e+11 - -1e+11]

999 cells (out of 185 * 129 target grid cells) were filled with smoke data
590193 smoke data was filtered by absent flag
No Smoke data was filtered by QC flags
Note 1: missing count is for smoke data, not cells
Note 2: No above log messages

/d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc/bin/point2grid /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1,2

DEBUG 4: regrid_goes_variable() -> Count: actual: 997, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 10, by absent: 590193, total: 590203

997 cells (out of 185 * 129 target grid cells) were filled with smoke data
590193 smoke data was filtered by absent flag
10 Smoke data was filtered by -qc 0,1,2

/d1/personal/hsoh/git/pull_request/MET_bugfix_2884_point2grid_no_qc/bin/point2grid /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc G212 goes_aod_smoke_adp_high2.nc -adp /d1/personal/hsoh/data/MET-2853/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1

DEBUG 4: regrid_goes_variable() -> Count: actual: 6, missing: 2758918, non_missing: 652344
DEBUG 4:    Filtered: by QC: 0, by adp QC: 62127, by absent: 590193, total: 652320

6 cells (out of 185 * 129 target grid cells) were filled with smoke data
62127 Smoke data was filtered by -qc 0,1

jprestop commented 1 month ago

Thanks, @hsoh-u! I took another look and see that I somehow missed these differences in the output, but they do exist. I appreciate you spelling it out for me.

dtcenter / MET