jeremyh
commented
1 year ago Running the extended tests:
The Landsat tests (which will will use the same fmask version) are all coming back pixel-identical.
Some of the extended S2 scenes (which are being bumped to the newer fmask) show some changes. I've included Fuqin's test scenes (which may be more a stress test?):
❯ ./cmp-fmask-outputs.py data-original-s2 data-devDifferences in S2B_MSIL1C_20210522T001109_N0300_R073_T55HEV_20210522T012307:
percent_class_distribution.clear changed from 95.13644248933763 to 95.13849954966967
percent_class_distribution.cloud changed from 0.8542606596700515 to 0.850670757606694
percent_class_distribution.cloud_shadow changed from 1.1232146624075343 to 1.1190939060649372
percent_class_distribution.snow changed from 0.08223928137194511 to 0.08375553390702313
percent_class_distribution.water changed from 2.803842907212843 to 2.807980252751667
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
11533 pixels (0.04%) differ with avg magnitude of 142.396861
Differences:
Clear -> Cloud: 38
Clear -> Cloud-Shadow: 3022
Clear -> Water: 347
Cloud -> Clear: 661
Cloud -> Cloud-Shadow: 540
Cloud -> Snow: 135
Cloud -> Water: 4
Cloud-Shadow -> Clear: 3111
Cloud-Shadow -> Cloud: 213
Cloud-Shadow -> Snow: 955
Cloud-Shadow -> Water: 1385
Snow -> Cloud: 7
Snow -> Cloud-Shadow: 626
Water -> Clear: 255
Water -> Cloud-Shadow: 234
Spatial distribution of differences:
| 1 | 15 | 2 | 1 | 0 | 3 | 1 | 3 | 1 | 8 |
| 869 | 1 | 5 | 5 | 2 | 2 | 3 | 3 | 6 | 3 |
| 3 | 7 | 5 | 1 | 2 | 1 | 4 | 7 | 10 | 11 |
| 9 | 7 | 3 | 6 | 7 | 7 | 10 | 6 | 31 | 14 |
| 6 | 9 | 1 | 4 | 3 | 12 | 11 | 19 | 147 | 7 |
| 3 | 11 | 7 | 4 | 15 | 10 | 11 | 7 | 9 | 5 |
| 11 | 6 | 1047 | 7 | 7 | 6 | 1 | 0 | 4 | 5 |
| 18 | 235 | 4603 | 3 | 3 | 2 | 2 | 0 | 0 | 1 |
| 49 | 601 | 3287 | 10 | 2 | 1 | 5 | 4 | 3 | 6 |
| 164 | 11 | 9 | 3 | 2 | 0 | 8 | 7 | 3 | 11 |Differences in S2A_OPER_PRD_MSIL1C_PDMC_20161204T065230_R002_V20161204T005702_20161204T005702:
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
Identical pixelsDifferences in S2A_MSIL1C_20211212T003701_N0301_R059_T54JWL_20211212T020702:
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
Identical pixelsDifferences in S2B_MSIL1C_20171208T003659_N0206_R059_T54JWL_20171208T034537:
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
Identical pixelsDifferences in S2B_MSIL1C_20230211T001109_N0509_R073_T55HEB_20230211T012907:
percent_class_distribution.clear changed from 98.91728073025052 to 98.9024341579673
percent_class_distribution.cloud changed from 0.17896562852128342 to 0.17791054337570458
percent_class_distribution.cloud_shadow changed from 0.632263023991097 to 0.6190044240618184
percent_class_distribution.snow changed from 0.018253046731016027 to 0.03446799176946136
percent_class_distribution.water changed from 0.2532375705060784 to 0.26618288282570046
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
6302 pixels (0.02%) differ with avg magnitude of 177.977309
Differences:
Invalid -> Clear: 1910
Invalid -> Cloud: 8
Invalid -> Cloud-Shadow: 7
Invalid -> Snow: 605
Invalid -> Water: 485
Clear -> Cloud-Shadow: 1380
Cloud -> Clear: 40
Cloud -> Cloud-Shadow: 2
Cloud-Shadow -> Clear: 1858
Cloud-Shadow -> Snow: 1
Cloud-Shadow -> Water: 6
Spatial distribution of differences:
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 300 | 324 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 2263 | 332 | 278 | 302 | 323 | 0 | 0 | 0 | 0 |
| 1027 | 0 | 0 | 0 | 0 | 15 | 341 | 276 | 285 | 236 |
Differences in S2A_MSIL1C_20220218T000241_N0400_R030_T55HFA_20220218T012533:
percent_class_distribution.clear changed from 96.68856015104845 to 96.8702856688648
percent_class_distribution.cloud changed from 0.5390742275411183 to 0.5393107698700319
percent_class_distribution.cloud_shadow changed from 1.1897554423673107 to 0.9909340471971183
percent_class_distribution.snow changed from 0.0015167239559081376 to 0.0015664999992532575
percent_class_distribution.water changed from 1.5810934550872149 to 1.5979030140687962
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
258278 pixels (0.86%) differ with avg magnitude of 103.800192
Differences:
Invalid -> Clear: 97
Invalid -> Cloud: 1
Invalid -> Snow: 14
Invalid -> Water: 23
Clear -> Cloud: 90
Clear -> Cloud-Shadow: 98401
Clear -> Water: 1
Cloud -> Clear: 15
Cloud -> Cloud-Shadow: 93
Cloud-Shadow -> Clear: 153266
Cloud-Shadow -> Cloud: 89
Cloud-Shadow -> Snow: 7
Cloud-Shadow -> Water: 5609
Snow -> Cloud-Shadow: 6
Water -> Cloud-Shadow: 566
Spatial distribution of differences:
| 7096 | 13 | 0 | 0 | 0 | 0 | 0 | 37700 | 46221 | 3050 |
| 146 | 1614 | 0 | 0 | 0 | 0 | 165 | 38235 | 53899 | 17468 |
| 3 | 0 | 0 | 0 | 0 | 0 | 837 | 14755 | 18320 | 4105 |
| 2062 | 963 | 0 | 0 | 0 | 0 | 0 | 0 | 500 | 771 |
| 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 1316 | 0 |
| 0 | 0 | 6221 | 0 | 88 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 1740 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| 0 | 0 | 982 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |(a previously-failing test case -- almost not real pixels)
Differences in S2B_MSIL1C_20230306T002059_N0509_R116_T55GCP_20230306T014524:
software_versions.eugl.version changed from 0.5.4 to 0.6.1
software_versions.fmask.version changed from 0.5.5 to 0.5.7
Identical pixels
uchchwhash
Note that the old code, when all fmask pixels were marked invalid, would place NaNs in all fmask metadata fields. I've kept this behaviour.