search

USGS-R / mda.lakes

Wisconsin Lake Modeling Aggregation
2 stars 11 forks source link

2016-03-25 validation data issues (depth of water temp) #88

Open jordansread opened 8 years ago

jordansread commented 8 years ago

related to #87 , but we have seemingly errant depth data the is coming from the Water Quality Portal.

After transforming the units, here are the distributions of the data:

depth.units   p.25   p.50     p.75  count
        (chr)  (dbl)  (dbl)    (dbl)  (int)
1        feet 1.8288 5.4864 10.02792  57365
2          ft 0.9144 0.9144  1.82880  93566
3          in 0.2540 0.3810  0.45720     27
4           m 1.0100 4.0590  9.00000 730459
5      meters 3.0000 8.0000 16.00000  53368
6          mm 0.0130 0.0130  0.01300      1

note that the depth.units is the original unit, and the values in the p.25, p.50, etc are the transformed measurements, all in m

A couple of things could be going on here:

  1. certain agencies/groups that use certain units may have errant data in here
  2. certain groups may use a certain unit (ft for example) and collect mostly surface measurements
  3. bad data is spread across agency identifiers and units randomly, or follows some kind of pattern we can use to ultimately make a judgement call to remove certain data.

Will be updating this issue w/ progress.

@lawinslow noted this difference between data coming from the WI DNR, and the WI section of the WQP: WQP image

DNR image

jordansread commented 8 years ago 
group_by(data.in, OrganizationFormalName) %>% summarize(unique.units=length(unique(ActivityDepthHeightMeasure.MeasureUnitCode)), count=length(ActivityDepthHeightMeasure.MeasureUnitCode)) %>% arrange(desc(count)) %>% data.frame
                                                 OrganizationFormalName unique.units  count
1                                    Minnesota Pollution Control Agency            3 692498
2                             Wisconsin Department of Natural Resources            4 461730
3                                   USGS Wisconsin Water Science Center            3  87138
4                        National Park Service Water Resources Division            3  20627
5                                    USGS Michigan Water Science Center            2  20059
6                          Michigan Department of Environmental Quality            2  11219
7                                             Grand Portage Reservation            1   6290
8                                     Fond du Lac Band of Chippewa (MN)            1   6233
9                                   USGS Minnesota Water Science Center            3   6158
10                                           Bay Mills Indian Community            2   3030
11 Lac Du Flambeau Band of Lake Superior Chippewa Indians Water Program            3   2795
12                       Forest County Potawatomi Community (Wisconsin)            2   2064
13                                  Little River Band of Ottawa Indians            1   1669
14                                EPA National Aquatic Resources Survey            1   1586
15                     MATCH-E-BE-NASH-SHE-WISH BAND OF POTAWATOMI (MI)            2   1477
16                                     The Stockbridge-Munsee Community            2   1232
17                       FOREST COUNTY POTAWATOMI COMMUNITY (Wisconsin)            2   1099
18                                                         Michigan DEQ            1    932
19                          GRAND PORTAGE Band of CHIPPEWA INDIANS (MN)            1    886
20                             STOCKBRIDGE-MUNSEE COMMUNITY (Wisconson)            2    811
21   Pokagon Band of Potawatomi Indians Department of Natural Resources            1    493
22                                                                 DENR            3    331
23                                        Prairie Island Community (MN)            1    234
24                         SD Dept of Environmental & Natural Resources            2    181
25                      MENOMINEE INDIAN TRIBE OF WISCONSIN (Wisconson)            2    162
26                                                         Red Lake DNR            1    122
27                                ONEIDA TRIBE OF WISCONSIN (Wisconsin)            1     54
28                              21IOWA-Iowa Dept. of  Natural Resources            1     49
29                              Sokaogon Chippewa Community (Wisconsin)            1     45
30                            Saginaw Chippewa Planning Department (MI)            1     17
31                                     Iowa Dept. of  Natural Resources            1     13
32                                    Lower Sioux Indian Community (MN)            1      7
33                               USGS North Dakota Water Science Center            1      7
34           Lac Du Flambeau Band of Lake Superior Chippewa Indians DNR            1      6
35                                     USGS Kansas Water Science Center            1      6
36                                                  GLEON Lake Observer            1      5
37         Sisseton-Wahpeton Sioux Tribe Lake Traverse Reservation (SD)            1      5
38                                    USGS Georgia Water Science Center            1      2
lawinslow commented 8 years ago

USGS Wisconsin Water Science Center 6 87138 Ha thought for sure the different units would be from an agency.

jordansread commented 8 years ago

I had screwed up on that, fixed

lawinslow commented 8 years ago

Ah, makes more sense.

jordansread commented 8 years ago

@lawinslow why are we seeing different things w/ the histograms? image

note this is all three states, not just WI.

lawinslow commented 8 years ago

Probably because you're looking at all three states? The comparison I showed was just overlapping Wilma lakes to aid the contrast (didn't really know what to expect). If you look at all maybe the bad Vals are diluted. I'll re-visit my question from yesterday, how many bad depth Vals do we need to cause an issue with rmse? 5%?

jordansread commented 8 years ago

Looks like "Wisconsin Department of Natural Resources" is a bit odd: image

jordansread commented 8 years ago

Focusing on WI-DNR, because of the large amount of near surface data.

Those data seem a little bit off for what we would expect in June: image

July: image

jordansread commented 8 years ago

vs image

lawinslow commented 8 years ago

Whoa, those low temperature surface values in the "ft" data seem pretty bad. Could we have stream data in there?

jordansread commented 8 years ago

WI DNR apparently uses ResultDepthHeightMeasure.MeasureUnitCode instead of ActivityDepthHeightMeasure.MeasureUnitCode

see: image

jordansread commented 8 years ago

Minnesota Pollution Control Agency uses ActivityDepthHeightMeasure.MeasureValue instead: image

lawinslow commented 8 years ago

That's such an awesome figure.

jordansread commented 8 years ago

All data after flipping between result and activity depth based on agency: image

(for all time)

jordansread commented 8 years ago

now the data look as we would expect: image

jordansread commented 8 years ago

In the end, this is what the problem looked like coming from WQP:

OrganizationFormalName ActivityDepth.count ResultDepth.count
1                             Wisconsin Department of Natural Resources               88969            356605
2                                     Fond du Lac Band of Chippewa (MN)                   0              6230
3  Lac Du Flambeau Band of Lake Superior Chippewa Indians Water Program                2037               758
4                      MATCH-E-BE-NASH-SHE-WISH BAND OF POTAWATOMI (MI)                1157               299
5                                                          Red Lake DNR                   0               122
6                                                   GLEON Lake Observer                   0                 5
7                                                          Michigan DEQ                   0                 1
8                               21IOWA-Iowa Dept. of  Natural Resources                   0                 0
9                                            Bay Mills Indian Community                2615                 0
10                                                                 DENR                 152                 0
11                                EPA National Aquatic Resources Survey                1585                 0
12                       Forest County Potawatomi Community (Wisconsin)                2030                 0
13                       FOREST COUNTY POTAWATOMI COMMUNITY (Wisconsin)                 800                 0
14                          GRAND PORTAGE Band of CHIPPEWA INDIANS (MN)                 886                 0
15                                            Grand Portage Reservation                6290                 0
16                                     Iowa Dept. of  Natural Resources                   0                 0
17           Lac Du Flambeau Band of Lake Superior Chippewa Indians DNR                   6                 0
18                                  Little River Band of Ottawa Indians                1669                 0
19                                    Lower Sioux Indian Community (MN)                   0                 0
20                      MENOMINEE INDIAN TRIBE OF WISCONSIN (Wisconson)                 160                 0
21                         Michigan Department of Environmental Quality                1643                 0
22                                   Minnesota Pollution Control Agency              691682                 0
23                       National Park Service Water Resources Division               20606                 0
24                                ONEIDA TRIBE OF WISCONSIN (Wisconsin)                   0                 0
25   Pokagon Band of Potawatomi Indians Department of Natural Resources                 493                 0
26                                        Prairie Island Community (MN)                 234                 0
27                            Saginaw Chippewa Planning Department (MI)                   0                 0
28                         SD Dept of Environmental & Natural Resources                   9                 0
29         Sisseton-Wahpeton Sioux Tribe Lake Traverse Reservation (SD)                   0                 0
30                              Sokaogon Chippewa Community (Wisconsin)                  45                 0
31                             STOCKBRIDGE-MUNSEE COMMUNITY (Wisconson)                 118                 0
32                                     The Stockbridge-Munsee Community                 867                 0
33                                    USGS Georgia Water Science Center                   0                 0
34                                     USGS Kansas Water Science Center                   0                 0
35                                   USGS Michigan Water Science Center               18914                 0
36                                  USGS Minnesota Water Science Center                5063                 0
37                               USGS North Dakota Water Science Center                   0                 0
38                                  USGS Wisconsin Water Science Center               86756                 0
jordansread commented 8 years ago

@jkreft-usgs you aware of this issue w/ some WQP profile data? What is the proper field for measurement depth in WQX?