Water Quality Graphs do not match Old Site

[hertelc]

I am comparing sites between the old Streamwebs site and the staging site. In reviewing the Water Quality graphs, there are some discrepancies in the site I'm looking at: "Wenaha River".

Staging: http://streamwebs-staging.osuosl.org/sites/wenaha-river/water/ Old: http://drupal.streamwebs.org/water-quality-data-graphing/?site=2686

1. Temperature graph: a. Old site is in C but the new graph shows F.
   b. The datapoints do not match up perfectly. i. Point for 10/16/13: Old site: 10.5; Staging: 12.75 ii. Point for 10/8/2014: Old site: 23.65 Staging: 53.285 c. Water Temperature is not graphed
2. Other graphs - datapoints do not match (4 vs. 1)
3. Should Salinity even show since there is no data? Showing Zero implies it was measured as such.

[hertelc]

@ramereth FYI. This needs attention.

[ramereth]

The data in the database seems to be correct so I suspect there is some logic error in the views somewhere.

[jerry-peng]

@hertelc @ramereth I've found the issues. Basically the front-end process the data from the samples, and convert the data into decimals and average them. Whenever a sample is not used, the data is "None" and mess up the decimals average. I fixed that. However, after processing the data, the front-end filters the data, so most of the 0s get filtered (as discussed with Renee before, not a problem). And after that, the front-end filters the outliers, and it causes problem because when the number of data sets are too small, all of them seem to get filtered since they are all "outliers". Should we delete this portion of the code?

As for the questions:

1. a, It was in Celsius but Renee wanted it to be Farenheiht so I converted them. This can be switched back. b, Due to the outlier filter function, the data set does not align. The imported data set is the same though. i. ii. Both are the same issue, the old site uses Celsius, while the new site uses Farenheiht. c. Water Temperature is not graphed because all data got filtered by the outlier filter function.

2. Filter function caused it

3. During past meetings, Renee told us to filter out those data because a lot of data from the old site has 0 for some reason (no option of "none"?), and those are considered to be garbage data so Renee wants it filtered out.

[hertelc]

@JerryPeng0112 thanks for your assessment! For item 3, is there a way to do a data pull to see how many are zeros that may be garbage? Tracy and Cait think that if there is a zero it should be graphed vs. an n/a or blank value.

[hertelc]

Why did the water temp get filtered out via the outliers?

[hertelc]

@JerryPeng0112 can you give me an example and the logic for the filter that I can share with Streamwebs folks to help them understand what is going on (and me, frankly). :)

[jerry-peng]

@hertelc I just make a PR. This fixes the issues of graphing. It turns out the this is caused by data processing function. In addition, I included Bailey's fix for outlier filter to return data when there is only 1 data set.

The outlier function's logic is as follows:

1. sort in ascending order.
2. split the data into 2 equal half, called top and bottom
3. get the median of bottom, called first quartile, and get median of top, called third quartile.
4. Take the difference between first quartile and third quartile, called irl
5. values outside of range (first quartile - 1.5 iqr) ~ (third quartile + 1.5 iqr) will be filtered.
6. Average data from the same day.

On the point of zero data, we can do it but it is hard to judge which ones are garbage data and which ones aren't. Ex: I've seen pH 0 before, or water temperature 0 degree Celsius, which is impossible. Th fix does not include changes to the zero filtering function

[hertelc]

This looks good in staging. Closing