Closed lindsayplatt closed 1 year ago
Plot timeseries of sediment percent using cloud percent for alpha to dull those with high cloud coverage.
Removing days with greater than 50% clouds did not alter the trends we were seeing before:
COOL!
I think this will get much much better with the new models. At the very least LS 5/7, 8/9, then Sentinel will be in separate models in hopes to account for the radiometric resolution (and slight band wavelength) differences.
Terrific news - that's exactly what I was hoping to hear. Wanted to get rolling on some code and exploration to start on our end!
Dumb question that I can probably answer with google...
HOW DO YOU MAKE THOSE FANCY 'code to recreate plot' THINGS?????
Surround your code with these:
<details>
<summary>THE TEXT YOU SEE</summary>
THE STUFF YOU WANT TO HIDE HERE (an empty line between the HTML is important if you have other markdown syntax)
</details>
That looks like this:
Wondering if monthly precip has similar dip in the 00s ... it is not as obvious.
Not obvious, but kinda looks like it's there??? Like, if you squint?!
Trying to see if there are trends for individual storm events. Using the 95th percentile daily flow to represent an intense rainfall event for now (inspired by EPA's technical guidance for stormwater control).
If you look at trends by season and HUC for the "intense rainfall" events, I see a dip in the 00s during summer which may help explain that dip in sediment trends ~since we have more summer dates than other times~ (EDIT: this is not true! They are all fairly equal). I will explore sediment trends by season next.
Lining up the sediment trends by season:
Initial correlation between sediment and precip for the known bloom dates.
There are ~1100 dates from the Landsat data between 1984 and 2023. When we match up the 33 known bloom dates (using a 3-day window for each bloom date), we only get 13 matches from the Landsat data. The correlation between precip and sediment area on those 13 dates results in a negative relationship. The more recent updates from https://github.com/rossyndicate/Superior-Plume-Bloom/pull/64, resulted in a slightly less negative relationship (right figure).
Left fig = earlier model from quick_gradientTreeBoost_landsat_stack_v2023-07-06.csv
file.
Right fig = most recent model from GTB_by_LS_mission_pass_v2023-07-25.csv
.
I adjusted the approach above slightly, so that the sediment values for each bloom date included 3 days ahead AND 3 days after. Barely any correlation but NOT NEGATIVE! The sediment values plotted along the y axis
are the average sediment percentages across the 7-days surrounding a bloom (averaged only when we had multiple sediment values that fell into the window).
Still waiting to really dig in and scrutinize these results after the shoreline contamination pieces are removed (https://github.com/rossyndicate/Superior-Plume-Bloom/issues/63) and looking forward to the Sentinel data to get added, not just Landsat (https://github.com/rossyndicate/Superior-Plume-Bloom/issues/61).
Suhweet.
Just so you have an idea of timeline, I'm working on the Sentinel right now. I have some output, but even though the accuracy was very high for Sen2, the applied data look absolutely terrible (labeled as mostly clouds). I'm hoping it was just a fat finger somewhere along the way. Hoping to un-stuck it by EOD. I'll include the shoreline contamination summaries after I get the Sen2 stuff figured out.
Something to keep in mind (and I haven't dug in to your code to look and see if you've dealt with this at all) not all of the bloom observations are within our AOI. Some are in strange bays on the North shore, some are in the harbor, etc. Just something to think about as you're playing with the obs data.
Another couple of thoughts: I'm not super familiar with flow patterns, but maybe also look out at 7 days prior cumulative precip? And/or only including 'storm events' in the cumulative data which I think is >2cm over 24h?
not all of the bloom observations are within our AOI
Ahhhh I had not taken that into account. Thanks for mentioning it!
also look out at 7 days prior cumulative precip
I had done this and 5 days prior without any noticeable difference.
only including 'storm events' in the cumulative data which I think is >2cm over 24h
Hmm now this is interesting. I used this sort of approach when just exploring precip trends a few comments above (see https://github.com/LimnoDataScience/plume_bloom_drivers/issues/11#issuecomment-1650062482) so I could pretty easily apply that here! Lemme see what I can do in this vein.
Doesn't seem like including "storm only" values changes things much but I did try it with 1, 3, 5 and 7 days ahead. I think I am going to explore individual HUC contributions next.
Here I am using the new data (see https://github.com/rossyndicate/Superior-Plume-Bloom/pull/67) and using the columns *_perc_extent_area
when extent == 'aoi no sc'
values, meaning the sediment values used are percentages of the AOI without any shoreline contamination.
Results using 3-, 5-, and 7-day ahead cumulative storm precipitation values.
Exploring sediment trends using percent extent without sediment contamination!
Precip-Q relationship check. Not looking great :( But maybe using flow instead of precip in the sediment correlations will help. EDIT: added scales="free_y"
and redid the plots. Also, there will be a lag between precip and flow, especially in bigger watersheds + snowmelt might be throwing these relationships off. Will explore that next.
Checking out sediment occurrence and discharge relationship since the precip-Q relationship is not as we expected. The plots display the river name for the HUC10 and NWIS site. They do match the arrangement of the previous correlation plots that display the HUC10 names.
All the rasters have exported, so I'm going to make those annual stacks that we talked about before and get them up on HydroShare (hopefully today). I'm guessing some sort of localized summary (of the area around the mouth of these rivers) will be more representative than the gigantic AOI we are currently using. We could probably start with just a buffer or 1000m or so around the point where these drain into Superior. I'd be surprised if we didn't see any patterns with a more localized approach.
Exploring sediment trends using percent extent without sediment contamination!
Data-processing code for these plots
Suggestion: Consider also limiting your plots to some percentage of aoi that was classified.
For precip:Q -
Could you break these out by season to see if there is a better/different relationship that could account for varying hydrologic processes?
Ha - you're jumping ahead :) Up in this comment (https://github.com/LimnoDataScience/plume_bloom_drivers/issues/11#issuecomment-1671709417) I've got a note about snowmelt potentially causing the funny relationship. I am jumping onto that soon :)
As for limiting to some percentage of aoi that was classified in those precip plots: I understood the *_perc_extent_area
columns to mean the classifiable AOI? For example, I am using extent == 'aoi no sc'
and assumed my percentage is now based on an area equal to the classifiable AOI minus sediment contamination. Is that right?
Excited to bring the rasters back into the workflow. So cool you figured something out with the export to make this reasonable! I agree that having some sort of region at the mouth of each river will be better. I wonder if we can actually statistically link cells to each river based on some kind of clustering algorithm using the sediment timeseries and the discharge?
Ha - you're jumping ahead :) Up in this comment (#11 (comment)) I've got a note about snowmelt potentially causing the funny relationship. I am jumping onto that soon :)
Your comment is exactly why I suggested it - I'm so curious.
As for limiting to some percentage of aoi that was classified in those precip plots: I understood the
*_perc_extent_area
columns to mean the classifiable AOI? For example, I am usingextent == 'aoi no sc'
and assumed my percentage is now based on an area equal to the classifiable AOI minus sediment contamination. Is that right?
This is right, I'm mostly thinking about limiting the input summaries to those that have 50% of the total area classified. That would be using the column 'perc_area_classified'. There are many images that only have a small portion of the AOI classified at all - each image date doesn't cover the same area.
Excited to bring the rasters back into the workflow. So cool you figured something out with the export to make this reasonable! I agree that having some sort of region at the mouth of each river will be better. I wonder if we can actually statistically link cells to each river based on some kind of clustering algorithm using the sediment timeseries and the discharge?
I honestly have no idea how I made it so much more efficient. Maybe it has something to do with the .geo column, because that's gone in the previous PR update too, and I have no idea why.
I would think about this even more simply, since the sediment all kind of becomes a single feature (except that dark near shore sediment, which only happens consistently in 1 of these 4 streams, I think) and clustering wouldn't necessarily work. I'd literally, for a first pass, just use a single point with a 1000m buffer and summarize the data in that area. We can get more nuanced if we start to actually see patterns emerge.
Ahhhh yes, OK. I see what you're saying now. 758 rows were dropped when I added filter(perc_extent_classified >= 50)
.
Takes out a smidge of noise!
And this is my fault for not providing good metadata for you. This is what happens when I'm rushing. ;)
Oh no worries! We're getting there. I'm also not doing code reviews like you have done right now, which is where you would probably suggest something like that. Wanted to get a handle on where we were going first but probably should have transitioned by now.
Alrighty, let's look at Precip-Q by season. This makes more sense to me
A million more times sensical!
And no worries about code review - when doing a lot of basic exploratory work, this is almost better on my end.
I've been exploring the lags between discharge and precip for each site. Based on this analysis:
Q
is more correlated with PPT
up to 3 days agoQ
is more correlated with PPT
up to 2 days agoQ
is correlated to the current day's PPT
Q
is more correlated with PPT
up to 5 days agoNext, I am going to adjust my previous precip-Q relationships.
It doesn't seem like adding the lags really made a huge difference. The storm filter made a bigger impact. If we were summing, maybe it would. I think using the lags in our analysis as we continue is still a good idea though because it is a characteristic of basin shape/size and also stream gage placement.
What I did was shift the Q to match with the precip from {lag}
days earlier and then plot Q vs PPT. Filtered to only precip storm days, too. Note that Siskiwit plots should not change when lag is added because their lag = 0.
Closed by #12
Setting this up as an exploration using the initial Yugo model output from July 6, which does not have shoreline contamination filtered. I think we should wait and see how these look with @steeleb's improved model output, but at first glance, the trend is maybe not what I would expect: decreases in sediment from 1980 to early 2000s but there is an increase starting around 2010 which is when bloom sightings started to pick up.
I'm not sure how much of the sediment in the 80s and 90s is influenced by lower resolution and quality on the satellites - you can see a difference in how LS5 and LS7 report sediment during their overlapping years in the 00s.
Below are some plots summarizing the percent sediment (% of the classifiable area that was sediment) and sediment area in hectares broken out by year and by mission. Not really much difference between using the % classifiable area and the actual sediment area.
Code to create first set of plots
```r library(tidyverse) yugo <- read_csv('quick_gradientTreeBoost_landsat_stack_v2023-07-06.csv') %>% # Keep only the percentages for sediment select(mission, date, light_nearshore_pct = lightNSSed_perc, dark_nearshore_pct = darkNSSed_perc, offshore_pct = offShoreSed_perc) %>% mutate(sediment_total_pct = light_nearshore_pct + dark_nearshore_pct + offshore_pct, year = year(date), year_mission = sprintf('%s_%s', year, mission)) p1<-ggplot(yugo, aes(x=date, y=sediment_total_pct)) + geom_smooth(color = 'grey') + geom_point(alpha = 0.5, aes(color = mission)) + theme_bw() + ylab('Daily % sediment') + xlab('Date') + ggtitle('Daily timeseries of % sediment') p2<-ggplot(yugo, aes(y=sediment_total_pct, x=year, group=year_mission, fill=mission)) + geom_boxplot(position = position_dodge(preserve = "single")) + theme_bw() + ylab('Daily % sediment') + xlab('Year') + ggtitle('Boxplots per year per mission for daily % sediment') p3<-ggplot(yugo, aes(y=sediment_total_pct, x=year, group=year)) + geom_boxplot(fill = 'grey') + theme_bw() + ylab('Daily % sediment') + xlab('Year') + ggtitle('Boxplots per year for daily % sediment') p4<-yugo %>% group_by(year) %>% summarize(avg_sediment = mean(sediment_total_pct)) %>% ggplot(aes(x = year, y = avg_sediment)) + geom_smooth(color = 'grey') + geom_point() + theme_bw() + ylab('Average annual % sediment') + xlab('Year') + ggtitle('Annual timeseries of % sediment') cowplot::plot_grid(p1, p4, p2, p3, nrow = 2, ncol = 2) ```Code to create second set of plots
```r library(tidyverse) # What if we did the same but with the `_ha` instead of %? yugo2 <- read_csv('quick_gradientTreeBoost_landsat_stack_v2023-07-06.csv') %>% # Keep only the areas for sediment select(mission, date, light_nearshore_ha = lightNSSed_ha, dark_nearshore_ha = darkNSSed_ha, offshore_ha = offShoreSed_ha) %>% mutate(sediment_total_ha = light_nearshore_ha + dark_nearshore_ha + offshore_ha, year = year(date), year_mission = sprintf('%s_%s', year, mission)) p1<-ggplot(yugo2, aes(x=date, y=sediment_total_ha)) + geom_smooth(color = 'grey') + geom_point(alpha = 0.5, aes(color = mission)) + theme_bw() + ylab('Daily sediment, ha') + xlab('Date') + ggtitle('Daily timeseries of sediment area') p2<-ggplot(yugo2, aes(y=sediment_total_ha, x=year, group=year_mission, fill=mission)) + geom_boxplot(position = position_dodge(preserve = "single")) + theme_bw() + ylab('Daily sediment, ha') + xlab('Year') + ggtitle('Boxplots per year per mission for daily sediment area') p3<-ggplot(yugo2, aes(y=sediment_total_ha, x=year, group=year)) + geom_boxplot(fill = 'grey') + theme_bw() + ylab('Daily sediment, ha') + xlab('Year') + ggtitle('Boxplots per year for daily sediment area') p4<-yugo2 %>% group_by(year) %>% summarize(avg_sediment = mean(sediment_total_ha)) %>% ggplot(aes(x = year, y = avg_sediment)) + geom_smooth(color = 'grey') + geom_point() + theme_bw() + ylab('Average annual sediment, ha') + xlab('Year') + ggtitle('Annual timeseries of sediment area') cowplot::plot_grid(p1, p4, p2, p3, nrow = 2, ncol = 2) ```