search

ioos / notebooks_demos

Notebook demonstrations and examples
https://ioos.github.io/notebooks_demos/
MIT License
19 stars 19 forks source link

Glider, Argo, and Hurricane Tracks for the 2020 hurricane Season #362

Closed ocefpaf closed 3 years ago

ocefpaf commented 3 years ago

Closes #361

nbviewer link: https://nbviewer.jupyter.org/github/ocefpaf/notebooks_demos/blob/HS20/notebooks/2020-11-18-Hurricane_Season-2020.ipynb

From https://github.com/ioos/notebooks_demos/pull/362#issuecomment-730499242

review-notebook-app[bot] commented 3 years ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

ocefpaf commented 3 years ago

@kbailey-noaa this is fetching both glider and storm data. The plot is a bit ugly b/c I want it "lightweight" for the animation tests, we can prettify it later.

The challenge now is to animate the hurricane as tracks and not points. It is not trivial b/c I need to do each line separate. I'm working on a better solution. Meanwhile, please take a look at the notebook and let me know if you have any feedback.

kbailey-noaa commented 3 years ago

@ocefpaf Thanks!! I see the data end on Nov 10. We should include Iota though, so need to add 1 more track. I asked Ben to comment here as well. Can you also include the Puerto Rico area? See Ben's image for extent: gliders 2020

benito334 commented 3 years ago

@ocefpaf This looks great! Is it possible to have the historical glider tracks and storm tracks disappear after a certain length of time? Say for gliders after 20 days and for the storms after 7 days?

benito334 commented 3 years ago

image

ocefpaf commented 3 years ago

@ocefpaf This looks great! Is it possible to have the historical glider tracks and storm tracks disappear after a certain length of time? Say for gliders after 20 days and for the storms after 7 days?

The short answer is yes but I never did something like that before. Looking into it...

ocefpaf commented 3 years ago

@ocefpaf Thanks!! I see the data end on Nov 10. We should include Iota though, so need to add 1 more track.

Extending it up to today then. We can re-run this as Iota develops.

I asked Ben to comment here as well. Can you also include the Puerto Rico area? See Ben's image for extent

Fixed in the last commit. Thanks!

kbailey-noaa commented 3 years ago

@ocefpaf @benito334 I was just staring at the code thinking about how I could replicate this in Arcmap and just discovered that in the first cell/block the start time is July 1 2020 but it should be June 1. ...but actually since the first storm (Arthur) was ~ May 16 should we move the start time up to around then?

benito334 commented 3 years ago

I recommend using Jun 1 as it is the date we use all our metrics for.

On Wed, Nov 18, 2020 at 3:12 PM kbailey-noaa notifications@github.com wrote:

@ocefpaf https://github.com/ocefpaf @benito334 https://github.com/benito334 I was just staring at the code thinking about how I could replicate this in Arcmap and just discovered that in the first cell/block the start time is July 1 2020 but it should be June 1. ...but actually since the first storm (Arthur) was ~ May 16 should we move the start time up to around then?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ioos/notebooks_demos/pull/362#issuecomment-729926377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH325NQD2EDOBQJ5WNNMJ6LSQQTDRANCNFSM4TXXKT5Q .

ocefpaf commented 3 years ago

I recommend using Jun 1 as it is the date we use all our metrics for.

OK, starting June 1st now. Changing the date range is fairly easy now. Still working on fading the past data... More on this later today.

kbailey-noaa commented 3 years ago

@ocefpaf Thanks! I see the animation. When do you want our feedback? I know this isn't prettified yet and you're working on the fading past data piece. Should we wait til then?

ocefpaf commented 3 years ago

@ocefpaf Thanks! I see the animation. When do you want our feedback? I know this isn't prettified yet and you're working on the fading past data piece. Should we wait til then?

If you have suggestions don't wait. Specially on making it prettier. (I'm terrible with that b/c I actually like the black and white coast and few lines/points.)

MathewBiddle commented 3 years ago

different colors for each glider?

benito334 commented 3 years ago

I would say no if we are able to pull in drifters and Argo.

On Thu, Nov 19, 2020 at 10:03 AM Mathew Biddle notifications@github.com wrote:

different colors for each glider?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ioos/notebooks_demos/pull/362#issuecomment-730433851, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH325NTSIQSII7MGR4FDJX3SQUXTXANCNFSM4TXXKT5Q .

kbailey-noaa commented 3 years ago

Ben, not entirely sure what you need vs what would be nice to have, plus I'm not clear on what is possible in this interface so I pose these as questions: Are we able to include a different basemaps? Speed up the animation? Should be attempt to replicate the glider groupings by color from Ben's image (NOAA/partner vs Navy vs Other mission)? Is it possible to label the different storm tracks somehow?

ocefpaf commented 3 years ago

I would say no if we are able to pull in drifters and Argo.

Argo should be OK to pull from ERDDAP in the same fashion we are getting glider. I agree with the colors, even fading the past data we will probably have more than 6 different instruments on the plot. And that is a "magic number" for how many colors we can distinguish in a plot.

benito334 commented 3 years ago

At this point anything better than a static image of gliders and storms is awesome. If we can add argo and drifts cool. If not, not a big deal. I'm attaching a kml of drifters that as time information.

Other thoughts Google earth type background, total animation time <30sec maybe, Each technology (glide, argo, drifter) should have their own group color. All gliders should be one color, all argo a different color etc Labeling storms would be very cool but not critical.

On Thu, Nov 19, 2020 at 10:56 AM kbailey-noaa notifications@github.com wrote:

Ben, not entirely sure what you need vs what would be nice to have, plus I'm not clear on what is possible in this interface so I pose these as questions: Are we able to include a different basemaps? Speed up the animation? Should be attempt to replicate the glider groupings by color from Ben's image (NOAA/partner vs Navy vs Other mission)? Is it possible to label the different storm tracks somehow?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ioos/notebooks_demos/pull/362#issuecomment-730468662, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH325NSUPZALI3CCUDNICPDSQU53RANCNFSM4TXXKT5Q .

kbailey-noaa commented 3 years ago

The Glider fade away looks great! @benito334 did you want to keep the storm tracks there the whole duration of the video? One idea is after the storm is 'done' the track disappears from view.

ocefpaf commented 3 years ago

The Glider fade away looks great! @benito334 did you want to keep the storm tracks there the whole duration of the video? One idea is after the storm is 'done' the track disappears from view.

I'm trying to implement the "storm fade away" too but that is a different beast. For the glider it is a trick with scatter plot and "aging" the old points with the alpha level, they are actually still there and it makes the animation a bit slower to create. However, the storms are lines, not scatter, making it harder know which is which and remove them correctly. Aging them would be awkward b/c ideally we would age/remove a full developed track and not as it progresses.

TL;DR it is not impossible and I have some idea on how to do it but it will take more time. So I'll spent the next iteration on https://github.com/ioos/notebooks_demos/pull/362#issuecomment-730499242.

kbailey-noaa commented 3 years ago

I didn't mean using the fade away trick for the storm track, though, but rather remove the whole track all at once at the end of it's life. Not sure if that clarification makes a difference, though! Just wanted to double check. If it costs processing time then might not be worth it.

What options do we have to beautify? Basemap seen in Ben's earlier posted image?

benito334 commented 3 years ago

The glider fading looks great! No need to age the storm tracks. I think just having them disappear completely after then are done would be fine.

ocefpaf commented 3 years ago

I didn't mean using the fade away trick for the storm track, though, but rather remove the whole track all at once at the end of it's life. Not sure if that clarification makes a difference, though! Just wanted to double check. If it costs processing time then might not be worth it.

Sorry. I was not clear before. The main issue to either remove the track once it is done or to fade it over time is b/c I don't track each line individually. They are plotted all together over the time axis. So, removing anything after 'x' amount of time would also erase "ongoing" storms. I have a few ideas to test before giving up...

What options do we have to beautify? Basemap seen in Ben's earlier posted image?

Pretty much anything we can get that is geo-referenced. I used cartopy's stock image in the last commit b/c it has some light colors and makes it easier to visualize things we plot on top of it.

This last commit also adds argo to the static plot. I'll animate it tomorrow.

PS: thanks @kbailey-noaa and @benito334! It has been a while that I had this much fun writing a notebook! Hopefully this will be useful.

kbailey-noaa commented 3 years ago

@ocefpaf I'm glad this has been a good experience!
Understood, re: disappearing tracks.

It's a little hard to actually see what's moving, so I wonder if we should reduce the size of the glider circles and/or increase the map dimensions overall to make everything a little easier to see. If I expand the map using the tool in the bottom right it's still a little fuzzy, I assume because of the resolution.
Are some of the orange dots not moving because they were removed / reached the end of their deployment? e.g. Miami area of Florida.

Not sure what kind of basemap @benito334 prefers. Do you want to leave as is or try some type of google-earthy/satellite type of basemap?

ocefpaf commented 3 years ago

@ocefpaf I'm glad this has been a good experience! Understood, re: disappearing tracks.

I did not give up. Just probably cannot make it in a short deadline.

It's a little hard to actually see what's moving, so I wonder if we should reduce the size of the glider circles and/or increase the map dimensions overall to make everything a little easier to see. If I expand the map using the tool in the bottom right it's still a little fuzzy, I assume because of the resolution.

Yep. Bigger plot and smaller circles helped a lot. Thanks! I added argo, so that really made a difference.

Are some of the orange dots not moving because they were removed / reached the end of their deployment? e.g. Miami area of Florida.

There is a bug in my code. I fixed one with the interpolation but there is still something odd that I could not figure out if it is real data or an artifact of my animation. I'll look closer frame-by-frame tomorrow.

kbailey-noaa commented 3 years ago

@ocefpaf @benito334 I like the background, personally. Looks good. I think the animation length is the next critical part. Ben, thoughts?

benito334 commented 3 years ago

@ocefpaf @benito334 I like the background, personally. Looks good. I think the animation length is the next critical part. Ben, thoughts?

Agree looks good! Is it possible to have the storm tracks fade after say 30 days? For speed, Is it possible to have two versions of the animation: 1) current speed 2) one that is 30s to 60s?

ocefpaf commented 3 years ago

Agree looks good! Is it possible to have the storm tracks fade after say 30 days?

Any kind of fading requires to track each one individually. I need to implement that in an efficient way. My first try blow up my laptop... Let me try again I think I have an idea on how to improve on that.

For speed, Is it possible to have two versions of the animation: 1) current speed 2) one that is 30s to 60s?

That is relatively easy and we can achieve it in two ways. Save a single speed and add the multiplier in the player (pretty much any modern player can do that), or we pre-save in the speed we want.

Here is a lower quality GIF at at 30 fps, which is ~20 s, for that movie:

animation-optimized

ocefpaf commented 3 years ago

Are some of the orange dots not moving because they were removed / reached the end of their deployment? e.g. Miami area of Florida.

Found the bug that led to that. The interpolation was "copying" the last point up to the end of the new time index. I limited it to 1 point. B/c we are using 6 hours interpolation the last glider/argo may actually be from 6 hours before, not bad for an animation but not OK for data analysis.

mwengren commented 3 years ago

@ocefpaf The glider animations look awesome! Will have to take a look at the code sometime to see what you did.

fgayanilo commented 3 years ago

@ocefpaf this is nice. will it be easy to add a time slider?

ocefpaf commented 3 years ago

@ocefpaf this is nice. will it be easy to add a time slider?

In gif or video format? No. We could re-do this as timestamped geojsons and put them on a slippy map. Quite hard at this point but definitely worth looking into it in the future That way we would have a timeslider. However, the video format will give you a progress bar, not the same thing but you can go forward/backward in the frames which kind of correspond to time.

@ocefpaf The glider animations look awesome! Will have to take a look at the code sometime to see what you did.

@mwengren don't tell anyone but I copied code from stackoverflow for that ;-p Jokes aside, it is cool not b/c of the animation per se but b/c of the frequency of the glider data. Argo is the same code but the frequency is lower, so not too appealing to look at. Glider gives you the "comet-like" animation.

MathewBiddle commented 3 years ago

I just tried to borrow some of this code to make an animation of ATN data. It looks like you have some local images you are using for cartopy. I can't seem to find where you download the file locally. Error message below: [Errno 2] No such file or directory: 'C:\\Users\\Mathew.Biddle\\cartopy_stockimages\\HYP_LR_SR_OB_DR.tif which can be downloaded from here, specifically the Cross Blended Hypso with Relief, Water, Drains, and Ocean Bottom.

kbailey-noaa commented 3 years ago

@ocefpaf I was looking at the animation in https://nbviewer.jupyter.org/github/ocefpaf/notebooks_demos/blob/HS20/notebooks/2020-11-18-Hurricane_Season-2020.ipynb and noticed at time 1:12 a track originates from land and then tracks into the ocean and doesn't disappear. I know you're still working on it but wasn't sure if you noticed that. -- it's Tropical storm Omar

kbailey-noaa commented 3 years ago

@ocefpaf I wanted to share the animation with some people. Do I point them to the above animation or the one here: https://nbviewer.jupyter.org/github/ocefpaf/notebooks_demos/blob/HS20/notebooks/2020-11-18-Hurricane_Season-2020.ipynb?

ocefpaf commented 3 years ago

@ocefpaf I wanted to share the animation with some people. Do I point them to the above animation or the one here: nbviewer.jupyter.org/github/ocefpaf/notebooks_demos/blob/HS20/notebooks/2020-11-18-Hurricane_Season-2020.ipynb?

The one in that link is the "most updated one." And if I push some changes it will be updated too. PS: I do have some updates to push, I'll try to send then ASAP.