Determine the best way to embed weather map information into NightWatch. Should we just embed another website or get the images and construct the movies ourselves.

[dyerlytle]

Weather Underground will stop providing an API for downloading data at the end of 2018.

Another possibility. Lowell Observatory can join the "Open Commons Consortium", "http://occ-data.org", which provides access to the GOES 16 satellite data. Might cost $50.

We can also join Amazon Web Services to get GOES 16 satellite data, I haven't figured the cost yet.

[astrobokonon]

@dyerlytle I'm pretty sure a good friend of mine was the one who got the OCC & NOAA collaboration going while she worked for U. Chicago/OSDC, so I'll check in with her to see if she has any suggestions. Not sure if they're really open to just being data providers since their focus (at least a few years ago) was more on computing and data processing.

[astrobokonon]

@dyerlytle This was her suggestion - https://registry.opendata.aws/unidata-goes/ or https://registry.opendata.aws/noaa-goes/.

It looks like Google Cloud has an equivalent https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16?pli=1, but I'll see if I can get a toy model going using one of the AWS source above.

[astrobokonon]

Toy model is now in https://github.com/LowellObservatory/Camelot/tree/master/GOESMcGOESface for playing around with.

Right now it's a working example that will list a specific band for the ABI-L2-CMIPC data of GOES-16. It's also time-since-present based, and will grab the last N hours worth of data.

ABI-L2-CMIPC:

• ABI is the instrument
• L2 is the processing level; L2 is the processed/derived stuff that we want
• CMIPC is the Cloud and Moisture Imagery Products for CONUS (rather than full disk or small/mesoscale regions)

Channel choice will probably be pretty easy, since we're targeting nighttime use; they're numbered in increasing central wavelength, so really only channels 7-16 will show anything after sunset since channel 7 is at ~3.9 microns and there's enough thermal emission by that point to see stuff. The fact sheets have more info.

The data are in netCDF format, which I haven't played with but it seems like there are simple enough tools out there in python nowadays to manipulate it. Seems like the CONUS files are at most ~5MB each, so it's not a killer thing to transfer either.

[astrobokonon]

Takes a few seconds to download a new image, and a few seconds for the first reprojection from the geosynchronous satellite view to this one (Lambert conformal conic). If you do it in a script over a bunch of files, only the first reprojection is a little slow and all further ones that match the area/extent parameters are faster because the calculation is cached automatically.

Final step is to add markers and roads for easier navigation within the image, but I think this is pretty sweet.

[astrobokonon]

@LowellObservatory/nightops here's what I can now produce now from the live/real time data of GOES-16 for any particular band/channel of the satellite. This particular example is for channel 13, the "Clean (LWIR)" band at 10.3 microns. I'm assuming that we don't actually care about the data/units and this is more of a visual thing - if there's a strong desire for actual valid brightness temperatures, now's the time to mention it.

The star is Lowell, and underneath of that are dots for the Mesa an the DCT respectively. The other purple/darker dots are Kitt Peak, Mt. Hopkins, and the LBT. I've got road data but haven't put it on the plots yet.

It takes just a few seconds to produce the above image from the raw data, so the idea would be to plot whatever we want ourselves rather than depend on stuff like this.

If you've got a favorite colormap, or think a zoomed in portion would be better, let me know. This particular colormap ("viridis" I think) was just the default one but any on this page could be applied quickly and easily.

[dyerlytle]

These plots look excellent! I think we should add the roads, at least the interstates. Next we need to determine exactly which bands or 'function of bands' to display, and then we'll need to set up a little movie of 10 or 12 frames to play in the Bokeh window. After that, on to radar data.....

[astrobokonon]

Initial feasibility version now complete; I added many roads. Since Lake Mary Rd. is in the same category as all the roads in Phoenix, to keep just the former I would have to spatially filter the road data...which is do-able, but probably overkill. A comparison of most roads vs. only interstates and federal roads is below.

Worth noting that plotting just the right image's set of roads is roughly 2x faster! That seals it for me.

The color map is probably the thing that needs the most attention for the next iteration. It almost certainly needs a divergent/nested map to differentiate ground vs. clouds, but then a more complicated color map interferes with map markers and road line color choices, and so on. TBD.

[dyerlytle]

Weather movies

Using OpenCV in Python, movies of sequences of images can be made. There are also various ways to make animated GIFs of a sequence of images. (These images were grabbed from the Grand Canyon Entrance Station web cam.). If we use a Bokeh server application for this, it might assemble a new animated gif every minute from the latest dozen or so weather images and show that in the DIV on the web page. (click on the image below to see the 46 frame animated GIF)

[astrobokonon]

Turns out that keeping things in data units (brightness temp.) is way easier because the color scale can than be physically defined by the average/usual range of ground temperatures (keeping them black/greyscale) and slapping a colored range on top of that to catch the clouds. That's pretty clever and works really, really well. I did this colormap by hand, but it makes sense; values are brightness temps (in Kelvin).

I'm sure some meteorologist somewhere is slapping their head saying "Duh..." but I'm an astronomer first, dammit, I spend most of my time trying to remove the Earth from my data.

[dyerlytle]

Animated GIFs in DIVs that update periodically

I'm liking the animated GIF idea for the weather maps. I find that I can make a DIV update itself periodically without reference to Bokeh, just by using some javascript. I can set a DIV to show an animated GIF of the weather map and have it update itself every minute so the newly created animated GIF gets loaded into the DIV. There would be a separate Python process running that would be making the new animated GIFs as the weather images are updated. We could even do this with plots but the bokeh server stuff is smoother for plots because the axes don't update, just the data. Example (DIV starts with Ocean Beach webcam image and then updates after 10 seconds to local animated GIF called "wave.gif". It then continues to update to the same animated GIF every 10 seconds (although this GIF is not changing))(I have this inserted into my Django website and it works great.):

<script>
      function refreshDiv() {
        document.getElementById("streamTitle").innerHTML =
      "<img src='/static/images/wave.gif'></img>" ;
      }

      window.setInterval(refreshDiv, 10000);
</script>

<div class="item weather item-h2" id="streamTitle">
      <img src='http://www.oceanbeach.org/images/sunset.jpg'></img>
</div>

Animated GIF called wave.gif:

[astrobokonon]

Latest images now available at: http://dctsleeperservice:8080/goes16/g16aws_latest.png

If you want to see all of them: http://dctsleeperservice:8080/goes16/pngs/

note: I updated these locations on 20181127 because of some necessary backend changes to make room for more containers/microservices like WebcamMcWebcamface

In the 'pngs' directory I haven't put a FIFO type setup into place yet, but that should be a relatively easy extension to get going to only keep the last X hours worth of data at any given time. As a stopgap, the "g16aws_latest.png" file should update every 5 minutes or so since the GOES CONUS data don't come any faster than that.

I'm also looking at creative ways to make an animated GIF or MP4 video; I've done the latter plenty of times and can just do what I always do (mencoder + ffmpeg on the command line), but I wanted to take a day to look at whether the situation has improved or not. Based on this StackOverflow conversation, imageio seems pretty promising but I haven't tried it out yet.

[dyerlytle]

Automatic updating of Animated GIFs using PIL, requests, image, glob, os, javascript.

I put together a little example of an updating animated GIF in "_proto/animate_gif". The three programs are described below:

A simple example of downloading images, constructing animated GIFs of recent data on a regular time interval, and displaying and updating these animaged GIFs in a web page.

"test-get.py" downloads images from a web cam in Bishop, California every ten seconds and puts them in the local directory. The files are named bishop00001.jpg and etc.

"update-gif.py" constructs an animated GIF from the most recent 30 images and puts it in "updateMovie.gif" every 60 seconds.

"showImage.html" is a very small web page that displays the above mentioned "updateMovie.gif" and updates itself every 60 seconds to grab the new movie. There is some code in there to fool the web browsers so they don't just reload the cached image. (When the web page is first loaded, it displays the most recent image from a web cam in ocean beach, CA, but soon updates to the most recent Bishop animated GIF.)

An example animated GIF from the Bishop web cam:

[astrobokonon]

@dyerlytle I made the GIF (and an MP4 movie, too) and I too ended up with a huge file to get a good size and/or good quality. Also put in a way to keep only the last 24h worth of data at any given time, but make GIFs of the last 6h.

Images will be about 300 KiB each, but a decent quality GIF clocked in at ~25 MiB. The video is smaller, by about a factor or 2-3. Ultimately I'm starting to wonder if we should look at some sort of javascript (probably a bokeh way, too) image sequence flipper that loads the much smaller individual frames and does it in-browser?

[astrobokonon]

GOESMcGOESface is up and running and stable for a period of time now approaching several months, so embedding the output of that is apparently the way we'll go first.

The only remaining thing to do is to figure out the image sequencer/display, since the GIFs are large to embed for all users and the movies are a irritating mess to make in a nice way with something simple/stable like FFMPEG.

I've moved on in my thinking from making movies, mainly because FFMPEG doesn't accept a list of files to turn into a movie, just wildcards. That means that if you want to make a movie of some subset of the time range, say the last X hours rather than the full 24-ish hour timerange I keep available, you have to copy/link the images to a new subdirectory, run FFMPEG (or mencoder, whatever) on that subdirectory with a wildcard, then delete it when you're done.

That's an awful lot of disk thrashing that just doesn't need to happen if we can just get an image gallery that updates/streams the latest images.

[astrobokonon]

Closing this since I think it's now clear that HAniS is the way to go, since it's built for this exact use case...and I've got it working already.