Cannot download (all) satellite images! (again)

[vacovo]

Hello, somewhere around Dec 4. cloudmap ceased to work, failing with error: Cannot download (all) satellite images! How can I make it work again?

[jmozmoz]

Yes, I know: A satellite stopped operation: https://www.ssec.wisc.edu/datacenter/mtsat_sched.html http://www.data.jma.go.jp/mscweb/notice/termination_201510.html

There is another source of satellite images: https://www.ssec.wisc.edu/data/ but unfortunately, the images have coast lines added. So I will search on a method to remove these and then try to replace the old download source. Also there seems to be no clear way to download images at of a certain time. You only get the latest.

[hallsofmirrors]

When you pull the gif's for the himawari sat from the server, use the fact that image band 9 has blue coastlines in full disk view (or any image band with cyan coastlines in west pacific view), and use imagemagick to create an alpha mask and interpolate, like so:

convert latest_himawari08_09_fd.gif -alpha set -fuzz 00% -transparent blue temp.png
convert temp.png -alpha extract coastmask.png
convert temp.png -mask coastmask.png -blur 7x65535 +mask -alpha off output.png

This masking should possibly be redone every time images update in case the coastlines drift between images, I'm not sure whether this will happen. I would expect the coast to be the same on different channels, if you need them, though. Thanks for keeping this awesome software up to date, I look forward to the maps updating properly soon

edit: Magic number 7x65535 seems to work better than original 1x65535 so pixel clusters in the coastline can blur to their centers. For other ir bands:

convert latest_himawari08_09_fd.gif -alpha set -fuzz 00% -transparent blue temp.png
convert temp.png -alpha extract coastmask.png
convert latest_himawari08_16_fd.gif -mask coast.png -transparent black temp.png
convert temp.png -mask coastmask.png -blur 7x65535 +mask -alpha off output.png

[jmozmoz]

Thank you for your help. I have already experimented with masking for coastlines in the branch for polar satellites: https://github.com/jmozmoz/cloudmap/tree/polar

Do you know how to do this without imagemagick but with pillow or opencv?

I will start looking into this from beginning this weekend.

[hallsofmirrors]

This code works, it's hacky but it gets the job done and I don't see any ghost coastlines.

First, get the mask by finding pixels that are blue and masking them (only works in image filter 09)...

import math
from PIL import Image
img = Image.open('latest_himawari08_09_fd.gif').convert('RGB')
mask = Image.new('RGB',(1100,1100),'black')
data = img.getdata()
maskdata = []
for i in data:
    if i[0] == 0 and i[1] == 0 and i[2] == 255:
        maskdata.append((255,255,255))
    else:
        maskdata.append((0,0,0))
mask.putdata(maskdata)
globe = Image.open('latest_himawari08_16_fd.gif').convert('RGB')                if card[++ni] and (maskdata[x-i,y+i] != (255,255,255)):

Now use the Image.load() to get pixel maps for the mask and for the alternate cloudmap you're using. For every pixel that is masked off, the code looks outward in 8 directions until it finds at least 1 pixel that isn't masked off, and copies them

pixdata = globe.load()
maskdata = mask.load()
for y in xrange(globe.size[1]):
    for x in xrange(globe.size[0]):
        if maskdata[x,y] == (255,255,255):
            n=0
            avg=0.0
            # handle the improbable condition where the search tries to look off the map
            rnge = 1099-y
            rnge = min(rnge, y)
            rnge = min(rnge, 1099-x)
            rnge = min(rnge, x)
            diag = math.sqrt(2)
            # card remembers whether we're looking for a pixel still
            card = True
            for i in xrange(1,rnge):
                ni = -1
                if maskdata[x+i,y] != (255,255,255):
                    avg += pixdata[x+i,y][0]
                    n += 1
                    card = False
                if maskdata[x-i,y] != (255,255,255):
                    avg += pixdata[x-i,y][0]
                    n += 1
                    card = False
                if maskdata[x,y+i] != (255,255,255):
                    avg += pixdata[x,y+i][0]
                    n += 1
                    card = False
                if maskdata[x,y-i] != (255,255,255):
                    avg += pixdata[x,y-i][0]
                    n += 1
                    card = False
                # now check diagonals... weight weights the average
                weight = i / diag
                diag += math.sqrt(2)
                if maskdata[x+i,y+i] != (255,255,255):
                    avg += pixdata[x+i,y+i][0] * weight
                    n += weight
                    card = False
                if maskdata[x+i,y-i] != (255,255,255):
                    avg += pixdata[x+i,y-i][0] * weight
                    n += weight
                    card = False
                if maskdata[x-i,y-i] != (255,255,255):
                    avg += pixdata[x-i,y-i][0] * weight
                    n += weight
                    card = False
                if maskdata[x-i,y+i] != (255,255,255):
                    avg += pixdata[x-i,y+i][0] * weight
                    n += weight
                    card = False
                if card == False:
                    break
            if n > 0:
                avg /= n
            avg = int(avg)
                # since clouds are greyscale, this is natural (as is only averaging red RGB values)
            pixdata[x,y] = (avg,avg,avg)
globe.save('python.png','png')
#globe.show()

This might/should probably work more precisely than the "smudge to 7 pixels out" imagemagick code, and reasonably quickly... hope this helps

[jmozmoz]

I think I found an relatively easy way to get the images: At http://www.jma.go.jp/en/gms/ one can get a "color" image of the infrared view. The actual cloud image is in the red channel (green is land, blue sea). Also there are images for the last view hours with meaningful filenames.

[jmozmoz]

I uploaded a new version to pypi which downloads the image of the Himawari satellite from the Japan Meteorological Agency website. Meanwhile Dundee also seems to add the images of the Himawari satellite, but there are still some problems. So this will be a temporary fix until the problems will have been solved there.

[eatdust]

Hi guys, the same has just happened end of march. There is a new satellite image but is not at the same longitude. It seems to be fine if I replace the first entry of the satellite file with this:

{ "type": "geoDundee", "longitude": 41.5, "limit": { "left": 17, "right": 911, "top": 17, "bottom": 911 }, "rescale": "ID", "base_url": "041.5E/MSG/", "suffix": "_MSG19" },

Not sure about the "9" channel though, just a guess :)

and I had to tweak the last one to this:

{ "type": "geoDundee", "longitude": 140.7, "limit": { "left": 11, "right": 1364, "top": 11, "bottom": 1364 }, "rescale": "ID", "base_url": "140.7E/MTSAT/", "suffix": "_MTSAT314" }

Cheers.

[jmozmoz]

I updated the 41.5E satellite and released it as new version 0.9.0

I also tried the other tweaks for the cut of settings, but I think, they do not improve the result (if I use the debug option which downloads the satellite images with boarders/coastlines).