Cloud Download

[maxsendq]

I have noticed that the update to the maps from Dundee at least, isn't consistent. The first time it is run on a day (8:30) it downloads the 0600 images. However it is hit and miss as to whether any further images are downloaded for the day. I have got a 0900 a couple of times, but I have to manually download 1200 and later images. Once downloaded the images are used correctly. For example, Running the script at 1300 and 1600 yesterday only used 0900 images were downloaded, although both the 1200 and 1500 images were available. I know nothing about python, but could the issue be in either timing or the crafting of the URL? I can't find a way to get those to be printed. I am running 64Bit 2.7 Python on a Win8.1 box. You can see in the screen grab, the script is sticking to 0600 but the 0900 is available

[chron0]

I can second that since I saw the very same behavior yesterday and also a couple of days ago. Python 2.7, Ubuntu 12.04 (x86_64).

[maxsendq]

I have spent the day running the job every 30mins and only at 1459 did the script download the 1200 images.

[chron0]

To me it seems like 1 hour 15 past official release time (as in the filenames) seems like the best spot to set a scheduler (5 or 10 minutes past often didn't work), so I've set the cloudmap repo generator script (https://github.com/apollo-ng/cloudmap) cron to 4:15, 7:15, 10:15 [...]. I also wonder why there is a 6 hour window at night (i.e. no 0000 image) :)

[maxsendq]

You also have to factor in the images wont appear at 0900 for the 0900 image. It seems to take around 60-90mins for the images to become available. But even when they do the script doesn't seem to pick it up. Which is why I am pointing the finger at a time issue. Which from experience can be a right pain.

[chron0]

Ack. That confirms my observation, 75 minutes past is working almost always (see the commit log of the repo). If it's something in the code, we should be able to reproduce it. But mostly it just runs for days without any hitch.

[jmozmoz]

The scripts looks for the latest time for which all images are available. The script is only downloading new satellite images, if they are not already saved on your disk. So - I think - it should not be a problem to run the script every hour (or so), because there is only very little traffic on the Dundee server (check if the file exist at https://github.com/jmozmoz/cloudmap/blob/master/cloudmap/create_map.py#L110).

I am working on an additional tool to only check for the latest images and log it. Then it will be possible to check for the availability of different images and resolutions.

[maxsendq]

Joachim,

My only other comment is that is doesn't get the 0000 image. I have left it running now for 96 hours, and it does seam consistently to only get the image 2 hours after the image time. Now I wasn't checking the actual time it was available, so it may just be a way that Dundee is doing things. But you definitely don't get the midnight image downloaded.

[jmozmoz]

Your are right, there is a bug with the 00:00UTC image (file names). I will fix this as soon as possible.

[chron0]

0.5.1 is deployed via pip and should hopefully throw a 0000 image tonight into https://github.com/apollo-ng/cloudmap/commits/master. Let's see and close this afterwards :)

[chron0]

Yes :) https://github.com/apollo-ng/cloudmap/commits/master - I can confirm getting 0000.

[jmozmoz]

Thank you for confirming the fix for the midnight problem. Regarding the time lag I guess this is just the processing time at Dundee.