Updated Alaska / ArcticDEM

[iandees]

The ArcticDEM explorer website seems to show a whole bunch of empty spots in Alaska that weren't there before. Specifically, Denali seems to be missing lots of important detail:

...and the data downloaded in #92 seems to be missing all of Alaska:

The FTP URL we downloaded from in that ticket (ftp://ftp.data.pgc.umn.edu/elev/dem/setsm/ArcticDEM/mosaic/v2.0/) seems to have a ton of very recent last-updated timestamps, so I'm thinking that at the time I processed it they just didn't have everything posted?

[iandees]

The problem is that my file listing script in #92 was looking for files that ended in .tar.gz. But it looks like some files (specifically the ones in Alaska?) end in .tar without any compression.

[iandees]

The website lists 2,462 mosaic tiles, the tile index shapefile lists 9,036 tiles, and the lftp file listing on the ftp site used in #92 listed just under 9,000. I don't really understand why all these numbers are different, but I'm going to use the file that lists the most data (the Shapefile tile index).

I opened the index Shapefile in Qgis, copied the attribute table to my text editor, and extracted the file URL out by itself. That file is in a gist here.

I'm transcoding the stuff we didn't get before (the stuff ending in .tar) with:

cat pgc-updated-listing.txt | grep .tar$ | \
while read url; do
    bn=$(basename ${url%.tar})
    AWS_PROFILE=openterrain AWS_ACCOUNT_ID=273461098965 make submit-job job=aws/transcode-job.json.hbs input=${url} output=s3://elevation-sources-transcoded/pgdc_5m/${bn} name=$(sed 's/\./_/' <<< $bn)
done

[iandees]

The transcoding finished last night, but it looks like the new data that was added in Russia for release 6 ends with .tar.gz so my transcode missed that data:

Purple is footprints database, green hashed polygons are footprints in the index shapefile for the dataset.

Release 6 is in the red here:

From ArcticDEM website.

Going to pull those in now.

[iandees]

I extracted the URLs for the data in release 6 mentioned above and put it in the above gist. Raw file here.

Using the same job submit script from above.

[iandees]

Finished transcoding, but also noticed that a swath of ~50 tiles are missing from their server:

I'm hoping to hear back from them some time this weekend.

[iandees]

The missing data was uploaded this morning and I transcoded it.

[iandees]

This data is all good to go in the database now, so I'm going to close this particular issue.