Add Canadian DEM data

[nvkelso]

From @nvkelso:

• http://geogratis.gc.ca/api/en/nrcan-rncan/ess-sst/C40ACFBA-C722-4BE1-862E-146B80BE738E.html

Canadian Digital Elevation Model Mosaic (CDEM) The Canadian Digital Elevation Model (CDEM) is part of Natural Resources Canada's altimetry system designed to better meet the users' needs for elevation data and products. The CDEM stems from the existing Canadian Digital Elevation Data (CDED). In these data, elevations can be either ground or reflective surface elevations. A CDEM mosaic can be obtained for a pre-defined or user-defined extent. The coverage and resolution of a mosaic varies according to latitude and to the extent of the requested area. Derived products such as slope, shaded relief and colour shaded relief maps can also be generated on demand.

Additional Information Date published November 6, 2012 Series Canadian Digital Elevation Model Mosaic (CDEM) NaN Issue 3.0 Purpose The CDEM plays the same role as contours and relief shading on conventional paper maps. The CDEM serves as key primary data in a range of applications critical to achieving sustainable development. These applications include environmental and ecological impact assessments, water flow and water quality analysis, climate change studies, forest regeneration planning and wildlife habitats. In addition, the CDEM can be used in the generation of three-dimensional graphics displaying terrain slope, profiles and lines of sight. Non-graphic applications include geoid calculations, terrain modelling, flood simulations and telecommunication studies.... show more Author Government of Canada, Natural Resources Canada, Earth Sciences Sector, Mapping Information Branch, GeoAccess Division Language This product is available in English and French Product type Elevation 5

Migrated from: https://github.com/valhalla/skadi/issues/41.

[mojodna]

Download link: http://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/

[mojodna]

This'll be a fun one to ingest because a) there are lots of subdirectories and b) everything is zipped.

Convenient commands:

# generate listings
wget -r --no-remove-listing --spider ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/

# read into a zip
gdalinfo /vsizip/cdem_dem_002C_tif.zip/cdem_dem_002C.tif

[mojodna]

wget stopped being able to connect (probably because it was repeatedly reconnecting). None of the other ftp clients I tried first could do recursive listings.

However, lftp to the rescue!

cdec.lftp:

open ftp.geogratis.gc.ca
cd /pub/nrcan_rncan/elevation/cdem_mnec
find
exit

lftp -f cdec.lftp > cdec.list

Result: https://gist.github.com/mojodna/c807e3fdbe618e99681aaef8e326f5e6

[mojodna]

Some images are only 1 or 4px high, which triggered some problems in footprint generation (root cause: https://github.com/mapbox/rasterio/pull/1025)

Transcoded versions in s3://mapzen-dynamic-tiler-test/cdem/0/

[iandees]

(I'm basically repeating what @mojodna said above but more explicitly. These were all run from an instance in AWS.)

Listing files

lftp \
    -c 'find' \
    ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/ \
| grep tif.zip > nrcan_cdem_listing.txt

Mirroring files to openterrain S3

cat nrcan_cdem_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/nrcan_cdem/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee nrcan_cdem_s3_objects.txt

Transcoding files

I got the nrcan_cdem_s3_objects.txt file onto my local computer, where I have the setup to submit transcoding jobs and ran:

cat nrcan_cdem_s3_objects.txt | \
while read filename; do
    bn=$(basename $filename)
    make submit-job job=aws/sample-transcode-job.json.hbs input=${filename} output=s3://elevation-sources-transcoded/nrcan_cdem/$(sed 's/_tif\.zip//' <<< $bn) name=$(sed 's/\./_/' <<< $bn)
done

[iandees]

(Transcoding is complete and footprints are inserted)

[iandees]

It seems like we downloaded an older version of this data. There's a newer, higher-resolution dataset at http://open.canada.ca/data/en/dataset/957782bf-847c-4644-a757-e383c0057995?

Download link for this dataset is http://ftp.maps.canada.ca/pub/elevation/dem_mne/highresolution_hauteresolution/dsm_mns/2m/

[nvkelso]

Let's track that newer data in a different issue: https://github.com/tilezen/joerd/issues/152