search

tilezen / joerd

Joerd can be used to download, merge and generate tiles from digital elevation data
MIT License
319 stars 49 forks source link

Europe 25 meter DEM #132

Open nvkelso opened 7 years ago

nvkelso commented 7 years ago
mojodna commented 7 years ago

A mirror of these files is in s3://mapzen-eudem/

Transcoded versions are in s3://mapzen-dynamic-tiler-test/EU-DEM/0/

iandees commented 7 years ago

Listing files

curl -s https://www.eea.europa.eu/data-and-maps/data/eu-dem \
  | grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' \
  | sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' \
  | grep entr_r_4258_1_arcsec_gsgrda-eudem-dem-europe_2012_rev1 \
  > eudem_listing.txt

Mirroring files to openterrain S3

cat eudem_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/eudem/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee eudem_s3_objects.txt

… This seems to fail because the download site isn't responding. I'll come back in a bit. …

iandees commented 7 years ago

They're looking into the down-ness:

https://twitter.com/EUEnvironment/status/872430176298303488

iandees commented 7 years ago

Looks to be back up, so I'm doing the mirroring step now.

iandees commented 7 years ago

Transcoding started:

aws s3 ls s3://elevation-sources-prod/eudem/ --recursive | \
grep tif$ | awk '{print $4}' | \
while read filename; do
    bn=$(basename ${filename%.*})
    make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname $filename)/${bn} name=${bn:0:50}
done
iandees commented 7 years ago

Some of these jobs are failing because they run out of memory. These are the failures. I'm cloning the jobs through the Batch console and giving them 3GB of memory: