search

tilezen / joerd

Joerd can be used to download, merge and generate tiles from digital elevation data
MIT License
324 stars 50 forks source link

Add 10 m elevation data for Norway #76

Open turban opened 8 years ago

turban commented 8 years ago

The dataset is available here: http://data.kartverket.no/download/content/digital-terrengmodell-10-m-utm-33

zerebubuth commented 8 years ago

Looks great, thanks!

nvkelso commented 8 years ago

Also /cc @mastermaps who is also interested and recommended:

turban commented 8 years ago

Please contact me if you need help to download the full dataset - as the default interface is not the best.

nvkelso commented 7 years ago

This would be better data than the 30m EU data in #132.

iandees commented 7 years ago

I've tried to manually click through the data download basket a few times and it resets when you miss any of the tiny squares. Very frustrating 😄.

If anyone wants to give it a shot and get a downloadable link for me, that'd be really helpful.

nvkelso commented 7 years ago

OMG, who designed that interface!?! I hope I clicked on all of them:

However:

Selection can not be longer than 2550 characters but is currently 9829 characters long.

So I'll try again later this afternoon. Unless @turban has a secret download link to share!

Terrengdata_7910_2_UTM33_10m_DEM.zip
Terrengdata_7810_1_UTM33_10m_DEM.zip
Terrengdata_7810_2_UTM33_10m_DEM.zip
Terrengdata_7710_1_UTM33_10m_DEM.zip
Terrengdata_7710_2_UTM33_10m_DEM.zip
Terrengdata_7711_4_UTM33_10m_DEM.zip
Terrengdata_7811_3_UTM33_10m_DEM.zip
Terrengdata_7710_3_UTM33_10m_DEM.zip
Terrengdata_7710_4_UTM33_10m_DEM.zip
Terrengdata_7810_3_UTM33_10m_DEM.zip
Terrengdata_7810_4_UTM33_10m_DEM.zip
Terrengdata_7910_3_UTM33_10m_DEM.zip
Terrengdata_7909_2_UTM33_10m_DEM.zip
Terrengdata_7909_3_UTM33_10m_DEM.zip
Terrengdata_7908_2_UTM33_10m_DEM.zip
Terrengdata_7908_3_UTM33_10m_DEM.zip
Terrengdata_7807_1_UTM33_10m_DEM.zip
Terrengdata_7808_4_UTM33_10m_DEM.zip
Terrengdata_7808_1_UTM33_10m_DEM.zip
Terrengdata_7809_4_UTM33_10m_DEM.zip
Terrengdata_7809_1_UTM33_10m_DEM.zip
Terrengdata_7806_1_UTM33_10m_DEM.zip
Terrengdata_7806_4_UTM33_10m_DEM.zip
Terrengdata_7806_3_UTM33_10m_DEM.zip
Terrengdata_7706_3_UTM33_10m_DEM.zip
Terrengdata_7706_4_UTM33_10m_DEM.zip
Terrengdata_7606_4_UTM33_10m_DEM.zip
Terrengdata_7606_3_UTM33_10m_DEM.zip
Terrengdata_7506_4_UTM33_10m_DEM.zip
Terrengdata_7506_3_UTM33_10m_DEM.zip
Terrengdata_7606_2_UTM33_10m_DEM.zip
Terrengdata_7607_3_UTM33_10m_DEM.zip
Terrengdata_7507_4_UTM33_10m_DEM.zip
Terrengdata_7506_1_UTM33_10m_DEM.zip
Terrengdata_7606_1_UTM33_10m_DEM.zip
Terrengdata_7607_4_UTM33_10m_DEM.zip
Terrengdata_7707_3_UTM33_10m_DEM.zip
Terrengdata_7706_2_UTM33_10m_DEM.zip
Terrengdata_7706_1_UTM33_10m_DEM.zip
Terrengdata_7707_4_UTM33_10m_DEM.zip
Terrengdata_7807_3_UTM33_10m_DEM.zip
Terrengdata_7806_2_UTM33_10m_DEM.zip
Terrengdata_7807_2_UTM33_10m_DEM.zip
Terrengdata_7808_3_UTM33_10m_DEM.zip
Terrengdata_7808_2_UTM33_10m_DEM.zip
Terrengdata_7809_3_UTM33_10m_DEM.zip
Terrengdata_7809_2_UTM33_10m_DEM.zip
Terrengdata_7709_1_UTM33_10m_DEM.zip
Terrengdata_7709_4_UTM33_10m_DEM.zip
Terrengdata_7708_1_UTM33_10m_DEM.zip
Terrengdata_7708_4_UTM33_10m_DEM.zip
Terrengdata_7707_1_UTM33_10m_DEM.zip
Terrengdata_7707_2_UTM33_10m_DEM.zip
Terrengdata_7708_3_UTM33_10m_DEM.zip
Terrengdata_7708_2_UTM33_10m_DEM.zip
Terrengdata_7709_3_UTM33_10m_DEM.zip
Terrengdata_7609_4_UTM33_10m_DEM.zip
Terrengdata_7608_1_UTM33_10m_DEM.zip
Terrengdata_7608_4_UTM33_10m_DEM.zip
Terrengdata_7607_1_UTM33_10m_DEM.zip
Terrengdata_7608_3_UTM33_10m_DEM.zip
Terrengdata_7608_2_UTM33_10m_DEM.zip
Terrengdata_7609_3_UTM33_10m_DEM.zip
Terrengdata_7607_2_UTM33_10m_DEM.zip
Terrengdata_7705_1_UTM33_10m_DEM.zip
Terrengdata_7705_4_UTM33_10m_DEM.zip
Terrengdata_7705_3_UTM33_10m_DEM.zip
Terrengdata_7705_2_UTM33_10m_DEM.zip
Terrengdata_7605_1_UTM33_10m_DEM.zip
Terrengdata_7605_4_UTM33_10m_DEM.zip
Terrengdata_7605_3_UTM33_10m_DEM.zip
Terrengdata_7605_2_UTM33_10m_DEM.zip
Terrengdata_7505_1_UTM33_10m_DEM.zip
Terrengdata_7505_4_UTM33_10m_DEM.zip
Terrengdata_7505_3_UTM33_10m_DEM.zip
Terrengdata_7505_2_UTM33_10m_DEM.zip
Terrengdata_7405_1_UTM33_10m_DEM.zip
Terrengdata_7405_4_UTM33_10m_DEM.zip
Terrengdata_7405_2_UTM33_10m_DEM.zip
Terrengdata_7405_3_UTM33_10m_DEM.zip
Terrengdata_7305_4_UTM33_10m_DEM.zip
Terrengdata_7305_3_UTM33_10m_DEM.zip
Terrengdata_7604_4_UTM33_10m_DEM.zip
Terrengdata_7604_2_UTM33_10m_DEM.zip
Terrengdata_7604_1_UTM33_10m_DEM.zip
Terrengdata_7604_3_UTM33_10m_DEM.zip
Terrengdata_7503_4_UTM33_10m_DEM.zip
Terrengdata_7503_2_UTM33_10m_DEM.zip
Terrengdata_7503_1_UTM33_10m_DEM.zip
Terrengdata_7503_3_UTM33_10m_DEM.zip
Terrengdata_7403_4_UTM33_10m_DEM.zip
Terrengdata_7403_1_UTM33_10m_DEM.zip
Terrengdata_7403_2_UTM33_10m_DEM.zip
Terrengdata_7403_3_UTM33_10m_DEM.zip
Terrengdata_7303_3_UTM33_10m_DEM.zip
Terrengdata_7303_4_UTM33_10m_DEM.zip
Terrengdata_7303_2_UTM33_10m_DEM.zip
Terrengdata_7303_1_UTM33_10m_DEM.zip
Terrengdata_7304_4_UTM33_10m_DEM.zip
Terrengdata_7304_1_UTM33_10m_DEM.zip
Terrengdata_7304_2_UTM33_10m_DEM.zip
Terrengdata_7304_3_UTM33_10m_DEM.zip
Terrengdata_7404_3_UTM33_10m_DEM.zip
Terrengdata_7404_2_UTM33_10m_DEM.zip
Terrengdata_7404_1_UTM33_10m_DEM.zip
Terrengdata_7504_4_UTM33_10m_DEM.zip
Terrengdata_7504_1_UTM33_10m_DEM.zip
Terrengdata_7504_2_UTM33_10m_DEM.zip
Terrengdata_7504_3_UTM33_10m_DEM.zip
Terrengdata_7404_4_UTM33_10m_DEM.zip
Terrengdata_7204_1_UTM33_10m_DEM.zip
Terrengdata_7204_4_UTM33_10m_DEM.zip
Terrengdata_7203_1_UTM33_10m_DEM.zip
Terrengdata_7203_4_UTM33_10m_DEM.zip
Terrengdata_7202_1_UTM33_10m_DEM.zip
Terrengdata_7202_4_UTM33_10m_DEM.zip
Terrengdata_7202_3_UTM33_10m_DEM.zip
Terrengdata_7202_2_UTM33_10m_DEM.zip
Terrengdata_7203_3_UTM33_10m_DEM.zip
Terrengdata_7203_2_UTM33_10m_DEM.zip
Terrengdata_7204_3_UTM33_10m_DEM.zip
Terrengdata_7204_2_UTM33_10m_DEM.zip
Terrengdata_7104_1_UTM33_10m_DEM.zip
Terrengdata_7104_4_UTM33_10m_DEM.zip
Terrengdata_7103_1_UTM33_10m_DEM.zip
Terrengdata_7103_4_UTM33_10m_DEM.zip
Terrengdata_7102_1_UTM33_10m_DEM.zip
Terrengdata_7101_2_UTM33_10m_DEM.zip
Terrengdata_7102_3_UTM33_10m_DEM.zip
Terrengdata_7102_2_UTM33_10m_DEM.zip
Terrengdata_7103_3_UTM33_10m_DEM.zip
Terrengdata_7103_2_UTM33_10m_DEM.zip
Terrengdata_7104_3_UTM33_10m_DEM.zip
Terrengdata_7104_2_UTM33_10m_DEM.zip
Terrengdata_7004_1_UTM33_10m_DEM.zip
Terrengdata_7004_4_UTM33_10m_DEM.zip
Terrengdata_7003_1_UTM33_10m_DEM.zip
Terrengdata_7003_4_UTM33_10m_DEM.zip
Terrengdata_7002_1_UTM33_10m_DEM.zip
Terrengdata_7002_4_UTM33_10m_DEM.zip
Terrengdata_7001_1_UTM33_10m_DEM.zip
Terrengdata_7001_4_UTM33_10m_DEM.zip
Terrengdata_7000_2_UTM33_10m_DEM.zip
Terrengdata_7001_3_UTM33_10m_DEM.zip
Terrengdata_7001_2_UTM33_10m_DEM.zip
Terrengdata_7002_3_UTM33_10m_DEM.zip
Terrengdata_7002_2_UTM33_10m_DEM.zip
Terrengdata_7003_3_UTM33_10m_DEM.zip
Terrengdata_7003_2_UTM33_10m_DEM.zip
Terrengdata_6903_1_UTM33_10m_DEM.zip
Terrengdata_6903_4_UTM33_10m_DEM.zip
Terrengdata_6902_1_UTM33_10m_DEM.zip
Terrengdata_6902_4_UTM33_10m_DEM.zip
Terrengdata_6901_1_UTM33_10m_DEM.zip
Terrengdata_6901_4_UTM33_10m_DEM.zip
Terrengdata_6900_1_UTM33_10m_DEM.zip
Terrengdata_6900_4_UTM33_10m_DEM.zip
Terrengdata_69m1_2_UTM33_10m_DEM.zip
Terrengdata_6900_3_UTM33_10m_DEM.zip
Terrengdata_6900_2_UTM33_10m_DEM.zip
Terrengdata_6901_3_UTM33_10m_DEM.zip
Terrengdata_6901_2_UTM33_10m_DEM.zip
Terrengdata_6902_3_UTM33_10m_DEM.zip
Terrengdata_6902_2_UTM33_10m_DEM.zip
Terrengdata_6903_3_UTM33_10m_DEM.zip
Terrengdata_6903_2_UTM33_10m_DEM.zip
Terrengdata_6803_1_UTM33_10m_DEM.zip
Terrengdata_6803_4_UTM33_10m_DEM.zip
Terrengdata_6802_4_UTM33_10m_DEM.zip
Terrengdata_6802_1_UTM33_10m_DEM.zip
Terrengdata_6801_1_UTM33_10m_DEM.zip
Terrengdata_6801_4_UTM33_10m_DEM.zip
Terrengdata_6800_1_UTM33_10m_DEM.zip
Terrengdata_6800_4_UTM33_10m_DEM.zip
Terrengdata_68m1_1_UTM33_10m_DEM.zip
Terrengdata_68m1_4_UTM33_10m_DEM.zip
Terrengdata_68m1_3_UTM33_10m_DEM.zip
Terrengdata_67m1_4_UTM33_10m_DEM.zip
Terrengdata_66m1_4_UTM33_10m_DEM.zip
Terrengdata_67m1_3_UTM33_10m_DEM.zip
Terrengdata_66m1_3_UTM33_10m_DEM.zip
Terrengdata_65m1_4_UTM33_10m_DEM.zip
Terrengdata_65m1_3_UTM33_10m_DEM.zip
Terrengdata_64m1_1_UTM33_10m_DEM.zip
Terrengdata_6400_3_UTM33_10m_DEM.zip
Terrengdata_6401_3_UTM33_10m_DEM.zip
Terrengdata_6400_2_UTM33_10m_DEM.zip
Terrengdata_6400_1_UTM33_10m_DEM.zip
Terrengdata_6401_1_UTM33_10m_DEM.zip
Terrengdata_6401_4_UTM33_10m_DEM.zip
Terrengdata_6400_4_UTM33_10m_DEM.zip
Terrengdata_65m1_2_UTM33_10m_DEM.zip
Terrengdata_6500_3_UTM33_10m_DEM.zip
Terrengdata_6500_4_UTM33_10m_DEM.zip
Terrengdata_65m1_1_UTM33_10m_DEM.zip
Terrengdata_66m1_2_UTM33_10m_DEM.zip
Terrengdata_6600_3_UTM33_10m_DEM.zip
Terrengdata_6600_4_UTM33_10m_DEM.zip
Terrengdata_67m1_2_UTM33_10m_DEM.zip
Terrengdata_6700_3_UTM33_10m_DEM.zip
Terrengdata_6700_4_UTM33_10m_DEM.zip
Terrengdata_6800_3_UTM33_10m_DEM.zip
Terrengdata_68m1_2_UTM33_10m_DEM.zip
Terrengdata_67m1_1_UTM33_10m_DEM.zip
Terrengdata_66m1_1_UTM33_10m_DEM.zip
Terrengdata_6500_2_UTM33_10m_DEM.zip
Terrengdata_6501_3_UTM33_10m_DEM.zip
Terrengdata_6501_2_UTM33_10m_DEM.zip
Terrengdata_6502_3_UTM33_10m_DEM.zip
Terrengdata_6502_2_UTM33_10m_DEM.zip
Terrengdata_6503_3_UTM33_10m_DEM.zip
Terrengdata_6503_4_UTM33_10m_DEM.zip
Terrengdata_6603_2_UTM33_10m_DEM.zip
Terrengdata_6603_1_UTM33_10m_DEM.zip
Terrengdata_6703_2_UTM33_10m_DEM.zip
Terrengdata_6703_1_UTM33_10m_DEM.zip
Terrengdata_6803_2_UTM33_10m_DEM.zip
Terrengdata_6803_3_UTM33_10m_DEM.zip
Terrengdata_6802_2_UTM33_10m_DEM.zip
Terrengdata_6702_1_UTM33_10m_DEM.zip
Terrengdata_6703_4_UTM33_10m_DEM.zip
Terrengdata_6702_2_UTM33_10m_DEM.zip
Terrengdata_6703_3_UTM33_10m_DEM.zip
Terrengdata_6603_4_UTM33_10m_DEM.zip
Terrengdata_6602_1_UTM33_10m_DEM.zip
Terrengdata_6602_2_UTM33_10m_DEM.zip
Terrengdata_6603_3_UTM33_10m_DEM.zip
Terrengdata_6502_1_UTM33_10m_DEM.zip
Terrengdata_6800_2_UTM33_10m_DEM.zip
Terrengdata_6801_3_UTM33_10m_DEM.zip
Terrengdata_6701_4_UTM33_10m_DEM.zip
Terrengdata_6700_1_UTM33_10m_DEM.zip
Terrengdata_6700_2_UTM33_10m_DEM.zip
Terrengdata_6701_3_UTM33_10m_DEM.zip
Terrengdata_6600_1_UTM33_10m_DEM.zip
Terrengdata_6601_4_UTM33_10m_DEM.zip
Terrengdata_6601_3_UTM33_10m_DEM.zip
Terrengdata_6600_2_UTM33_10m_DEM.zip
Terrengdata_6500_1_UTM33_10m_DEM.zip
Terrengdata_6501_4_UTM33_10m_DEM.zip
Terrengdata_6501_1_UTM33_10m_DEM.zip
Terrengdata_6502_4_UTM33_10m_DEM.zip
Terrengdata_6602_3_UTM33_10m_DEM.zip
Terrengdata_6601_2_UTM33_10m_DEM.zip
Terrengdata_6601_1_UTM33_10m_DEM.zip
Terrengdata_6702_3_UTM33_10m_DEM.zip
Terrengdata_6602_4_UTM33_10m_DEM.zip
Terrengdata_6701_1_UTM33_10m_DEM.zip
Terrengdata_6701_2_UTM33_10m_DEM.zip
Terrengdata_6702_4_UTM33_10m_DEM.zip
Terrengdata_6802_3_UTM33_10m_DEM.zip
Terrengdata_6801_2_UTM33_10m_DEM.zip
turban commented 7 years ago

Secret already shared with Ian :-)

iandees commented 7 years ago

Listing files

curl \
    -u "bengler:data" \
    -s \
    http://data.kartverket.no/betatest/terrengdata/10m/utm33/ | \
grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' | \
sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' | \
grep Terren > kartverket_listing.txt

Mirroring files

cat kartverket_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/kartverket/$f"; curl -u "bengler:data" -s http://data.kartverket.no/betatest/terrengdata/10m/utm33/$f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee kartverket_s3_objects.txt

Transcoding files

aws s3 ls s3://elevation-sources-prod/kartverket/ --recursive | \
grep DEM.zip$ | \
awk '{print $4}' | \
while read filename; do
    bn=$(basename ${filename%.*}); 
    make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname $filename)/${bn} name=${bn:0:50};
done