search

tilezen / joerd

Joerd can be used to download, merge and generate tiles from digital elevation data
MIT License
318 stars 49 forks source link

Add 5 m data for Australia #77

Open jdugge opened 8 years ago

jdugge commented 8 years ago

Geoscience Australia provides a 5 m Digital Elevation Model (DEM) of Australia derived from LiDAR. It's licensed under CC-BY-4.0, dataset covers 245000 km² of "Australia's populated coastal zone; floodplain surveys within the Murray Darling Basin, and individual surveys of major and minor population centres"

zerebubuth commented 8 years ago

Thanks!

nvkelso commented 7 years ago

Looks like there's a new 1m and 2m DEM for New South Wales, see #121.

iandees commented 7 years ago

Data download URLs:

iandees commented 7 years ago

Listing files

curl -s http://www.ga.gov.au/metadata-gateway/metadata/record/gcat_22be4b55-2465-4320-e053-10a3070a5236 \
  | grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' \
  | sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' \
  | grep state-mosaics \
  | sort \
  | uniq \
  > geoscience_au_listing.txt

Mirroring files

I skipped mirroring because these were already on S3 and are rather large.

cat geoscience_au_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/geoscience_au/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee geoscience_au_s3_objects.txt

Transcoding files

cat geoscience_au_listing.txt | \
while read filename; do
    bn=$(basename ${filename%.*})
   make submit-job job=aws/transcode-job.json.hbs input=${filename} output=s3://elevation-sources-transcoded/geosciences_au/${bn} name=${bn:0:50}
done
iandees commented 7 years ago

These seem to be failing intermittently, probably because the ECS instance is running out of disk space. I'll check these off as they successfully run.

iandees commented 7 years ago

…these are still failing, even when running on a separate instance with plenty of space. It's not clear to me why they're failing, but it seems that gdal_translate is just stopping at some point:

Processing https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip to s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW...
Downloading https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip...
Transcoding...
Transcoding zip:///tmp/tmp.DXfBGJtkdW.22be4b55-2465-4320-e053-10a3070a5236_NSW.zip!NSW/z54/mdbaz54ag.tif
Transcoding bands...
Input file size is 84000, 104800
0...10.delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.tif
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.tif.msk
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW_footprint.json
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.vrt
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW_thumb.png
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.json
mojodna commented 7 years ago

That behavior is consistent with things I've seen when there hasn't been enough memory available. GDAL tries to be sensible about what it's using and seems to end up never finishing. This is why it's wrapped with timeout in places. Try increasing the amount of memory available?

iandees commented 7 years ago

@mojodna I'm doing this on a machine with 64GB of memory and don't see any memory pressure while I'm running it. I wonder if Docker limits memory usage by default?

mojodna commented 7 years ago

I don't think it does, so maybe this is something else... Hmm.

iandees commented 7 years ago

After trying this again on a bare EC2 instance I tried downloading a DEM to my laptop. Unzipping the file that got downloaded with the MacOS X expander created a .cpgz file instead of the expected .tif, so I tried it from the command line and got:

$ unzip 22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
Archive:  22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
   skipping: TAS/tasz55.tif          need PK compat. v4.5 (can do v2.1)
  inflating: TAS/Ancillary/metadata/Digital Elevation Model (DEM) 5 Metre Grid of Australia derived from LiDAR.pdf

So... maybe it's just a broken zipfile?

nvkelso commented 7 years ago

This data is all over the place and missing most of Australia. Since there is no good, easily accessible file for Australia let's travel over to New Zealand where LINZ is straight forward.

OliverColeman commented 5 years ago

Is this project still active? Is there any plan for getting better resolution data for Australia at present?

iandees commented 5 years ago

It's active, but the Earth doesn't change all that much so we don't update it frequently.

Some Australian data made it into the most recent build, but the data we found here didn't cover the entire country. I think there's just a few small pieces that we found. We'd love to see sources for better data in Australia!

andrewharvey commented 5 years ago

As far as I can tell from http://elevation.fsdf.org.au

NSW provides state wide ranging from 1m-5m CC BY. TAS is not under an open license. VIC provides state wide 10m CC BY (http://www.delwp.vic.gov.au/vicmap) and coastal areas 0.5m-1m is not under an open license. QLD not under an open license

I've been doing the clip and ship process for about 1/3rd of NSW, and have this sitting in a private Digital Ocean Space, which I can share if anyone is interested in bypassing the Clip and Ship process.

OliverColeman commented 5 years ago

Does this not provide 5m coverage for all of Australia that is CC BY? http://gaservices.ga.gov.au/site_9/rest/services/DEM_LiDAR_5m/MapServer

iandees commented 5 years ago

It's not easy to get bulk data out of an Esri REST map service. I think that service uses the data mentioned on this ticket above that doesn't successfully unzip.

I'm downloading the data from here: https://data.gov.au/dataset/ds-ga-22be4b55-2465-4320-e053-10a3070a5236/details / https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/89644 (which may or may not be an updated version of what I linked to above). Once I get it downloaded I'll see if the zip files are still broken.