search

tilezen / joerd

Joerd can be used to download, merge and generate tiles from digital elevation data
MIT License
319 stars 49 forks source link

EU AWS S3 bucket #215

Open StrandedKitty opened 7 months ago

StrandedKitty commented 7 months ago

We've encountered a problem with slow response times when fetching tiles from the US-based public bucket elevation-tiles-prod (https://registry.opendata.aws/terrain-tiles/) using HTTP. It often takes ~100ms to fetch a single tile. We think that the main problem is that this bucket is hosted in the United States while a lot of our users are from the EU.

There's a mention of a EU-based mirror bucket (elevation-tiles-prod-eu) but it doesn't work. It's not possible to access it neither using HTTP nor S3.

Is it possible by any chance for us to access the dataset and mirror it ourselves? We've tried mirroring this public bucket by moving tiles 1-by-1, but unfortunately there's so many small files that it's not feasible to do it like this.

iandees commented 7 months ago

At this time we only offer the single bucket in the one region. You're welcome to copy the data over to another bucket in another region if needed. As you found, there are lots of files, but if you parallelize the process it should be possible to copy them over in a few days. AWS also offers tools to migrate data between S3 buckets that you could look into. The bucket is public, so you should already have the access you need to perform the mirror.

StrandedKitty commented 7 months ago

@iandees Thanks for the quick response! What's the total size of the bucket?