search

nvkelso / natural-earth-vector

A global, public domain map dataset available at three scales and featuring tightly integrated vector and raster data.
https://www.naturalearthdata.com/
Other
1.78k stars 368 forks source link

Move file hosting to S3 via AWS public dataset #445

Closed nvkelso closed 2 years ago

nvkelso commented 3 years ago

The data has been hosted by FSU (huge thank you!) for the last 11 years.

NACIS (Natural Earth's formal project sponsor) is in discussions with AWS to host the data files in their public dataset program.

nvkelso commented 3 years ago

AWS Public Dataset approval has been granted. Need to setup new org and S3 bucket next (with CW logging enabled for analytics), post files, update the download manager, and document direct S3 download links.

nvkelso commented 3 years ago

The S3 bucket has been setup and initial data transfer of v4 released files has kicked off.

barronh commented 3 years ago

This is great news! What is the status of documentation? I'd suggest making a registry entry that documents these buckets. I think you can just fork the registry repo, add a yaml file with meta data, and make a PR. The biggest commitment is filling in the template.

https://registry.opendata.aws/

thanks for all you do!

nvkelso commented 3 years ago

The related PR (WIP) is https://github.com/awslabs/open-data-registry/pull/853.

nvkelso commented 3 years ago

That PR is ready to merge now.

But looking at the path structure of the S3 bucket it has 10m_cultural instead of 10m/cultural so I need to have a closer look, they should match before switching over naciscdn.org to S3 instead of FREAC.

nvkelso commented 3 years ago

The AWS public data registry PR has been merged and the data is live. But it needs a little bit more automation for updates, and really versioned releases.

nvkelso commented 3 years ago

I'm adding the missing 10m and 50m rasters now.

nvkelso commented 2 years ago

Versioned will live in dirs like:

And latest stable release will live at the root:

The same will be true for the naciscdn links:

And latest stable release will live at the root: