search

protomaps / go-pmtiles

Single-file executable tool for working with PMTiles archives
BSD 3-Clause "New" or "Revised" License
356 stars 49 forks source link

pmtiles upload is unreliable for large files #89

Open bdon opened 11 months ago

bdon commented 11 months ago

Look into S3 retries

gimse commented 9 months ago

Why is "pmtiles upload" needed? Is it not possible just to use the aws cli?

bdon commented 9 months ago

It isn't hard to implement as it's part of gocloud and it means one can make complete use of pmtiles with a S3-compatible bucket, a set of credentials, and a single CLI tool. But maybe the overlap of pmtiles users and those with access to awscli or rclone is 100%.

Last I checked the AWS CLI requires an installer step and multiple files which is more complex than a single-file binary like go-pmtiles. Thoughts?

jamesscottbrown commented 6 months ago

Personally I find pmtiles upload useful: the AWS CLI is a bit heavy to install, and rclone is more cumbersome to use. I thought rclone required first defining the S3 bucket that you want to use as a remote in a config file; IIUC, this is now longer true and you can now use the :backend:path/to/dir syntax, but it's still much harder to figure out the correct command for rclone than for pmtiles upload.

But maybe the overlap of pmtiles users and those with access to awscli or rclone is 100%.

I have access to both of these tools, but not necessarily from the machine or VM where I am generating the tiles. For example, I sometimes need to generate tiles containing data in a PostgreSQL DB that I can't connect to directly from my workstation because of firewall rules.

lauriemann commented 5 months ago

Just my .02, but I'd never used rclone until yesterday when I needed to copy up my .pmtiles files, but I found it pretty simple to use the docker version of rclone to upload to R2, so not a lot of value in spending time building yet another copy mechanism, time is better spent on other parts of the pmtiles project