pmtiles latency

[bdon]

Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets.

If PMTiles implements splitting to <10 MB files, it can be a valid alternative to running servers.

Can you add more information on what you claim in the README here?

I made this test script on a 100GB+ full planet dataset:

curl -H "Range: bytes=0-100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles --output tmp -w "%{time_total}"

curl -H "Range: bytes=100000000000-100000100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles --output tmp -w "%{time_total}"

And I get identical timing results, including SSL handshake etc.

I understand specific storage providers like https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fprotomaps%2Fopenstreetmap%2Ftiles%2Fv3.pmtiles may have high latency - I'll investigate that case, but it shouldn't be true across every static storage provider.

[hyperknot]

Hey Brandon!

So let me explain the testing in detail. Since bandwidth is crucial for my application, what I tested in detail was only the Cloudflare buckets.

1. I uploaded the full planet file to a local, European public Cloudflare bucket and opened a map page in browser. It was basically unusable. Some tiles loaded within a few seconds, some tiles loaded after 10 seconds, some tiles seemed to take forever.

Having said that, it was 6-8 month ago, Cloudflare could have improved their stack, but I doubt it's a better experience now.

2. On your side, I tested this page before the HN launch: https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fprotomaps%2Fopenstreetmap%2Ftiles%2Fv3.pmtiles Now, with the HN launch, it's smooth and fast, I believe S3 probably distributed it to many servers or keeps it in cache, or S3 might be caching range requests which CloudFlare doesn't.

But still, I made a screenshot about a random tile on a random xray on https://maps.protomaps.com/builds/

This is what I call latency. These values were much longer just a few days ago, they could really take 3-5 seconds. You can see the blue part is the content download, that's really fast, even if it goes across Europe - US-West. The problem is the "Waiting for server response" part.

[hyperknot]

I can imagine testing latency could be done properly like this:

1. Pick a random tile
2. Pick a random build from https://maps.protomaps.com/builds/
3. Time TTFB on that tile on that build. Do this once per minute or similar, and make a graph from the results.

Of course I don't know if data.source.coop and build.protomaps.com are the same provider or not.

[bdon]

Yes, I can confirm that Cloudflare is exceptionally slow, I have some scripts to test that: https://github.com/bdon/cloudflare-r2-latency

https://maps.protomaps.com/builds/ runs in Cloudflare because it hosts the daily automated build. That powers the "diff" comparison testing between build versions, and also acts as a distribution point, since I encourage users to download the whole 100+ GB or use pmtiles extract to only get the region they need. So I can't afford to pay for egress fees (a single full download would cost me $2 on AWS) and the daily builds are a bit of a "special case" compared to what a user deploying themselves would see on another platform.

Your experience might be different on most other storage platforms, like OVH, Scaleway (Europe), DigitalOcean, AWS, Google Cloud or Azure storage. Would you be willing to evaluate those using your tools and your location relative to datacenters, etc?

[hyperknot]

If you give me a full example I'm happy to help, but I think this should be measured by a machine. I recommend what I wrote here: https://github.com/hyperknot/openfreemap/issues/16#issuecomment-2373342395

Basically a graph of TTFB over time, measured once per minute.

Also, I think it really should measure in a full planet file, as maybe these cloud platforms have a hard limit on which file-size they distribute across their servers. For example say a 5 MB file is requested frequently => gets distributed to 100 hosts. 90 GB file is requested frequently => gets distributed to 2 hosts.

Having said that I'll clarify my claim and add that on Cloudflare the latency is bad.

[hyperknot]

I've updated the README. I didn't realise that requests also costs money on Cloudflare buckets, so knowing that now this is actually the real reason.

[bdon]

I've updated the README. I didn't realise that requests also costs money on Cloudflare buckets, so knowing that now this is actually the real reason.

Unfortunately, cloud platforms can be prohibitively expensive for production usage of map tiles, as map tiles are only 405 bytes on average. This means 1 TB of traffic = 2.7 billion requests.

I don't think that is realistic either, because most people spend their time looking at populated areas and not parts of the ocean. I've observed a mean requested tile size of about 100kb in our logs.

[bdon]

Here are 3 full planet tilesets:

AWS S3 in us-central-1 (Frankfurt):

https://protomaps-latency.s3.eu-central-1.amazonaws.com/v4.pmtiles

Hetzner Helsinki served by Caddy:

https://protomaps.dev/~bdon/v4.pmtiles

And a different 100GB+ tileset in AWS S3 us-west-2 (Oregon):

https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles

All of those can be:

• viewed in the pmtiles viewer
• fetch a single tile from CLI:

time pmtiles tile https://protomaps-latency.s3.eu-central-1.amazonaws.com/v4.pmtiles 0 0 0 > tmp

That TTFB includes SSL handshake, etc.

• another bandwidth test is fetching the first 8 zoom levels of the archive:

 pmtiles extract https://protomaps-latency.s3.eu-central-1.amazonaws.com/v4.pmtiles z7.pmtiles --maxzoom=7

[hyperknot]

You are right, it should be done on a deduplicated real world access pattern, only that'd make a realistic test.

How is your total bandwidth divided by total requests on those buckets, for average file size?

[hyperknot]

I did a quick statistics on 2800 requests, the average file size of those were 125 kB.

[hyperknot]

@bdon, I tested all your links and they work quite well. I see that AWS works well with range requests.

It was data.source.coop which was very slow for me during the previous weeks when I tested it a few times. I mean loading of a full page map was like 8-10 seconds.

Anyway, hosting on AWS is out of the question for me because of the crazy data egress costs. Maybe some small clients can fit in the free tier, but with AWS there is always the risk of a bad bot finding your website and creating a thousand dollar surprise bill.

So in summary, the only solution which might work for me is Cloudflare bucket + Cloudflare worker, where egress is free. Do you by any chance have a full planet test on a Cloudflare worker?

[bdon]

How is your total bandwidth divided by total requests on those buckets, for average file size?

The bucket is only accessed for CDN cache misses, not cache hits. So it doesn't directly match what you are returning to browsers.

Looking at a log of the most recent 10,000 CDN cache misses the mean tile size is 22 KB

Looking at how much I paid for a specific deployment in Google Cloud, I paid $5.88 in egress and $3.35 in Class B operations. That's ~294 GB in bandwidth and 8.3 million requests, or about 35 kilobytes for mean tile size on a cache miss.

People spend more time looking at populated areas with larger tiles so 100+ kb for a mean tile response including hits makes sense.

[bdon]

So in summary, the only solution which might work for me is Cloudflare bucket + Cloudflare worker, where egress is free. Do you by any chance have a full planet test on a Cloudflare worker?

A site like https://bdon.github.io/s2js-demos/ uses the hosted API I run at api.protomaps.com, which has API keys to curb abuse. That returns responses from Cloudflare Workers, so I don't pay for egress on hits. You could set up workers to proxy from non-Cloudflare buckets, like a dedicated server, so you only pay egress for misses.

Here is a fresh Workers + R2 bucket deploy I just made following https://docs.protomaps.com/deploy/cloudflare

https://maps.protomaps.com/#map=5.27/41.346/-72.863&theme=light&lang=en&tiles=https://cool-test.protomaps-dev.workers.dev/z7.json (only z0-z7)

there is a separate discussion of breaking up files on Cloudflare R2 here: https://github.com/protomaps/PMTiles/discussions/465

[hyperknot]

This works well, I think the key is the z7 limit. Cloudflare really suffers from big files, I think they just don't cache those requests or don't distribute it across multiple hosts on their backend. The smaller it gets, the more chance it'll be distributed. At least this is my theory about their technical side.

[bdon]

Cloudflare really suffers from big files, I think they just don't cache those requests or don't distribute it across multiple hosts on their backend. The smaller it gets, the more chance it'll be distributed.

I haven't seen any evidence to support this theory (see discussion 465 above), it's just slow for small and large files.

Cloudflare R2 does not do any distribution as far as I know, and each bucket only exists in a single region that you can choose upon bucket creation. Workers however run at the POP closest to the user (also depending on subscription level, see http://cloudflare-test.judge.sh) so you can encounter high latency situations when the POP is distant from the R2 bucket.

A newer storage system that does distribute static objects across multiple regions is http://tigrisdata.com which I'm working on adding the the docs. Anyways, thanks for clarifying and testing and congrats on your launch!

[hyperknot]

I get at least $104 in Google Cloud pricing calculator for 1 TB in egress fees, so I'm quite surprised you only paid $5.88. Maybe the free allowance is still having a big affect on your amount.

As you can see, my philosophy is to be cloud-free. I'm basically only using Cloudflare buckets for the big file downloads, end-users never touch anything cloud related. I believe this is the only way to have control over costs, there are so many horror stories of people being burned by unexpected cloud costs.

Having said this, if Cloudflare fixes their behaviour (basically they need to start measuring this and addressing it in their eng team) then I might migrate to PMTiles in the future.