search

systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.4k stars 225 forks source link

Weekly update world osm mbtiles #569

Open timprepscius opened 8 months ago

timprepscius commented 8 months ago

Thank you for spending so much time on this project. I am using it at the moment to build the world. 16GB ram and 200GB "--store" space. Hopefully it will work.

I was wondering whether it would be possible to set up a weekly build of the world. Because:

  1. You obviously have the machines to do it. I'm assuming you have a machine with a ton of ram.
  2. There may be free options to host it.
  3. This may be valuable in the future to see how cities and roads change over the years.

With #2 I'm referring to something like: https://archive.org/details/osm-vector-mbtiles

This would also serve as a test of your system to see if changes affect memory and cpu performance.

In any case- thank you again for your work on this project.

systemed commented 8 months ago

I like the idea. I couldn't really commit to doing it with my hardware though - the machine I have is actually here in my study and hence uses my home connectivity and power, rather than being colocated.

But it might be worth revisiting and looking at options when some of the work to reduce tilemaker's memory footprint (#499) is complete.

timprepscius commented 8 months ago

Great.

In my projects, I have found the xcode memory profiling tools to be excellent. (when compared to the linux assortment - I don't know about windows). You can also run Xcode from within a linux hosted osx docker via sickcodes.

I know this is a pretty random comment, however, understanding memory usage is sometimes non-trivial, and the tools can help a lot.

I wish you luck!

henryreed commented 4 weeks ago

Hi @timprepscius. You mentioned that there may be free options to host the data. Were you thinking of just archive.org or are there better options? Separately, would you want both mbtiles and pmtiles, or would pmtiles only suffice?

timprepscius commented 3 weeks ago

Hey there,

I actually upgraded and bought like 128gb of memory in order to generate the tilesets. I am only using MBTiles at the moment. I would need to write more code for PMTiles, which I don't have time at the moment. (although, maybe there is a mbtile front send server for pmtile data?)

Umm, I was thinking of archive.org I think. Oh, and I think amazon has some sort of "public data" archive.

I have a question:

During dev, I generated my own MBTiles and created the huge sqlite file. I then run it off of my computer computer (actually using a low speed usb drive) and ssh pipe from the server. It's not publicly accessible.

This works great for dev, but going to prod, especially as my internet connection sometimes goes down when it rains...

I'm thinking of doing this:

  1. Generate a disk image containing the SQLite database.
  2. Upload that disk image to Cloudflare R2
  3. Use s3fs to mount that bucket
  4. Mount the file system.
  5. Use the file system to drive the mbtile server.

On a scale of 1-10, how dumb is this? This would lower my costs from $50 a month for a server with the space for the tile set to $5 ($2 for the server, $x for the R2)

Most things are cached on the server anyhow, so it wouldn't be accessing R2 that often.

I know this is off topic, but just searching for a sanity check.

systemed commented 3 weeks ago

I'd look at pmtiles rather than mbtiles if you're doing that - it's specifically designed for serving from the cloud. Have a look at https://protomaps.com/ which explains how it all works.