Closed Deam94 closed 1 year ago
systemed
commented
2 years ago Is there a particular reason you want to create z18 tiles?
Generally with vector tiles, you'll create up to z14, but these z14 tiles will have all the information needed to render at z18 (and beyond). See https://github.com/systemed/tilemaker/blob/master/docs/VECTOR_TILES.md#the-format
Deam94
commented
2 years ago Yes i need z18 tiles for my application. It propably would work to create mbtiles for each country, but that would be a lot of work foreach country on planet.
Is there no workaround for "out of memory" issue ? Is this issue caused because of limited memory of 10gb ?
systemed
commented
2 years ago I would genuinely be interested to know what your application is as I've never heard of anyone needing z18 vector tiles before!
tilemaker will have a lot of overhead creating z18 tiles for such a large area, and so the memory usage will be high. If you're just using a raw extract from (say) Geofabrik, outlying islands will probably cause a lot of sea tiles to be generated, so you could save some memory by supplying a bounding box that's closer to the main landmass only.
Deam94
commented
2 years ago I dont display the map in my application but need the data for research, z18 tiles have a perfect size, z17 are already to big.
Might it would help to reduce data content, lower details ? I dont need sharp edges etc. For me it would be fine if a street run completly straight inside of a z18 tile instead of making small curves...
systemed
commented
2 years ago It might help a bit, but ultimately tilemaker keeps an index of the features that are needed in each individual tile, and the more tiles you have, the more memory that needs.
For research/analysis, the most usual approach is to put OSM data into a Postgres database (using osm2pgsql or similar) and then query that using the PostGIS analysis functions - tilemaker isn't really designed for this sort of use-case I'm afraid.
Deam94
commented
2 years ago According info page osm2pgsql take a lot of memory aswell (~64gb memory for 64gb osm file) and according my fast research the output dont seem to be in a similar format like tilemaker is able to (z18,y,x,data; which is perfect for me, tilemaker is awesome). Osmium-renumber takes a lot of memory aswell else --compact mode would have been an option and i havent seen a osmium-renumber prepared planet.osm.pbf file or continents.
Well i will stick to tilemaker and use the --bbox option to generate parts of each continent step by step. I think thats the best solution for me.
Is there a way to lower details to reduce overhead a little bit ? What does -DFAT_WAYS does ? Or maybe can you link me a documentation of available build options ? Thanks a lot.
systemed
commented
2 years ago -DFAT_WAYS isn't needed any more. The only build option is -DFAT_TILE_INDEX which I think you're probably using already - that's the one that enables output at z17+. --bbox sounds like the best solution for now, I think!
ImreSamu
commented
2 years ago I dont display the map in my application but need the data for research, z18 tiles have a perfect size, z17 are already to big.
https://github.com/mapbox/vector-tile-spec/blob/master/2.1/README.md
Deam94
commented
2 years ago @ImreSamu thanks for your post. I need z18 tiles informations locally inside my mbtiles file with the correct tile_x(z18),tile_y(z18) and data(z18) value. It would require a tool to rewrite the whole database and split up each z14 tile into many small tiles(~256 ?) and fetch the matching data to it. I have not found anything like this and could imagine my setup isnt good enough for conversion of the whole planet.
I have build tilemaker with -DFAT_TILE_INDEX which allows to create z18 tiles on the fly.
I was able to build whole south america thanks to bbox option and 5 runs.
ImreSamu
commented
2 years ago @Deam94
( for research purposes ) in addition to postgis/osm2pgsql .. most people using osm-qa-tiles - Zoom12 Vector tiles and it has a ~1cm precision.
I need z18 tiles informations locally inside my mbtiles file
I don't know what other program gets this z18 data, but I would check if it supports overzooming.
Deam94
commented
2 years ago In the end i have created a shell script and made z18 vector tiles for each country, which worked perfectly beside russia. Now im merging them with DB browser.
I am not sure if this is a issue with tilemaker or hardware/software.
I am trying to create level 18 tiles of south america(3gb) with highway data on a virtual machine (virtualbox). The virtual machine spec: cpu 4cores i7 4790k memory 10gb ddr3 hard drive ssd(40gb)&hdd(250gb)
I have tried a few different settings like less memory, fixedspace drive, different drives, ssd, always in combination with --store ... the result is always the same: "Out of memory: Killed process"
The process on ssd does not take a lot, node creation and sorting, same as ways is finished in 15 minutes, but instantly after i get the error. --store creates around 10gb+ of space on ssd from 3gb osm.pbf.
For very small regions its working fine.
Log: