Open joto opened 2 months ago
Thank you for creating the issue @joto, I have begun to look into implementing this into render_expired
and have come up with some ideas for preventing duplicate work based on your comment and reading the linked comments that I am testing out.
I'm not sure what you mean here though:
We could just write out expire list on zoom level 16 instead.
as metatiles do not contain tiles spanning multiple zoom levels, Could you please explain what you mean there?
What I mean is that instead of writing out, say z19,x453726,y39265, I write out z16,x56715,y4908 in the expire list. z16,x56715,y4908 is the metatile that contains z19,x453726,y39265 (and many neighboring tiles). Then render_expired
expires the metatile which contains z19,x453726,y39265. Basically we have to remove the code in render_expired
which calculates the matatile from the tile it is given and instead we give it the metatile directly.
It seems like that is basically what is already being done, more or less, here's an example command:
printf "16/56715/4908\n17/113420/9816\n18/226860/19632\n19/453726/39265\n" | \
render_expired --min-zoom 3 --max-zoom 19 --verbose --touch-from 0
16/56715/4908
) and a loop is started on it for zoom levels 3 to 19 (min-zoom
to max-zoom
):
** Message: Read valid line: 16/56715/4908
** Message: Starting loop on 19/453720/39264 for zoom levels 3 to 19
requested
, and the metatiles do not exist on my disk, so they are skipped
(processing would indeed occur if they were to exist)
** Message: Processing: 19/453720/39264
** Message: Skipping 'file:///var/cache/renderd/tiles/default/19/96/233/201/86/128.meta' (metatile does not exist)
** Message: Processing: 18/226860/19632
** Message: Skipping 'file:///var/cache/renderd/tiles/default/18/48/116/108/43/128.meta' (metatile does not exist)
** Message: Processing: 17/113430/9816
** Message: Skipping 'file:///var/cache/renderd/tiles/default/17/16/178/182/21/8.meta' (metatile does not exist)
** Message: Processing: 16/56715/4908
** Message: Skipping 'file:///var/cache/renderd/tiles/default/16/0/209/211/130/136.meta' (metatile does not exist)
** Message: Processing: 15/28357/2454
** Message: Skipping 'file:///var/cache/renderd/tiles/default/15/0/96/233/201/0.meta' (metatile does not exist)
** Message: Processing: 14/14178/1227
** Message: Skipping 'file:///var/cache/renderd/tiles/default/14/0/48/116/108/8.meta' (metatile does not exist)
** Message: Processing: 13/7089/613
** Message: Skipping 'file:///var/cache/renderd/tiles/default/13/0/16/178/182/0.meta' (metatile does not exist)
** Message: Processing: 12/3544/306
** Message: Skipping 'file:///var/cache/renderd/tiles/default/12/0/0/209/211/128.meta' (metatile does not exist)
** Message: Processing: 11/1772/153
** Message: Skipping 'file:///var/cache/renderd/tiles/default/11/0/0/96/233/136.meta' (metatile does not exist)
** Message: Processing: 10/886/76
** Message: Skipping 'file:///var/cache/renderd/tiles/default/10/0/0/48/116/8.meta' (metatile does not exist)
** Message: Processing: 9/443/38
** Message: Skipping 'file:///var/cache/renderd/tiles/default/9/0/0/16/178/128.meta' (metatile does not exist)
** Message: Processing: 8/221/19
** Message: Skipping 'file:///var/cache/renderd/tiles/default/8/0/0/0/209/128.meta' (metatile does not exist)
** Message: Processing: 7/110/9
** Message: Skipping 'file:///var/cache/renderd/tiles/default/7/0/0/0/96/136.meta' (metatile does not exist)
** Message: Processing: 6/55/4
** Message: Skipping 'file:///var/cache/renderd/tiles/default/6/0/0/0/48/0.meta' (metatile does not exist)
requested
, and the metatiles do exist on my disk, so they are touched
(as per --touch-from 0
)
** Message: Processing: 5/27/2
** Message: Touching 'file:///var/cache/renderd/tiles/default/5/0/0/0/16/128.meta'
** Message: Processing: 4/13/1
** Message: Touching 'file:///var/cache/renderd/tiles/default/4/0/0/0/0/128.meta'
** Message: Processing: 3/6/0
** Message: Touching 'file:///var/cache/renderd/tiles/default/3/0/0/0/0/0.meta'
17/113420/9816
) and a loop is again started on it for zoom levels 3 to 19 (min-zoom
to max-zoom
):
** Message: Read valid line: 17/113420/9816
** Message: Starting loop on 19/453680/39264 for zoom levels 3 to 19
requested
, and the metatiles do not exist on my disk, so they are skipped
(processing would indeed occur if they were to exist)
** Message: Processing: 19/453680/39264
** Message: Skipping 'file:///var/cache/renderd/tiles/default/19/96/233/201/54/0.meta' (metatile does not exist)
** Message: Processing: 18/226840/19632
** Message: Skipping 'file:///var/cache/renderd/tiles/default/18/48/116/108/27/128.meta' (metatile does not exist)
** Message: Processing: 17/113420/9816
** Message: Skipping 'file:///var/cache/renderd/tiles/default/17/16/178/182/5/136.meta' (metatile does not exist)
** Message: Processing: 16/56710/4908
** Message: Skipping 'file:///var/cache/renderd/tiles/default/16/0/209/211/130/8.meta' (metatile does not exist)
requested
** Message: Already requested metatile containing '15/28355/2454', moving on to next input line
18/226860/19632
) and nothing needs to be done as its metatile and all above and below it have already been requested
:
** Message: Read valid line: 18/226860/19632
** Message: Starting loop on 19/453720/39264 for zoom levels 3 to 19
** Message: Already requested metatile containing '19/453720/39264', moving on to next input line
19/453726/39265
) and again nothing needs to be done as its metatile and all below it have already been requested
:
** Message: Read valid line: 19/453726/39265
** Message: Starting loop on 19/453726/39265 for zoom levels 3 to 19
** Message: Already requested metatile containing '19/453726/39265', moving on to next input line
When a tile is expired on, say, zoom level 19, it is marked in the expire list in that zoom level 19. But really we are using metatiles, so always the whole metatile is getting expired. If there are a lot of tiles in the same metatile, they are all in the expire list. In the worst case there are 8x8=64 times the number of entries in the expire list than we would really need. We could just write out expire list on zoom level 16 instead.
So I propose some kind of option to
render_expires
(and mayberender_list
) that says: interpret zoom levels as meta tile zoom levels, not tile zoom levels.See also: https://github.com/osm2pgsql-dev/osm2pgsql/issues/2190