Bug: building visualization only partly / depending on zoom level/ but buildings are always clickable

[traveller195]

Describe the problem there are some rendering problems after importing building data.

So, in some zoom levels, only parts of the buildings are displayed. After zoom in, all buildings appear.

But the building which were not visualized in the map, also are clickable and get the red stroke after the manually selection by mouse click.

some are after zoom in. Now, you can see the building in the map

also without rendered buildings, there are clickable:

Expected behavior to display all buildings on all zoom levels

Screenshots see above

More information What I made already:

• first I used buildings for whole Saxony (approx. 2.6 millions), now I reduced it to only city of Dresden (approx. 160.000 buildings) but the problem is the same
• I initially changed the bbox extent for tile renderer from London to Dresden. is it here important to use only e.g. coordinates with not to many digits after the decimal point?

@matkoniecz @tomalrussell are there any other performance parameter for tile renderer e.g. which could help here to improve the perfomance of the visualizations of buildings?

thanks in advance :-)

[traveller195]

also I changed now the extent from London to Saxony (which I forgot before), but unfortunately it could not solve the rendering problem see https://github.com/colouring-cities/colouring-dresden/commit/63c633fe7bf2b1842e73813babbd22f48bd9f2bc

[matkoniecz]

https://user-images.githubusercontent.com/57902113/220087052-5cbca7ec-0b25-4c3e-992a-606cc9edffea.png - this looks like in progress rendering with some tiles not yet rendered

Can you look at console log or at running threads on server (for example with top command?)? I expect that you will see in progress rendering work.

Can you look at number of cached tile images and check is it growing?

Note that initial display is especially performance heavy as tiles are not yet cached.

is it here important to use only e.g. coordinates with not to many digits after the decimal point?

it should not matter

[traveller195]

I also changed now the boundaries in /var/www/colouring-dresden/app/public/geometries the behavior of displayed buildings changed a bit. but it is similar to the initial issue.

thanks for your hints. I will check it

[traveller195]

@matkoniecz in firefox the console log / after F12 shows a yellow warning "Speicherverbrauch von "will-change" ist zu hoch. Die Budget-Grenze ist die Fläche des Dokuments multipliziert mit 3 (1150720 px). Vorkommen von "will-change", die das Budget überschreiten, werden ignoriert."

like this here: https://www.joomshaper.com/forum/question/3331

[traveller195]

500 Internal Server Error

[matkoniecz]

What happens if you wait and open this specific tile image in browser, from this 500 error failing links?

500 error code is sometimes returned when server is overloaded and it is possible when low-powered server is used or user pans across never cached area and server cannot keep up.

(ideally it would be fixed in CC code so that it returns less cryptic errors AND/OR is more performant)

"Speicherverbrauch von "will-change" ist zu hoch. Die Budget-Grenze ist die Fläche des Dokuments multipliziert mit 3 (1150720 px). Vorkommen von "will-change", die das Budget überschreiten, werden ignoriert."

almost certainly unrelated one

[traveller195]

so I tried it to open

http://colouring.dresden.ioer.de/tiles/building_attachment_form/18/141075/87705.png?rev=0

(currently I guess only availabl at IOER intranet. But I want to open it soon, also to be able to show you etc. the colouring dresden version)

In browser, an white empty tile appears:

[matkoniecz]

but it loaded without error, right?

empty tile is a normal case here in area without buildings like https://cl-staging.uksouth.cloudapp.azure.com/tiles/building_attachment_form/18/130947/87168.png?rev=7686617 from park.

And in case where buildings have no provided data.

Can you try editing Adjacency/configuration field in Typology? Then tile overlay will have some buildings.

Can you reopen the initial problematic area? Does it load more tiles now? If not - try to flush local cache.

[traveller195]

we did some tests (and I will soon also test your suggested cases!)

just quick here:

• I changed http to https for OpenStreetMap base map service, now the page is more secure only using https
• while loading a new Colouring Dresden platform, the server in the backend goes up to approx. 60% CPU usage, but also extending it from 3 to 4 cores could not fix the problem

In network analysis from firefox Browser, I can see, that some tiles (mostly from OSM) got an succesful response. But others, most of the Colouring Dresden tile server, get not response. When I zoom in, the building will be displayed in tiles.

I was wondering, whether it coul problematic, that first there will be loaded a lot a huge data in geojson files from London, we will not need for Dresden! it is initial about 50MB data. see screenshot And, perhaps, the problem ist, loading those London planning datasets takes to much time and create a timeout for the tile service ? so, that only a few or no building tiles get a response?

@matkoniecz @tomalrussell Could this be the problem? What is the easist way to handle it? Just replacing the geojson files /public/geometries by the Dresden data? or by some empty dummy geojson files?

thanks in advance and I will do your suggested test, as well @matkoniecz

[matkoniecz]

And, perhaps, the problem ist, loading those London planning datasets takes to much time and create a timeout for the tile service ?

If you have not run planning data load script then your database should have no planning permission data in database at all (distinct from that geojsons you mention later)

Planning data stored in database should affect only planning data tile layers

I was wondering, whether it coul problematic, that first there will be loaded a lot a huge data in geojson files from London, we will not need for Dresden! it is initial about 50MB data. see screenshot

This should not interfere with tile generation, but you can try replacing this area data with empty geojsons

[matkoniecz]

If I would be randomly guessing performance issues: is it possible that you would benefit from reindexing data?

It is possible that in CL data was loaded then one of migrations introduced indexes or reindexing was run manually, and here all migrations were run then data was imported?

[matkoniecz]

But others, most of the Colouring Dresden tile server, get not response.

Is cached tile count increasing at least? Or are no new tiles generated at all?

[traveller195]

If I would be randomly guessing performance issues: is it possible that you would benefit from reindexing data?

@matkoniecz I did a reindex for buildings and geometries using the files for migrations (first down, then up again). To create new (SPATIAL) INDEX for those tables

after loading and displaying the map ... the tiles will be loaded 'from cache' so it should work

mh. it is confusing.

In the light mode of the map, I can see the buildings tiles in some zoom level. (not in all, if I zoom out all disappear). but in the dark mode on the some zoom level, the buildings disappear partly and will be shown only after zoom in again

[traveller195]

@matkoniecz Update: I derived and joined building height data for all buildings within Dresden, Germany. So, it is now better visible if a tile is loaded or not. please go to category "Größe" = size... (you will know it better than me ) ;-)

The first thought for me was, that we have - I guess - 3 different layers of tiles here:

1.) base map raster tiles from OpenStreetMap or OS 2.) buidling tiles showing all buildings as gray polygons 3.) coloured buildings, showing the selected building feature

here, wen can see it:

it is in the old city of Dresden: 1.) you can see some buildings which are not included to the official buidling dataset I used in the bottom left corner (= typical OSM style from OSM tiles) 2.) you can see the "first" building tiles in gray, which are alost everywhere (and god to see in light mode... in dark mode, I am not sure if those tiles are shown) 3.) we cann see the magenta/ pink coloured buildings showing building height in the top of the screenshot... this behaviour we know already from tests in the past (compare other comments here in this issue)

[matkoniecz]

The first thought for me was, that we have - I guess - 3 different layers of tiles here: 1.) base map raster tiles from OpenStreetMap or OS 2.) buidling tiles showing all buildings as gray polygons 3.) coloured buildings, showing the selected building feature

yes, and also borough labels as fourth one

And if you enable historic map layer (I guess that its link and attribution is the next candidate for configuration...) you will get fifth one.

3.) we cann see the magenta/ pink coloured buildings showing building height in the top of the screenshot... this behaviour we know already from tests in the past (compare other comments here in this issue)

Yes, something is definitely going wrongly here

[matkoniecz]

Update: I derived and joined building height data for all buildings within Dresden, Germany.

have you cleared cache of rendered (empty) tiles for this layer?

Cache is flushed when edited manually but direct database operation will NOT clear cached tiles, so empty ones will continue to edit unless some manual edits were made.

[traveller195]

now I also cleared cache of browser. And it wouls the same behaviour.

And I can als add, that in city center there is a rechtangle (with some irregular smaller rectangles in addition) without pink building height tiles. but outside, it is shown

Can you see the same when you laod Colouring Dresden?

[traveller195]

additional note: most of the tests I made with firefox browser (only sometimes I changed to MS Edge): now again, I test it with MS Edge and there similarities and differences: being on the same zoom level like in firefox, I can also see that rectangle in the center without pink tiles. But when I zoom out in Edge one level, here all pink tiles will be shown (in Firefox nothing is displayed here)

okay... mh so , the tile service should work good. But maybe it is more a caching/ browser problem?

if you go to colouring.dresden.ioer.de is the map view fine for your? all tiles are shown?

[matkoniecz]

now I also cleared cache of browser. And it wouls the same behaviour.

Have you cleared tiles cached on the server?

[tomalrussell]

Have you cleared tiles cached on the server?

I had the same question - it looks to me like some central tiles were rendered and cached before you joined the building heights data.

See the configured cache location in your ecosystem config, like here: https://github.com/colouring-cities/colouring-dresden/blob/1e560352925155e0b9ede01a5712228a4af52c9c/ecosystem.config.template.js#L20

The server-side tile cache is created and configured here: https://github.com/colouring-cities/colouring-dresden/blob/8d251b27ae45b2a3b5122457f29c5f1526c16433/app/src/tiles/rendererDefinition.ts#L41-L53

Also - as an opinionated comment - I had a strong policy of always making all edits to add building data through the API (however quick and easy it would be to use direct database access) - this means that all edits are logged and attributable to a user, the cache is automatically expired, and any other data validation that is implemented in the application code can run and check the edit.

[traveller195]

@tomalrussell good point, thanks. I also was wondering if it is the best to add data without edit history/ logs.

[traveller195]

@matkoniecz @tomalrussell so, I cleared now tilecache on server side (before, I always just cleared caching in browser client)

rm -r tilecache/*

and I guee, this was the problem. So, now It worked as excepted. Hopefully, there will be no other problems.

Thanks for your support. I could learn a lot about the app