search

camptocamp / ngeo

Library combining OpenLayers and AngularJS
https://camptocamp.github.io/ngeo/master/examples
MIT License
135 stars 86 forks source link

Huge performance issues with many layergroups #2377

Closed rbovard closed 6 years ago

rbovard commented 7 years ago

On mobile, when many layergroups are displayed, it's almost impossible to navigate:

  1. Open https://map-demo.nyon.ch/v2/mobile/theme/amenagement_territoire
  2. Activate all layergroups
  3. Navigate --> KO

On desktop with the same config it's smooth:

  1. Open https://map-demo.nyon.ch/v2/theme/amenagement_territoire
  2. Activate all layergroups
  3. Navigate --> OK

It's a big regression here, any idea about those differences?

sbrunner commented 7 years ago

I don't see a big deference, witch device/navigator do you use?

rbovard commented 7 years ago
fredj commented 7 years ago

I confirm that the mobile interface is not usable when all the layers are displayed (nexus 5x)

sbrunner commented 7 years ago

For me it's working better with Firefox then with Crome on the same phone...

rbovard commented 7 years ago

Related to #2378

fredj commented 7 years ago

It's also reproductible with the demo: https://testgmf.sig.cloud.camptocamp.net/2.1/mobile/theme/Demo

fredj commented 7 years ago

This may be related to the high DPI requests to mapserver. On my Nexus 5X, the device pixel ratio is 3; so the images are 9 times bigger.

Maybe a GPU memory limit ?

rbovard commented 7 years ago

As I said before, I have the issue on mobile and PC.

sbrunner commented 7 years ago

In conclusion Chome on Android has poor performance when many layers are displayed on the canvas...

rbovard commented 7 years ago

@sbrunner Did you read my comment here https://github.com/camptocamp/ngeo/issues/2377#issuecomment-282700249

sbrunner commented 7 years ago

Yes but I can't reproduce it on desktop...

rbovard commented 7 years ago

Did you try with my permalink (with taking care of #2386) because with all layergroups of theme amenagement_territoire loaded, it's catastrophic.

rbovard commented 7 years ago

I just retested on desktop and mobile (Nexus 5x - Android 7.1.1) with 2.1.2 and it still catastrophic.

gnerred commented 7 years ago

Strange... If you go on Cartoriviera in theme "Aménagement" with a mobile device :

https://map.cartoriviera.ch/mobile/theme/amenagement_du_territoire

And you activate all layers (number of layers is more or less the same than in Nyon), it runs relative normally without huge performance issue. Can you confirm ? Why ?

rbovard commented 7 years ago

The diffrence is that you have only 2 layergroups:

gnerred commented 7 years ago

Ah OK sorry, I did not well understand the original issue. So I haven't this performance issue in Cartoriviera, because I haven't any theme with a lot of layergroups.

rbovard commented 7 years ago

@gnerred Lucky you ;)

@fredj #2378 was not supposed to fix this issue?

gnerred commented 7 years ago

I also have a lot of layergroups, but of second level. So it seems that this issue is only related to layergroups of first level.

rbovard commented 7 years ago

Yes because first level layergroup are 1 GetMap query = 1 big image to display and sub level layergroups are grouped in only 1 GetMap query --> so there is not performance issue in this case.

In cxpg it's the same logical, 1 GetMap query by first level group and it works fine! So it's ngeo which can not handle correcty multiple WMS layers...

sbrunner commented 7 years ago

@rbovard with ngeo and cgxp we should have the same number of GetMap request, do you have some examples permalink?

rbovard commented 7 years ago

@sbrunner It's what I said since the beginning, there are same number of GetMap's, but ngeo cannot handle 6 layers...

The steps to reproduce are in my initial message.

sbrunner commented 7 years ago

I have 6 requests in booth case, then it's working :-)

rbovard commented 7 years ago

Yes 6 GetMap's in both case, but with ngeo it's impossible to navigate which is the current issue...

Please see with @fredj if it's not clear.

fredj commented 7 years ago

Version 2.x and 1.x loads the same number of layer. The only difference is that in version 2.x, the device pixel ratio is taken into account: larger images are requested to the server.

I've run the performance debugger tools on https://map-demo.nyon.ch/v2/mobile/theme/amenagement_territoire with all the layers groups (nexus 5x).

It seems that the bottleneck is the image decoding by the browser: img_decode

fredj commented 7 years ago

I've tried the load https://map-demo.nyon.ch/v2/mobile/theme/amenagement_territoire in chrome on a Windows 7 and it's working perfectly.

@rbovard can you please give us the content of this url: chrome://gpu and open a debugger and the value of window.devicePixelRatio

rbovard commented 7 years ago 
Graphics Feature Status
Canvas: Hardware accelerated
Flash: Hardware accelerated
Flash Stage3D: Hardware accelerated
Flash Stage3D Baseline profile: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Enabled
Native GpuMemoryBuffers: Software only. Hardware acceleration disabled
Rasterization: Software only, hardware acceleration unavailable
Video Decode: Hardware accelerated
Video Encode: Hardware accelerated
VPx Video Decode: Software only, hardware acceleration unavailable
WebGL: Hardware accelerated
WebGL2: Hardware accelerated
Driver Bug Workarounds
clear_uniforms_before_first_program_use
decode_encode_srgb_for_generatemipmap
disable_direct_composition
disable_discard_framebuffer
disable_dxgi_zero_copy_video
disable_framebuffer_cmaa
exit_on_context_lost
force_cube_complete
scalarize_vec_and_mat_constructor_args
texsubimage_faster_than_teximage
Problems Detected
VPx decoding isn't supported before Windows 10 anniversary update.: 616318
Disabled Features: accelerated_vpx_decode
GPU rasterization should only be enabled on NVIDIA Pascal and Maxwell, Intel Broadwell+, and AMD RX-R2 GPUs for now.: 643850
Disabled Features: gpu_rasterization
Some drivers are unable to reset the D3D device in the GPU process sandbox
Applied Workarounds: exit_on_context_lost
TexSubImage is faster for full uploads on ANGLE
Applied Workarounds: texsubimage_faster_than_teximage
Clear uniforms before first program use on all platforms: 124764, 349137
Applied Workarounds: clear_uniforms_before_first_program_use
Always rewrite vec/mat constructors to be consistent: 398694
Applied Workarounds: scalarize_vec_and_mat_constructor_args
ANGLE crash on glReadPixels from incomplete cube map texture: 518889
Applied Workarounds: force_cube_complete
Framebuffer discarding can hurt performance on non-tilers: 570897
Applied Workarounds: disable_discard_framebuffer
Direct composition flashes black initially on Win <10: 588588
Applied Workarounds: disable_direct_composition
Zero copy DXGI video hangs on shutdown on Win < 8.1: 621190
Applied Workarounds: disable_dxgi_zero_copy_video
Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198
Applied Workarounds: disable_framebuffer_cmaa
Zero-copy NV12 video displays incorrect colors on NVIDIA drivers.: 635319
Applied Workarounds: disable_dxgi_zero_copy_video
Disable KHR_blend_equation_advanced until cc shaders are updated: 661715
Decode and Encode before generateMipmap for srgb format textures on Windows: 634519
Applied Workarounds: decode_encode_srgb_for_generatemipmap
Native GpuMemoryBuffers have been disabled, either via about:flags or command line.
Disabled Features: native_gpu_memory_buffers
Version Information
Data exported   25/04/2017 à 10:28:05
Chrome version  Chrome/57.0.2987.133
Operating system    Windows NT 6.1.7601 SP1
Software rendering list version 12.13
Driver bug list version 9.29
ANGLE commit id c1a5d16e964a
2D graphics backend Skia/57 ae9cc5d3588d52f4b371b55845704b25d88cf06d
Command Line Args   Files (x86)\Google\Chrome\Application\chrome.exe" --flag-switches-begin --flag-switches-end
Driver Information
Initialization time 46
In-process GPU  false
Passthrough Command Decoder false
Sandboxed   false
GPU0    VENDOR = 0x10de, DEVICE= 0x10d8
Optimus false
AMD switchable  false
Desktop compositing Aero Glass
Diagonal Monitor Size of \\.\DISPLAY2   22.9"
Diagonal Monitor Size of \\.\DISPLAY1   22.9"
Driver vendor   NVIDIA
Driver version  9.18.13.3343
Driver date 7-22-2014
Pixel shader version    4.1
Vertex shader version   4.1
Max. MSAA samples   8
Machine model name  
Machine model version   
GL_VENDOR   Google Inc.
GL_RENDERER ANGLE (NVIDIA NVS 300 Direct3D11 vs_4_1 ps_4_1)
GL_VERSION  OpenGL ES 3.0 (ANGLE 2.1.0.c1a5d16e964a)
GL_EXTENSIONS   GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc_srgb GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
Disabled Extensions GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent
Window system binding vendor    Google Inc. (adapter LUID: 0000000000007914)
Window system binding version   1.4 (ANGLE 2.1.0.c1a5d16e964a)
Window system binding extensions    EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource
Direct rendering    Yes
Reset notification strategy 0x8252
GPU process crash count 0
Compositor Information
Tile Update Mode    One-copy
Partial Raster  Enabled
GpuMemoryBuffers Status
ATC Software only
ATCIA   Software only
DXT1    Software only
DXT5    Software only
ETC1    Software only
R_8 Software only
RG_88   Software only
BGR_565 Software only
RGBA_4444   Software only
RGBX_8888   Software only
RGBA_8888   Software only
BGRX_8888   Software only
BGRA_8888   Software only
YVU_420 Software only
YUV_420_BIPLANAR    Software only
UYVY_422    Software only
Diagnostics
0
b3DAccelerationEnabled  true
b3DAccelerationExists   true
bAGPEnabled true
bAGPExistenceValid  true
bAGPExists  true
bCanRenderWindow    true
bDDAccelerationEnabled  true
bDriverBeta false
bDriverDebug    false
bDriverSigned   false
bDriverSignedValid  false
bNoHardware false
dwBpp   32
dwDDIVersion    10
dwHeight    1080
dwRefreshRate   60
dwWHQLLevel 0
dwWidth 1920
iAdapter    1
lDriverSize 18241624
lMiniVddSize    0
szAGPStatusEnglish  Enabled
szAGPStatusLocalized    Activé
szChipType  NVS 300
szD3DStatusEnglish  Enabled
szD3DStatusLocalized    Activé
szDACType   Integrated RAMDAC
szDDIVersionEnglish 10.1
szDDIVersionLocalized   10.1
szDDStatusEnglish   Enabled
szDDStatusLocalized Activé
szDXVAHDEnglish Supported
szDXVAModes ModeMPEG2_A ModeMPEG2_C ModeVC1_C ModeWMV9_C
szDescription   NVIDIA NVS 300
szDeviceId  0x10D8
szDeviceIdentifier  {D7B71E3E-5398-11CF-837E-6F281FC2C435}
szDeviceName    \\.\DISPLAY1
szDisplayMemoryEnglish  4050 MB
szDisplayMemoryLocalized    4050 MB
szDisplayModeEnglish    1920 x 1080 (32 bit) (60Hz)
szDisplayModeLocalized  1920 x 1080 (32 bit) (60Hz)
szDriverAssemblyVersion 9.18.13.3343
szDriverAttributes  Final Retail
szDriverDateEnglish 12/17/2015 18:23:23
szDriverDateLocalized   17.12.2015 18:23:23
szDriverLanguageEnglish English
szDriverLanguageLocalized   Anglais
szDriverModelEnglish    WDDM 1.1
szDriverModelLocalized  WDDM 1.1
szDriverName    nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um
szDriverNodeStrongName  oem111.inf:NVIDIA_SetA_Devices.NTamd64.6.1:Section126:9.18.13.3343:pci\ven_10de&dev_10d8&subsys_0862103c
szDriverSignDate    
szDriverVersion 9.18.0013.3343
szKeyDeviceID   Enum\PCI\VEN_10DE&DEV_10D8&SUBSYS_0862103C&REV_A2
szKeyDeviceKey  \Registry\Machine\System\CurrentControlSet\Control\Video\{0AA5C0DA-23E9-462D-98A4-90368E0630D0}\0000
szManufacturer  NVIDIA
szMiniVdd   n/a
szMiniVddDateEnglish    n/a
szMiniVddDateLocalized  n/a
szMonitorMaxRes 
szMonitorName   Moniteur Plug-and-Play générique
szNotesEnglish  No problems found.
szNotesLocalized    Aucun problème n’a été détecté.
szOverlayEnglish    Supported
szRankOfInstalledDriver 00E00001
szRegHelpText   
szRevision  
szRevisionId    0x00A2
szSubSysId  0x0862103C
szTestResultD3D7English Not run
szTestResultD3D7Localized   Non exécuté
szTestResultD3D8English Not run
szTestResultD3D8Localized   Non exécuté
szTestResultD3D9English Not run
szTestResultD3D9Localized   Non exécuté
szTestResultDDEnglish   Not run
szTestResultDDLocalized Non exécuté
szVdd   n/a
szVendorId  0x10DE
1
b3DAccelerationEnabled  true
b3DAccelerationExists   true
bAGPEnabled true
bAGPExistenceValid  true
bAGPExists  true
bCanRenderWindow    true
bDDAccelerationEnabled  true
bDriverBeta false
bDriverDebug    false
bDriverSigned   false
bDriverSignedValid  false
bNoHardware false
dwBpp   32
dwDDIVersion    10
dwHeight    1080
dwRefreshRate   60
dwWHQLLevel 0
dwWidth 1920
iAdapter    0
lDriverSize 18241624
lMiniVddSize    0
szAGPStatusEnglish  Enabled
szAGPStatusLocalized    Activé
szChipType  NVS 300
szD3DStatusEnglish  Enabled
szD3DStatusLocalized    Activé
szDACType   Integrated RAMDAC
szDDIVersionEnglish 10.1
szDDIVersionLocalized   10.1
szDDStatusEnglish   Enabled
szDDStatusLocalized Activé
szDXVAHDEnglish Supported
szDXVAModes ModeMPEG2_A ModeMPEG2_C ModeVC1_C ModeWMV9_C
szDescription   NVIDIA NVS 300
szDeviceId  0x10D8
szDeviceIdentifier  {D7B71E3E-5398-11CF-837E-6F281FC2C435}
szDeviceName    \\.\DISPLAY2
szDisplayMemoryEnglish  4050 MB
szDisplayMemoryLocalized    4050 MB
szDisplayModeEnglish    1920 x 1080 (32 bit) (60Hz)
szDisplayModeLocalized  1920 x 1080 (32 bit) (60Hz)
szDriverAssemblyVersion 9.18.13.3343
szDriverAttributes  Final Retail
szDriverDateEnglish 12/17/2015 18:23:23
szDriverDateLocalized   17.12.2015 18:23:23
szDriverLanguageEnglish English
szDriverLanguageLocalized   Anglais
szDriverModelEnglish    WDDM 1.1
szDriverModelLocalized  WDDM 1.1
szDriverName    nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um
szDriverNodeStrongName  oem111.inf:NVIDIA_SetA_Devices.NTamd64.6.1:Section126:9.18.13.3343:pci\ven_10de&dev_10d8&subsys_0862103c
szDriverSignDate    
szDriverVersion 9.18.0013.3343
szKeyDeviceID   Enum\PCI\VEN_10DE&DEV_10D8&SUBSYS_0862103C&REV_A2
szKeyDeviceKey  \Registry\Machine\System\CurrentControlSet\Control\Video\{0AA5C0DA-23E9-462D-98A4-90368E0630D0}\0001
szManufacturer  NVIDIA
szMiniVdd   n/a
szMiniVddDateEnglish    n/a
szMiniVddDateLocalized  n/a
szMonitorMaxRes 
szMonitorName   Moniteur Plug-and-Play générique
szNotesEnglish  No problems found.
szNotesLocalized    Aucun problème n’a été détecté.
szOverlayEnglish    Supported
szRankOfInstalledDriver 00E00001
szRegHelpText   
szRevision  
szRevisionId    0x00A2
szSubSysId  0x0862103C
szTestResultD3D7English Not run
szTestResultD3D7Localized   Non exécuté
szTestResultD3D8English Not run
szTestResultD3D8Localized   Non exécuté
szTestResultD3D9English Not run
szTestResultD3D9Localized   Non exécuté
szTestResultDDEnglish   Not run
szTestResultDDLocalized Non exécuté
szVdd   n/a
szVendorId  0x10DE
Log Messages
GpuProcessHostUIShim: The GPU process exited normally. Everything is okay.
rbovard commented 7 years ago

With all layergroups activated

Windows 7, Chrome:

Windows 7, Chrome (Nexus 5x emulated)

fredj commented 7 years ago

Everything looks hardware accelerated ...

Did you try with Edge?

rbovard commented 7 years ago

Works fine on W10/Edge...

rbovard commented 7 years ago

Any news on this one? I tested with 2.1.4 and still impossible to navigate with Chrome (mobile and desktop)...

Ping @ybolognini: it's clearly a blocker for 2.1 migration.

rbovard commented 7 years ago

Just tested 2.1.8 and it's better on desktop but not on mobile, which is problematic because it concerns the mobile version...

rbovard commented 7 years ago

@fredj Any news on this one?

rbovard commented 7 years ago

To try to fix this migration blocker, I edited all my themes to have a maximum of 3 layergroups (4 for Aménagement du territoire because WMS and WMTS cannot be grouped) and it works fine!

Thème Aménagement du territoire

3 layergroups = 3 WMS queries (Plans approuvés is a WMTS group)

image

image

So that's good, I don't have this performance issue on ngeo with Chrome!

But...

Thème Pompiers

(private theme so no permalink)

2 layergroups = 17 WMS queries (!)

image

image

Why 17 WMS queries?! It makes the interface freeze like hell...

Conclusion

After reflexion it seems that for private layers, a WMS query is done for each layer and not for each layergroup...

So @fredj @sbrunner @yjacolin @ybolognini, is it possible to fix this blocker, please?

sbrunner commented 7 years ago

It shouldn't have any deference for private layers, are all your layers on the same OGC server ?

rbovard commented 7 years ago

Good point, some are source for image/jpeg, other source for image/png...

Any idea why?

rbovard commented 7 years ago

It's seems not to be related to the FORMAT

image

image

rbovard commented 7 years ago

What's the condition to set the ogcserver in the migration script? Because I have 3% of my layers with source for image/jpeg.

All my "Pompiers" layers are in those 3%, but not all are private layers and not all my private layers are in it...

sbrunner commented 7 years ago

For the FORMAT is an issue that the image format isn't take in account...

sbrunner commented 7 years ago

@rbovard I'm not sure to understand your last comment, with the migration script all the LayerWMS should have an OGC server ...

rbovard commented 7 years ago

Yes they have one. But 3% have source for image/jpeg and the rest source for image/png

My question what's the criteria to assign an ogcserver during the migration? Because for me they should be all source for image/png

sbrunner commented 7 years ago

You probably have some LayerV1 with the format image/jpeg https://github.com/camptocamp/c2cgeoportal/blob/2.2/c2cgeoportal/scripts/themev1tov2.py#L213-L217

rbovard commented 7 years ago

Right. But that's very strange, because the dropdown in the admin interface is grayed so I don't understand why it's taken in account...

Admin (id=640): image

Admin (id=641): image

Database: image

fredj commented 7 years ago

see #3024 for the image_type issue

rbovard commented 7 years ago

So the workaround for this issue is to avoid to have to many layergroups per theme...

@sbrunner I let you open the 2 issues we found:

sbrunner commented 7 years ago

https://github.com/camptocamp/c2cgeoportal/pull/3197 will fix not null image_type in layerv1

sbrunner commented 6 years ago

With the last changes we get a reasonable result when we don't have more than 4-5 OpenLayers layer. But we will keep it open.

rbovard commented 6 years ago

Note: This issue is only with Chrome

sbrunner commented 6 years ago

https://jira.camptocamp.com/browse/GSGMF-532