Open lucernae opened 5 years ago
Update on the investigations:
I replicated the behaviour on my own machine with the following specs:
CPU: Skylake 4 core, 4.0 GHz
RAM: 16 GB
Disk: plenty/not a problem
OS: Ubuntu, but GeoSAFE run on Rancher, the same way with prod environment.
The crash did happen after the second analysis. The second analysis itself was finished with no problem, but the whole computer crash when I tried to view the layer. Thus, I conclude the problem lies with QGIS-Server rendering.
It turns out the reason of the memory leak is because for each container, there are several Apache worker thread on it (pretty normal actually). However for this kind of file, which is 100MB geojson layer, a thread takes time to open the file and render it. While this thread is currently working, another thread tried to access the layer (to render a different tile location), it somehow raised permission denied and died without cleanup. This process happens, again and again accumulating dead memory until the container itself is destroyed.
To check if this is coming from apache or QGIS own code. I tried to load the layer on QGIS desktop and memory it uses is a whopping 8GB for a mere 100MB geojson layer.
Wow....
It is only using around 500 MB if the geojson is converted to shapefile. So I guess the problem comes from the data format. Maybe strings type is not handled properly.
Because the problem lies with the data format and how QGIS handled it, we can use the following alternatives for short term solutions:
Long term solutions:
Short term solution no. 2 seems practical, but we need to build/test the new QGIS-Server backend container optimized for rancher like this.
Problem
This issue might relate with other issue because of it's generic symptoms.
Saravana tried to run analysis with a modest size of building layers (60MB of shapefile). Even if the extent is big, the analysis was complete in around 20minutes or so, after my recent patch to GeoSAFE. However the problem happens when he is trying to run a second analysis. The server crashed. This bug is reproducible consistently.
Here's the excerpt of the conversation in the email (of my conclusion):
The eventual conclusion is QGIS-Server backend was the cause and not optimized to handle the memory leak.
Proposed solution will follow after further investigations.