Electron process error with mapbox-gl

[smit1678]

Current deployment runs into an electron error with mapbox-gl. Unclear if this is related to a version or dependency.

Docker output from the renderer:

debug: renderer received message {"email":"nate@hotosm.org","map":{"lat":57.71096311750432,"lng":11.95305096853906,"zoom":12.399017242352361,"startDate":1500807600000,"endDate":1532343600000,"interval":"months","style":{"background":"light","features":[{"base":{"line-color":{"a":0.7,"b":187,"g":147,"r":42},"line-width":1},"name":"roads","enabled":true,"highlight":{"line-color":{"a":0.3,"b":255,"g":255,"r":255},"line-width":1},"baseEnabled":true,"highlightEnabled":true},{"base":{"line-color":{"a":0.3,"b":109,"g":85,"r":214},"line-width":1},"name":"buildings","enabled":true,"highlight":{"line-color":{"a":0.3,"b":255,"g":255,"r":255},"line-width":1},"baseEnabled":true,"highlightEnabled":true}]}},"format":"mp4","speed":1,"size":"1280x720","dir":"8b815230ec7002d3a3cc90ccfba3b440"}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: 2017-07-23 11:00:00 (0/12 [months])
 {}
info: electron process: error: renderer timeouted, exiting... {}
 {}
debug: electron rendering finsished {}
error: exited with 1 {}

[dakotabenjamin]

To add to this: Once this error occurs, any future render requests are ignored. They show up in the api logs, but nothing in the renderer logs.

[szymonkaliski]

So I remember seeing this when the docker container with electron was running low on RAM, could you try increasing memory that docker can allocate?

Is this happening on your local machine, or some hosted env?

Also, is there a reproducible way to trigger this?

[szymonkaliski]

Also, I just noticed develop branch, so where is that error happening? In master or develop?

If in develop, then I suspect this commit might be an issue: https://github.com/hotosm/visualize-change/commit/227936a8f7177bf9bc627ee4203202c513a8abff

[smit1678]

If in develop, then I suspect this commit might be an issue: 227936a

Yeah I thought we had reverted that. That was a shot in the dark because of the error was being seen both on Elastic Beanstalk and the Digital Ocean box.

But I think the work from @dakotabenjamin last night thought that memory might be an issue because the issue was popping up in other places.

Any recommendations on how we manage memory across containers?

[szymonkaliski]

Any recommendations on how we manage memory across containers?

Not sure what you mean here, sorry. But I'd recommend turning on swap on the boxes you run this on, and monitoring to see if that's actually the problem.

Ideally we could find some kind of reproducible crash, that we can backtrace and fix this issue...

[dakotabenjamin]

We can manage memory with the Dockerrun.aws.json file- I gave each container 256MiB of memory. I will try increasing the memory allocation for the renderer. What do you recommend?

[szymonkaliski]

Oh much more. So what the renderer is doing is: crating virtual X server, running Electron with WebGL inside of it, and software-rendering the window content (mapboxgl map with lines). I'm pretty sure we'd need more for all the textures. I'm afraid this could be trial&error process, but I'd start with 1GB.

[dakotabenjamin]

What other containers do you think will require a bump in memory? I was thinking I should put them all at least to 512MiB.

[szymonkaliski]

Other containers are pretty small, I wouldn't worry about them. If we are working in some kinds of constraints, I think it's better to give as much as we can to the renderer, and leave others small. The node API is super simple, the frontend is basically running static http server in production mode, ngnix, rabbitmq and postgres don't need much. Only the renderer is resource-hungry.

[dakotabenjamin]

We have a t2.large with 8gb, so I set the max memory allocation for the renderer to 2GiB and everything else 512MiB. If that doesn't work I will adjust accordingly.

[szymonkaliski]

👍 let me know how it goes!

[dakotabenjamin]

I bumped it up to 4GB but unfortunately it still fails:

debug: renderer received message {"email":"dakota.benjamin@hotosm.org","map":{"lat":34.391820000000024,"lng":132.45209,"zoom":12.49999999999995,"startDate":1531843200000,"endDate":1532448000000,"interval":"days","style":{"background":"light","features":[{"name":"roads","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":42,"g":147,"b":187,"a":0.7},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}},{"name":"buildings","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":214,"g":85,"b":109,"a":0.3},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}}]}},"format":"gif","speed":1,"size":"1280x720","dir":"2edc4eb019f8968b12c398a89b4e65e8"}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: 2018-07-17 16:00:00 (0/7 [days])
 {}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}

editing this to add on, I am sometimes able to get it to work I've been playing around with format/size (although I'm not finding a pattern yet):

debug: renderer received message {"email":"dakota.benjamin@hotosm.org","map":{"lat":34.391820000000024,"lng":132.45209,"zoom":12.49999999999995,"startDate":1531843200000,"endDate":1532448000000,"interval":"days","style":{"background":"light","features":[{"name":"roads","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":42,"g":147,"b":187,"a":0.7},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}},{"name":"buildings","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":214,"g":85,"b":109,"a":0.3},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}}]}},"format":"gif","speed":1,"size":"1280x720","dir":"2edc4eb019f8968b12c398a89b4e65e8"}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: 2018-07-17 16:00:00 (0/7 [days])
 {}
info: electron process: 2018-07-17 16:00:00 (0/7 [days])
 {}
info: electron process: 2018-07-18 16:00:00 (1/7 [days])
 {}
info: electron process: 2018-07-19 16:00:00 (2/7 [days])
 {}
info: electron process: 2018-07-20 16:00:00 (3/7 [days])
 {}
info: electron process: 2018-07-21 16:00:00 (4/7 [days])
 {}
info: electron process: 2018-07-22 16:00:00 (5/7 [days])
 {}
info: electron process: 2018-07-23 16:00:00 (6/7 [days])
 {}
info: electron process: 2018-07-24 16:00:00 (7/7 [days])
 {}
info: electron process: debug: rendering finished {}
 {}
debug: electron rendering finsished {}

This one finished despite errors. Here's the gif:

[dakotabenjamin]

So it doesn't seem to be writing the buildings and roads:

[szymonkaliski]

This is very strange:

1. does it happen on master?
2. can you reproduce on a local machine, or only on AWS?

[dakotabenjamin]

I think @mataharimhairi was experiencing the same issue on master and locally.

[mataharimhairi]

Mine seems a little different as it appears to write the building and roads, but runs into an error half way through rendering the images for stitching. This was run on locally.

renderer_1  | [1] info: electron process: 2018-01-01 04:00:00 (0/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-01-01 04:00:00 (0/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-01-08 04:00:00 (1/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-01-15 04:00:00 (2/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-01-22 04:00:00 (3/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-01-29 04:00:00 (4/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-02-05 04:00:00 (5/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-02-12 04:00:00 (6/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-02-19 04:00:00 (7/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-02-26 04:00:00 (8/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-03-05 04:00:00 (9/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-03-12 04:00:00 (10/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-03-19 04:00:00 (11/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: 2018-03-26 04:00:00 (12/25 [weeks])
renderer_1  | [1]  {}
renderer_1  | [1] info: electron process: error: renderer timeouted, exiting... {}
renderer_1  | [1]  {}
renderer_1  | [1] debug: electron rendering finsished {}
renderer_1  | [1] error: exited with 1 {}
renderer_1  | [1] debug: renderer received message {"email":"mhairi.ohara@hotosm.org","map":{"lat":-7.013472026244955,"lng":110.42295059946855,"zoom":12.000000000000062,"startDate":1514779200000,"endDate":1530331200000,"interval":"weeks","style":{"background":"dark","features":[{"base":{"line-color":{"a":0.7,"b":194,"g":227,"r":80},"line-width":1},"name":"roads","enabled":true,"highlight":{"line-color":{"a":0.3,"b":255,"g":255,"r":255},"line-width":1},"baseEnabled":true,"highlightEnabled":true},{"base":{"line-color":{"a":0.3,"b":28,"g":173,"r":248},"line-width":1},"name":"buildings","enabled":true,"highlight":{"line-color":{"a":0.3,"b":255,"g":255,"r":255},"line-width":1},"baseEnabled":true,"highlightEnabled":true}]}},"format":"mp4","speed":1,"size":"1024x768","dir":"7a1cda0d84f127da389dc22d6e05c152"}

[mataharimhairi]

[szymonkaliski]

Ok, can someone send me some config that reliably fails so I can have a look? I'll try to do this in upcoming days.

[szymonkaliski]

So, I bumped up electron to most recent version and played a little with options passed to electron, I managed to be able to reliably render 50+ frames for the most zoomed-out zoom level we support.

The work is in fix/renderer-crashes branch, together with bit more logging; please test (remember that you have to rebuild the docker images with yarn dev:build) and if it still crashes, please let me know with the log output from console :)

PS. Memory for me usually hovers around 1.5-2GB when rendering, and CPU usage can be at 200% for the renderer process.

[awright]

(Thanks @szymonkaliski for looking into this.)

[dakotabenjamin]

It seems to be working now:

debug: renderer received message {"email":"dakota.benjamin@hotosm.org","map":{"lat":-6.173292000000103,"lng":106.84103600000026,"zoom":12.5,"startDate":1524844800000,"endDate":1532620800000,"interval":"months","style":{"background":"light","features":[{"name":"roads","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":42,"g":147,"b":187,"a":0.7},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}},{"name":"buildings","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":214,"g":85,"b":109,"a":0.3},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}}]}},"format":"gif","speed":1,"size":"1280x720","dir":"a3f58e29bc9cc9fe4394716d1b068272"}
info: electron process: This page appears to be missing CSS declarations for Mapbox GL JS, which may cause the map to display incorrectly. Please ensure your page includes mapbox-gl.css, as described in https://www.mapbox.com/mapbox-gl-js/api/.
 {}
info: electron process: { [Error] message: '' }
 {}
info: electron process: { [Error] message: '' }
 {}
info: electron process: { [Error] message: '' }
{ [Error] message: '' }
{ [Error] message: '' }
{ [Error] message: '' }
 {}
info: electron process: 2018-04-27 16:00:00 (0/2 [months])
 {}
info: electron process: 2018-04-27 16:00:00 (0/2 [months])
 {}
info: electron process: 2018-05-27 16:00:00 (1/2 [months])
 {}
info: electron process: 2018-06-27 16:00:00 (2/2 [months])
 {}
info: electron process: debug: rendering finished {}
 {}
debug: electron rendering finsished {}
debug: renderer replying to server {"msg":"{\"email\":\"dakota.benjamin@hotosm.org\",\"dir\":\"a3f58e29bc9cc9fe4394716d1b068272\",\"format\":\"gif\"}"}
debug: renderer received message {"email":"dakota.benjamin@hotosm.org","map":{"lat":-6.173576298754838,"lng":106.83053729371954,"zoom":14.11304775289034,"startDate":1524844800000,"endDate":1532620800000,"interval":"months","style":{"background":"light","features":[{"name":"roads","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":42,"g":147,"b":187,"a":0.7},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}},{"name":"buildings","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":214,"g":85,"b":109,"a":0.3},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}}]}},"format":"gif","speed":1,"size":"1280x720","dir":"0732e674dcb717b48c8143b4152346ea"}
info: electron process: This page appears to be missing CSS declarations for Mapbox GL JS, which may cause the map to display incorrectly. Please ensure your page includes mapbox-gl.css, as described in https://www.mapbox.com/mapbox-gl-js/api/.
 {}
info: electron process: { [Error] message: '' }
 {}
info: electron process: 2018-04-27 16:00:00 (0/2 [months])
 {}
info: electron process: 2018-04-27 16:00:00 (0/2 [months])
 {}
info: electron process: 2018-05-27 16:00:00 (1/2 [months])
 {}
info: electron process: 2018-06-27 16:00:00 (2/2 [months])
 {}
info: electron process: debug: rendering finished {}
 {}
debug: electron rendering finsished {}
debug: renderer replying to server {"msg":"{\"email\":\"dakota.benjamin@hotosm.org\",\"dir\":\"0732e674dcb717b48c8143b4152346ea\",\"format\":\"gif\"}"}

http://visualize-change-test-clone.us-east-1.elasticbeanstalk.com//renders/a3f58e29bc9cc9fe4394716d1b068272/render.gif

I think the lack of roads and buildings may be a separate issue, as I was able to build locally and run without a problem. It's only on the AWS EB deployment that I'm having issues.

[szymonkaliski]

So for the lack of roads/buildings I'd first confirm if ./docker/data/tiles/tiles.mbtiles actually exists?

[dakotabenjamin]

AH I know what's wrong. I had to move the ./docker/data/* files to a separate files system (EFS if you're familiar with aws). Is there a hard-coded link to that directory in the renderer?

[dakotabenjamin]

The tiles do appear here: http://visualize-change-test-clone.us-east-1.elasticbeanstalk.com/edit

Maybe its because the renderer is unable to access the tiles here ?

[szymonkaliski]

Is there a hard-coded link to that directory in the renderer?

Renderer access tiles the same way that frontend (through mapboxgl endpoints), so if they display on the frontend, they should display in the renderer as well.

[dakotabenjamin]

OK- So just to be clear about my process and how it differs from the docker-compose:

1. We build the images and run the containers using docker-compose on a circleci server (process defined here)
2. Those images are committed and pushed to quay.io
3. We deploy a multi-container docker app using AWS Elastic Beanstalk, which uses the Elastic Container Service to pull the remote images above and run them in a container instance. Instead of using a docker-compose file, the task to run the containers is defined in Dockerrun.aws.json, which connects the volumes, sets up the bridge network, and applies the environment variables to each container.
4. The static data (postgres db, rabbit, capture, world tiles) are stored an a mounted Elastic File System at /efs on the container instance, and it is properly volume-mounted in the Dockerrun.aws.json file to the respective containers.

From what I understand of Docker Compose and ECS, both create a network bridge so that containers can access other containers from ports exposed in the Dockerfile.

However, I needed to specify each linked container in the Dockerrun.aws.json file with the links parameter; I had not linked the api container to renderer.

@smit1678 please test on visualize-change-dev.hotosm.org and close if fixed.