Greengrass daemon kills service workers and shuts down.

[dzimine]

Greengrass daemon fails after starting. At first, it shows some errors that seem to imply network misconfiguration - can't resolve localhost. Than, the daemon shuts down.

The logs imply that the service lambda functions are shut down by greengrass daemon for exceeding max memory limit, but AWS folks claim it's not the reason. I run greengrass in a docker container on ubuntu xenial per https://github.com/dzimine/aws-greengrass-docker. Everything was working well until now, I'm quite puzzled what could cause the change.

Docker version 18.05.0-ce, build f150324 GGC version: 1.5.0. LOGS:

...
[2018-07-09T15:59:12.976244487Z] Failed to invoke PutLogEvents on local Cloudwatch, logGroup: /GreengrassSystem/router, error: RequestError: send request failed
caused by: Post http://localhost:8000/2016-11-01/cloudwatch/logs/: dial tcp: lookup localhost on 10.0.2.3:53: no such host, response: {

}
[2018-07-09T15:59:14.710684596Z] Failed to invoke PutLogEvents on local Cloudwatch, logGroup: /GreengrassSystem/connection_manager, error: RequestError: send request failed
caused by: Post http://localhost:8000/2016-11-01/cloudwatch/logs/: dial tcp: lookup localhost on 10.0.2.3:53: no such host, response: {

}
[2018-07-09T15:59:12.51Z][ERROR]-worker ungracefully killed arn:aws:lambda:::function:GGDeviceCertificateManager 35c6cf07-c8cb-4bd9-7c63-f6a2b626dbcd &{312 256 0xc4201d2870}
[2018-07-09T15:59:12.71Z][INFO]-Function arn:aws:lambda:::function:GGTES Worker ebb5b46c-22ca-449f-63d1-f29ecf482990 Max Memory Used 13004 KB
[2018-07-09T15:59:12.71Z][ERROR]-worker ungracefully killed arn:aws:lambda:::function:GGTES ebb5b46c-22ca-449f-63d1-f29ecf482990 &{298 256 0xc4201d2870}
[2018-07-09T15:59:12.914Z][INFO]-Function arn:aws:lambda:::function:GGShadowSyncManager Worker 3c76188a-5801-4035-7d77-639d68a33823 Max Memory Used 11552 KB
[2018-07-09T15:59:12.915Z][ERROR]-worker ungracefully killed arn:aws:lambda:::function:GGShadowSyncManager 3c76188a-5801-4035-7d77-639d68a33823 &{326 256 0xc4201d2870}
[2018-07-09T15:59:13.11Z][INFO]-Function arn:aws:lambda:::function:GGShadowService Worker e2073291-a603-4383-5a1a-abfd21fda9ef Max Memory Used 13092 KB
[2018-07-09T15:59:13.11Z][ERROR]-worker ungracefully killed arn:aws:lambda:::function:GGShadowService e2073291-a603-4383-5a1a-abfd21fda9ef &{339 256 0xc4201d2870}
[2018-07-09T15:59:13.51Z][INFO]-Function arn:aws:lambda:::function:GGRouter Worker 1dbe6a6a-9f6c-4678-67b2-060bd698b744 Max Memory Used 11268 KB
[2018-07-09T15:59:13.511Z][ERROR]-worker ungracefully killed arn:aws:lambda:::function:GGRouter 1dbe6a6a-9f6c-4678-67b2-060bd698b744 &{271 256 0xc4201d2870}
[2018-07-09T15:59:13.511Z][FATAL]-Failed to restart system worker. GGC will stop. Reached limit for restarting function arn:aws:lambda:::function:GGDeviceCertificateManager, restart reason: 2

[dzimine]

Current theory: it is caused by the cloudwatch failures (lookup errors).

Tried: docker run with --networking=host, use docker-compose - no luck.

[kwe712]

I have the exact same problem, also in vagrant. As this is the only thing i have running in that vm, the max memory limit really doesn't seem to be the issue. --networking=host and docker-compose didn't work for me, either. But i can start the greengrass daemon on its own, without docker run. Unfortunately i have no experience with Cloudwatch so i don't know where to go from here...

[japel]

I'm encountering the same issue :( ... Does anyone know how to workaround this?

[kempstonjoystick]

I'm hitting this too.

[str3tch]

joining the conversation. Used this a while back, was all working well (vagrant on mac). today, not working, errors as reported above :(

[dzimine]

@str3tch That's exactly what had happened for me: it worked on all machines, then on some, now I can't make it work again. Care to try with Greengrass v1.4.0?