search

citusdata / docker

:ship: Docker images and configuration for Citus
Apache License 2.0
249 stars 101 forks source link

High CPU utilisation of woker container. #186

Open mahajanankur opened 4 years ago

mahajanankur commented 4 years ago

I found the membership-manager docker is utilising a lot CPU due to which master and workers are also heating up the environment.

When I stop the membership-manager docker, everything goes to normal.

This might be due to membership-manager has connected to Citus cluster and firing the events repeatedly.

citus_stats

hanefi commented 4 years ago

Thanks @mahajanankur for reporting this.

~I created two PRs that aims to resolve this issue:~ ~1. https://github.com/citusdata/membership-manager/pull/9 implements a polling mechanism in Membership Manager, so that (a) it will detect the readiness of the coordinator node, (b) properly report that it is ready to accept new Citus worker services~

~2. https://github.com/citusdata/docker/pull/187 (a) updates the docker images to properly detect dependencies, (b) introduce Compose V3 definitions~

~Once they are merged, this issue should be resolved~

Edit: this issue seems unrelated to my fixes

mahajanankur commented 4 years ago

@hanefi, just to inform you, the issue is still there even after shutting down the membership-manager container. Server with load citus_stats_2

Idle server citus_stats_3

Update: The Citus docker ecosystem pooling the data frequently due to which CPU and I/O gets chocked. This happens even if there is no load or even no schema.