[bitnami/mongodb] Zombie mongosh processes after startup

[simon-jouet]

Name and Version

bitnami/mongodb 13.4.1

What steps will reproduce the bug?

I deployed a single node (for the time being) mongodb replicaset. After a reboot of the server I SSH into the machine and get notified of zombie processes.

The zombie processes are only timestamped when the machine is starting and after that, I can see mongosh being executed by the mongodb container without problems.

Are you using any custom parameters or values?

Very bare bone configuration

image:
  registry: docker.io
  repository: bitnami/mongodb
  tag: 6.0.2-debian-11-r11
  pullPolicy: IfNotPresent
  debug: false

architecture: replicaset

volumePermissions:
  enabled: false

persistence:
  enabled: true

replicaCount: 1

arbiter:
  enabled: false

auth:
  enabled: true
  existingSecret: mongodb

What is the expected behavior?

No zombie processes

What do you see instead?

When connecting over SSH in the banner message

  => There are 2 zombie processes.

Checking the processes

$ sudo ps aux | grep 'Z'

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
abc         8389  2.2  0.0      0     0 ?        Z    11:43   0:01 [mongosh mongodb] <defunct>
abc         8390  0.0  0.0      0     0 ?        Z    11:43   0:00 [grep] <defunct>

Checking the start time of the machine

$ uptime -s
2022-11-05 11:42:29

Additional information

Using pstree -p I can see that the zombie process is from the mongodb pod

           ├─containerd-shim(5888)─┬─mongod(5976)─┬─grep(8390)
           │                       │              ├─mongosh mongodb(8389)
           │                       │              ├─{mongod}(6228)
   [truncated]

[javsalgar]

Hi,

Could you add more details of the resources of the cluster? We are aware that mongosh has a significant performance impact (reported upstream).

[simon-jouet]

Hi @javsalgar,

Do you mean the compute/process resources of the nodes? If that's the case the load is extremely minimal as this is a brand new deployment (I have other clusters with older versions and much higher load without issues).

Anyway, the CPU usage is a few percent (2.7% right now) and the RAM usage is at 12%, the single node has 16GB of RAM and a 4 core E3-1230 v6 (OVH).

There are very few pods booting up with kubernetes, at the moment it's only calico for the network, cert-manager, ingress-nginx and harbor as well as obviously mongodb.

Best regards, Simon

[aoterolorenzo]

After some research I found this issue which seems a pretty close behaviour, reported in Mongodb's issue tracker:

"another user was running a dockerized instance of mongod on start-up and was prohibiting the service'd mongod from starting at boot-time"

Could this be related to your issue?

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[ey3ball]

I'm seeing this too, it looks we get these defunct process because the liveness probe is so slow to respond that it ultimately gets killed.

The upstream issue mentioned by @javsalgar seems to be this one maybe ? https://jira.mongodb.org/browse/MONGOSH-1240

I have hardware that is relatively slow, I guess you could get the same behaviour with a more capable server that's sporadically under big loads.

[ey3ball]

It seems that mongosh has been here a while and that anything remotely recent is affected (I tried going back to the 10.x series of this chart and mongodb 4.4.x)

For now the only "solution" (considering I don't plan on upgrading my hardware) is simply to disable the probes :

livenessProbe:
  enabled: false
readinessProbe:
  enabled: false

This is obviously far from ideal, but after this change performance gets back to normal and there are no more defunct / zombie process created

[aoterolorenzo]

Thank you for the contribution and the extra information on your personal case! I will close the issue then, hoping other people could feed from it.

[ey3ball]

I understand that the mongosh performance hog is really on mongodb's side, but maybe there could be another way for the chart to check for liveness ? Having to disable it altogether seems a bit extreme.

It looks like mongosh has been that way since its inception, so that seems unlikely to improve.

It's just my two cents but as a very random mongodb user I find it is very surprising that these probes have such a huge impact. I have admittedly a relatively small instance (gcp - e2-small), but it's still pretty standard cloud stuff. The instance is completely idle and it goes very quickly from 0 to 200% CPU usage as soon as I install the chart with the default probing parameters

[marcosbc]

Hi @ey3ball, currently we are relying on the official tools provided by MongoDB to check the liveness of the cluster. We are open to improving our chart, so if you have any idea on how this could be improved, feel free to propose it and/or contribute it via a PR.

This has already been reported as an upstream issue in MONGOSH-1240, as mentioned earlier. As you can see, the MongoDB development team is already aware of this issue, as in fact they have created a specific epic for this issue in MONGOSH-1314, although they are unable to provide any ETA.

For now, my opinion is that it is best to wait for this issue to be fixed on the MongoDB side. But as mentioned earlier, any improvements are welcome. The only thing to keep in mind is that we will not be able to support the legacy mongo tool anymore, due to being deprecated/unmaintained.

[koss822]

I have the same issue. When I have it with other containers it was due to a missing initd (there are a few projects for initd for docker), once I installed some type of initd into container problems with defunct processes ended. Regarding the upstream issue MONGOSH-1240, it is related to poor performance not zombie process, so this should be reopened.

[aoterolorenzo]

Hi @koss822, thanks for comment!

I understand that MONGOSH-1240 could not be in complete line with the issue, but in any case as @marcosbc mentioned, we completely rely the probes onto the upstream script set, so both the problem an the solutions should be reported and solved on upstream side, so I don't see a need of reopen this issue.

[koss822]

Hi @aoterolorenzo

I checked the upstream issue, it is related to mongo. Regarding probes, this is not related to probes, or slow mongo or slow upstream, this is related purely to bitnami image.

Bitnami image should contain init process. See for example Dockerfile

You should implement dumb-init or other tool

# OPTIONAL: Install dumb-init (Very handy for easier signal handling of SIGINT/SIGTERM/SIGKILL etc.)
RUN wget https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64.deb
RUN dpkg -i dumb-init_*.deb
ENTRYPOINT ["dumb-init"]

This would solve issue with zombie proccess.

Please google e.g. - "zombie process docker init"

This is not related to mentioned slow probes or any problem with mongo, this is purely related to bitnami image.

Please reopen.

[aoterolorenzo]

Hi @koss822,

The thing is, a dumb-init would only kill these zombie process by itself, but that's still just a workaround to solve the fact of the initial issue: there are zombie processes, and the probes don't detect that process deaths before a new one goes on, which instead of a pod reboot, ends in a new process with background zombies.

As I completely understand your point, and this would workaround the issue, I ask myself what would happen if we start applying workarounds like this to our containers. About the difficulty of tracking upstream issues to remove this workarounds as soon a real fix is out. And how many logic and dependencies would remain from this workarounds after non announced fixes.

Also, in this specific case, we are adding a new dependency here to the container, and this implies increasing our attack surface agains. So, conceptually, it's not so easy as adding a package and apply a workaround.

Anyway, I will discuss it with the team to search for consensus regarding this.

[koss822]

Hi @aoterolorenzo ,

I understand, increasing complexity and adding security dependencies might be an issue.

Just to note, regarding zombie processes, this is quite common, same happened to me with Mozilla Firefox and other big stable projects. Even if Mongo will fix this issue it might happen again in future.

What concerns me a little bit more that on my server I have now 40+ zombie processes.

[keeshoekzema]

I ran into this problem as well and did some digging:

The first problem that causes the zombie processes is that the startupProbe has a default timeout of 1 second. When i next started testing the probe, it usually ran in 0.9-1.2 seconds, meaning that everytime the probe ran more than 1 second it created a zombie process because the shell got killed and the mongosh process remained. I worked around this by increasing the startupProbe.timeoutSeconds from the default 1 seconds to 5 seconds.

But i was wondering why it was so slow, so i started stracing the mongosh process. Most of the time went into setting up a connecting to a random https service running in AWS and after some digging in the docs i found it was most likely the telemetry reporting (which i didn't know existed). After disabling telemetry (mongosh --quiet -eval 'disableTelemetry()') my mongosh command ran in 0.5 seconds.

This could be fixed by either setting the startupProbe.timeoutSeconds to a higher number or by disabling telemetry by default at startup (it is saved in the /data/db/mongosh directory so it should persist)

[pmhahn]

This still is unsolved for me: Graylog is using docker.io/bitnami/mongodb:6.0.4-debian-11-r0. After some time I now have 38.677 (!) processes running in that container:

1001     3815518  3.1  1.6 4929836 2132852 ?     Ssl  Jan04 889:30  \_ /opt/bitnami/mongodb/bin/mongod --config=/opt/bitnami/mongodb/conf/mongodb.conf
1001     2885255  0.0  0.0      0     0 ?        Z    Jan04   0:02      \_ [mongosh mongodb] <defunct>
1001     2885256  0.0  0.0      0     0 ?        Z    Jan04   0:00      \_ [grep] <defunct>
...
1001     3043775 68.8  0.0      0     0 ?        Z    16:02   0:04      \_ [mongosh mongodb] <defunct>
1001     3043776  0.0  0.0      0     0 ?        Z    16:02   0:00      \_ [grep] <defunct>

This is caused by the rediness probe:

    Liveness:       exec [/bitnami/scripts/ping-mongodb.sh] delay=30s timeout=10s period=20s #success=1 #failure=6
    Readiness:      exec [/bitnami/scripts/readiness-probe.sh] delay=5s timeout=5s period=10s #success=1 #failure=6

which does this:

#!/bin/bash
…
    mongosh $TLS_OPTIONS --port $MONGODB_PORT_NUMBER --eval 'db.hello().isWritablePrimary || db.hello().secondary' | grep 'true'

The scripts is executed by shell bash will fork two sub-processes mongosh and grep. If these group of processes do not finish within timeout=5s, the bash process gets killed and its exit status is collected by k8s. But as it is the parent process for mongosh and grep, they loose their parent and get re-parented to PID 1, which is the main mongod process! As it has no handler for SIGCHLD and/or is not written to work as a init process, these 2 processes will turn into zombie processes when they terminate, remaining in that state forever as no-one collects their exit status. (zombie processes are dead processes, which have terminated, but their exit status has not yet been collected by their parent process)

IMHO the bug is in the bitnami image as it uses shell pipelining in the readiness probe. I see two options:

1. Use dumb-init like proposed above in https://github.com/bitnami/charts/issues/13365#issuecomment-1527337975
2. Change readiness-probe.sh to use some other mechanism to check for rediness.

PS: Please also change /bitnami/scripts/ping-mongodb.sh to use exec to replace the shell process with the mongosh process, so that k8s kills it directly instead of the intermediate bash process only.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[gituser8796]

Hi @aoterolorenzo, is there a suggested workaround from your end?