Closed ledroide closed 6 months ago
tianon
commented
6 months ago This sounds like you probably exposed your MongoDB container to the public internet, and it's likely compromised. :grimacing:
I would suggest you shut it down ASAP and consider all data in it leaked or compromised. :see_no_evil:
ledroide
commented
6 months ago This sounds like you probably exposed your MongoDB container to the public internet, and it's likely compromised. 😬 I would suggest you shut it down ASAP and consider all data in it leaked or compromised. 🙈
@tianon Thanks for this warning. However :
securityContext.readOnlyRootFilesystem: true and had been restarted many times since we have notices these requests.From what I see, there is something in this container configuration that is fetching some service(s) hosted in AWS. Note that this Mongo container is not running on AWS hosting.
LaurentGoderre
commented
6 months ago @ledroide what is the image digest (sha) of the image you are using?
tianon
commented
6 months ago also, any chance you're using mongosh inside that container? apparently it enables telemetry by default (and we should probably update our docs to point that out :disappointed:): https://www.mongodb.com/docs/mongodb-shell/telemetry/
tianon
commented
6 months ago (maybe you could use something like tshark to log the DNS requests to try and map those IPs to hostnames to verify?)
ledroide
commented
6 months ago also, any chance you're using
mongoshinside that container? apparently it enables telemetry by default (and we should probably update our docs to point that out 😞): https://www.mongodb.com/docs/mongodb-shell/telemetry/
Trying :
mongodb@mongo-76db777b55-qv5pl:/$ mongosh
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.1.5
Using MongoDB: 7.0.6
Using Mongosh: 2.1.5
test> config.get('enableTelemetry')
true
test> disableTelemetry()
Telemetry is now disabled.
test> config.get('enableTelemetry')
falseAfter 45 minutes, no external access is reported. @tianon it looks like you have found the cause of this issue.
Unfortunately I am unable to find a way to configure my container with telemetry disabled. Here are supported environment variables. No telemetry.
Maybe is there a way to mount a file mongosh.conf - from a configMap - somewhere in the container ?
ledroide
commented
6 months ago I guess the probes are triggering the telemetry, because running a mongosh command :
readinessProbe:
exec:
command:
- /usr/bin/mongosh
- --eval
- "db.adminCommand('ping').ok"
- localhost:27017/test
- --quietI could not find a clean way to configure telemetry in mongosh, so I put this workaround in the podspec :
containers:
- name: mongo
image: docker.io/library/mongo:7.0.6-jammy
args:
- --dbpath
- /data/db
lifecycle:
postStart:
exec:
command:
- /usr/bin/mongosh
- --nodb
- --eval
- "disableTelemetry()"This writes the config here :
kubectl exec -ti deploy/mongo -- bash
mongodb@mongo-7c9d665b54-99dpt:/$ cat ~/.mongodb/mongosh/config | jq .{
"userId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"telemetryAnonymousId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"enableTelemetry": false,
"disableGreetingMessage": true
}15 minutes away from restarting the container, no external requests. It looks that the workaround solves the issue.
However, an environment variable to switch off telemetry would be appreciated.
tianon
commented
6 months ago That all tracks because the function would probably only disable it for the lifetime of the mongosh command -- unfortunately, for more than that you'll need to file a feature request with MongoDB directly (we're just packagers of what they publish). :sweat_smile:
tianon
commented
6 months ago Oh, sorry, misread your latest -- it does persist, that's interesting. I agree though that an environment variable would be awesome, but isn't something we would implement here. :smile:
summary
Network tracker shows denied outgoing requests from mongo container to external destinations. Mongo seem to run fine however.
What are these requests ? What is the purpose ? How do we configure Mongo to avoid these requests ?
context and versions
hubble output
External access is denied because only DNS is allowed for egress NetworkPolicies, so the error is legit.
Incomplete list of denied requests ->
<address>:<port>.All these IP addresses are part of "Amazon Technologies Inc."