Closed CocolinoFan closed 3 months ago
selsta
commented
3 months ago You are still using v0.18.2.2 which is a year old, so it's not possible that monerod itself changed. Can you try to start monerod with restricted rpc to see if it's some external program that does the RPC call to start mining?
CocolinoFan
commented
3 months ago When trying to reproduce the bug, monerod was acting odd (not shutting down with Ctrl+C, exit command taking minutes); p2pool was not starting either, complaining that port 3333 was already in use (by monerod somehow I think). But, it did not start mining on its own again (at least in the two minutes I kept it open). After a reboot, everything seems normal. I will report back if it happens again, but looks unlikely to be a hacking situation. Thank you for the idea of running monerod with RPC disabled. In hindsight, I should probably not have opened a new issue.
v0.18.3.3 is out 👀
selsta
commented
3 months ago If open RPC causes the mining to start automatically there is a risk that some program or malware is sending the RPC call to start mining. You might be able to increase the log level and see to which address is being mined, though not sure if that's getting printed at a higher log level. You'd have to try.
CocolinoFan
commented
3 months ago Well, I've installed the latest version of monerod from my distribution's repository and re-downloaded the blockchain. After one day monerod starts mining on it's own again.
2024-06-10 03:22:24.066 I difficulty: 294057258027
2024-06-10 03:50:26.067 W WARNING: no two valid DNS TXT records were received
2024-06-10 05:48:27.015 W WARNING: no two valid DNS TXT records were received
2024-06-10 07:46:22.444 W WARNING: no two valid DNS TXT records were received
2024-06-10 09:44:18.377 W WARNING: no two valid DNS TXT records were received
2024-06-10 10:09:34.853 W WARNING: no two valid DNS TXT records were received
2024-06-10 11:42:17.076 W WARNING: no two valid DNS TXT records were received
2024-06-10 13:40:20.742 W WARNING: no two valid DNS TXT records were received
2024-06-10 14:09:44.823 I Miner thread was started [0]
2024-06-10 14:09:44.823 I Miner thread was started [1]
2024-06-10 14:09:44.823 I Miner thread was started [8]
2024-06-10 14:09:44.823 I Miner thread was started [2]
2024-06-10 14:09:44.823 I Miner thread was started [3]
2024-06-10 14:09:44.823 I Miner thread was started [11]
2024-06-10 14:09:44.823 I Miner thread was started [12]
2024-06-10 14:09:44.823 I Miner thread was started [13]
2024-06-10 14:09:44.823 I Miner thread was started [6]
2024-06-10 14:09:44.823 I Miner thread was started [7]
2024-06-10 14:09:44.824 I Miner thread was started [5]
2024-06-10 14:09:44.824 I Miner thread was started [15]
2024-06-10 14:09:44.824 I Miner thread was started [4]
2024-06-10 14:09:44.824 I Miner thread was started [14]
2024-06-10 14:09:44.824 I Miner thread was started [10]
2024-06-10 14:09:44.824 I Miner thread was started [9]I was ready this time. Typing mining_status gave me
Mining at 931 H/s with 16 threads
PoW algorithm: RandomX
Mining address: 47aMSgJnHgcTefoPB2fAkKWEZWZPRYjgci67aQnFRrNXKAtW7uveU2BSFuxzpky9ntWnKAkZemKgfLA3zzGdnvcT4xGNPJnNeedless to say I don't know that address, unless monerod is a wallet as well and somehow made a default wallet and started mining in it? But I don't see any commands related to wallet management.
I start monerod with the command monerod --zmq-pub tcp://127.0.0.1:18083 --out-peers 32 --in-peers 64 --add-priority-node=p2pmd.xmrvsbeast.com:18080 --add-priority-node=nodes.hashvault.pro:18080 --disable-dns-checkpoints --enable-dns-blocklist as instructed in the p2pool documentation
I have not stooped the node, so let me know if I should run any commands or look in any logs.
selsta
commented
3 months ago There is no code in monerod that would create an address and mine to it, I assume some software on your computer is sending a RPC call to start mining.
CocolinoFan
commented
3 months ago That is very alarming. How would I check this? I have Gentoo Linux 6.9.3
moneromooo-monero
commented
3 months ago If you set log level to, say... 2, you should have all you need to check where the request to mine comes from. So:
set_log 2 stop_mining
Then wait for mining to start again. After it does, check logs for "start_mining". You should get a line with "calling /start_mining" in it. A bit earlier in that line, you will see the IP and port the request is coming from.
If it is 127.0.0.1, then it is a process on your machine. You can run:
netstat -anpt
to see what's running with network connections. If it's some malware, it might not be running network at the time though, but if something is connected to your daemon, you'll see it. Wallets are connected to it, and it might be a wallet if you setup background mining (you'd have been asked when creating a wallet). Exit wallet software to make sure it's not that. It'd be a known wallet address here though.
If the origin IP is not 127.0.0.x (or anything in the 127 range), then it's coming from the internet (unless a non routable IP from your local network, but I assume you'd know if you had set that up).
If you have some setup to masquerade/forward ports on your machine, it might be coming from outside and appear to be coming from the local machine. Again, you should know if you set something like that up too.
CocolinoFan
commented
3 months ago Thank you, I understand. I set the log level 2 and I will output everything to a file. Is gonna be a bit hard to know when exactly the mining starts (log level 2, text scrolling by fast) but I will see the CPU use. I'm only using the official Monero GUI wallet .AppImage and my network setup is a typical home network, nothing exotic going on.
moneromooo-monero
commented
3 months ago The log is also written to ~/.bitmonero/bitmonero.log so it can be grepped later. There is also a bit more info in it.
CocolinoFan
commented
3 months ago Caught it! bitmonero.log.tar.gz monerod_manual_log_2.txt.tar.gz
cat monerod_log_2.txt | grep -i3 "start_mining"
2024-06-10 18:12:08.035 D Percent used: 89.6441 Percent threshold: 90.0000
2024-06-10 18:12:08.038 D Queueing 3 transaction(s) for Dandelion++ fluffing
2024-06-10 18:12:08.109 D SSL detection buffer, 9 bytes: 80 79 83 84 32 47 115 116 97
2024-06-10 18:12:08.109 I HTTP [127.0.0.1] POST /start_mining
2024-06-10 18:12:08.109 I [127.0.0.1:46724 INC] calling /start_mining
2024-06-10 18:12:08.111 D Filling block template, median weight 300000, 28 txes in the pool
2024-06-10 18:12:08.111 D DB map size: 233496887296
2024-06-10 18:12:08.111 D Space used: 209316081664
--
2024-06-10 18:12:08.138 I Ignoring battery
2024-06-10 18:12:08.138 I Miner thread was started [1]
2024-06-10 18:12:08.138 I Miner thread was started [2]
2024-06-10 18:12:08.138 D /start_mining processed with 0/29/0ms
2024-06-10 18:12:08.138 I Miner thread was started [5]
2024-06-10 18:12:08.138 I Miner thread was started [6]
2024-06-10 18:12:08.138 I Miner thread was started [7]127.0.0.1:46724 👀
Of course I can't find anything open with netstat -anpt | grep 46724
moneromooo-monero
commented
3 months ago The caller will have closed the connection. If you can compile your own, you could...
in src/rpc/core_rpc_server.cpp, look for:
core_rpc_server::on_start_mining
One of the first lines in this function reads:
CHECK_CORE_READY();
Right after it, insert this:
sleep(1000);
Build again. Run again. Wait till monerod freezes. It will not go full CPU, it'll just wait 1000 seconds, giving you time to look at what's connected to it.
moneromooo-monero
commented
3 months ago Might have to "#include
CocolinoFan
commented
3 months ago I've successfully added the sleep(1000); and compiled the code (had to do it manually as the default make release did not work). Unfortunately the sleep(1000) did not work, it just started mining on its own again (it is possible I compiled wrong).
But I came up with another way of figuring out what program is starting the mining:
1) Start monerod and make it output to a file
monerod --zmq-pub tcp://127.0.0.1:18083 --out-peers 32 --in-peers 64 --add-priority-node=p2pmd.xmrvsbeast.com:18080 --add-priority-node=nodes.hashvault.pro:18080 --disable-dns-checkpoints --enable-dns-blocklist| tee ~/output.txt`2) Set set_log 2 in monerod
3) Run this script in the same directory where output.txt is (preferably as root)
#!/bin/bash
# Start tailing the output.txt file
tail -F output.txt | while read line; do
# Check if the line contains the specific pattern
if echo "$line" | grep -q '\[127.0.0.1:[0-9]* INC\] calling /start_mining'; then
echo "Start to mine call detected!"
echo "$line"
echo ""
# Extract the port number
port_number=$(echo "$line" | grep -oP '(?<=127\.0\.0\.1:)\d+(?= INC)')
echo "Port used: $port_number"
echo ""
if [ ! -z "$port_number" ]; then
# Run netstat for the extracted port number
netstat_output=$(netstat -anpt | grep "$port_number")
ps_number=$(echo "$netstat_output" | awk '{print $7}' | cut -d'/' -f1)
ps_output=$(ps auxww | grep "$ps_number")
# Print the netstat output
echo "netstat output:"
echo "$netstat_output"
echo ""
echo "ps output:"
echo "$ps_output"
fi
fi
doneWhen monerod starts mining it should output information about the program that started the mining. It worked for me for the official Monero GUI Wallet. With this setup, monerod crashes often tho, so is not ideal.
I gave up on trying to find out what program was starting the mining for me. I'm doing a clean re-install of my OS.
In any case, this is expected behavior from monerod no bug or issue.
ki9us
commented
3 months ago I have two machines running monerod in docker, one local and one VPS, and they both started doing this on the same date this issue was opened (June 7). Neither have open RPCs. One was running monero:latest and the other used an older version I built from the source.
I just upped the log level and it didn't take more than a minute for the mining to start up again. I found that the call was coming from "inside the house" too. In this case 172.27.2.2 is the container's IP and 172.27.2.1 is the gateway to the host machine.
$ docker logs monerod | grep -A4 -B4 start_mining
2024-06-16 13:27:14.726 D New server for RPC connections, SSL autodetection
2024-06-16 13:27:14.726 D Spawned connection #301 to 0.0.0.0 currently we have sockets count:2
2024-06-16 13:27:14.727 D connection type 1 172.27.2.2:18081 <--> 172.27.2.1:57364 (via 172.27.2.1:57364)
2024-06-16 13:27:14.728 D SSL detection buffer, 9 bytes: 80 79 83 84 32 47 115 116 97
2024-06-16 13:27:14.728 I HTTP [172.27.2.1] POST /start_mining
2024-06-16 13:27:14.728 I [172.27.2.1:57364 INC] calling /start_mining
2024-06-16 13:27:14.731 D Filling block template, median weight 300000, 103 txes in the pool
2024-06-16 13:27:14.732 D DB map size: 90026735616
2024-06-16 13:27:14.732 D Space used: 79798337536
2024-06-16 13:27:14.732 D Space remaining: 10228398080
--
2024-06-16 13:27:14.806 I Ignoring battery
2024-06-16 13:27:14.806 I Miner thread was started [4]
2024-06-16 13:27:14.806 I Miner thread was started [5]
2024-06-16 13:27:14.806 I Miner thread was started [1]
2024-06-16 13:27:14.806 D /start_mining processed with 0/78/0ms
2024-06-16 13:27:14.806 D Destructing connection #300 to 0.0.0.0
2024-06-16 13:27:14.807 I Miner thread was started [8]
2024-06-16 13:27:14.808 I Miner thread was started [7]
2024-06-16 13:27:14.808 I Miner thread was started [6]I actually have a third machine running monerod but not listed on monero.fail and it was not affected.
I find it hard to believe that both my machines and @CocolinoFan's were compromised at the same time. I'm a little suspect that someone figured out how to send RPC calls through port 18080. But I suppose then the calls would have come from 172.27.2.2.
I'd be interested to learn:
I have zero time to look into this further so for now I disabled RPC. In about a week I could do some more debugging such as packet sniffing or trying to pinpoint the program making the call.
selsta
commented
3 months ago @ki9us mining_status shows you the address.
ki9us
commented
3 months ago The request hung a few times but after several tries I got this:
{
"active": true,
"address": "47aMSgJnHgcTefoPB2fAkKWEZWZPRYjgci67aQnFRrNXKAtW7uveU2BSFuxzpky9ntWnKAkZemKgfLA3zzGdnvcT4xGNPJn",
"bg_idle_threshold": 0,
"bg_ignore_battery": false,
"bg_min_idle_seconds": 0,
"bg_target": 0,
"block_reward": 601211460000,
"block_target": 120,
"difficulty": 326217709391,
"difficulty_top64": 0,
"is_background_mining_enabled": false,
"pow_algorithm": "RandomX",
"speed": 1069,
"status": "OK",
"threads_count": 16,
"untrusted": false,
"wide_difficulty": "0x4bf417374f"
}I don't recognize it of course. Different from the address in #5523.
selsta
commented
3 months ago From what I can tell @CocolinoFan's node was not on monero.fail, it wasn't even publicly accessible.
selsta
commented
3 months ago @ki9us were the two docker nodes that started mining set to have a restricted RPC?
ki9us
commented
3 months ago No, I will try one with --restricted-rpc and see if it comes back.
selsta
commented
3 months ago Having nodes publicly accessible without restricted RPC means anyone can send a start mining command. Though I don't fully know your setup and you said the RPC call came from your host machine, so not sure if that's what is going on here.
ki9us
commented
3 months ago As with OP's setup, the port is firewalled off. Checking from a remote VPS off the network:
Starting Nmap 7.80 ( https://nmap.org ) at 2024-06-16 10:21 MDT
Nmap scan report for gf4.pw (64.57.57.12)
Host is up.
rDNS record for 64.57.57.12: 64.57.57.12-dyn.gojade.org
PORT STATE SERVICE
18081/tcp filtered unknown
Nmap done: 1 IP address (1 host up) scanned in 2.15 secondsBut the call came from the host machine, which does have access to the RPC.
CocolinoFan
commented
3 months ago ki9us
commented
3 months ago Ok, I found the leak and it is my fault and it is embarrassing.
I set up this same unrestricted node as a web-compatible public rpc. Port 18081 was blocked but the rpc was accessible at https://xmr.gf4.pw/ through an nginx proxy.
selsta
commented
3 months ago @CocolinoFan can you rule out that you had some network misconfiguration that allowed external access to your node? To me it seems like this isn't malware but instead someone scanning the internet for publicly accessible nodes and sending RPC commands to start mining.
CocolinoFan
commented
3 months ago That's the thing, I can't rule it out. When I first setup my node I was a beginner and I red somewhere you should forward port 18081 (i think, I am not 100% sure if it was forwarded or not). I am doing some more testing now. With and without port 18081 accessible over the internet and, with and without the default starting parameters for p2pool. I will report back if anything interesting happens.
selsta
commented
3 months ago Usually people recommend to port forward 18080 so that others can connect to your node. If you have 18081 accessible from the internet without restricted RPC it means anyone can start mining.
ki9us
commented
3 months ago After I stopped the nginx proxy, the malicious mining commands stopped. Port 18081 was always closed when the node was unrestricted. So the threat actor was definitely accessing it through the web proxy
In my experience and that of others, nginx has a tendency to ignore the default_server directive. So while I'm pretty sure that the node was only accessible at xmr.gf4.pw, it might have leaked out to [IP]:80 or [IP]:443.
We were mining to the same address, therefore we had the same attacker, likely with the same TTPs. That is, your system was probably not compromised because mine wasn't. @CocolinoFan, did you have any web server or proxy running on your system? If so, grep your config files for 18081.
CocolinoFan
commented
3 months ago I did use to have an Apache website, but was moved to a different machine and network weeks before "the attack".
I've reinstalled my OS, but recreated the monerod setup (port 18081 accessible over the internet, mining in the same address, with the same IP), but monerod did not start mining on its own again.
selsta
commented
3 months ago @CocolinoFan if 18081 is accessible over the internet you have to make the node restricted, otherwise it's a matter of time until this happens again
CocolinoFan
commented
3 months ago I understand. And if it does start mining on it's own again I will be very happy. I would know, without a shadow of a doubt, that it was just an exposed port 18081 and nothing more serious.
Similar to issue 5523. All of a sudden I notice my computer's utilization is at 100%. I check what is using the resources and it was monerod. Looking at the console output I see:
I shutdown monerod and start it again. And monerod started mining again. I am running monerod for years this never happened before. Console output from the second monerod run:
No additional relevant software was installed and no monerod settings have been changed. If relevant I am also running, on the same computer: xmrig, p2pool and bitcoind.