Closed linas closed 3 years ago
OK, so I measured share submissions and payouts to three pools; one pool which blocked the scammer just a few days ago, and two which have not (yet). I submitted shares for 2-3 days, from small hash-power machines (so that it would average out). Here are the results.
First, the baseline: QRL block difficulties are variable; from 29 December 2020 to 1 Jan 2021, the difficulties and payouts low, high were:
difficulty reward QRL/hash date
737988332 5.3342 7.228e-9 11AM 1 Jan
938979454 5.3361 5.683e-9 10PM 30 Dec
Thus, a miner operating over these days might expect rewards in this range. Here are the actual results:
start date end date submitted balance QRL/Hash Pool
30 Dec 17:00 1 Jan 17:00 120512453 0.5265 4.369e-9 miningocean.org
30 Dec 04:00 31 Dec 19:00 193467739 1.0224 5.285e-9 herominers.com
28 Dec 04:00 30 Dec 16:00 413303777 1.6050 3.883e-9 pool2mine pre-ban
30 Dec 22:00 1 Jan 16:00 132890290 0.8399 6.320e-9 pool2mine post-ban
This shows the timer interval over which I mined, the hashes submitted, the rewards collected, and the average reward per hash. Times are in GMT, rounded to the hour. All three pools are seeing spiking behavior (actually, eight pools are, more about that below). I alerted the pool2mine operator, who blocked the spiker. Thus, for pool2mine, I can show pre-ban and post-ban rewards. Post-ban, the rewards are within the expected range. More-or-less on-the-button. Pre-ban rewards were a disaster. The other two pools are paying out at well below the expected value.
Again, with the spiking:
Of the three, herminers is the biggest pool, and is thus the hardest to spike, and thus pays out the best rewards (although significantly less than expected). miningocean is getting hit worse. pool2mine was getting crushed, before the ban.
Here is what "spiking" looks like:
Note the orange graph upper left. The biggest visible spike has a label: 1/1/2021 5:50:49 PM Pool hashrate: 26.36 MH/sec
Recap: if someone can mine at 2x or 3x the network hash rate, then why don't they just mine solo? Why do they hit up 8+ different pools for a few minutes at a time? Answer: they get more credits by hitting up small pools, and getting rewards out of proportion to actual hashes computed.
Who is the guilty party? Who are the affected pools? There are at least two. One is:
Q010300059eb9badfdd06657299051697472919faf89baa864351e83cd5aec68c1f9957391977ef
He's hit (at least) eight pools. Here are screenshots. Look for Q010300...91977ef
in each screenshot, notice how many hashes he's submitted (grand total) - he's always the biggest miner in the pool. Notice the hash rate, (usually 0.0 H/sec) and the "Last Share submitted" date (usually hours or days ago)
Bonus: here are all the QRL pools. Note that most of them are seeing spikes.
What's the detailed mechanism for pulling this off? I dunno. I have noticed only one tiny hint of a problem. You can do this experiment too.
N0
SZ
(for most people, this should be between 25K and 100K)NB
N1
N1 - N0
this is how many hashes the pool gave you credit for.SZ * NR
. This is how many hashes you submitted.When I do this, the number of hashes is sometimes different, by maybe 3 or 18 or something. Out of a hundred-million hashes, whats 3 or 18? Who cares? Well, but this shows the pool is recording a number that is different from the actual number of submitted hashes. Which is wrong. And its wrong not by some even number, but by some weird fraction that came out of thin air.
Now, I can't test to see what happens if I submitted a few billion hashes, but I'm thinking that the math error is going to be ... significant ... significant enough to be worth exploiting. Just guessing here. Circumstantial evidence. But, you too can run these experiments. You too can reproduce these results.
p.s. I have not seen spikes on monero pools. However, this does not mean that monero pools are not vulnerable to to something similar, maybe on a smaller scale, or something.
FWIW, contacted all the pools that I could. Admins of both miningocean and herominers are in full denial; they're saying "that is just how it works".
i feel like this will stay a monologue my friend, this code is unsupported or supported by people that ask you for $ even to answer a simple question. Sad individuals
@linas so where is the issue, redis storing the shares, redis accumulating the shares, truncation, miner calculating shares differently to pool.? 3-18 difference is negligible
systems very easy.
so i've given this some thought. maybe you could run a pool and insert some code into the pool.js module that records all shares in a file.
somewhere around here. https://github.com/dvandal/cryptonote-nodejs-pool/blob/master/lib/pool.js#L1076
add something like this
log('warn', logSystem, 'miners hashes %s', [job.difficulty])
then run you test and accumulate shares from file
would be nice if miner could have share logging turned on, then we could see where the issue occurs
think this would highlight several things
It would improve the "believability" of this issue, if someone could reproduce this, especially, my last comment https://github.com/dvandal/cryptonote-nodejs-pool/issues/620#issuecomment-753369116 - I've spoken to one person who is in complete denial, but is unwilling to perform basic tests (citing "proprietary modifications" to this code base).
As to where the problem might be: I've instrumented the miner (xmrig) to see how it works; I don't think its the miner; when it prints accepted (1305/0) diff 500054
that really is correct; especially when working with fixed difficulty. So in this case, the pool should record exactly 1305*500054 hashes (I haven't been checking this one... I guess I should.)
Being off by 3 or 18 out of a hundred-million hashes smells more like a floating-point rounding error, rather than (say) a thread race condition. Thread-race conditions (e.g. some read-modify-write was not locked/atomic) would wipe out an entire share, not a fraction of a share.
Fuuu... in a different pool, I asked it to send me fixed-sized difficulty. I picked exactly 500000. It sends me 500054 ... why? I dunno, I assumed some kind of prime-number nonce magic or something. When the miner finds something, it reports accepted (nnn/0) diff 50054
. If I look at the pool, it increments by 500000 and not 500054 .. what happened to those extra 54 hashes? (That works out to 0.0108% which is way too small to be a "pool fee" or "donations to core team") ... is my miner crazy, or is the pool crazy? This is ... irritating... the amounts are small, but why is there any difference at all?
If I get a free moment I might setup a test pool. Het the definitive answer.
Get Outlook for Androidhttps://aka.ms/ghei36
From: Linas Vepštas notifications@github.com Sent: Sunday, January 3, 2021 3:40:33 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com Cc: muscleman musclesonvacation@hotmail.com; Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Fuuu... in a different pool, I asked it to send me fixed-sized difficulty. I picked exactly 500000. It sends me 500054 ... why? I dunno, I assumed some kind of prime-number nonce magic or something. When the miner finds something, it reports accepted (nnn/0) diff 50054. If I look at the pool, it increments by 500000 and not 500054 .. what happened to that extra 54 hashes? (That works out to 0.0108% which is way too small to be a "pool fee" or "donations to core team") ... is my miner crazy, or is the pool crazy? This is ... irritating... the amounts are small, but why is there any difference at all?
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/dvandal/cryptonote-nodejs-pool/issues/620#issuecomment-753679942, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABDK6E5YCSIGRH3TZNCYJGTSYDP5DANCNFSM4VM4XFUA.
Here's another screen grab showing very regular, but absolutely immense oscillations of the network hash power:
Found that at https://explorer.theqrl.org/ -- assuming 60 seconds/block the period is about 95 minutes ... why would there be a factor of 10x or 20x oscillation of network hash power, that is extremely regular? What would cause/motivate miners to "pile on" like that? Why would this be a beneficial strategy?
Answer Q: When is the best time to mine? A: When the block difficulty is low. Q: And when is that? A: Well, you look at the difficulty. If its low, hit the network with 2x to 4x the network hash rate, mine a block every 5-20 seconds. The QRL network notices this, and gradually auto-adjusts the difficulty to make it harder. This takes about 30-60 minutes to happen. Once the difficulty gets hard, you quit, and go back to mining monero. About 30 minutes later, the QRL network difficulty drops, making it easy to mine, again. Rinse and repeat.
Why does this work? Well, if you constantly mined QRL with 2x to 4x the current network hash rate, then the block difficulty would slowly adjust to the new network hash rate -- the difficulty would be 2x to 4x harder. With current QRL network settings, it would take about a day(?) to level out at the new difficulty. But if you can aim a firehose at it, and pulse it, like this, the difficulty stays low. Mining stays easy.
To put it differently: if you control 2x to 4x the network hash rate, and you pulse it, then mining is 2x or 4x easier, than if you mined at a constant rate. For the same amount of hash power, you get 2x to 4x more coin! What a deal!
Somewhere, the QRL specs say that "enough blocks are left for 200 years" but that assumes that the mining rate is approx constant from day to day. i.e. that the network difficulty auto-adjustment is able to respond to mining rate changes. Clearly, it can't: its responding much too slowly. Someone will mine out that 200-year supply in just a decade or so, cause they can mine at 5-20 seconds/block.
Anyway, I think that is the answer for the wild hash-rate swings. Someone is pumping the network, mining while its easy, then backing off soon as the difficulty starts increasing in response. Here's my own, independent graph of time-to-block-discovery, showing the 90-minute cycles.
Actually a pool operator that I maintain software for had 400gh on monero the other day for less than 10mins. We believe there is a massive farm or facility with a Russian ip that is doing this. Seems limited to CPU algos.
Get Outlook for Androidhttps://aka.ms/ghei36
From: Linas Vepštas notifications@github.com Sent: Sunday, January 3, 2021 4:32:15 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com Cc: muscleman musclesonvacation@hotmail.com; Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Here's another screen grab showing very regulat, but absolutely immense oscillations of the network hash power: [qrl-hash-power]https://user-images.githubusercontent.com/94368/103490300-9273ae80-4de0-11eb-9f81-f6edd6cf6c42.png
Found that at https://explorer.theqrl.org/ -- assuming 60 seconds/block the period is about 95 minutes ... why would there be a factor of 10x or 20x oscillation of network hash power, that is extremely regular? What would cause/motivate miners to "pile on" like that? Why would this be a beneficial strategy?
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/dvandal/cryptonote-nodejs-pool/issues/620#issuecomment-753685748, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABDK6E6DP72LCHAMHFB2HMDSYDV67ANCNFSM4VM4XFUA.
Also with pps they'll get the majority of the block reward. So I'm about to do pplns
Get Outlook for Androidhttps://aka.ms/ghei36
From: Gary Rusher musclesonvacation@hotmail.com Sent: Sunday, January 3, 2021 4:38:35 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com; dvandal/cryptonote-nodejs-pool reply@reply.github.com Cc: Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Actually a pool operator that I maintain software for had 400gh on monero the other day for less than 10mins. We believe there is a massive farm or facility with a Russian ip that is doing this. Seems limited to CPU algos.
Get Outlook for Androidhttps://aka.ms/ghei36
From: Linas Vepštas notifications@github.com Sent: Sunday, January 3, 2021 4:32:15 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com Cc: muscleman musclesonvacation@hotmail.com; Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Here's another screen grab showing very regulat, but absolutely immense oscillations of the network hash power: [qrl-hash-power]https://user-images.githubusercontent.com/94368/103490300-9273ae80-4de0-11eb-9f81-f6edd6cf6c42.png
Found that at https://explorer.theqrl.org/ -- assuming 60 seconds/block the period is about 95 minutes ... why would there be a factor of 10x or 20x oscillation of network hash power, that is extremely regular? What would cause/motivate miners to "pile on" like that? Why would this be a beneficial strategy?
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/dvandal/cryptonote-nodejs-pool/issues/620#issuecomment-753685748, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABDK6E6DP72LCHAMHFB2HMDSYDV67ANCNFSM4VM4XFUA.
Do mining huge across many pools may keep the network diff flatter while getting blocks at many pools
Get Outlook for Androidhttps://aka.ms/ghei36
From: Gary Rusher musclesonvacation@hotmail.com Sent: Sunday, January 3, 2021 4:40:08 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com; dvandal/cryptonote-nodejs-pool reply@reply.github.com Cc: Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Also with pps they'll get the majority of the block reward. So I'm about to do pplns
Get Outlook for Androidhttps://aka.ms/ghei36
From: Gary Rusher musclesonvacation@hotmail.com Sent: Sunday, January 3, 2021 4:38:35 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com; dvandal/cryptonote-nodejs-pool reply@reply.github.com Cc: Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Actually a pool operator that I maintain software for had 400gh on monero the other day for less than 10mins. We believe there is a massive farm or facility with a Russian ip that is doing this. Seems limited to CPU algos.
Get Outlook for Androidhttps://aka.ms/ghei36
From: Linas Vepštas notifications@github.com Sent: Sunday, January 3, 2021 4:32:15 PM To: dvandal/cryptonote-nodejs-pool cryptonote-nodejs-pool@noreply.github.com Cc: muscleman musclesonvacation@hotmail.com; Comment comment@noreply.github.com Subject: Re: [dvandal/cryptonote-nodejs-pool] Share miscalculation/share scamming (#620)
Here's another screen grab showing very regulat, but absolutely immense oscillations of the network hash power: [qrl-hash-power]https://user-images.githubusercontent.com/94368/103490300-9273ae80-4de0-11eb-9f81-f6edd6cf6c42.png
Found that at https://explorer.theqrl.org/ -- assuming 60 seconds/block the period is about 95 minutes ... why would there be a factor of 10x or 20x oscillation of network hash power, that is extremely regular? What would cause/motivate miners to "pile on" like that? Why would this be a beneficial strategy?
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/dvandal/cryptonote-nodejs-pool/issues/620#issuecomment-753685748, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABDK6E6DP72LCHAMHFB2HMDSYDV67ANCNFSM4VM4XFUA.
noticed the same shenanigans on my pools:
i have set fixed diff in the miner at 200000: "user":"cm.....Z642RC1x.200000"
Pool side: (Thread 2) Accepted trusted share at difficulty 200000/206880
but on the miner side i get: new job from pool.xmrminers.club:4444 diff 200007 algo cn-pico height 630147
So there's def something odd going on
well, just checked it out. if you using my pool software. thats version < 2.0.0 there ain't an issue.
So if your looking to mine on one of these pools.
There appears to be reasonable evidence that someone has figured out how to confuse the calculation of shares for (an older version) of this pool software. It appears to affect at least ~two~ eight different pools. It's a doozey. Here we go.
An earlier version of this bug is reported here: tevador/RandomX#200 because I mistakenly assumed this was a RandomX weakness; it almost surely is not, its a pool weakness. To recap:
QRL, the "Quantum Resistant Ledger" uses RandomX. It's a small coin, about 14 MH/sec grand total network hashrate. Pool2mine is a small pool for QRL, about 84KH/sec. So, about 3x or 4x per day, someone joins this pool, mines at a rate of 2-5MH/sec for 2-5 minutes, and then leaves. When they mine, they pick up almost every block, and every block they pick up has a difficulty of 1% to 50%.
At first I thought: "a hah, someone is guessing the easy blocks and sucking them up". Nope. What I now think is happening is that someone is aiming a firehose of shares: about 100x more than what this pool normally sees, and approx 1/3rd of the total network hash-rate, and is tangling up the share calculator to credit them with more than they actually shared. (I mean, if this was a RandomX weakness, they would just solo-mine, instead of attaching to a pool.) This miner is probably a regular Monero miner (Monero has a network hash rate of 1.5GH/sec, so a 3MH/sec miner is not that big. But for QRL, 3MH/sec is huge.)
For Pool2mine you can see evidence for this as follows. Here's some recent activity:
So -- 27 December is a good example. This miner shows up at the indicated time, with maybe 5MH/sec(? approx?) hash power (this is maybe(?) 1/3rd(?) of total hash power for this coin), then, starting with block 1322615 they mined almost every block in a row, every 5 to 30 seconds, and then they leave about 7 minutes later. The network rate is supposed to be one block per minute.
You can view the above at the Pool Blocks page. The hash rate spikes are visible at the Dashboard. Note that the spikes there are are the hash-rate averaged over half an hour; the instantaneous rate is much higher. The miner is Q010300...91977ef and is visible in the top ten charts
So, how much is being lost? Here's my experiment:
If this is an old bug or a known bug, let me know. pool2mine seems to be running a Feb 11 2020 version of this s/w. No clue what version miningocean is using.