search

oliverw / miningcore

Miningcore is a high-performance Mining Pool Software for Linux and Windows.
https://store.miningcore.pro
MIT License
717 stars 660 forks source link

Other pools get way more blocks. Configuration issue for shares? #150

Closed Kukks closed 6 years ago

Kukks commented 6 years ago

Comparing my lyra2rev2 pool to other pools(yiimp based), they seem to get more blocks for the same ratio of hashrate. Is there some configuration I should be adjusting?

Pool is reporting the following: 

Miners: 29
Hashrate: 3.99 GH/s
Valid Shares Per Second: 3
Current Block: 65176
Peers: 99
Network Difficulty 1.998543824405046 K
Network Hashrate 169.24 GH/s

block explorer for straks: https://straks.info current difficulty around 3000-4000 my config:

      "4500": {
        "listenAddress": "0.0.0.0",
        "name": "SimpleDifficulty",
        "difficulty": 32.0,
        "varDiff": {
          "minDiff": 0.5,
          "targetTime": 15.0,
          "retargetTime": 60.0,
          "variancePercent": 90.0
        }
      },
      "7710": {
        "listenAddress": "0.0.0.0",
        "name": "SimpleDifficulty",
        "difficulty": 32.0,
        "varDiff": {
          "minDiff": 0.5,
          "targetTime": 15.0,
          "retargetTime": 60.0,
          "variancePercent": 30.0
        }
      },
      "4501": {
        "listenAddress": "0.0.0.0",
        "name": "HardDifficulty",
        "difficulty": 500.0,
        "varDiff": {
          "minDiff": 10.0,
          "targetTime": 15.0,
          "retargetTime": 60.0,
          "variancePercent": 50.0
        }
      }
    },

https://mpool.mooo.com/site/mining https://datapaw.net/site/mining https://www.cryptopool.xyz/site/mining

oliverw commented 6 years ago

We need to get to the bottom of this. Could it be VarDiff related? Did you try connecting a miner to your own pool and compare the difficulty assigned to the difficulty when mining on the other pools you've mentioned?

Kukks commented 6 years ago

I'll try and test tonight and see if I can find a difference. I did switch the config of variancePercent to 90 from 30 and am seeing a little better performance but that's only speculation. The block time on straks is 60 seconds, could that affect anything?

I just noticed my networks stats of the pool reporting this:

   "networkStats": {
      "networkType": "Main",
      "networkHashRate": 181716534221.55481,
      "networkDifficulty": 2046.5088761907671,
      "lastNetworkBlockTime": "2018-01-04T07:17:01.5742892Z",
      "blockHeight": 65176,
      "connectedPeers": 99,
      "rewardType": "POW"
    }

The block height seems to be stuck. This is probably another separate issue.

oliverw commented 6 years ago

@Kukks Could you try this and report back?

 "blockRefreshInterval": 333,
 "jobRebroadcastTimeout": 10,
Kukks commented 6 years ago

Just applied it now to the pool. Current block only updated on restart. http://5.189.159.254:4000/api/pools/straks1 @oliverw

oliverw commented 6 years ago

@Kukks The config changes were related to the block finding frequency. Could you try it? The stuck block height is a bug.

Kukks commented 6 years ago

Pool has been running on these settings since you posted. 3 blocks since then. Network hash is at 106gh/s and I only have ~2 so will see. @oliverw

oliverw commented 6 years ago

@Kukks Is that an improvement?

Kukks commented 6 years ago

I'll give it till end of day to compare to yesterday. But it seems to be doing better than yesterday

On Thu, Jan 4, 2018 at 2:06 PM, Oliver Weichhold notifications@github.com wrote:

@Kukks https://github.com/kukks Is that an improvement?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-355277538, or mute the thread https://github.com/notifications/unsubscribe-auth/ABu-_mZ39E7repPzgRP7RZxuRNuWAzrIks5tHMzqgaJpZM4RR8wp .

oliverw commented 6 years ago

Ok, please keep me posted. I take this very seriously.

Kukks commented 6 years ago

Will do. I do think adjusting the vardiff did a big difference though.

On Thu, Jan 4, 2018 at 2:39 PM, Oliver Weichhold notifications@github.com wrote:

Ok, please keep me posted. I take this very seriously.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-355284350, or mute the thread https://github.com/notifications/unsubscribe-auth/ABu-_pFfXQE4TVR09hSt1X6Ffc7nS-80ks5tHNSlgaJpZM4RR8wp .

Kukks commented 6 years ago

@oliverw Spoke too soon. No blocks found since 13:25 :(

Kukks commented 6 years ago

@oliverw So it's been with no blocks for almost 18 hours now. My miners have fled like there was a plague. :(

oliverw commented 6 years ago

I know that feeling all too well ;(

Lower jobRebroadcastTimeout to 3

Kukks commented 6 years ago

Done.

On Fri, Jan 5, 2018 at 8:53 AM, Oliver Weichhold notifications@github.com wrote:

I know that feeling all too well ;(

Lower jobRebroadcastTimeout to 3

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-355493495, or mute the thread https://github.com/notifications/unsubscribe-auth/ABu-_kEpaXgp_P_b0DX9NlWkTKCV5VYfks5tHdTjgaJpZM4RR8wp .

Kukks commented 6 years ago

@oliverw I've noticed that there are only 2 other pools left and they seem to have ~60gh/s and ~40gh/s in a ~100gh/s network hashrate. Could it simply be because of this?

oliverw commented 6 years ago

There's always a problem with hashrate monopolies (or duopolies) like this, yes.

oliverw commented 6 years ago

You could also try running the latest dev branch commit which contains some network low-level optimizations.

martin-key commented 6 years ago

Sorry for messing up here. Z-Nomp (node-stratum-pool) seem to suffer from the same problems in Bitcoin Gold. I have been digging a lot into this and:

I have been considering that there can be a problem in the daemon since other pool operators state that there is no such problem with other Z currencies, but however I have not found anything that could cause this kind of "bad luck".

oliverw commented 6 years ago

@StarbuckBG Are you referring to your z-node-stratum-pool derivate created for BTG?

martin-key commented 6 years ago

Yes: https://github.com/poolgold/z-nomp-bitcoin-gold https://github.com/StarbuckBG/node-stratum-pool

martin-key commented 6 years ago

Initially we thought that it is a Redis issue, as Redis is single thread and in the way znomp is made it is really easy to fill it with data and it has major issues on the fetches. Hence we have made changes and it didn't improved. Rest of info is provided above.

oliverw commented 6 years ago

Well, Miningcore is heavily multi-threaded I would be surprise if its a performance issue.

martin-key commented 6 years ago

The devs of pool.gold will migrate to it over the next week. Please check your gitter I have dropped some messages. I can give you some access so we can debug and improve under heavy load.

On 7 Jan 2018, at 10:58, Oliver Weichhold notifications@github.com wrote:

Well, Miningcore is heavily multi-threaded I would be surprise if its a performance issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-355808819, or mute the thread https://github.com/notifications/unsubscribe-auth/ADrjrRmSO7sdUYheyAXkQtJfO_5oK9XYks5tIIcpgaJpZM4RR8wp.

oliverw commented 6 years ago

@StarbuckBG They will migrate to what?

martin-key commented 6 years ago

To Miningcore (:

On 7 Jan 2018, at 11:05, Oliver Weichhold notifications@github.com wrote:

@StarbuckBG https://github.com/starbuckbg They will migrate to what?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-355809143, or mute the thread https://github.com/notifications/unsubscribe-auth/ADrjrUVxDkHAtBsSMjVlQLo1rvkHrct9ks5tIIjngaJpZM4RR8wp.

oliverw commented 6 years ago

@StarbuckBG Will do. One question though. I thought pretty much all pools are running https://github.com/StarbuckBG/node-stratum-pool and pool.gold is the biggest BTG pool. How can they have an issue with found blocks? And even if they do wouldn't everyone else have the same problem?

martin-key commented 6 years ago

On average pool has about 12% lower luck than the one supposed to have (based on 1 month calculations). All the z-nomp based pools are having that problem. There is a chat for investigating this problem in the BTG Slack with all the pool operators. It would be helpful if you join as well. And pool.gold is the 3rd largest BTG pool. First and second are respectively supernova and miningpoolhub. Both of them are using the python based pool stratum. This is why 8 devs are looking for the problem for the last 4 weeks and it is still unknown.

There was a problem with the block validation (valid blocks shown invalid) and I have risen it as an issue to the repo owner (https://github.com/joshuayabut/equihashverify/issues/3 https://github.com/joshuayabut/equihashverify/issues/3). It was taking away 1 ever 25 blocks on average and I have found that by logging the shares with higher difficulty than the target for the network. Then I have found in the logs that some of the “block candidates” are rejected because of invalid equihash solution, but on the other hand when I validate manually the solution is correct.

Znomp is good for small pools, for example if you have farm, but it is not for a product large pool. That is why all the pools that have more than 500 users must decide wether to switch to new software or not.

On 7 Jan 2018, at 11:10, Oliver Weichhold notifications@github.com wrote:

@StarbuckBG https://github.com/starbuckbg Will do. One question though. I thought pretty much all pools are running https://github.com/StarbuckBG/node-stratum-pool https://github.com/StarbuckBG/node-stratum-pool and pool.gold is the biggest BTG pool. How can they have an issue with found blocks? And even if they do wouldn't everyone else have the same problem?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-355809329, or mute the thread https://github.com/notifications/unsubscribe-auth/ADrjrYnntuMeyxxCf1e57byKHf7so9xYks5tIIoAgaJpZM4RR8wp.

oliverw commented 6 years ago

@StarbuckBG Send me a slack invite.

martin-key commented 6 years ago

Please check here - https://join.slack.com/t/bitcoin-gold/shared_invite/enQtMjY1MzkzMzUxNjY4LWM1YmQ4MjZhZTQxMWE1ZDQyNjA4N2QwZTkyZjYzMjhiMzdlMmVkNjQ3NzZlZDdmMDE4NWIyY2JmYzdjYmE2MzA

I’m there as @Martin, and I have a status “Out Sick”. There are impersonators, so beware.

Thanks for all the help.

Sent from my iPhone 7 plus

On Jan 7, 2018, at 11:44 AM, Oliver Weichhold notifications@github.com wrote:

@StarbuckBG Send me a slack invite.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

oliverw commented 6 years ago

Signed in to slack as CoinFoundry

martin-key commented 6 years ago

Messaged there.

Sent from my iPhone 7 plus

On Jan 7, 2018, at 12:01 PM, Oliver Weichhold notifications@github.com wrote:

Signed in to slack as CoinFoundry

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Allar commented 6 years ago

I'm in the same boat. I got my own shitcoin, shares are being accepted, but in the same timeframe that p2pool and nomp had X mining effort, miningcore with the same mining effort solved almost 5% of blocks. I have a relatively closed environment and am able to coordinate mining efforts, and it doesn't seem to be an issue like 'another mining pool is huge and therefore is taking all the blocks'. I can hotswap MiningCore out on a pool server with nomp/p2pool, and end up seeing a lot more solved blocks with either.

martin-key commented 6 years ago

Can you please check debug.log of the coin daemon and grow for acceptBlock.

I had problems with incorrect SegWit blocks, then with Orphaned blocks and went to zeroMQ external stratum option and it is working beautifully.

Sent from my iPhone 7 plus

On Jan 15, 2018, at 4:14 AM, Michael Allar notifications@github.com wrote:

I'm in the same boat. I got my own shitcoin, shares are being accepted, but in the same timeframe that p2pool and nomp had X mining effort, miningcore with the same mining effort solved almost 5% of blocks. I have a relatively closed environment and am able to coordinate mining efforts, and it doesn't seem to be an issue like 'another mining pool is huge and therefore is taking all the blocks'. I can hotswap MiningCore out on a pool server with nomp/p2pool, and end up seeing a lot more solved blocks with either.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Kukks commented 6 years ago

@StarbuckBG How did you set up the external stratum config and with what software?

martin-key commented 6 years ago

  1. You must get one of the node-stratum-pool repos for the coin that you mine - https://github.com/coinfoundry https://github.com/coinfoundry

  2. In the node-stratum-pool you must make a config.json file. You can follow the instructions on the repo for the config file, but be aware that is must be in a JSON file, not on main.js file. Also add the “coin” : {} information inside the config JSON.

  3. Add this to config.json:

    "publisherSocket": "tcp://<StratumMachineIP(this machine. It must be the external IP)>:4001", "publisherTopic": “topic1”

  4. Inside COIN config of miningcore config file you add "externalStratum": true, "externalStratumZmqSocket" : "tcp://<ip from point 3>:4001", "externalStratumZmqTopic" : “topic1"

  5. you start with node path/to/lib/index.js path/to/config.json (mind that the path to config.json must be relative to index.js, not the current directory).

Be aware that if you enable external stratum, people will not be able to connect to the current pool. Only 1 is working - wether external stratum or miningcore stratum.

On 15 Jan 2018, at 9:31, Andrew Camilleri notifications@github.com wrote:

@StarbuckBG https://github.com/starbuckbg How did you set up the external stratum config and with what software?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coinfoundry/miningcore/issues/150#issuecomment-357602570, or mute the thread https://github.com/notifications/unsubscribe-auth/ADrjrZjawgy3cNX0MKJvww79v2LxPU5Kks5tKv7tgaJpZM4RR8wp.

Kukks commented 6 years ago

@StarbuckBG Ah, so the external stratum setting only works with nomps' stratum. It wouldn't be able to hook up to say, yiimp?

oliverw commented 6 years ago

@Kukks The external stratum stuff is for a couple of NOMP pool derivates (this, this and this) I was experimenting with when researching an issue similar to this one.

martin-key commented 6 years ago

@Oliver BTG pool runs fine on the external stratums with the provided steps to setup.

When you have take please check the last issue I have opened. I can give you access to the server if needed. I’m able to reproduce it each time.

Sent from my iPhone 7 plus

On Jan 15, 2018, at 10:10 AM, Oliver Weichhold notifications@github.com wrote:

@Kukks The external stratum stuff is for a couple of NOMP pool derivates (this, this and this this I was experimenting with when researching an issue similar to this one.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Gadflyii commented 6 years ago

So what this fixed, or is it still an open issue?

Kukks commented 6 years ago

I'm leaving this open as I can't tell right now since my hashrate is too low for the current Straks network hashrate to make assumptions. I still think there is some sort of performance issue with finding blocks when compared to the yiimp solution.

oliverw commented 6 years ago

@Kukks I'm calling for some concerted effort to figure this out.

The only way to solve this mystery is probably multiple people doing code analysis over the two areas I've mentioned.

Gadflyii commented 6 years ago

@oliverw I will start to go over the job management and test it

Allar commented 6 years ago

I'm not smart enough to be involved, but I've had this reoccuring issue reported (haven't been able to reproduce myself) that may be related to job management?

Using nomp/p2pool, the following behavior did NOT occur, but appears to be happening with MiningCore.

Miners who mine actively for long periods of time will stop getting payments, and also stop gaining pending shares, even though they are actively mining. Restarting the miner and reconnecting to the pool generally fixes the problem. Multiple people on my pool have reported this and have transaction logs (backed by our block explorer) that show no payments when they ought to be still being paid and earning shares. The mining pool still counts these miners as connected to the pool however when pulling the worker count stat.

Regarding total block count, we've switched everyone to our miningcore server and we no longer have gaps in mining blocks, but it appears our average block difficulty is generally lower as well than when we used other pool software. Still not really sure whats going on, and maybe its just a quirk of our blockchain, but I'm definitely getting more consistent reports of bad behavior by the pool from miners than with any other pool server.

oliverw commented 6 years ago

Miners who mine actively for long periods of time will stop getting payments, and also stop gaining pending shares, even though they are actively mining.

Well, I had to deal with a lot of those complaints on coinfoundry.org (formerly poolmining.org) and in retrospective every single case was either that the person stopped mining at some point and had a misconfigured miner. Take our Dash pool for for example. The pool has paid out close to half a million dollars in coins so far and ASIC miners are very, very concerned about payments and ROI. You can bet that if there was problem with the payment system, those guys would have drawn the pitchforks and left long ago.

oliverw commented 6 years ago

I wanted to get to the bottom of this. Therefore I decided to test this on a production pool with high enough and consistent hashrate. Therefore I chose Coinfoundry's Dash Pool. Throughout the past two weeks I switched back and forth between Miningcore's internal stratum and an external one which is a stripped down version of the node-stratum-pool that only processes shares and block submissions.

The net result was that I was unable to find any significant advantage of using node-stratum-pool over Miningcore's internal stratum. In fact judging from the average block effort, Miningcore had a slight advantage. One thing to remember is that you should utilize streaming block updates by all means. I've updated the Wiki to reflect this (see zmqBlockNotifySocket and wsPort).

Gadflyii commented 6 years ago

@oliverw Quick question, how do you configure MC to use the node-stratum-pool ?