Stale Shares

[tuaris]

I'm guessing that these are stale shares? I get a lot (I mean a lot) of them every once in a while with a fast miner

2013-12-28 16:58:20,142 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:20,142 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:20,850 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:20,850 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:21,234 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:21,235 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:21,595 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:21,596 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:21,862 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:21,863 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:22,066 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:22,066 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:22,149 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:22,149 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:23,056 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:23,057 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:23,343 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:23,343 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:23,641 INFO block_updater # Checking for new block.
2013-12-28 16:58:23,716 INFO template_registry # Job id '16e6' not found
2013-12-28 16:58:23,717 INFO interfaces # False (0) INVALID scrypt1300.jjmccalip
2013-12-28 16:58:24,454 INFO template_registry # Job id '16e6' not found

What would cause this and what would be the best way to prevent it? Is it a system resource issue? (need more RAM or something) Or do I need to tweak the VDIFF_MIN_VDIFF_TARGET_TIME and VDIFF_MAX_TARGET variables a bit?

[ahmedbodi]

thats just a new block nothing can be changed im afraid

[tuaris]

I tweaked the difficulty variables a bit and the number of stale shares has been reduced. I did the following:

disabled VDIFF_X2_TYPE lowered VDIFF_MAX_TARGET and VDIFF_TARGET_TIME increased VDIFF_VARIANCE_PERCENT

I wonder if it's possible for the service to detect these and adjust these values on the fly on it's own?

[ahmedbodi]

no, that just reduces the shares, it has 0 affect on stale shares

[educatedwarrior]

Very interesting. I was doing the same thing in my config file using the same reasoning. If you lower the vdiff_target_time, wouldn't that cause workers to submit more shares per block based on their hashpower, thus reducing the percentages of stale shares?

Also let's say a coin had a target block time of 40 seconds and your vdiff_target_time was set to 15 seconds. Would not 1 out of 3 shares be stale theoretically?

Logical reasoning in my mind says so, so I am struggling to understand why these settings have 0 affect on stale shares.

[educatedwarrior]

The answer to this question I believe lies in the understanding of how mining works.

Either

1. When work is sent out to the workers, all workers are working to solve the same block simultaneously. OR
2. When work is sent out to the workers, workers may receive work to solve different outstanding blocks and the block that is solved first is placed on the block chain.

I think it is number 1, but I don't know for sure. Maybe guru Ahmed could shed some light.

[tuaris]

I think its definitely 1.

Anyway, after studying the code a little, I think the stales are mostly a result of having MERKLE_REFRESH_INTERVAL too low. I have it set to 60. Perhaps it should be higher if a block is suppose to be found every 10 minutes?

I guess what I am asking is getting a new block template for the same block useful? Or why not just get a new template only when the "new block notification" is triggered?

I appologize if any of this sounds simple or stupid, I'm only vaguely familiar with the concepts behind all this.

[obigal]

The answer is 1.

MERKLE_REFRESH_INTERVAL = has no effect on stales, this checks to see if there are new transactions that are trying to make it into a block 30 seconds is a good number 60 is fine as well not too low but not too high

Let me try to explain how mining works, or more specifically stale shares.

All miners are working on the same block at the same time.

When a new block is found on the network stratum immediately broadcasts the new work to miners.

Your miner may have already submitted work for the last block to the stratum server before it has actually received the new work. (cgminer has a setting to submit stales or not turn it on and see the results)

Once stratum receives it it is no longer valid since it is from the previous block and absolutely nothing from it can be used so it is thrown away and marked as stale. (meaning from the last block)

reducing vdiff_target_time will actually increase stales, because the number of stales is in relation to the difficulty lower difficulty = more shares = more potential stale shares

The absolute best way to combat stale shares is to have the block notify script working properly.

In reality there is nothing that can be done there is always going to be some latency between the server and client and information crossing paths at the same time before one knows about the other.

[ahmedbodi]

thanks obigal, been really busy lately need to get back to work on this

[educatedwarrior]

@obigal , could you tell me how I could find out more about the "block notify script"? Is this totally separate from Ahmed's stratum-mining script?

[ahmedbodi]

nope

[educatedwarrior]

@ahmedbodi Okay, I found it. You telling me I've been doing this all wrong. I found this embedded in Ahmed's script. So, I have to load blocknotify when loading the daemon?

Send notification to Stratum mining instance on localhost that there's new bitcoin block You can use this script directly as an variable for -blocknotify argument: ./litecoind -blocknotify="blocknotify.sh --password admin_password" This is also very basic example how to use Stratum protocol in native Python

[ahmedbodi]

yup

[educatedwarrior]

Well, thanks --- been running pools for over 3 months now and didn't know anything about it. I've been pushing the minimum difficulty up on the pools to reduce stale shares, because I didn't know about this -- and miners were screaming at me.

[nevrending]

@educatedwarrior Dumb question: What is "admin_password" there?

[educatedwarrior]

@obigal and @ahmedbodi , why do you get stale shares when a block is not found. I believe you also get stale shares when stratum sends out the same work unit to mulitiple workers ... when the work unit is solved, the other miner working on the same work unit gets stale shares. How do you combat this scenario?

[tuaris]

I've realized that the stale share problem comes down to two things:

1) Fast coins - Not much you can do, it's simply going to happen. It can be helped a little by setting up a proper block notify script. 2) Latency - Reduce it as much as possible between all points. From block notification to server performance, and the connection between client and server.

The most common cause to stale shares is latency. A client sitting half way around the world with a slow connection is going to get lots of stale shares. The same is true server side. If your server is on a high latency low bandwidth connection, then all your clients will get stale shares. Examples of high latency "pipes" are residential class cable and DSL.