Testnet stuck on block 27070

[nomnombtc]

Hello,

it seems someone mined a bunch of blocks yesterday, which maybe(?) triggered the BIP102 rule that it rejects blocks that are too small. But whoever did it did not follow up with an actual bigger block, so two of my nodes rejected it with "bad-blk-length-toosmall" banned a ton of nodes and are stuck now on block 27070.

I also have a BitcoinUnlimited node up on testnet5 which has no such reject builtin and this one is now on block 32767...

2017-07-09 21:26:53 UpdateTip: new best=0000000035a7b078c8b54e33b496dcbd66f8d52049da3684d80291d1cc13f29a height=27070 version=0x20000000 log2_w
ork=56.568487 tx=1081252 date='2017-07-09 21:26:51' progress=1.000000 cache=0.3MiB(1034tx)
2017-07-09 21:26:53 ERROR: AcceptBlock: bad-blk-length-toosmall, size limits failed (code 16)
2017-07-09 21:26:53 ERROR: ProcessNewBlock: AcceptBlock FAILED
2017-07-09 21:26:53 ERROR: AcceptBlockHeader: block 0000000016b2af58fc30fef380a4cec7262858e68aaa0bad37a9c419257d0636 is marked invalid
2017-07-09 21:26:53 ERROR: invalid header received
2017-07-09 21:26:53 ProcessMessages(headers, 82 bytes) FAILED peer=10

[christophebiocca]

Can you confirm that your node banned its peers for relaying a too-small block? Run getpeerinfo and paste the output here if you can't interpret it?

[nomnombtc]

I think it banned mostly nodes that were running slightly older versions like 1.14.1, or also my BU node (5.135.186.15). (because they kept extending on this chain without rejecting the block, the BU node is now on block 32826)

getpeerinfo: https://pastebin.com/Au8CDaDs

listbanned:

[
  {
    "address": "5.135.186.15/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "52.168.136.58/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "52.213.126.249/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "82.201.92.138/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "101.37.33.210/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "101.99.31.77/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "104.239.175.108/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "115.75.4.185/32",
    "banned_until": 1499722044,
    "ban_created": 1499635644,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "153.169.136.25/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "195.154.69.36/32",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }, 
  {
    "address": "2001:41d0:8:c30f::1/128",
    "banned_until": 1499722014,
    "ban_created": 1499635614,
    "ban_reason": "node misbehaving"
  }
]

[christophebiocca]

It turns out the DOS-ban is intentional, unlike in the first draft of the hard-fork-on-block-X PR.

Given that, what you're seeing is the intentional outcome of wipeout protection, except that no blocks are being built by segwit2x nodes for the testnet. The chain is forked permanently (at least for 1MB nodes, BU might switch back and forth based on which chain is the longest).

[nomnombtc]

Ok, so the code seems to work like it should. This is still a strange event, I would have expected that the miners are prepared that this condition could happen, so they would be ready to mine the >1MB block in time.

I guess a miner needs to create a big block now for the chain to continue...

[jheathco]

On https://testnet5.blockchain.info/ it appears to be continuing to follow the fork with 1mb blocks while http://btcfaucet.ix28uktqsp.us-west-2.elasticbeanstalk.com/ (which I've compiled to be running the latest beta release) appears to be on the chain stuck on block 27070.

Assuming as @nomnombtc stated we're just waiting for a miner to create a single block > 1mb to continue this blockchain. I'm not sure what solution was selected to ensure we've got this covered when live - assuming we'll have enough of a backlog of transactions in the mempool doesn't seem like a reliable option.

[betawaffle]

How long has it been stuck? Is anybody actually mining 2x on testnet5?

[jheathco]

There are transactions in the mempool and I'm assuming there are still miners on testnet5 - likely just waiting on enough transactions to occupy a > 1mb block.

[betawaffle]

Does anyone have mempool stats?

[sneurlax]

Should have used the hardfork bit instead of a hacky kludge 🙃

[betawaffle]

Have there seriously no miners or testers for the past nearly 24 hours!?

[JaredR26]

Should have used the hardfork bit instead of a hacky kludge

This is only a problem on testnet where there are not enough transactions to reach the limit easily without scripting spam.

[betawaffle]

And nobody planned on scripting spam (aka. testing)?

[GratefulTony]

So basically, we're supposed to disregard this bad testing outcome [FAIL] because the test environment wasn't properly thought out?

[JaredR26]

Pay attention to the facts people. The non-segwit2x test chain was 5,697 blocks ahead 7 hours ago. Someone jumped on testnet with a modern ASIC miner under the non-segwit2x codebase and mined nearly 6000 blocks in less than 1 day. Someone is screwing with the chain intentionally before the devs intended to start the test.

So yes, you're supposed to disregard attacks that can't possibly happen on mainnet being done by trolls. Or you can just continue to attack the project for lulz, no one with any critical thinking skills is going to listen to you after looking at the facts.

[GratefulTony]

So how are we going to test this under adversarial conditions? or do we conclude that "testing is impossible"?

[JaredR26]

So how are we going to test this under adversarial conditions? or do we conclude that "testing is impossible"?

Please describe an adversarial condition where the main blockchain could get 6,000 blocks (three difficulty changes) in a 24 hour period with an empty memory pool. After you do so, we can discuss how to test that situation.

[buzztiaan]

lol, issues a 5 usd altcoin can overcome are too big for btc1

[jheathco]

@GratefulTony once the first block of > 1mb is created the blockchain will continue as normal. What @JaredR26 stated makes perfect sense.

[betawaffle]

Why the delay in testing?

[GratefulTony]

@jheathco

Adversarial conditions on mainnet will be entirely different when real money is on the line-- these kids are pulling your feathers for lulz--Don't expect reality to be any kinder than a test tube, even if this particular failure mode is unlikely.

My question is: what is the "Plan B" for testing this code?

[JaredR26]

@betawaffle

Why the delay in testing?

I'm not sure, as I'm not privy to the testing plan (and agree that that part should be more public and clear).

If you and @GratefulTony disagree that this scenario is not a feasible adversarial condition for mainnet, or disagree with my speculation on the cause of the issue, can you clarify why?

If instead either of you agree with my response that testing for adversarial conditions that cannot possibly occur on mainnet is not necessary, can you state so? There's so much anger and deceit in this debate - from every side - that a lack of clarity will only contribute to more anger and deceit in the future from at least one side of the debate. Fair?

[GratefulTony]

@JaredR26

I'm trying to be as productive as possible here: what's plan B to test the code if testnet doesn't play nice? Get more real miners to mine testnet?

[betawaffle]

@JaredR26 I agree that's not an adversarial scenario for mainnet. The concern I have is this testing plan which AFAICT, either has no transactions, or hasn't started yet (which doesn't make much sense).

[NiKiZe]

@GratefulTony No plan B is needed, Just one >1MB block once there is enough transactions to fill it. Worst case scenario on mainnet is that it will work as expected!

[NiKiZe]

One could only conclude that the test so far is a success, S2x refuses blocks that is not >1MB when they need to be. Not enough transactions is an non issue on mainnet. The outcome will only be clear after the needed size of transactions has been reached.

[JaredR26]

My question is: what is the "Plan B" for testing this code?

I suspect the test will need to be restarted as the devs won't be able to see the effects of the split in real-time anymore. To prevent another attack they may need to plan a different testnet starting over with better parameters to get it done quickly / control when it activates. Unfortunately that testnet may need to be done in secret (Thanks anonymous attacker, you've done a great service here!).

I'm trying to be as productive as possible here: what's plan B to test the code if testnet doesn't play nice? Get more real miners to mine testnet?

You still didn't either agree with what I said, or state what part you disagreed with... Can you do that?

I don't think that adding miners to testnet is the solution, getting into a testnet arms race with anonymous attackers wouldn't be productive. I'm open to suggestions or maybe they will weigh in(I'm not privy to the actual plans), but the only ideas that come to mind initially would be tighter control over the activation process in testnet code and restarting, or a secret new testnet. Neither are great solutions.

@JaredR26 I agree that's not an adversarial scenario for mainnet. The concern I have is this testing plan which AFAICT, either has no transactions, or hasn't started yet (which doesn't make much sense).

Thank you for stating so. I agree that the lack of clarity around the testing plan isn't ideal. Unfortunately it may become more secret now to prevent a repeat of this from interfering with observations... Making it more open and still thorough within the time limit might be a harder path, sadly.

[betawaffle]

@JaredR26 You mean a private testnet? That will boost confidence, for sure.

[JaredR26]

@GratefulTony No plan B is needed, Just one >1MB block once there is enough transactions to fill it. Worst case scenario on mainnet is that it will work as expected!

I'm not sure that this is sufficient testing. I'd much rather be watching / debugging multiple nodes/screens at the moment of hardfork to see what happens, when, and to whom.

One could only conclude that the test so far is a success, S2x refuses blocks that is not >1MB when they need to be.

That's one test and one potential failure mechanism. I doubt it was intended, though, and it is quite far from a thorough test of the variety of outcomes.

[betawaffle]

As a bitcoin user, "trust us, we tested it" isn't going to fly.

[JaredR26]

@JaredR26 You mean a private testnet? That will boost confidence, for sure.

Agreed, not ideal. If we have better suggestions we should list them to help the plan. Maybe they could whitelist specific miners under their control on testnet? Wouldn't help though if the whitelist was public as the attacker could mine to the whitelisted targets.

As a bitcoin user, "trust us, we tested it" isn't going to fly.

Based on your statements, I don't think "segwit2x" would fly with you either, so that statement doesn't really mean anything. But I agree that openness would be better if at all possible, but openness would probably be worse than either missing deadlines significantly or poor testing if those were the only options.

[jgarzik]

To summarize the scenario, the expected behavior of the HF trigger point was activated. The chain rules require a >1M block at this point.

It is normal for test networks to have minimal mining power. This implies that anyone can connect a miner to the Bitcoin Core testnet or the segwit2x testnet and disrupt the chain. This is a known attribute of test networks, which does not occur on mainnet.

In this case, someone accelerated a test plan scheduled far in the future to trigger immediately. A very low cost annoyance attack, in sum.

It was expected when the segwit2x effort began that folks would try to disrupt segwit2x testing, and make it harder to test the software. This falls within the realm of expected behaviors.

[mkwia]

@jgarzik do you intend to use private testnets?

[betawaffle]

I agree, this matches exactly what I would expect this code to do. The only problem I see is that nobody is creating the necessary transactions. Why has nobody attempted to use the testnet for so long?

[JaredR26]

In this case, someone accelerated a test plan scheduled far in the future to trigger immediately. A very low cost annoyance attack, in sum.

Is the test going to be repeated as intended, or is that portion of it considered complete?

[christophebiocca]

51 blocks actual testing, AFAICT.

[JaredR26]

Why has nobody attempted to use the testnet for so long?

It's testnet, they'd have to create 2000+ transactions(or equivalent) in between every single block. Since the test wasn't planned anytime soon apparently, I doubt anyone would have seen a need to do that. The hardfork itself is farther out and probably second priority to the segwit activation testing.

[justvanbloom]

So this nontest testnet Szenario is a fault by Setup? Impossible on mainnet to have blocks below 1mb?

[betawaffle]

they'd have to create 2000+ transactions(or equivalent) in between every single block.

They only have to do that once right now.

[JaredR26]

So this nontest testnet Szenario is a fault by Setup? Impossible on mainnet to have blocks below 1mb?

Only one block is required to be >1mb. What's impossible about this situation is having a nearly-empty mempool for hours on end. We're lucky if the mempool is nearly empty for even one block today, much less hours.

they'd have to create 2000+ transactions(or equivalent) in between every single block.

They only have to do that once right now.

Considering Jeff first responded 12 minutes ago and there are several devs involved... Give them time. I doubt Jeff can even answer my/our questions about the plan right now before he talks to the other individuals involved. Coordinating people and making decisions takes time, give him some time.

[jgarzik]

To @ all re test plan, as @christophebiocca hints, there are changes to the test implementation needed for further activation scenario testing (ref #51 et al). This would necessitate updating all test clients with incompatible changes. In short, test network disruption was expected, even in absence of help from jokers on the Internet.

@mkwia: Several of us already use private testnets to make first-pass testing before pushing PRs to github for review.

However, a public testnet is very important. We will use that unless jerks on the Internet make it impossible to use a public testnet.

DDoS'ing of alternate implementations is a common technique in the Bitcoin space, sadly.

@justvanbloom: See "Hard Fork on Block X", https://github.com/btc1/bitcoin/issues/29

[betawaffle]

As far as I can tell, the jerks on the internet haven't made this impossible to test at all. The only thing that has happened is the appearance of failure because testing hasn't started yet (plus jerks on the internet). No need to abandon testnet5, unless you're trying to test anything before the hardfork.

[dooglus]

Could we please disable the Issues page for this repository?

By definition anyone having any kind of issue with btc1 fails to recognize its inherent perfection and is therefore an enemy of the project.

[chjj]

Haven't been paying attention to btc1 development lately. @jgarzik, from what I'm seeing, it looks like the wipeout protection rule was added to the validation code but not the mining code? Either that, or the primary miner on the testnet right now did not upgrade? This doesn't look like an attack to me. I'd be the first to call it out if it was, since I'm all about preventing attacks.

edit: Also, I don't think the >1mb wipeout protection rule is a good idea anyway. It makes mining straight up weird (i.e. what does the rpc do if the mempool is empty and the HF block needs to be mined? require a longpoll of the endpoint? fill the coinbase with garbage data?). There's probably a better way to do wipeout protection.

[johanndt]

This doesn't look like an attack to me.

@chjj There definitely was some sort of attack. It was because the hash-rate suddenly jumped by an order of magnitude. 6000 blocks mined in under 24 hours. Whereas normal conditions would be the usual one every 10 minutes.

However as you say, that doesn't explain why a proper new larger block hasn't been mined yet. If as you say there is no wipeout protection in the mining side, then we need to come up with some mechanism to force the mining of a larger block even if there's not enough transactions. I believe on main net there is a very large transaction crafted with a mining bounty - maybe something similar could be injected into test net.

[christophebiocca]

@johanndt: This was discussed in #29.

Can't use a hardcoded transaction because the funder could double spend the inputs.

As of right now default block selection code will not make its own bloat-txes to get over the 1MB limit.

[chjj]

@johanndt, if it was an attack, it was only an attack on the lack of necessary mining code in btc1, which isn't an attack IMO. This is a testnet, so that is a valid way of disclosing a bug.

[jgarzik]

@chjj is correct that the mining code does not synthetically force a >1M block at fork point X, if the mempool lacks 1M of traffic.

The behavior is that the chain will pause until sufficient transactions arrive to build a large block, then life continues as normal. This is easy to achieve on mainnet, and only slightly more cumbersome on testnet.

Overall it is a good field test so far.

[kek-coin]

P2P banning for following an invalid chain tip/relaying invalid blocks was encountered during BIP148 development as well, the BTC1 project may want to benefit from UASF/bitcoin#25.

[nyhele]

So let me get this straight. SegWit2x will hardfork because in their mind the network is overloaded. But they may have to create fake transactions or pause their chain until transactions build up because there wont be enough come fork time. That seems a bit ironic.

[ghost]

I'm still awake to see the moment where the UASF devs (which are supposed to be the self-centered fanatics) send a tip about coding properly using publicly available code.