The network seems to be getting corrupted, due to the nodes sleeping when they receive no gossip for a prolonged period of time. This causes some transactions to persist indefinitely in the mempool. We're investigating the cause.
The network is looking more stable now. A couple of issues we encountered that are perhaps worth noting:
the snark fee hint in the command line tool suggests different units than are required.
creating a faucet for a private network requires some thought. The faucet account balance will need to be a delegated to a BP otherwise it will throw off block-production. However, this might then cause the delegated BP to dominate the network. One way round this is to have all BP accounts have far more than the faucet so that the delegation has little impact on the dynamics. Why a non-bp account should ever be selected to produce a block feels like something worth investigating/fixing.
the effect of a lot of fixes (e.g. delegating a balance, changes in total currency from coinbase) weren't seen because they don't take effect until two epochs after the transaction has been snarked, which is about two weeks given the default set up.
The network seems to be getting corrupted, due to the nodes sleeping when they receive no gossip for a prolonged period of time. This causes some transactions to persist indefinitely in the mempool. We're investigating the cause.