search

raiden-network / raiden

Raiden Network
https://developer.raiden.network
Other
1.84k stars 378 forks source link

Resilient transaction polling #4529

Closed hackaugusto closed 5 years ago

hackaugusto commented 5 years ago

Tests are failing because the transaction receipt is disappearing.

Geth 1.9.0 Raiden g50c090cdb

Problem Definition

The scenario tests from the weekend failed with invalid transaction, check gas price, which is raised here:

https://github.com/raiden-network/raiden/blob/28438bc88bd65743c87108deb579897ab8b07e87/raiden/network/rpc/client.py#L667-L678

This exception is raised after the following happens in order (using update transfer as an example):

Possible root causes

Recently transaction polling was changed to use receipts https://github.com/raiden-network/raiden/issues/4497 and Geth client upgraded to version 1.9.0 https://github.com/raiden-network/raiden/pull/4410

Notes

That used to happen when a transaction gas price was too low, Geth would add the transaction to the poll and then remove it (additionally it would reject all transactions from the same account). This behavior, however, was observed with the eth_getTransaction RPC call, and not the eth_getTransactionReceipt.

Dominik1999 commented 5 years ago

Here some examples of successful transactions with the Raiden exception raise Exception("invalid transaction, check gas price") Exception: invalid transaction, check gas price

found in test ms1:

found in test ms3:

found in test pfs4:

found in test pfs5:

LefterisJP commented 5 years ago

Hm so for now someone should investigate why the transaction receipt could appear with a value for a block number and then dissappear.

I am not sure if the gas price message is still relevant.

Also how is this polling code resilient to mini reorgs? This is another case I can imagine this problem popping up.

ulope commented 5 years ago

@hackaugusto asked me to give some background why the switch was made.

In the scenario player we used to use a very similar construction to the one that was in JSONRPCClient.poll() (i.e. poll web3.eth.getTransaction() until the blockNumber key in the return dict is no longer None).

The problem is that there were cases where the blockNumber was returning a value even though the tx was still pending. This lead to failures when code tried to interact with a tx result (e.g. created contract) fast enough after the prematurely exited wait loop.

According to the eth rpc spec this shouldn't happen but I observed it multiple times (unfortunately I can't recall if that only happened with a specific client).

Using getTransactionReceipt() solved this problem in the SP. There the nodes seem to behave according to the spec.

I assume the reason why we never encountered this problem in Raiden is because of the confirmation block count which adds additional wait that masks this issue.

LefterisJP commented 5 years ago

a related issue would be this: https://github.com/raiden-network/raiden/issues/4513 to track geth and parity's response on how this is supposed to behave

hackaugusto commented 5 years ago

I just tried it out with parity, I got the same error:

Traceback (most recent call last):
  File "/home/hack/work/scenario_player_venv/lib/python3.7/site-packages/raiden/network/proxies/utils.py", line 121, in log_transaction
    yield
  File "/home/hack/work/scenario_player_venv/lib/python3.7/site-packages/raiden/network/proxies/token_network.py", line 277, in new_netting_channel
    partner, settle_timeout, log_details
  File "/home/hack/work/scenario_player_venv/lib/python3.7/site-packages/raiden/network/proxies/token_network.py", line 343, in _new_netting_channel
    receipt = self.client.poll(transaction_hash)
  File "/home/hack/work/scenario_player_venv/lib/python3.7/site-packages/raiden/network/rpc/client.py", line 675, in poll
    raise Exception("invalid transaction, check gas price")
Exception: invalid transaction, check gas price
{"raiden": "0.100.5.dev37+g50c090cdb", "raiden_db_version": 23, "python_implementation": "CPython", "python_version": "3.7.3", "system": "Linux 64bit_ELF 5.1.16.a-1-hardened", "architecture": "x86_64", "distribution": "source", "event": "Starting Raiden", "logger": "raiden.ui.runners", "level": "info", "timestamp": "2019-08-05 13:02:30.320672"}
{"node": "0xee4e3d3Bfe8e910e09e77116950Ea213a5360e22", "available_nonce": 1, "client": "Parity-Ethereum//v2.5.1-beta-adabd81-20190514/x86_64-linux-gnu/rustc1.34.1", "event": "JSONRPCClient created", "logger": "raiden.network.rpc.client", "level": "debug", "timestamp": "2019-08-05 13:02:31.748751"}
LefterisJP commented 5 years ago

Also got that with geth 1.9.1 in pretty much the first transaction my node tried to send out. That means the nodes don't behave as we expect them to with getTransactionReceipt() and this needs to change asap.

hackaugusto commented 5 years ago

@Dominik1999 just replicated the problem. It was caused because of a reorg, the rough outline is:

LefterisJP commented 5 years ago

nice! As I mentioned above indeed this code did not seem to take reorgs into account whatsoever. I guess waiting for the confirmation blocks there and allowing for rewriting of the receipt would be the way to go?

Dominik1999 commented 5 years ago

ok so here is what we had, when we reached the exception: tx_receipt: None but

last_result: 

(Pdb++) last_result
AttributeDict({'blockHash': HexBytes('0x9bfc8718ca55632096d061d8038ee3251a95e39a74f704b38fd46aff576125ef'), 'blockNumber': 1066523, 'contractAddress': '0x1593392f4629f8255262820cF6F369A129532cbC', 'cumulativeGasUsed': 1464315, 'from': '0xb232e968923c7efb813c5a10e3ffb4e92768f625', 'gasUsed': 1464315, 'logs': [], 'logsBloom': HexBytes('0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'), 'root': None, 'status': 1, 'to': None, 'transactionHash': HexBytes('0x89fd1d9ed0f3a08d7843cc4c4c12b40c6d0400b7e183890df4ce31cee9b031b6'), 'transactionIndex': 0})

then we got for 

(Pdb++) self.web3.eth.getTransactionReceipt(transaction_hash)
AttributeDict({'blockHash': HexBytes('0xe1f22aa967d58b7aef4c5b4e94afd66d63bd72438cc62a79f7cf769258929013'), 'blockNumber': 1066524, 'contractAddress': '0x1593392f4629f8255262820cF6F369A129532cbC', 'cumulativeGasUsed': 1464315, 'from': '0xb232e968923c7efb813c5a10e3ffb4e92768f625', 'gasUsed': 1464315, 'logs': [], 'logsBloom': HexBytes('0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'), 'root': None, 'status': 1, 'to': None, 'transactionHash': HexBytes('0x89fd1d9ed0f3a08d7843cc4c4c12b40c6d0400b7e183890df4ce31cee9b031b6'), 'transactionIndex': 0})

and finally 

(Pdb++) self.web3.eth.getTransaction(transaction_hash)
AttributeDict({'blockHash': HexBytes('0xe1f22aa967d58b7aef4c5b4e94afd66d63bd72438cc62a79f7cf769258929013'), 'blockNumber': 1066524, 'chainId': None, 'condition': None, 'creates': '0x1593392f4629f8255262820cf6f369a129532cbc', 'from': '0xb232E968923c7EFB813c5a10E3FFb4E92768f625', 'gas': 1464315, 'gasPrice': 1015370030, 'hash': HexBytes('0x89fd1d9ed0f3a08d7843cc4c4c12b40c6d0400b7e183890df4ce31cee9b031b6'), 'input': '0x60806040526040805190810160405280600481526020017f48302e31000000000000000000000000000000000000000000000000000000008152506006908051906020019062000051929190620001e3565b503480156200005f57600080fd5b50604051620014f9380380620014f9833981018060405260808110156200008557600080fd5b8101908080519060200190929190805190602001909291908051640100000000811115620000b257600080fd5b82810190506020810184811115620000c957600080fd5b8151856001820283011164010000000082111715620000e757600080fd5b505092919060200180516401000000008111156200010457600080fd5b828101905060208101848111156200011b57600080fd5b81518560018202830111640100000000821117156200013957600080fd5b505092919050505083600160003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002081905550836000819055508160039080519060200190620001a4929190620001e3565b5082600460006101000a81548160ff021916908360ff1602179055508060059080519060200190620001d8929190620001e3565b505050505062000292565b828054600181600116156101000203166002900490600052602060002090601f016020900481019282601f106200022657805160ff191683800117855562000257565b8280016001018555821562000257579182015b828111156200025657825182559160200191906001019062000239565b5b5090506200026691906200026a565b5090565b6200028f91905b808211156200028b57600081600090555060010162000271565b5090565b90565b61125780620002a26000396000f3fe608060405234801561001057600080fd5b50600436106100ec576000357c01000000000000000000000000000000000000000000000000000000009004806332424aa3116100a957806395d89b411161008357806395d89b41146103f9578063a9059cbb1461047c578063cae9ca51146104e2578063dd62ed3e146105df576100ec565b806332424aa3146102fa57806354fd4d501461031e57806370a08231146103a1576100ec565b806306fdde03146100f1578063095ea7b31461017457806318160ddd146101da57806323b872dd146101f857806327e235e31461027e578063313ce567146102d6575b600080fd5b6100f9610657565b6040518080602001828103825283818151815260200191508051906020019080838360005b8381101561013957808201518184015260208101905061011e565b50505050905090810190601f1680156101665780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6101c06004803603604081101561018a57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291905050506106f5565b604051808215151515815260200191505060405180910390f35b6101e26107e7565b6040518082815260200191505060405180910390f35b6102646004803603606081101561020e57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291905050506107f0565b604051808215151515815260200191505060405180910390f35b6102c06004803603602081101561029457600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610b54565b6040518082815260200191505060405180910390f35b6102de610b6c565b604051808260ff1660ff16815260200191505060405180910390f35b610302610b83565b604051808260ff1660ff16815260200191505060405180910390f35b610326610b96565b6040518080602001828103825283818151815260200191508051906020019080838360005b8381101561036657808201518184015260208101905061034b565b50505050905090810190601f1680156103935780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6103e3600480360360208110156103b757600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610c34565b6040518082815260200191505060405180910390f35b610401610c7d565b6040518080602001828103825283818151815260200191508051906020019080838360005b83811015610441578082015181840152602081019050610426565b50505050905090810190601f16801561046e5780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6104c86004803603604081101561049257600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff16906020019092919080359060200190929190505050610d1b565b604051808215151515815260200191505060405180910390f35b6105c5600480360360608110156104f857600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291908035906020019064010000000081111561053f57600080fd5b82018360208201111561055157600080fd5b8035906020019184600183028401116401000000008311171561057357600080fd5b91908080601f016020809104026020016040519081016040528093929190818152602001838380828437600081840152601f19601f820116905080830192505050505050509192919290505050610e84565b604051808215151515815260200191505060405180910390f35b610641600480360360408110156105f557600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff1690602001909291905050506111a4565b6040518082815260200191505060405180910390f35b60038054600181600116156101000203166002900480601f0160208091040260200160405190810160405280929190818152602001828054600181600116156101000203166002900480156106ed5780601f106106c2576101008083540402835291602001916106ed565b820191906000526020600020905b8154815290600101906020018083116106d057829003601f168201915b505050505081565b600081600260003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055508273ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff167f8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925846040518082815260200191505060405180910390a36001905092915050565b60008054905090565b600081600160008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020541015151561084057600080fd5b81600260008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002054101515156108cb57600080fd5b6000821115156108da57600080fd5b81600160008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002054101580156109a5575081600260008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205410155b80156109b15750600082115b15610b485781600160008573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828254019250508190555081600160008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828254039250508190555081600260008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020600082825403925050819055508273ffffffffffffffffffffffffffffffffffffffff168473ffffffffffffffffffffffffffffffffffffffff167fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef846040518082815260200191505060405180910390a360019050610b4d565b600090505b9392505050565b60016020528060005260406000206000915090505481565b6000600460009054906101000a900460ff16905090565b600460009054906101000a900460ff1681565b60068054600181600116156101000203166002900480601f016020809104026020016040519081016040528092919081815260200182805460018160011615610100020316600290048015610c2c5780601f10610c0157610100808354040283529160200191610c2c565b820191906000526020600020905b815481529060010190602001808311610c0f57829003601f168201915b505050505081565b6000600160008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020549050919050565b60058054600181600116156101000203166002900480601f016020809104026020016040519081016040528092919081815260200182805460018160011615610100020316600290048015610d135780601f10610ce857610100808354040283529160200191610d13565b820191906000526020600020905b815481529060010190602001808311610cf657829003601f168201915b505050505081565b600081600160003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205410158015610d6c5750600082115b15610e795781600160003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828254039250508190555081600160008573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020600082825401925050819055508273ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff167fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef846040518082815260200191505060405180910390a360019050610e7e565b600090505b92915050565b600082600260003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002081905550600060608573ffffffffffffffffffffffffffffffffffffffff1633863087604051602401808573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018481526020018373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200180602001828103825283818151815260200191508051906020019080838360005b83811015610fd3578082015181840152602081019050610fb8565b50505050905090810190601f1680156110005780820380516001836020036101000a031916815260200191505b50955050505050506040516020818303038152906040527f8f4ffcb1000000000000000000000000000000000000000000000000000000007bffffffffffffffffffffffffffffffffffffffffffffffffffffffff19166020820180517bffffffffffffffffffffffffffffffffffffffffffffffffffffffff83818316178352505050506040518082805190602001908083835b6020831015156110ba5780518252602082019150602081019050602083039250611095565b6001836020036101000a0380198251168184511680821785525050505050509050019150506000604051808303816000865af19150503d806000811461111c576040519150601f19603f3d011682016040523d82523d6000602084013e611121565b606091505b509150915081151561113257600080fd5b8573ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff167f8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925876040518082815260200191505060405180910390a36001925050509392505050565b6000600260008473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205490509291505056fea165627a7a72305820f77074ae8e2dff5a8a100ec8a553ae368d8cd36ba23f18307e341b62c70f3b45002900000000000000000000000000000000000000000000000000000000000027100000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000c00000000000000000000000000000000000000000000000000000000000000003544b4e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003544b4e0000000000000000000000000000000000000000000000000000000000', 'nonce': 51, 'publicKey': HexBytes('0x0625560d462c3d03a55930c222257fbb6ef5cd8732eb083015ddbf056b3ab421c08e7488b5a5d1fa93ea21b9dbd0dee88f1200eb69ba0c49a3dac8f398fbbebe'), 'r': HexBytes('0xd60d8076f97859a5b2c0a180351b9e8435d0e485f77822a0b857dfea0143e621'), 'raw': HexBytes('0xf9164b33843c85512e831657fb8080b915f960806040526040805190810160405280600481526020017f48302e31000000000000000000000000000000000000000000000000000000008152506006908051906020019062000051929190620001e3565b503480156200005f57600080fd5b50604051620014f9380380620014f9833981018060405260808110156200008557600080fd5b8101908080519060200190929190805190602001909291908051640100000000811115620000b257600080fd5b82810190506020810184811115620000c957600080fd5b8151856001820283011164010000000082111715620000e757600080fd5b505092919060200180516401000000008111156200010457600080fd5b828101905060208101848111156200011b57600080fd5b81518560018202830111640100000000821117156200013957600080fd5b505092919050505083600160003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002081905550836000819055508160039080519060200190620001a4929190620001e3565b5082600460006101000a81548160ff021916908360ff1602179055508060059080519060200190620001d8929190620001e3565b505050505062000292565b828054600181600116156101000203166002900490600052602060002090601f016020900481019282601f106200022657805160ff191683800117855562000257565b8280016001018555821562000257579182015b828111156200025657825182559160200191906001019062000239565b5b5090506200026691906200026a565b5090565b6200028f91905b808211156200028b57600081600090555060010162000271565b5090565b90565b61125780620002a26000396000f3fe608060405234801561001057600080fd5b50600436106100ec576000357c01000000000000000000000000000000000000000000000000000000009004806332424aa3116100a957806395d89b411161008357806395d89b41146103f9578063a9059cbb1461047c578063cae9ca51146104e2578063dd62ed3e146105df576100ec565b806332424aa3146102fa57806354fd4d501461031e57806370a08231146103a1576100ec565b806306fdde03146100f1578063095ea7b31461017457806318160ddd146101da57806323b872dd146101f857806327e235e31461027e578063313ce567146102d6575b600080fd5b6100f9610657565b6040518080602001828103825283818151815260200191508051906020019080838360005b8381101561013957808201518184015260208101905061011e565b50505050905090810190601f1680156101665780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6101c06004803603604081101561018a57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291905050506106f5565b604051808215151515815260200191505060405180910390f35b6101e26107e7565b6040518082815260200191505060405180910390f35b6102646004803603606081101561020e57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291905050506107f0565b604051808215151515815260200191505060405180910390f35b6102c06004803603602081101561029457600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610b54565b6040518082815260200191505060405180910390f35b6102de610b6c565b604051808260ff1660ff16815260200191505060405180910390f35b610302610b83565b604051808260ff1660ff16815260200191505060405180910390f35b610326610b96565b6040518080602001828103825283818151815260200191508051906020019080838360005b8381101561036657808201518184015260208101905061034b565b50505050905090810190601f1680156103935780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6103e3600480360360208110156103b757600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610c34565b6040518082815260200191505060405180910390f35b610401610c7d565b6040518080602001828103825283818151815260200191508051906020019080838360005b83811015610441578082015181840152602081019050610426565b50505050905090810190601f16801561046e5780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6104c86004803603604081101561049257600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff16906020019092919080359060200190929190505050610d1b565b604051808215151515815260200191505060405180910390f35b6105c5600480360360608110156104f857600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291908035906020019064010000000081111561053f57600080fd5b82018360208201111561055157600080fd5b8035906020019184600183028401116401000000008311171561057357600080fd5b91908080601f016020809104026020016040519081016040528093929190818152602001838380828437600081840152601f19601f820116905080830192505050505050509192919290505050610e84565b604051808215151515815260200191505060405180910390f35b610641600480360360408110156105f557600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff1690602001909291905050506111a4565b6040518082815260200191505060405180910390f35b60038054600181600116156101000203166002900480601f0160208091040260200160405190810160405280929190818152602001828054600181600116156101000203166002900480156106ed5780601f106106c2576101008083540402835291602001916106ed565b820191906000526020600020905b8154815290600101906020018083116106d057829003601f168201915b505050505081565b600081600260003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055508273ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff167f8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925846040518082815260200191505060405180910390a36001905092915050565b60008054905090565b600081600160008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020541015151561084057600080fd5b81600260008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002054101515156108cb57600080fd5b6000821115156108da57600080fd5b81600160008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002054101580156109a5575081600260008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205410155b80156109b15750600082115b15610b485781600160008573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828254019250508190555081600160008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828254039250508190555081600260008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020600082825403925050819055508273ffffffffffffffffffffffffffffffffffffffff168473ffffffffffffffffffffffffffffffffffffffff167fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef846040518082815260200191505060405180910390a360019050610b4d565b600090505b9392505050565b60016020528060005260406000206000915090505481565b6000600460009054906101000a900460ff16905090565b600460009054906101000a900460ff1681565b60068054600181600116156101000203166002900480601f016020809104026020016040519081016040528092919081815260200182805460018160011615610100020316600290048015610c2c5780601f10610c0157610100808354040283529160200191610c2c565b820191906000526020600020905b815481529060010190602001808311610c0f57829003601f168201915b505050505081565b6000600160008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020549050919050565b60058054600181600116156101000203166002900480601f016020809104026020016040519081016040528092919081815260200182805460018160011615610100020316600290048015610d135780601f10610ce857610100808354040283529160200191610d13565b820191906000526020600020905b815481529060010190602001808311610cf657829003601f168201915b505050505081565b600081600160003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205410158015610d6c5750600082115b15610e795781600160003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828254039250508190555081600160008573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020600082825401925050819055508273ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff167fddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef846040518082815260200191505060405180910390a360019050610e7e565b600090505b92915050565b600082600260003373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008673ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002081905550600060608573ffffffffffffffffffffffffffffffffffffffff1633863087604051602401808573ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018481526020018373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200180602001828103825283818151815260200191508051906020019080838360005b83811015610fd3578082015181840152602081019050610fb8565b50505050905090810190601f1680156110005780820380516001836020036101000a031916815260200191505b50955050505050506040516020818303038152906040527f8f4ffcb1000000000000000000000000000000000000000000000000000000007bffffffffffffffffffffffffffffffffffffffffffffffffffffffff19166020820180517bffffffffffffffffffffffffffffffffffffffffffffffffffffffff83818316178352505050506040518082805190602001908083835b6020831015156110ba5780518252602082019150602081019050602083039250611095565b6001836020036101000a0380198251168184511680821785525050505050509050019150506000604051808303816000865af19150503d806000811461111c576040519150601f19603f3d011682016040523d82523d6000602084013e611121565b606091505b509150915081151561113257600080fd5b8573ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff167f8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925876040518082815260200191505060405180910390a36001925050509392505050565b6000600260008473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205490509291505056fea165627a7a72305820f77074ae8e2dff5a8a100ec8a553ae368d8cd36ba23f18307e341b62c70f3b45002900000000000000000000000000000000000000000000000000000000000027100000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000c00000000000000000000000000000000000000000000000000000000000000003544b4e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003544b4e00000000000000000000000000000000000000000000000000000000001ca0d60d8076f97859a5b2c0a180351b9e8435d0e485f77822a0b857dfea0143e621a023ad7962b3d3324e821d2ec9be3185561ac9ea7ae616699058df810c86b8a976'), 's': HexBytes('0x23ad7962b3d3324e821d2ec9be3185561ac9ea7ae616699058df810c86b8a976'), 'standardV': 1, 'to': None, 'transactionIndex': 0, 'v': 28, 'value': 0})
hackaugusto commented 5 years ago

I guess waiting for the confirmation blocks there and allowing for rewriting of the receipt would be the way to go?

I think we should just revert the changes, the previous code was correct and handled the corner case of a gas price too low.

LefterisJP commented 5 years ago

Hmmm I think we are kind of rushing here. We had a problem as seen by @ulope which prompted us to do this.

If we revert blindly:

  1. We reintroduce that problem [ :disappointed: ]
  2. Lose the performance boost of less RPC calls. [ :disappointed: ]
  3. Get out of our current bad situation. [ :smiley: ]

Couldn't we just try to find a way to handle the reorgs here?

For example as I said above the receipt can be rewritten in the case of a reorg unless the gas price is too low, in which case getTransaction would also return None. Something like below:

            # if the transaction was added to the pool and then removed
            if tx_receipt is None and last_result is not None:
                tx = self.web3.eth.getTransaction(transaction_hash)
                if not tx:
                    raise Exception("invalid transaction, check gas price")

Additionally this whole problem uncovered some test cases which involve reorgs which are not tested by our test suite. We should create tests that cover this and not have to catch this with the Scenario Player again. It seems it happens very often ... out there ... which is very strange since our tests did not catch this :confused:

LefterisJP commented 5 years ago

@hackaugusto thanks for the non-reverting fix. Let's see how this goes.

Looking at my comment above:

It seems it happens very often ... out there ... which is very strange since our tests did not catch this

After thinking about this a bit more I think it's obvious why this was not caught by our tests. Both with parity and with geth our tests create a PoA testnet. As such mining of blocks and inclusion of transactions in them is not handled the same way as in the PoW mainnet/testnets.