docs: Sharding Spec clarity. macOS and Docker Setup. Troubleshooting steps

ltfschoen commented 6 years ago

Where it explains process of assigning a Validator to a specific Shard. Added clarity to show how a Validator becomes a Collator, and that they create Collations on Shards (not Blocks as they are referred to on the Main Chain)
Add clarity to argument name of addHeader
Add explanation of how coinbase is created
Add clarity that period_start_prevhash is the block hash of the "last block"
Minor grammar tweak (i.e. sent not changed to not sent)
Clarity in pseudocode to show where it is referring to remaining_gas
Add example Setup steps on macOS
Add Troubleshooting section showing how to overcome common macOS error caused by SSL dependency

ltfschoen commented 6 years ago

@mhchia I've updated the Pull Request with this commit https://github.com/ethereum/sharding/pull/51/commits/a352b847c729be056fcba33c6b990ed1643f5350 addressing your comments. It appears to work with 3.6.4 Release Candidate 1 3.6.4rc1 and I've overcome all the OpenSSL errors now. But when I run the Unit Tests with pytest sharding/tests/ (using Python version 3.6.4rc1), all of the tests pass except for one of them. I'm not sure if it's actually just a test wasn't passing previously anyway, other whether it was a unit test that should already be passing.

Could anyone else who is using the latest version of the Sharding repo please confirm if the test that is shown as failing for me is actually passing for them?

I already know that I need to be using Python 3.6+ because if I switch to Python 3.5.0 and try to run the Unit Tests I get the error E Exception: Requires python3.6+ since Viper requires Python 3.6.

I even tried changing back to the earliest version of Python 3.6 (i.e. Python 3.6.0) with pyenv install 3.6.0, and then installed the dependencies with python setup.py install, and then installed with specific pyethereum branch and commit hash (as mentioned in the Readme) with USE_PYETHEREUM_DEVELOP=1 python setup.py develop and then installed more dependencies (as mentioned in the Readme) pip install -r dev_requirements.txt, but it still gives the error below

$ pytest sharding/tests/
===================================================== test session starts =====================================================
platform darwin -- Python 3.6.4rc1, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /Users/Ls/code/blockchain/clones/sharding, inifile:
plugins: timeout-1.0.0, cov-2.5.1, catchlog-1.2.2
collected 44 items                                                                                                            

sharding/tests/test_collation.py .                                                                                      [  2%]
sharding/tests/test_collator.py .......                                                                                 [ 18%]
sharding/tests/test_main_chain.py .....F                                                                                [ 31%]
sharding/tests/test_receipt_consuming_tx_utils.py .                                                                     [ 34%]
sharding/tests/test_shard_chain.py ........                                                                             [ 52%]
sharding/tests/test_state_transition.py .......                                                                         [ 68%]
sharding/tests/test_stateless_collator.py .                                                                             [ 70%]
sharding/tests/test_tester.py ...                                                                                       [ 77%]
sharding/tests/test_used_receipt_store.py .                                                                             [ 79%]
sharding/tests/test_used_receipt_store_utils.py .                                                                       [ 81%]
sharding/tests/test_validator_manager.py ..                                                                             [ 86%]
sharding/tests/test_validator_manager_utils.py ......                                                                   [100%]

========================================================== FAILURES ===========================================================
___________________________________________________ test_longest_chain_rule ___________________________________________________

    def test_longest_chain_rule():
        # Initial chains
        shard_id = 1
        t = chain(shard_id)

        # [block 1]
        block_1 = t.mine(1)
        log.info('[block 1] CURRENT BLOCK HEAD:{}'.format(encode_hex(t.chain.head_hash)))

        t.set_collation(
            shard_id,
            expected_period_number=t.chain.get_expected_period_number())

        # [block 2]: includes collation A -> B
        t.tx(tester.k1, tester.a2, 1, data=b'', shard_id=shard_id)
        collation_AB = Collation(t.collate(shard_id, tester.k0))
        block_2 = t.mine(5)
        log.info('[block 2] CURRENT BLOCK HEAD:{}'.format(encode_hex(t.chain.head_hash)))
        log.info('[block 2] CURRENT SHARD HEAD:{}'.format(encode_hex(t.chain.shards[shard_id].head_hash)))
        assert t.chain.shards[shard_id].get_score(t.chain.shards[shard_id].head) == 1

        # [block 2']: includes collation A -> B
        # Change main chain head
        t.change_head(block_1.hash)
        # Clear tester
        t.set_collation(
            shard_id,
            expected_period_number=collation_AB.header.expected_period_number,
            parent_collation_hash=collation_AB.header.parent_collation_hash)
        # tx of shard 1
        t.tx(tester.k1, tester.a2, 1, data=b'', shard_id=shard_id)
        collation_AB_2 = Collation(t.collate(shard_id, tester.k0))
        # tx of main chain
        t.tx(tester.k1, tester.a4, 1, data=b'')
        assert collation_AB.hash == collation_AB_2.hash
        t.mine(5)
        log.info('[block 2\'] CURRENT BLOCK HEAD:{}'.format(encode_hex(t.chain.head_hash)))
        log.info('[block 2\'] CURRENT SHARD HEAD:{}'.format(encode_hex(t.chain.shards[shard_id].head_hash)))
        assert t.chain.shards[shard_id].get_score(t.chain.shards[shard_id].head) == 1
        assert t.chain.get_score(t.chain.head) == 37

        # [block 3']: includes collation B -> C
        # Clear tester
        expected_period_number = t.chain.get_expected_period_number()
        t.set_collation(shard_id, expected_period_number)
        # tx of shard 1
        t.tx(tester.k1, tester.a2, 1, data=b'', shard_id=shard_id)
        # tx of main chain
        t.tx(tester.k1, tester.a4, 1, data=b'')
        collation_BC = Collation(t.collate(shard_id, tester.k0))
>       t.mine(5)

sharding/tests/test_main_chain.py:169: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sharding/tools/tester.py:281: in mine
    assert self.chain.add_block(self.block)
sharding/main_chain.py:101: in add_block
    while b.header.number >= int(self.db.get(b'GENESIS_NUMBER')):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ethereum.db._EphemDB object at 0x10be92cc0>, key = b'GENESIS_NUMBER'

    def get(self, key):
>       return self.db[key]
E       KeyError: b'GENESIS_NUMBER'

/Users/Ls/.pyenv/versions/3.6.4rc1/lib/python3.6/site-packages/ethereum/db.py:27: KeyError
---------------------------------------------------- Captured stdout call -----------------------------------------------------
Initializing chain from provided state
INFO:eth.block  Block pre-sealed, 2564963 gas used 
INFO:eth.chain  Adding to head head=c6745cf3
INFO:eth.chain  Added block 1 (446b1608) with 6 txs and 2564963 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=446b1608
INFO:eth.chain  Added block 2 (cec1ab90) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=cec1ab90
INFO:eth.chain  Added block 3 (8549c1b6) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=8549c1b6
INFO:eth.chain  Added block 4 (ed680141) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=ed680141
INFO:eth.chain  Added block 5 (7b07e394) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=7b07e394
INFO:eth.chain  Added block 6 (1c052cf8) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 202212 gas used 
INFO:eth.chain  Adding to head head=1c052cf8
INFO:eth.chain  Added block 7 (49e07165) with 2 txs and 202212 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=49e07165
INFO:eth.chain  Added block 8 (33407a23) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=33407a23
INFO:eth.chain  Added block 9 (b113d25e) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=b113d25e
INFO:eth.chain  Added block 10 (c0c628db) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=c0c628db
INFO:eth.chain  Added block 11 (88b52465) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=88b52465
INFO:eth.chain  Added block 12 (cc49c424) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=cc49c424
INFO:eth.chain  Added block 13 (5daa7d06) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=5daa7d06
INFO:eth.chain  Added block 14 (fa6a8895) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=fa6a8895
INFO:eth.chain  Added block 15 (8034054f) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=8034054f
INFO:eth.chain  Added block 16 (5785e2d4) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=5785e2d4
INFO:eth.chain  Added block 17 (b51c120c) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=b51c120c
INFO:eth.chain  Added block 18 (342fa793) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=342fa793
INFO:eth.chain  Added block 19 (9195098b) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=9195098b
INFO:eth.chain  Added block 20 (ceb73727) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=ceb73727
INFO:eth.chain  Added block 21 (c6cb65bd) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=c6cb65bd
INFO:eth.chain  Added block 22 (e583595f) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=e583595f
INFO:eth.chain  Added block 23 (2fdbf57d) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=2fdbf57d
INFO:eth.chain  Added block 24 (e11bf42c) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=e11bf42c
INFO:eth.chain  Added block 25 (7dd2218f) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=7dd2218f
INFO:eth.chain  Added block 26 (61614956) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=61614956
INFO:eth.chain  Added block 27 (27389302) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=27389302
INFO:eth.chain  Added block 28 (a3c431c9) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=a3c431c9
INFO:eth.chain  Added block 29 (a1a4d39b) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=a1a4d39b
INFO:eth.chain  Added block 30 (c99ef694) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=c99ef694
INFO:eth.chain  Added block 31 (2be547ea) with 0 txs and 0 gas 
INFO:sharding.shard_chain   Initializing chain from provided state 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=2be547ea
INFO:eth.chain  Added block 32 (ad10a13b) with 0 txs and 0 gas 
INFO:test.shard_chain   [block 1] CURRENT BLOCK HEAD:ad10a13bb5cd251ec837e838510610cbeaeef3c8c9c1d805d772a435e80d97f2 
INFO:sharding.shard_state_transition    Collation pre-sealed, 21000 gas used 
INFO:sharding.shard_chain   Receiving collation(da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673) which its parent is in db: 0000000000000000000000000000000000000000000000000000000000000000 
INFO:sharding.shard_chain   collation_score of da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673 is 1 
INFO:sharding.shard_chain   Added collation (da00e9d8) with 1 txs 
INFO:eth.block  Block pre-sealed, 182467 gas used 
INFO:eth.chain  Adding to head head=ad10a13b
INFO:eth.chain  Added block 33 (71f357b7) with 1 txs and 182467 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=71f357b7
INFO:eth.chain  Added block 34 (94fbf3a6) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=94fbf3a6
INFO:eth.chain  Added block 35 (8a55d0fc) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=8a55d0fc
INFO:eth.chain  Added block 36 (ba20775d) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Adding to head head=ba20775d
INFO:eth.chain  Added block 37 (cd08da7f) with 0 txs and 0 gas 
INFO:test.shard_chain   [block 2] CURRENT BLOCK HEAD:cd08da7fdddc7bd0e0835263c42d2d435702766625f63414968ab6ee19e12c76 
INFO:test.shard_chain   [block 2] CURRENT SHARD HEAD:da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673 
INFO:sharding.shard_state_transition    Collation pre-sealed, 21000 gas used 
INFO:sharding.shard_chain   Receiving collation(da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673) which its parent is in db: 0000000000000000000000000000000000000000000000000000000000000000 
INFO:sharding.shard_chain   collation_score of da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673 is 1 
INFO:sharding.shard_chain   Added collation (da00e9d8) with 1 txs 
INFO:eth.block  Block pre-sealed, 203467 gas used 
INFO:eth.chain  Receiving block 33 (cdb3e32c) not on head (cd08da7f), adding to secondary post state ad10a13b 
INFO:eth.chain  Added block 33 (cdb3e32c) with 2 txs and 203467 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Receiving block 34 (2ed60b5b) not on head (cd08da7f), adding to secondary post state cdb3e32c 
INFO:eth.chain  Added block 34 (2ed60b5b) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Receiving block 35 (ae061922) not on head (cd08da7f), adding to secondary post state 2ed60b5b 
INFO:eth.chain  Added block 35 (ae061922) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Receiving block 36 (2fd630d7) not on head (cd08da7f), adding to secondary post state ae061922 
INFO:eth.chain  Added block 36 (2fd630d7) with 0 txs and 0 gas 
INFO:eth.block  Block pre-sealed, 0 gas used 
INFO:eth.chain  Receiving block 37 (e30e8e1a) not on head (cd08da7f), adding to secondary post state 2fd630d7 
INFO:eth.chain  Added block 37 (e30e8e1a) with 0 txs and 0 gas 
INFO:test.shard_chain   [block 2'] CURRENT BLOCK HEAD:cd08da7fdddc7bd0e0835263c42d2d435702766625f63414968ab6ee19e12c76 
INFO:test.shard_chain   [block 2'] CURRENT SHARD HEAD:da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673 
INFO:sharding.shard_state_transition    Collation pre-sealed, 21000 gas used 
INFO:sharding.shard_chain   Receiving collation(b99bedd6fc78c87a662d39266d64764e3d4a3dd6ecf4204609a38d3012e194cc) which its parent is in db: da00e9d8dcb252d067cffe47cdd731cba964cce916b7a984dad52b77af660673 
INFO:sharding.shard_chain   collation_score of b99bedd6fc78c87a662d39266d64764e3d4a3dd6ecf4204609a38d3012e194cc is 2 
INFO:sharding.shard_chain   Added collation (b99bedd6) with 1 txs 
INFO:eth.block  Block pre-sealed, 191012 gas used 
INFO:eth.chain  Receiving block 38 (25a5bfdd) not on head (cd08da7f), adding to secondary post state e30e8e1a 
====================================================== warnings summary =======================================================
None
  pytest-catchlog plugin has been merged into the core, please remove it from your requirements.

-- Docs: http://doc.pytest.org/en/latest/warnings.html
====================================== 1 failed, 43 passed, 1 warnings in 85.38 seconds =======================================

hwwhww commented 6 years ago

@ltfschoen See the travis setting of #53, that should pass the tests.

If you dopython setup.py develop before you do USE_PYETHEREUM_DEVELOP=1 python setup.py develop, the commit setting may NOT overwrite the latest ethereum and viper. In that case, you need to uninstall ethereum and viper and then use the USE_PYETHEREUM_DEVELOP flag.

ltfschoen commented 6 years ago

@hwwhww I tried following your advise by proceeding with the following steps shown below, but that one test still fails. Any other ideas?

Show versions installed - https://pip.pypa.io/en/stable/reference/pip_list/

pip freeze | grep -i -E "ethereum|viper|setuptools"

ethereum==2.1.0
viper==0.0.2

Install uninstall and reinstall dependencies - https://github.com/ethereum/sharding/pull/53

python -m pip uninstall setuptools -y;
python -m pip install setuptools==37;
python -m pip uninstall ethereum -y;
python -m pip install ethereum==2.1.0;
python -m pip uninstall viper -y;
USE_PYETHEREUM_DEVELOP=1 python setup.py develop;
python -m pip install -r dev_requirements.txt;

Run Unit Tests
```
pytest sharding/tests/
```

hwwhww commented 6 years ago

@ltfschoen Can you try without python -m pip install ethereum==2.1.0;?

ltfschoen commented 6 years ago

@hwwhww I've fixed all the issues now in these two commits. All the tests are passing: https://github.com/ethereum/sharding/pull/51/commits/7e5fb4d4a378bf2b41632d64f21ed6b610a28e98, and https://github.com/ethereum/sharding/pull/51/commits/5e4fd7e72f110425f2a83040283bca059c9a6d9f

ltfschoen commented 6 years ago

@hwwhww I've made those further changes and fixed numbering of the steps in this commit https://github.com/ethereum/sharding/pull/51/commits/a71ede1e855fa948e775e66626f7b3750f9df66c

ltfschoen commented 6 years ago

@mhchia @hwwhww I've created a Docker configuration to hopefully help with ease of adoption across platforms, tested that it works with all tests passing, and included instructions in the Readme, in this commit https://github.com/ethereum/sharding/pull/51/commits/4fb12224c3cf48640e2981890f8521e47b435e41

hwwhww commented 6 years ago

The old PoC codebase is archived.

ethereum / sharding

docs: Sharding Spec clarity. macOS and Docker Setup. Troubleshooting steps #51