M2 Test output - Githubissues

JB273 commented 4 years ago

Running Test M2 on Jupyter notebook 6.0.3 , output is not correct;

Test-M2: Median is correctly computed for even and odd number of available slots enter the number of blocks you want to test: 10 current and last block numbers 11796 11817 wait until the new epoch begins at block number ... 11818 new epoch first block reached 11818 will begin testing 1iteration will begin ... current block number 11818

KeyError Traceback (most recent call last)

in ----> 1 M2_test() in M2_test() 31 if i['metrics'] == None: 32 continue ---> 33 by_shard_metrics = i['metrics']['by-shard'] 34 slots = len(by_shard_metrics) 35 delegation = total_delegation / slots KeyError: 'by-shard'

ivorytowerdds commented 4 years ago

the same issue! see https://github.com/harmony-one/harmony-log-analysis/issues/1

ivorytowerdds commented 4 years ago

I pass twice today. @JB273 maybe you can use the new script to test, if you also pass, then we can close this one!

sophoah commented 4 years ago

@ivorytowerdds @JB273 M2_test in the new script is still not working even after change.

In master repo, test_case.py M2_test uses:

        # calculate the median manually
        slot_winners = getMedianRawStakeSnapshot()['epos-slot-winners']
        stake = []
        for i in slot_winners:
            stake.append((float(i['eposed-stake'])))
        cal_median = float(get_median(stake))

i've changed that to my local copy :

        # calculate the median manually
        validator_infos = getAllValidatorInformation()
        stake = []
        for i in validator_infos:
            total_delegation = i['total-delegation']
            if i['metrics'] == None:
                continue
            by_key_metrics = i['metrics']['by-bls-key']
            slots = len(by_key_metrics)
            delegation = total_delegation / slots
            for i in range(slots):
                stake.append(delegation)
        cal_median = float(get_median(stake))

however it looks like the manual calculation vs the one returned by the network is always different:

=============== Starting M2_test ===============

INFO:economic-test:Test-M2: Median is correctly computed for even and odd number of available slots
INFO:economic-test:current and last block numbers: 34508, 34541
INFO:economic-test:current epoch: 908, begin testing...
WARNING:economic-test:Test-M2: Fail
WARNING:economic-test:calculated median: 2.6519582272405003e+24
WARNING:economic-test:rpc median: 2.5e+24

OR

=============== Starting M2_test ===============

INFO:economic-test:Test-M2: Median is correctly computed for even and odd number of available slots
INFO:economic-test:current and last block numbers: 34489, 34503
INFO:economic-test:current epoch: 907, begin testing...
WARNING:economic-test:Test-M2: Fail
WARNING:economic-test:calculated median: 2.5e+24
WARNING:economic-test:rpc median: 2.6519582272405e+24

Code looks correct after change but the number of total slot taken seems to be the issues between our manual calculation and the one coming from the network.

ivorytowerdds commented 4 years ago

I check the M2 script. It's what we wrote had early last week https://github.com/harmony-one/harmony-log-analysis/blob/master/notebooks/economic-test/old_version_test/M2-test.ipynb. But since it's failed sometimes, Ganesha asked me to write using the current method, use the getMedianRawStakeSnapshot. Does it fail for you? For me, it doesn't fail for about 5 tests.

I will change the script back to the getMedianRawStakeSnapshot temporarily

harmony-one / harmony-log-analysis

M2 Test output #2