Reduce test fixtures size to speed up tests

[ixje]

Current behavior

travis-ci takes > 15 minutes to run all tests. One reason is that the tests rely on a fixture database that's > 500MB in size. The fixture database is basically a zipped up testnet blockchain folder. This big size is the result of having testcases deployed on the regular TestNet, which then have many unnecessary blocks added to the fixtures in between different tests created on that network.

We believe that the fixture size can be easily reduced to < 50 MB by recreating all tests on a special private TestNet. This testnet should be based on neo-privatenet-docker The image should only be spun up when adding new tests that requires events or interactions on a full blockchain (e.g. smart contract storage tests) to keep the database to the minimal size necessary.

TODO:

• [x] create an inventory of all tests to be ported to the private testnet (hint: look at what derives from: [BlockchainFixtureTestCase](https://github.com/CityOfZion/neo-python/blob/master/neo/Utils/BlockchainFixtureTestCase.
• [x] migrate all tests to a new private testnet docker image (based on neo-privatenet-docker)
• [x] make testnet docker image publicly available
• [x] create new fixture file, get it on CoZ AWS (ask @localhuman), update all test cases.
• [x] create PR to neo-python

[dauTT]

Thanks @jseagrave21 for your hard work! :+1: I will have a look at the remaining test cases tomorrow.

[dauTT]

Hi @ixje ,

there are few test cases left. This one is really tough:

    def test_get_storage_item(self):
        contract_hash = '16f1559c3c27d66d087bef936804105457617c8a'
        storage_key = binascii.hexlify(b'totalSupply').decode('utf-8')
        req = self._gen_rpc_req("getstorage", params=[contract_hash, storage_key])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(res['result'], '00f938dbba05')
        actual_val = int.from_bytes(binascii.unhexlify(res['result'].encode('utf-8')), 'little')
        self.assertEqual(actual_val, 6300100000000)

I have try to replicate the test case on the privnet by running this transaction:

neo> testinvoke b9fbcff6e50fd381160b822207231233dd3c56c2 totalSupply []         
Used 0.32 Gas 

-------------------------------------------------------------------------------------------------------------------------------------
Test invoke successful
Total operations: 347
Results [{'type': 'ByteArray', 'value': '00a031a95fe300'}]
Invoke TX GAS cost: 0.0
Invoke TX fee: 0.0001
-------------------------------------------------------------------------------------------------------------------------------------

Enter your password to continue and invoke on the network

[password]> **********                                                          
[I 180830 23:01:20 Transaction:618] Verifying transaction: b'bad4face9c794900569f861a0386ccea1c3d99fda5c3602ab9d213d981524fe2' 
Relayed Tx: bad4face9c794900569f861a0386ccea1c3d99fda5c3602ab9d213d981524fe2 

However the transaction does not produce the desire

storage_key = binascii.hexlify(b'totalSupply').decode('utf-8')

If I look at all the key in my privnet:

def test_storage():
    from neo.Implementations.Blockchains.LevelDB.DBCollection import DBCollection
    from neo.Implementations.Blockchains.LevelDB.DBPrefix import DBPrefix
    from neo.Core.State.StorageItem import StorageItem
    import binascii 
    blockchain = LevelDBBlockchain("/home/dau/.neopython/docker-privnet/Chains/backup/fixtures/test_chain") 

    Blockchain.RegisterBlockchain(blockchain)
    _db = Blockchain.Default()._db
    sn=_db.snapshot()
    storages = DBCollection(_db, sn, DBPrefix.ST_Storage, StorageItem)
    for k in storages.Keys:
        value =storages.TryGet(k).Value
        print(k, "\n", value, "\n new k, v \n")

Output:

new k, v 

 b'\xc2V<\xdd3\x12#\x07"\x82\x0b\x16\x81\xd3\x0f\xe5\xf6\xcf\xfb\xb9H\x03;X\xefT|\xbfT\xc8\xee/r\xa4-[`<\x00\xaf' 
 b'\x00B\xe4\xb52' 

 new k, v 

 b'\xc2V<\xdd3\x12#\x07"\x82\x0b\x16\x81\xd3\x0f\xe5\xf6\xcf\xfb\xb9in_circulation' 
 b'\x00\xa01\xa9_\xe3\x00' 

 new k, v 

 b'\xc2V<\xdd3\x12#\x07"\x82\x0b\x16\x81\xd3\x0f\xe5\xf6\xcf\xfb\xb9initialized' 
 b'\x01' 

they are ending either with no "human readable" suffix or with suffix in ['in_circulation', 'initialized']. Do you have any hint how I can generate a transaction with storage key ending with suffix totalSupply?.

I believe the answer of this question may have to do with the ico_template:

NEP5_METHODS = ['name', 'symbol', 'decimals', 'totalSupply', 'balanceOf', 'transfer', 'transferFrom', 'approve', 'allowance']

[ixje]

@dauTT actually the one ending in in_circulation is the storage item that returns totalSupply. I'm going to assume that back when that test was created I believe the contract was this old version I still have a copy of here (if you look at those 2 lines it will make sense)

        if operation == 'totalSupply':
            return storage.get(token.in_circulation_key)

A newer version can be found here Which almost has the same code

    elif operation == 'totalSupply':
        return Get(ctx, TOKEN_CIRC_KEY)

where TOKEN_CIRC_KEY refers to

TOKEN_CIRC_KEY = b'in_circulation'

So if we take the value belonging to the in_circulation key from your output b'\x00\xa01\xa9_\xe3\x00', and convert that we get

In [1]: int.from_bytes(b'\x00\xa01\xa9_\xe3\x00', 'little')
Out[1]: 250000000000000

looks acceptable to me, no?

[dauTT]

Thanks @ixje, your explanation makes sense.

Finally we manage to migrate all the test cases to the privnet. :sweat_smile:

Before I can hand over to you all the test cases for reviewing, I need to run once again all the tests with the latest neo-python. @ixje , could you confirm that the latest neo-python is 0.7.8-dev master branch?

There are two test cases that will need to be updated everything time we add new blocks to the privnet fixtures blockchain:

I found the following:

1. reference

   def test_a_initial_setup(self):
   
       self.assertEqual(self._blockchain.Height, 758986)

2. reference

   def test_get_bestblockhash(self):
       req = self._gen_rpc_req("getbestblockhash", params=[])
       mock_req = mock_request(json.dumps(req).encode("utf-8"))
       res = json.loads(self.app.home(mock_req))
       self.assertEqual(res['result'], '0x6e9d1698ba2cd43432b6df1fca4bfbc53b2492a29401d1411699ddbd56cf8559')

   Do we want to keep them?

[dauTT]

@ixje ,

you can review all the test cases here: link

Following files have been modified:

        modified:   ../Implementations/Blockchains/LevelDB/test_LevelDBBlockchain.py
    modified:   ../Implementations/Blockchains/LevelDB/tests/test_leveldb.py
    modified:   ../Settings.py
    modified:   ../SmartContract/tests/test_smart_contract.py
    modified:   ../SmartContract/tests/test_smart_contract2.py
    modified:   ../SmartContract/tests/test_smart_contract3.py
    modified:   ../Utils/BlockchainFixtureTestCase.py
    modified:   ../VM/tests/test_interop_blockchain.py
    modified:   ../api/JSONRPC/test_json_invoke_rpc_api.py
    modified:   ../api/JSONRPC/test_json_rpc_api.py
    modified:   ../api/REST/RestApi.py
    modified:   ../api/REST/test_rest_api.py

Latest image created: docker pull dautt/neo-privnet-unittest:v0.0.21

Smart Contracts deployed on the image: UnitTest-SM.zip

Fixtures to upload to AWS:

1. fixtures_v8.tar.gz
2. notif_fixtures_v8.tar.gz (both fixture files are in total about 7MB)

All the test cases have been tested using neo-python 0.7.8-dev from the master branch.

[jseagrave21]

@dauTT Nice to see it coming together!

[ixje]

@dauTT super nice! I'm excited to go through all this, ~14 MB sounds amazing 🥇 As for the 2 tests, yes let's keep those in. It's a small adjustment to make whenever new tests are added, and still useful to quickly verify proper loading of the right blockchain fixtures. I might be able to go through them on Wednesday, otherwise no sooner then Friday (just to give an indication)

[ixje]

test cases @dauTT the tests in your link look good to me , great job (@jseagrave21 too)👌 You probably want to run make lint before submitting the PR because I think i see some spaces that violate the automatically enforced code style checker.

fixture testnet data The collection of smart contracts in the zip also looks fine. The readme can use a little bit of work. I know what you did and i can make sense of most data in the readme, but i think a first time reader might not or have trouble following. I think a structure along the following lines with a bit more words would be great

Intro
- neo-python fixtures are based of a private unit test net <link>
- it currently contains the following smart contracts ...
- a default wallet with the following credentials is in place...

How to add new fixture tests
- pull & run private net (howto...)
- use wallet with the following credentials as needed
- add test, pr, ask for image update

For instructions like

rm -rf /home/dau/.neopython/Chains/privnet

try to make them generic / applicable to all readers i.e.

rm -rf ~/.neopython/Chains/privnet

file placing for PR As the BlockChainFixtureTestCase resides under the directory neo/Utils I think the right place to put the contents of UnitTest-SM.zip could be something like neo/Utils/fixtures-testnet/

The fixture files should have been uploaded to CoZ AWS by now so you should be good to PR and have travis pass (if not then i'll restart the build once they're up, no worries). I'll have to discuss how we want to port/backup the docker image but we can at least proceed for now.

[localhuman]

S3 Files: https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/fixtures_v8.tar.gz https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/notif_fixtures_v8.tar.gz https://s3.us-east-2.amazonaws.com/cityofzion/fixtures/UnitTest_SM_v8.zip

Thanks for all the hard work on this, I can't imagine yet how much more enjoyable testing will be 🥇

[jseagrave21]

@dauTT let me know if you need any help with anything else before submitting the PR. Or if you would like me to review anything. I am glad I could help and would be happy to in any way I can. Great work!

[dauTT]

Thanks @jseagrave21 for the help. I am a little bit confuse about submitting the PR.

@ixje / @jseagrave21, Shall I submit a PR to the Development branch or to the Master branch? (If it is the Development branch, we need perhaps to update the test cases again before submitting a PR)

If I run, one by one, the test modules which we have migrated to the privnet, they are all passing using the latest version of neo-python v0.7.8. However when I run this command:

python -m unittest discover neo

Some of them are failing and there are too many failures and errors.

Ran 353 tests in 86.778s
FAILED (failures=17, errors=11)

It seems there are still come conflicts which need to resolve. @jseagrave21, if you have time you can also look at this and let me know what is your opinion. Thanks!

[jseagrave21]

@dauTT We are supposed to submit the PR to the Development Branch.

[jseagrave21]

Can you link to or share the report so we can see what the errors are? Maybe a Google Doc?

[jseagrave21]

I am building a fresh environment so I can recreate the failures/errors but I want to avoid double work.

[dauTT]

Hi @jseagrave21 , I have this log file: log.txt,

My first impression is that when we run all the tests together, the fixtures are coming not all from the privnet but also from the testnet. There are some conflicts. Need to investigate further...

[jseagrave21]

@dauTT Here are the results from my tests:

Log1.txt

I noticed in test_json_rpc_api.py that the same issue where getblock returns the wrong block has returned. Maybe you're right about the wrong fixtures being loaded.

[jseagrave21]

@dauTT When I was building my environment, I created a fresh pull of the CoZ Development branch and then modified all the files you mention here by copy and pasting from your neo-python repository "latest...." branch. I did not download anything else because I was assuming that since the files have been uploaded to AWS, then we should be able to pull down from there automatically, during make test

I also didn't do anything with the contents of UnitTest-SM.zip. Please let me know if I need to take further action to create an accurate environment.

[jseagrave21]

Also, since several of the errors involve UserWalletTestCase, should that have been updated?

[dauTT]

@jseagrave21 , It seems that you have added a lot of test cases related to the BlockchainFixtureTestCase class. This include UserWalletTestCase as well. :joy:

Unfortunately, I think we need to postpone the PR and update again the inventory of test cases related to BlockchainFixtureTestCase from the development branch

@ixje, @jseagrave21 There are two options in my opinion to get this task done:

1. Perhaps it will be a good idea to fix a deadline for this task and until then nobody should add any test cases which are related to BlockchainFixtureTestCase class otherwise we may end up to have the same problem in future.
2. We go ahead with the PR and then we will need to fix quickly all the tests which are failing.The risk is that we are not fast enough and the build process will not work for days. This situation will put too much pressure on us.

I honestly prefer option 1. Do you agree?

[jseagrave21]

@dauTT I think option 1 is best. But, even if we go ahead with the PR, they will not approve it until we have a passing build.

[jseagrave21]

That is why @ixje has delayed some of the other PRs. Because some are dependent on the WalletFixtureTestCase.

[jseagrave21]

@dauTT I don't think it will be too hard to fix the errors but I am concerned about the errors we have already fixed that I mentioned earlier (getblock returning 9 when 10 was queried)

[dauTT]

@jseagrave21 , in regards to the conflict of fixtures that I was talking above it is related to the fact that some other test cases (e.g, VerifiableTestCase) which are not related to the BlockchainFixtureTestCase class are using the location

os.path.join(settings.DATA_DIR_PATH, 'fixtures/test_chain')

For some strange reason the setup and teardown of some test cases didn't work properly and in the end the fixtures are mixed up. The mixing of fixtures causes errors we have already fixed (e.g, getblock returning 9 when 10 was queried). This can be resolved by tweaking BlockchainFixtureTestCase but it will better to fix the problem at the source.

To my knowledge the following is the updated list of test cases to migrate to the privnet. These test cases depend on BlockchainFixtureTestCase and WalletFixtureTestCase:

0. test_withdraw.py (won't fix)
1. test_LevelDBBlockchain.py (Done, potentially to update)
2. test_leveldb.py (Done, potentially to update)
3. test_user_wallet.py (Done)
4. test_create_user_wallet.py (Done)
5. test_interop_blockchain.py (Done, potentially to update)
6. test_rest_api.py (Done, potentially to update)
7. test_json_invoke_rpc_api.py (Done, potentially to update)
8. test_json_rpc_api.py (Done, potentially to update)
9. test_token_commands.py (Done)
10. test_send_command.py (Done)
11. test_wallet_commands.py (Done)
12. test_claim_command.py
13. test_node_leader.py
14. test_smart_contract.py (Done, potentially to update)
15. test_payable.py
16. test_breakpoints.py
17. test_storage_find.py
18. test_smart_contract3.py (Done, potentially to update)
19. test_smart_contract2.py (Done, potentially to update)
20. test_gas_costs.py
21. test_migrate_destroy.py

So there are still some work to do. @jseagrave21 , shall I assign half of the remaining test cases to me and half to you?

[jseagrave21]

@dauTT okay that sounds good!

[jseagrave21]

Is there anything that we need help with? Like the tweaking of BlockchainFixtureTestCase or fixing it at the source? So we can have three people (including @ixje) working on it?

[dauTT]

Let us involve @ixje for difficult task as we will need his help of course. :smile: I have assigned the first items to me an the remaining to you. Thanks!

[ixje]

I've asked those with merging rights to not merge any PRs that rely on or inherit from BlockchainFixtureTestCase until we get this one PR'ed. What else can i do for now? @dauTT are you on discord? If not, joining could make communicating a bit faster/easier at this stage

[jseagrave21]

@dauTT okay, I will start at the bottom and work my way up

[jseagrave21]

@dauTT test_migrate_destroy.py is good. test_gas_costs.py is good test_smart_contract2.py is good test_smart_contract3.py is good test_storage_find.py is good test_breakpoints.py is good test_payable.py is good test_smart_contract.py is good test_node_leader.py is good

[jseagrave21]

@dauTT So, test_claim_command.py completely fails and I think I know why. The WalletFixtureTestCase references wallets with addresses which do not exist in your fixture. You need to add the wallets you have been using to the WalletFixtureTestCase.

I think the easiest way to do this would be to open np-prompt and create a wallet labeled testwallet.db3 with the password testpassword and then import the wif for the address AK2nJJpJr6o664CWJKi1QRXjqeic2zRp8y and delete the other (original) address. Then you could replace the testwallet.db3 which already resided in fixtures/. Then create another wallet labeled testwallet2.db3 with the password testwallet and import the wif for the address AXpNr3SDfLXbPHNdqxYeHK5cYpKMHZxMZ9 and delete the other (orginal) address. Then you could replace the testwallet2.db3 in fixtures/. I am not sure if you have a third address you were using. If not, you should make a new wallet to replace testwallet3.db3 and send some assets to it. If you do this, then we shouldn't have to completely change all code for the class UserWalletTestCase(BlockchainFixtureTestCase) But we will have to change a lot.

@ixje Could you double check what I am recommending to see if it makes sense? Essentially we are rebuilding the WalletFixtureTestCase.

[ixje]

Yes I think you need to 'rebuild' the wallets in WalletFixtureTestCase. These wallets will rely on blockchain data coming from fixtures_v7.tar.gz which is unlikely to match the new v8 blockchain data.

ps: with 'rebuild' i don't mean the command, but making sure the wallet addresses in the wallet actually exists in the blockchain data of the v8 fixtures. Including those addresses having funds, transactions and whatever is needed for the tests.

[dauTT]

Thanks @jseagrave21 , @ixje. I will do that.

[jseagrave21]

@dauTT sounds good. I would love to help with this part but I don't have the WIFs for any of the addresses used and I am not sure of all the addresses which are active.

As soon as the WalletFixtureTestCase is migrated, please post a readme with the addresses and their associated wallets and passwords and hashes (Look at test_claim_command.py to see what kind of information is referenced when a UserWalletTestCase is set up). The first wallet should have a watch_address set so the second wallet addresses. We might need to update these fixtures to meet the demands of particular tests but that will at least get us rolling.

Afterwards, I will start over to make sure the UserWalletTestCase in each test references the correct info and also passes.

[dauTT]

Hi @jseagrave21 ,

here are the wallets that we are using for the privnet: neo-wallet.zip

1) neo-privnet.wallet. (pwd: coz) 2) neo-test.wallet (pwd: 1234567890) 3) neo-test2.wallet (pwd: 1234567890)

You can proceed with your work, and later on hopefully we can merge everything together.

[jseagrave21]

Okay, I added the three wallets to the "fixtures" folder and I have updated WalletFixtureTestCase and the following tests:

• test_claim_command.py
• test_migrate_destroy.py
• test_gas_costs.py
• test_breakpoints.py
• test_wallet_commands.py

@dauTT I am going to try to do a PR merge into your "latest..." branch because I have to stop now and I want to be on the same page.

At this point all tests from the bottom (21) through test_wallet_commands.py (11) are migrated.

Here is the PR to your base @dauTT : https://github.com/dauTT/neo-python/pull/1 cc @ixje

[dauTT]

As promised, I have created a document which explain why it is better to change our testing setup. Let me know what you think and let us decide what it is the next step. If we go with the current setup we will be finish with the migration in a matter of few days. With the proposed new setup perhaps one week.

testing setup proposal .docx

[jseagrave21]

@dauTT I like what you proposed and I think I see what you are getting at. However, I think we can minimize the number of new wallets needed. Here is my counter-proposal.

testing.setup.proposal2.docx

[dauTT]

@ixje / @jseagrave21 , you can find the most updated files here: link

I have modifed the following files:

        modified:   fixtures/testwallet.db3 (autogenerated, no need for review)
    modified:   neo/Implementations/Blockchains/LevelDB/test_LevelDBBlockchain.py (okay)
    modified:   neo/Implementations/Blockchains/LevelDB/tests/test_leveldb.py (okay)
    modified:   neo/Implementations/Wallets/peewee/test_create_user_wallet.py 
    modified:   neo/Implementations/Wallets/peewee/test_user_wallet.py
    modified:   neo/Network/test_node_leader.py 
    modified:   neo/Prompt/Commands/tests/test_claim_command.py (not okay, to investigate why in the fixture we have negative unavailable_bonus! look at test_1_no_available_claim and test_2_wallet_with_claimable_gas)
    modified:   neo/Prompt/Commands/tests/test_send_command.py
    modified:   neo/Prompt/Commands/tests/test_token_commands.py
    modified:   neo/Prompt/Commands/tests/test_wallet_commands.py
    modified:   neo/Settings.py (okay)
    modified:   neo/SmartContract/tests/StorageTest.avm (autogenerated, no need for review)
    modified:   neo/SmartContract/tests/test_breakpoints.py
    modified:   neo/SmartContract/tests/test_gas_costs.py
    modified:   neo/SmartContract/tests/test_migrate_destroy.py
    modified:   neo/SmartContract/tests/test_smart_contract.py  (okay)
    modified:   neo/SmartContract/tests/test_smart_contract2.py  (okay)
    modified:   neo/SmartContract/tests/test_smart_contract3.py  (okay)
    modified:   neo/Utils/BlockchainFixtureTestCase.py (okay)
    modified:   neo/Utils/VerifiableTestCase.py
    modified:   neo/Utils/WalletFixtureTestCase.py
    modified:   neo/VM/tests/test_interop_blockchain.py (okay)
    modified:   neo/api/JSONRPC/test_json_invoke_rpc_api.py (okay)
    modified:   neo/api/JSONRPC/test_json_rpc_api.py (@ixje, @jsagrave to enrich the test cases as new ones were added in the meantime: test_getbalance_invalid_params, test_validate_address, test_getbalance_invalid_params, test_valid_multirequest, test_multirequest_with_1_invalid_request)
    modified:   neo/api/REST/RestApi.py (okay)
    modified:   neo/api/REST/test_rest_api.py (okay)
    deleted:    neo/contrib/nex/test_withdraw.py (okay, @ixje confirmed)

and added the following wallets:

fixtures/neo-test-coz.wallet (pwd=coz)
fixtures/neo.test1-bc.wallet (pwd=1234567890)
fixtures/neo-test2-bc.wallet (pwd=1234567890)
fixtures/neo-test1-w.wallet (pwd=1234567890)
fixtures/neo-test2-w.wallet (pwd=1234567890)
fixtures/neo-test3-w.wallet (pwd=1234567890)

Latest image created:

docker pull dautt/neo-privnet-unittest:v0.0.22

Fixtures to upload to AWS:

fixtures_v8.tar.gz notif_fixtures_v8.tar.gz

(both fixture files are in total about 7MB)

All the test cases have been tested using neo-python v0.7.9-dev from the development branch. They all pass if we execute one test at the time by running these commands:

python -m unittest neo/Implementations/Blockchains/LevelDB/test_LevelDBBlockchain.py
python -m unittest neo/Implementations/Blockchains/LevelDB/tests/test_leveldb.py
python -m unittest neo/Implementations/Wallets/peewee/test_user_wallet.py
python -m unittest neo/Implementations/Wallets/peewee/test_create_user_wallet.py
python -m unittest neo/VM/tests/test_interop_blockchain.py
python -m unittest neo/api/REST/test_rest_api.py
python -m unittest neo/api/JSONRPC/test_json_invoke_rpc_api.py
python -m unittest neo/api/JSONRPC/test_json_rpc_api.py
python -m unittest neo/Prompt/Commands/tests/test_send_command.py
python -m unittest neo/Prompt/Commands/tests/test_wallet_commands.py
python -m unittest neo/Prompt/Commands/tests/test_claim_command.py
python -m unittest neo/Network/test_node_leader.py
python -m unittest neo/SmartContract/tests/test_smart_contract.py
python -m unittest neo/SmartContract/tests/test_payable.py
python -m unittest neo/SmartContract/tests/test_breakpoints.py
python -m unittest neo/SmartContract/tests/test_storage_find.py
python -m unittest neo/SmartContract/tests/test_smart_contract3.py
python -m unittest neo/SmartContract/tests/test_smart_contract2.py
python -m unittest neo/SmartContract/tests/test_gas_costs.py
python -m unittest neo/SmartContract/tests/test_migrate_destroy.py

Hovever if we run this command:

python -m unittest discover neo

we get the following unexpected result:

======================================================================
ERROR: test_initial_setup (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 62, in test_initial_setup
    self.assertTrue(GetBlockchain().GetBlock(0).Hash.To0xString(), '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099')
AttributeError: 'NoneType' object has no attribute 'Hash'

======================================================================
FAIL: test_add_header (Implementations.Blockchains.LevelDB.tests.test_initial_db.LevelDBTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/Implementations/Blockchains/LevelDB/tests/test_initial_db.py", line 54, in test_add_header
    self.assertEqual(self._blockchain.CurrentHeaderHash, b'd42561e3d30e15be6400b6df2f328e02d2bf6354c41dce433bc57687c82144bf')
AssertionError: b'b3181718ef6167105b70920e4a8fbbd0a0a56aacf460d70e10ba6fa1668f1fef' != b'd42561e3d30e15be6400b6df2f328e02d2bf6354c41dce433bc57687c82144bf'

======================================================================
FAIL: test_block_non_verbose (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 239, in test_block_non_verbose
    self.assertEqual(block.Index, 2003)
AssertionError: 2002 != 2003

======================================================================
FAIL: test_get_block_int (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 182, in test_get_block_int
    self.assertEqual(GetBlockchain().GetBlock(1).ToJson()['hash'], '0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273')
AssertionError: '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099' != '0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273'
- 0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099
+ 0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273

======================================================================
FAIL: test_get_block_sysfee (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 214, in test_get_block_sysfee
    self.assertEqual(res['result'], 1560)
AssertionError: 1070 != 1560

======================================================================
FAIL: test_getblockhash (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 104, in test_getblockhash
    self.assertEqual(expected_blockhash, res["result"])
AssertionError: '0x049db9f55ac45201c128d1a40d0ef9d4bdc58db97d47d985ce8d66511a1ef9eb' != '0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273'
- 0x049db9f55ac45201c128d1a40d0ef9d4bdc58db97d47d985ce8d66511a1ef9eb
+ 0x55f745c9098d5d5bdaff9f8f32aad29c904c83d9832b48c16e677d30c7da4273

======================================================================
FAIL: test_getrawtx_1 (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 322, in test_getrawtx_1
    self.assertEqual(res['blockhash'], '0x6088bf9d3b55c67184f60b00d2e380228f713b4028b24c1719796dcd2006e417')
AssertionError: '0x0e9c0a7e88aeaf217654107041986f4491b447590cfc3d060304cef1b8536ea0' != '0x6088bf9d3b55c67184f60b00d2e380228f713b4028b24c1719796dcd2006e417'
- 0x0e9c0a7e88aeaf217654107041986f4491b447590cfc3d060304cef1b8536ea0
+ 0x6088bf9d3b55c67184f60b00d2e380228f713b4028b24c1719796dcd2006e417

======================================================================
FAIL: test_prompt_open_wallet (bin.test_prompt.PromptTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/bin/test_prompt.py", line 26, in test_prompt_open_wallet
    self.assertIn(checktext, text)
AssertionError: 'Opened' not found in "open wallet fixtures/testwallet.db3\r\ntestpassword\r\n\x1b[32m[I 180915 21:17:39 LevelDBBlockchain:114]\x1b[39m Created Blockchain DB at /home/dau/.neopython/Chains/SC234 \r\n\x1b[32m[I 180915 21:17:46 NotificationDB:73]\x1b[39m Created Notification DB At /home/dau/.neopython/Chains/Test_Notif \r\n\x1b[0m\x1b[?7h\x1b[0;38;5;4mNEO\x1b[0;38;5;10m cli. Type \x1b[0;38;5;197m'help' \x1b[0;38;5;10mto get started\x1b[0m\r\r\n\x1b[0m\r\n\r\n\x1b[?1l\x1b[6n\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[Jneo>  \x1b[0m\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n                                                                               \r\x1b[79C \r\x1b[7A\x1b[5C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[5Dneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[40C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[40D\x1b[0m\x1b[Jneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l\x1b[?1l\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[24C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[24D\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l"

----------------------------------------------------------------------
Ran 347 tests in 68.850s

FAILED (failures=7, errors=1)

I believe the major reason for the above failures, is related to this test case that I have added:

ERROR: test_initial_setup (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/api/JSONRPC/test_json_rpc_api.py", line 62, in test_initial_setup
    self.assertTrue(GetBlockchain().GetBlock(0).Hash.To0xString(), '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099')
AttributeError: 'NoneType' object has no attribute 'Hash'

I tried to fix it but unsuccessfully. So I need help here. For some unknown reason the Genesis block is replaced by None when we run all the tests together, and therefore the final Blockchain is corrupted. (Strangely this behavior does not happen when we run the test one by one). If we can fix this one, most of the cases will be fix by themselves.

So how can we divide the remaining work?

I thought as follow, but please feel free to do as you wish. :smile:
1) @ixje can perhaps fix the issue related to this command python -m unittest discover neo 2) @ixje can review the test cases which depend only WalletFixtureTestCase. The one which depend on BlockChainFixtureTestCase have been already reviewed by @ixje and threfore there is no need to review them again. I have labeled them as "okay". 3) @jseagrave21 can update the test case depending only BlockChainFixtureTestCase. I believe there is only this one test_json_rpc_api.py but I maybe wrong. 4) @jseagrave21 can look at the test case test_claim_command.py and see why we have negative unavailable_bonus.

[jseagrave21]

@dauTT Here is the migrated test_json_rpc_api.py including the new tests for sendtoaddress since @localhuman just merged that PR.

https://github.com/jseagrave21/neo-python/blob/DauTT-branch/neo/api/JSONRPC/test_json_rpc_api.py

Additionally, please note that the my "DauTT-branch" is completely up to date with your "latest2" branch, and it is also up-to-date with the latest CoZ Development branch. So, you should be able to merge my branch into your branch if you'd like.

[jseagrave21]

For the other failures in test_json_rpc_api.py when running make test there must be something wrong with the way that discover neo is being implemented. For example, here is one of the failures:

FAIL: test_block_non_verbose (api.JSONRPC.test_json_rpc_api.JsonRpcApiTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/c/users/jseag/neo/neo-python-travis/neo/api/JSONRPC/test_json_rpc_api.py", line 241, in test_block_non_verbose
    self.assertEqual(block.Index, 2003)
AssertionError: 2002 != 2003

Yet, the block being queried in test_json_rpc_api.py is 2003:

def test_block_non_verbose(self):
        req = self._gen_rpc_req("getblock", params=[2003, 0])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertIsNotNone(res['result'])

        # we should be able to instantiate a matching block with the result
        output = binascii.unhexlify(res['result'])
        block = Helper.AsSerializableWithType(output, 'neo.Core.Block.Block')
        self.assertEqual(block.Index, 2003)
        self.assertEqual(len(block.Transactions), 1)

So, the discover neo unittest is off by one block. cc @ixje

[jseagrave21]

@dauTT When I ran test_claim_command.py individually, I do not receive any errors, so I think the error you are referring to must be related to the problem we are having with discover neo.

[dauTT]

@jseagrave21 ,

thanks for fixing test_json_rpc_api.py. I will update my branch accordingly.

In regards to test_claim_command.py even though everything pass, there are two tests which I am not happy with:

test_1_no_available_claim
test_2_wallet_with_claimable_gas

Look at those test cases, I have added some hints towards fixing the issue. Basically when we stop the image and create the fixture, we lose somehow a bit of information in the fixture and we end up to have negative unavailable_bonus. You can see this by debugging GetAvailableClaimTotal in the fixture and in the image. The deviation happens exactly here when coinheight.end = 12350 in this assignment :

endamount = Blockchain.Default().GetSysFeeAmountByHeight(coinheight.end - 1)

For instance looking at the test case, test_1_no_available_claim, If you open the wallet neo-test3-w.wallet and run the the command wallet when the blockchain is fully sync at 12350, you will see "unavailable": "0.0002685".

Opened wallet at /home/dau/workspace/python/github.com/neo/dautt/neo-python/fixtures/neo-test3-w.wallet
neo> wallet                                                                                                                                              
[I 180916 10:14:39 UserWallet:538] Script hash b'\xc4\xc1\xb0\xcf\xa8\x7f\xcb\xacE\x98W0\x16d\x11\x03]\xdf\xed#' <class 'bytes'>
Wallet {
    "path": "/home/dau/workspace/python/github.com/neo/dautt/neo-python/fixtures/neo-test3-w.wallet",
    "addresses": [
        {
            "version": 0,
            "script_hash": "AZiE7xfyJALW7KmADWtCJXGGcnduYhGiCX",
            "frozen": false,
            "votes": [],
            "balances": {
                "0x602c79718b16e442de58778e148d0b1084e3b2dffd5de6b7b16cee7969282de7": "3.0",
                "0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b": "25.0"
            },
            "is_watch_only": false
        }
    ],
    "height": 12350,
    "percent_synced": 100,
    "synced_balances": [
        "[NEO]: 25.0 ",
        "[NEOGas]: 3.0 "
    ],
    "public_keys": [
        {
            "Address": "AZiE7xfyJALW7KmADWtCJXGGcnduYhGiCX",
            "Public Key": "03989f7417da540a8ce00195738249291cba058102a12d2df1b00e2a826d8bd061"
        }
    ],
    "tokens": [],
    "claims": {
        "available": "0.0",
        "unavailable": "0.0002685"
    }
} 

However in the fixture we find "unavailable": "-0.0003665". This is really a strange behavior.. :smile:

[jseagrave21]

@dauTT, I am not sure about how to address this issue though perhaps it is related to #538 ?

[jseagrave21]

@dauTT I reran make test several times. I changed test_intial_setup to this:

def test_initial_setup(self):
        expected_blockhash = '0x996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099'
        req = self._gen_rpc_req("getblockhash", params=[0])
        mock_req = mock_request(json.dumps(req).encode("utf-8"))
        res = json.loads(self.app.home(mock_req))
        self.assertEqual(expected_blockhash, res["result"])

Instead of the expected result, I consistently get 0xb3181718ef6167105b70920e4a8fbbd0a0a56aacf460d70e10ba6fa1668f1fef which I cannot find when I query with block in np-prompt. So, it looks like block 0 is being replaced by a different block that doesn't exist in the fixture at some point during the discover neo test.

[ixje]

@dauTT 1) I solved the unittest discovery difference. You'll have to merge this PR: https://github.com/CityOfZion/neo-python/pull/616

2) Having this call is good https://github.com/dauTT/neo-python/blob/latest2-move-unittest-to-privnet-478/neo/Settings.py#L232-L234 But we do not want to copy the settings from the privnet protocol file. We should create a new one and check that in (e.g. protocol.unittest-net.json). It can be a copy of the privnet one (content wise), but they cannot be the same file. Reason being is that people should be free to change their privatenet protocol to whatever matches their needs, however they might touch e.g. the validators or magic and that will mess up the test cases.

3) this test will still fail https://github.com/CityOfZion/neo-python/blob/951de3a902a398ce6b178481e837ce96014affbd/neo/Implementations/Blockchains/LevelDB/tests/test_initial_db.py#L45-L56

You'll have to update the setUpClass()to add:

def setUpClass(self):
        settings.setup_fixturenet()
        Blockchain.DeregisterBlockchain()

and update the test to the following:

    def test_add_header(self):
        hexdata = binascii.unhexlify(self.block_one_raw)
        block_one = Helper.AsSerializableWithType(hexdata, 'neo.Core.Block.Block')

        if settings.MAGIC == 56753:
            self.assertEqual(self._blockchain.CurrentHeaderHash, b'996e37358dc369912041f966f8c5d8d3a8255ba5dcbd3447f8a82b55db869099')
        else:
            self.assertEqual(self._blockchain.CurrentHeaderHash, b'd42561e3d30e15be6400b6df2f328e02d2bf6354c41dce433bc57687c82144bf')

        self.assertEqual(self._blockchain.HeaderHeight, 0)

        self._blockchain.AddBlock(block_one)
        self.assertEqual(self._blockchain.HeaderHeight, 1)

that will fix the errors and actually make the test do what the testname suggests 😂

[dauTT]

Thanks @ixje / @jseagrave21 , great team work!!! :smile:

I have created a clean branch with all the PR (611, 616) merged together along with the update file test_json_rpc_api from @jseagrave21 branch. Here is the link

Fixtures to upload to AWS:

notif_fixtures_v8.tar.gz fixtures_v8.tar.gz

Tested the code by running:

make lint
make test

Everything pass except this one:

======================================================================
FAIL: test_prompt_open_wallet (bin.test_prompt.PromptTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dau/workspace/python/github.com/neo/dautt/neo-python/neo/bin/test_prompt.py", line 26, in test_prompt_open_wallet
    self.assertIn(checktext, text)
AssertionError: 'Opened' not found in "open wallet fixtures/testwallet.db3\r\ntestpassword\r\n\x1b[32m[I 180919 00:31:55 LevelDBBlockchain:113]\x1b[39m Created Blockchain DB at /home/dau/.neopython/Chains/SC234 \r\n\x1b[32m[I 180919 00:31:58 NotificationDB:73]\x1b[39m Created Notification DB At /home/dau/.neopython/Chains/Test_Notif \r\n\x1b[0m\x1b[?7h\x1b[0;38;5;4mNEO\x1b[0;38;5;10m cli. Type \x1b[0;38;5;197m'help' \x1b[0;38;5;10mto get started\x1b[0m\r\r\n\x1b[0m\r\n\r\n\x1b[?1l\x1b[6n\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[Jneo>  \x1b[0m\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n\r\r\n                                                                               \r\x1b[79C \r\x1b[7A\x1b[5C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[5Dneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[40C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[40D\x1b[0m\x1b[Jneo> open wallet fixtures/testwallet.db3                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l\x1b[?1l\x1b[?2004h\x1b[?25l\x1b[0m\x1b[?7l\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[24C\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?25l\x1b[?7l\x1b[24D\x1b[0m\x1b[J[password]> ************                                                       \r\x1b[79C \r\x1b[0m\r\r\n\x1b[J\x1b[?7h\x1b[0m\x1b[?12l\x1b[?25h\x1b[?2004l"

----------------------------------------------------------------------
Ran 352 tests in 71.690s

FAILED (failures=1)

This was also failing before we did any changes. It has nothing to do with BlockChainFixtureTestCase nor WalletFixtureTestCase. I am not sure if this is just happening on my local machine or not.

@jseagrave21 , I believe you know very well this test. Can you please have a look at it? Thanks. :smile:

[jseagrave21]

@dauTT okay let me take a look

[jseagrave21]

@dauTT since we didn't change the test and testwallet.db3 is still accessible with testpassword, I suspect it is just the time out. I added 5 sec to the test here. Feel free to add time if you need to. https://github.com/jseagrave21/neo-python/blob/DauTT-branch/neo/bin/test_prompt.py

[dauTT]

Yes, your right, increasing the time out help. So all the test cases are fine. :+1: