only one txn works

[jkrauska]

Setup:

Start one daemon as a 'seed' node on one host

cli daemon -propose false

Start second daemon as a 'joiner' node on another host pointing to see node as initial peer.

cli daemon -propose true -peer seed.o1test.net:8303

They seem to connect to each other.

Trigger:

1. Check balances on two accounts -- looks ok.
2. Perform transaction from one account to another.
3. Check balances again, confirm new balances after txn. (looks fine)
4. Perform second transaction.
5. Check balances third time, and still see state from step 3.

Attaching some logs that had extra info captures from @cmr (thx).

trigger notes:

$ trade.py
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     10000
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      100
/usr/local/bin/cli client send-txn -amount 33 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool
Sleeping 10
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     9967
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      133
/usr/local/bin/cli client send-txn -amount 37 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool
Sleeping 10
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     9967
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      133
/usr/local/bin/cli client send-txn -amount 25 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool

[jkrauska]

join.log.gz seed.log.gz

[jkrauska]

Sorry for the wide blast here, but it's a pretty big blocker to doing any 'real' scaling tests.

[es92]

I made an integration test for this yesterday, going to be working on figuring it out today #542

[deepthiskumar]

I am working on a PR that fixes this (I think). I encountered this issue when running full-test with snarks on. There was a lib snark error that Izzy fixed and then there was an rpc error that I haven't figured out the root cause of, but made a workaround-fix. I am also working on another PR that is required for this issue to be fixed (having non-zero fee_excess at the top). I'll include more details in the PR.

[jkrauska]

tried deb from #542 and it works

admin@ip-172-31-18-66:~$ trade.py
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     10000
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      100
/usr/local/bin/cli client send-txn -amount 48 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool
Sleeping 10
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     9952
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      148
/usr/local/bin/cli client send-txn -amount 24 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool
Sleeping 10
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     9928
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      172
/usr/local/bin/cli client send-txn -amount 16 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool
Sleeping 10
--------------------------------------------------------------------------------
/usr/local/bin/cli client get-balance -address JgN/txgzOFPrOy8khSJSdPyFHdT+t05dgljjcaRvjXSC6ZFfcfkXAA
high     9912
/usr/local/bin/cli client get-balance -address JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
low      188
/usr/local/bin/cli client send-txn -amount 31 -from JgGg7PfX+SuEc8siVxn+OVeXBXcN3p0Wed0plH+tRn4+RduAlfUY -receiver JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA
Successfully enqueued txn in pool
Sleeping 10

also added run-start-worked PUBLICKEY to the join script.

cli daemon -propose true -peer seed.o1test.net:8303 -run-snark-worker JgMuF0Dkx1tw6eAX2om4kJ+4Ch4PsBmtaaeX9o0y26fsnUBLrUlPAA

thx.