Test the effect of alpha value on the confirmation rates

[jakubcech]

Description

We need to test IRI with several different alpha values and spammer and map the effect it has on the confirmation rates. For example, issue a 100+ transactions with each different alpha value and map the results.

The result of this is the mapping of alpha values and how they affect the confirmation rates. The value can be changed with TIPSELECTION_ALPHA in the iota.ini file that you put in IRI's working directory.

We can use https://github.com/iota-tangle-io/dtlg for this.

[romansemko]

Hercules works a little differently, but the tipsel algorithm is 95% same (apart of the txs validation, for example). I was playing with different alpha values and can confirm that the value of 0.01 (instead of 0.001) works slightly better, especially in high TPS environment.

Maybe this value could also be set automatically baed on the tangle speed (together with the depth value as explained in #901)?

[Johnny-Milkshakes]

Has this been tested on the spamnet yet? I think comparing the two nets and how they react differently to the different alpha values might be very insightful. I would also be happy to help organize community spam events for either net.

[jakubcech]

Thanks for the input @romansemko! The dynamic setting of the value is something we've discussed, and this will all be based off of the results yielded by this.

I agree @Johnny-Milkshakes that it could be insightful, the main goal here is to see how this behaves on the Mainnet. Thanks for the offer, I'll reach out if we need some extra spamming for this. 👍

[DyrellC]

Ran a bunch of spam tests on the mainnet yesterday and today. The results for 1.5.2 ranged from 10% with 0.1 to 34% with 0.01 and 52% with 0.001. I wasn't able to get very consistent results while running those tests over though, so I updated to 1.5.3 this morning and ran the battery again.

Results below are from the DTLG on a vm node: 1.5.3 Alpha 0.1: @100 txs: conf rate = 31%, No visible blowballs @200 txs: conf rate = 53%, No visible blowballs

1.5.3 Alpha 0.01: @100 txs: conf rate = 46%, No visible blowballs @200 txs: conf rate = 58%, No visible blowballs

1.5.3 Alpha 0.001: @100 txs: conf rate = 67%, No visible blowballs @200 txs: conf rate = 80%, No visible blowballs

[DyrellC]

The tangle was under an avg 20 tps load at the time of these tests

[jakubcech]

By Dyrell Alpha tests

Alpha tests were run on several nodes running several releases of IRI. Spam was directed using the DTLG tool with an average 0.25 TPS. For the testing of 1.5.2, the TPS load on the network was averaging over 100, so this might have lowered the confirmation rate, although tests conducted over the few days prior did not show promising conf rates either. For the testing of 1.5.3 and Gal's branch, the network was under an average load of 20 TPS.

-On release 1.5.2, confirmation rates did not rise above 10% with alpha 0.1, 34% with alpha 0.01, and 52% with alpha 0.001. -On release 1.5.3, confirmation rates stabilised at 53% with alpha 0.1, 58% with alpha 0.01, and 80% with alpha 0.001. -On Gal's fix branch, confirmation rates stabilised above 80% for alphas 0.01 and 0.001 (92% in one case)

Check consistency tests

Check consistency was called on a recently sent transaction on both a 1.5.3 node and a node using Gal's branch. The tests conducted were:

-Sending a transaction from one node, and checking consistency on the same node (both 1.5.3 and checking consistency) -Sending a transaction from one node running 1.5.3 and checking consistency on a node running Gal's fix -Sending a transaction from one node running Gal's fix and checking consistency on a node running 1.5.3

In all cases the check_consistency would return true.

[TeeVeeEss]

Not sure if still needed, but I had some spare time this afternoon. Last tx (tag XEEVEESPAM99999999999999999) was [(https://thetangle.org/transaction/AJJQ9IKEYF9FOCSRNRFFRBIRRFYPJEHMMTEWUULUEEJMDLWDYVKEUWLYEEDPIVYDCMUFHE9BRLXKZ9999)] Results: ` TIPSELECTION_ALPHA = 0.1: 2018/08/12 16:30:26 Duration: 8m7.6313522s Count: 203 Milestone Trunk: 0 Milestone Branch: 0 Bad Trunk: 0 Bad Branch: 0 Both: 0 Fetched: 250 Cached: 160 {"txs_succeeded":201,"txs_failed":0,"bad_branch":0,"bad_trunk":0,"bad_trunk_and_branch":0,"milestone_trunk":0,"milestone_branch":0,"tps":0.4176552903313878,"error_rate":0,"cached_tx":247,"new_tx":157,"confirmed_tx":66,"confirmation_rate":32.83582089552239}

TIPSELECTION_ALPHA = 0.01: 2018/08/12 16:50:21 Duration: 7m24.8316426s Count: 207 Milestone Trunk: 0 Milestone Branch: 1 Bad Trunk: 0 Bad Branch: 0 Both: 0 Fetched: 265 Cached: 153 {"txs_succeeded":205,"txs_failed":0,"bad_branch":0,"bad_trunk":0,"bad_trunk_and_branch":0,"milestone_trunk":0,"milestone_branch":1,"tps":0.46879562871465696,"error_rate":0,"cached_tx":262,"new_tx":152,"confirmed_tx":4,"confirmation_rate":1.951219512195122}

TIPSELECTION_ALPHA = 0.001 2018/08/12 17:09:33 Duration: 9m10.7954027s Count: 205 Milestone Trunk: 1 Milestone Branch: 1 Bad Trunk: 0 Bad Branch: 0 Both: 0 Fetched: 249 Cached: 163 {"txs_succeeded":203,"txs_failed":0,"bad_branch":0,"bad_trunk":0,"bad_trunk_and_branch":0,"milestone_trunk":1,"milestone_branch":1,"tps":0.3718834076987729,"error_rate":0,"cached_tx":248,"new_tx":162,"confirmed_tx":9,"confirmation_rate":4.433497536945813}

Default (no TIPSELECTION_ALPHA in iota.ini): 2018/08/12 17:35:15 Duration: 13m47.1327853s Count: 359 Milestone Trunk: 0 Milestone Branch: 1 Bad Trunk: 0 Bad Branch: 0 Both: 0 Fetched: 442 Cached: 280 {"txs_succeeded":357,"txs_failed":0,"bad_branch":0,"bad_trunk":0,"bad_trunk_and_branch":0,"milestone_trunk":0,"milestone_branch":1,"tps":0.4337077830915288,"error_rate":0,"cached_tx":438,"new_tx":278,"confirmed_tx":89,"confirmation_rate":24.92997198879552} Nodeinfo: { "appName": "IRI", "appVersion": "1.5.3", "duration": 0, "jreAvailableProcessors": 4, "jreFreeMemory": 345066896, "jreMaxMemory": 3728736256, "jreTotalMemory": 1190133760, "jreVersion": "1.8.0_181", "latestMilestone": TRIWLGSJURJYAHZGHLSYATHXHRZUGMAKBELHDCUUU9A9PWZT9MEWTOHFJWNCKMFIIAVIHVTXMXWRA9999", "latestMilestoneIndex": 675861, "latestSolidSubtangleMilestone": TRIWLGSJURJYAHZGHLSYATHXHRZUGMAKBELHDCUUU9A9PWZT9MEWTOHFJWNCKMFIIAVIHVTXMXWRA9999", "latestSolidSubtangleMilestoneIndex": 675861, "milestoneStartIndex": 590000, "neighbors": 3, "packetsQueueSize": 0, "time": 1534088898392, "tips": 755, "transactionsToRequest": 0 }`