Payment for cycle 527 was off, overpaid delegates

[ericlavoie]

In cycle 527 I had the same bug everyone had with KT1 address, I updated TRD and created a failed filed file with the missing payment, following Nicolas script

echo "address,type,amount,hash,paid,description" > failed/527.csv
cat done/527.csv | grep undefined | sed s/undefined,.*$/fail,Error/ >> failed/527.csv

The payment identified correctly 23 missing delegate with KT1 for which payment failed, and it was processed and paid. After this was done I checked the balance paid overall versus what the baker earned, and we overpaid, the calculation was off.

I am using latest TRD, mainnet, command line to run the baker is: .../tezos-reward-distributor/src/main.py -E http://x.x.x.x:yyyy -N MAINNET -R -5 -P tzkt -b /home/xxxxxxx/Hayek -O 10

I attached the calculation file for cycle 527 527.csv

[baketzforme]

I can confirm. The calculations.csv file is showing an expected payout to KT1 accounts that is half of the value that is being reported as paid in the done.csv file.

[baketzforme]

Andrew Best in the Baker Slack asked about the KT1 payout bug (#619) and pasted the following log for cycles 527 and 528:

2022-09-23 20:38:18,499 - consumer0 - DEBUG - Payment done for address KT19bjUk54JimXKiEM569CgSXR3C3mjGjGPE type D amount 155,894 mutez paid UNDEFINED
2022-09-23 20:38:18,499 - consumer0 - DEBUG - Payment done for address KT1Xoo3KC18TjW5dSLeQwVkk9pkCL26hZfva type D amount 9,947 mutez paid UNDEFINED
2022-09-23 20:42:42,831 - consumer0 - DEBUG - Payment done for address KT1Xoo3KC18TjW5dSLeQwVkk9pkCL26hZfva type D amount 9,947 mutez paid UNDEFINED
2022-09-23 20:42:42,831 - consumer0 - DEBUG - Payment done for address KT19bjUk54JimXKiEM569CgSXR3C3mjGjGPE type D amount 155,894 mutez paid UNDEFINED
2022-09-26 18:32:24,200 - consumer0 - DEBUG - Payment done for address KT19bjUk54JimXKiEM569CgSXR3C3mjGjGPE type D amount 78,837 mutez paid UNDEFINED
2022-09-26 18:32:24,200 - consumer0 - DEBUG - Payment done for address KT1Xoo3KC18TjW5dSLeQwVkk9pkCL26hZfva type D amount 4,681 mutez paid UNDEFINED

This shows a couple of duplicate entries for cycle 527. If the above was written to the done.csv file as shown then there would be duplicate payouts that got copied over to the failed.csv when entering the commands given by Nicolas.

Thankfully the uncaught payout failures (#621) that led to this may be fixed soon, but in my opinion TRD should never write payout details to a file multiple times. Will #623 be sufficient to prevent this from happening again or does TRD need some additional checks or precautions implemented?

[Vlad1mir-D]

Personally I don't think this is an issue because TRD itself isn't writing succeeded payments into "failed" report - that's the reason why you got you successful payment duplicated.

[baketzforme]

It seems my theory above was not accurate. Lionel in the baking Slack says each payout was only found once in the done.csv, and the failed.csv he created only listed each payout once, yet TRD still paid out twice the amount.

So there's still the mystery of why TRD paid twice the amount shown in the failed.csv file.

[ericlavoie]

The issue is resolved with subsequent fixes. I will close this.