Broadcast are stuck whilst running performance testing against 1.3.0-rc1

[EnriqueL8]

As part of https://github.com/hyperledger/firefly/issues/1470, I've noticed that the broadcast where not getting confirmed or just stuck.

I ran a first test and this first screenshot already shows something odd from 577e8c47680c6230209a74829921a9c427766af8:

The pending graph is negative, so we have more confirmed than sent odd.

I then pulled down 1.3.0-rc1 and run another test to see

It just gets stuck and again more confirmed than submitted.. I opened up the UI and all the broadcast messages were stuck in Pending but the private message were going through.

Digged into the logs and found the following:

[2024-03-27T15:05:42.508Z] DEBUG Pin 0000000122 batch unavailable: batch=2b81fa92-4bbe-4129-9b2f-0eaa6f0d971f pinIndex=0 hash=37a8eec1ce19687d132fe29051dca629d164e2c4958ba141d5f4133a33f0688f masked=false dbtx=BSehIPrl ns=default pid=1 role=aggregator

and

[2024-03-27T15:05:42.698Z] DEBUG Message b350b174-2d02-447d-9d65-0da0ce655860 pinned at sequence 173 blocked by earlier context 37a8eec1ce19687d132fe29051dca629d164e2c4958ba141d5f4133a33f0688f at sequence 166 dbtx=ZzYFZpfK ns=default pid=1 role=aggregator

Clearly there is a Batch that is stuck somewhere, I then traversed more logs and found:

[2024-03-27T15:05:03.388Z] DEBUG Pin 0000000001 batch unavailable: batch=08457241-5137-4ec6-bdd9-44a84b825810 pinIndex=0 hash=aeaccb6cb5744020c5d9988e99d55a3df1c3e25f647a8a31ab3feeaa56edc3d7 masked=false dbtx=HzZm9RND ns=default pid=1 role=aggregator

Unfortunately I cleared the above run, and ran the same commit of 1.3.0-rc1 and got the same problem, but found some news logs

[2024-03-27T17:32:54.440Z]  INFO -> BatchPinComplete batch=14c83d3e-3ba0-4e07-9d8c-88b0181e2e9f txn=000000000521/000000/000000 signingIdentity=0x11d2e10b90312f378b5925b2241fa06e68338706 dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.441Z] DEBUG SQL<- query transactions (0.45ms) dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.441Z] DEBUG Transaction '523d0362-0852-4d30-8073-b4d24b5d406e' not found dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.442Z] DEBUG SQL<- insert transactions sequences=[9] (0.43ms) dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.442Z]  INFO <- BatchPinComplete batch=14c83d3e-3ba0-4e07-9d8c-88b0181e2e9f txn=000000000521/000000/000000 signingIdentity=0x11d2e10b90312f378b5925b2241fa06e68338706 dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.446Z] DEBUG SQL<- insert blockchainevents sequences=[49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69] (2.54ms) dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.450Z] DEBUG SQL<- insert pins sequences=[264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340] (2.33ms) dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.451Z] DEBUG SQL<- query batches (0.55ms) dbtx=5W3OnAw0 ns=default pid=1 role=event-manager
[2024-03-27T17:32:54.451Z] DEBUG Batch '14c83d3e-3ba0-4e07-9d8c-88b0181e2e9f' not found dbtx=5W3OnAw0 ns=default pid=1 role=event-manager

Two things that I then investigated:

• Looked in the Postgres DB and the batch was there, so that not found could have been at creation? not sure...
• I seem to notice that the Batch Pin event has less contexts than the amount of Pins associated to that BatchPin. I think this is where I would explore next!

[EnriqueL8]

Fixed by creating a token pool correctly in the Firefly Performance CLI https://github.com/hyperledger/firefly-perf-cli/pull/79

[EnriqueL8]

Actually fixed by https://github.com/hyperledger/firefly-transaction-manager/pull/116