search

LF-Decentralized-Trust-labs / paladin

Programmable privacy for EVM
https://lf-decentralized-trust-labs.github.io/paladin
Apache License 2.0
18 stars 6 forks source link

Intermittent issue in `TestProduceLatestInFlightStageContextRetrieveGas` #267

Open peterbroadhurst opened 1 month ago

peterbroadhurst commented 1 month ago

I've seen this a couple of times, but I haven't dug into it.

Stack shows TestProduceLatestInFlightStageContextRetrieveGas is the test failing

Note in below output, big clues look like:

        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:661 +0x1c0
[2024-10-03T14:56:12.233-04:00] ERROR Panic error detected for transaction 0xa47781b615fb07aaeca3546d2f401583e181b021:1, when executing: retrieveGasPrice, error: runtime error: invalid memory address or nil pointer dereference role=public_tx_mgr
goroutine 149 [running]:

and also

[2024-10-03T14:56:12.236-04:00] DEBUG Transaction with ID 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 entering status update, current status: pending, target status: suspending role=public_tx_mgr
fatal error: concurrent map writes

[0.187ms] [rows:0] SELECT `public_txns`.`signer_nonce`,`public_txns`.`from`,`public_txns`.`nonce`,`public_txns`.`created`,`public_txns`.`key_handle`,`public_txns`.`to`,`public_txns`.`gas`,`public_txns`.`fixed_gas_pricing`,`public_txns`.`value`,`public_txns`.`data`,`public_txns`.`suspended`,`Completed`.`signer_nonce` AS `Completed__signer_nonce`,`Completed`.`created` AS `Completed__created`,`Completed`.`tx_hash` AS `Completed__tx_hash`,`Completed`.`success` AS `Completed__success`,`Completed`.`revert_data` AS `Completed__revert_data`,`Binding`.`signer_nonce` AS `Binding__signer_nonce`,`Binding`.`transaction` AS `Binding__transaction`,`Binding`.`tx_type` AS `Binding__tx_type` FROM `public_txns` LEFT JOIN `public_completions` `Completed` ON `public_txns`.`signer_nonce` = `Completed`.`signer_nonce` LEFT JOIN `public_txn_bindings` `Binding` ON `public_txns`.`signer_nonce` = `Binding`.`signer_nonce` WHERE from = '4b4b44f24a3859fa6d6e85029c6c9749951f4b01' AND to = 'b5e150140390eed4da8e785eb1fb4566b00e3d97' AND "Completed"."tx_hash" IS NULL AND "Binding"."signer_nonce" IS NULL AND data IS NULL LIMIT 1
[2024-10-03T14:56:12.232-04:00] ERROR TransferGasFromAutoFuelingSource error occurred when getting pending fueling tx for address: 0xb5e150140390eed4da8e785eb1fb4566b00e3d97, error: pop role=public_tx_mgr
[2024-10-03T14:56:12.232-04:00] DEBUG Creating new enterprise transaction handler
[2024-10-03T14:56:12.232-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.232-04:00] DEBUG Enterprise transaction handler created
[2024-10-03T14:56:12.232-04:00] DEBUG Initializing enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.232-04:00] DEBUG Balance manager calcMethod setting: max role=public_tx_mgr
[2024-10-03T14:56:12.232-04:00] DEBUG Initialized enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Balance manager calcMethod setting: max role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Calculate the amount to be topped up for address &{Address:0x136d506ccf8ab1fa5e270f176826a59906fb33a2 Balance:+100 SpentTransactionCount:2 MinCost:+50 MaxCost:+150 Spent:+200} ; autoFueling config: &{ethClient:0x140005674f0 pubTxMgr:0x140005b6dc0 balanceCache:0x14000626040 source:autofueler sourceAddress:0x140004bb380 minSourceBalance:<nil> proactiveFuelingTransactionTotal:1 proactiveFuelingCalcMethod:max minDestBalance:<nil> maxDestBalance:<nil> minThreshold:<nil> destinationAddressesFuelingTracked:map[] trackedFuelingTransactions:map[] destinationAddressesFuelingTrackedMux:{state:0 sema:0} addressBalanceChangedMap:map[] addressBalanceChangedMapMux:{state:0 sema:0}} role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Requesting top up for address 0x136d506ccf8ab1fa5e270f176826a59906fb33a2 using calculated amount: 100 based on spent: 200 role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG TransferGasFromAutoFuelingSource no existing tracking fueling request for  destination address: 0x136d506ccf8ab1fa5e270f176826a59906fb33a2 role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving balance for address 0x0566625614845ae755764a66f947e9771155a228  role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving balance for address 0x0566625614845ae755764a66f947e9771155a228 from connector role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] ERROR Failed retrieving balance for address 0x0566625614845ae755764a66f947e9771155a228 from connector due to: pop role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] ERROR TransferGasFromAutoFuelingSource failed to get balance of source: 0x0566625614845ae755764a66f947e9771155a228 role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from fixed gas price
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from fixed gas price
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from fixed gas price
[2024-10-03T14:56:12.233-04:00]  WARN Cannot get gas price due to PD011917: The gasPrice 'invalid' could not be parsed. Must be a numeric string, or an object with 'gasPrice' field, or 'maxFeePerGas'/'maxPriorityFeePerGas' fields (EIP-1559), error: invalid character 'i' looking for beginning of value
[2024-10-03T14:56:12.233-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from fixed gas price
[2024-10-03T14:56:12.233-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from node eth call
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from node eth call
[2024-10-03T14:56:12.233-04:00] ERROR Failed to retrieve gas price from the node
[2024-10-03T14:56:12.233-04:00] DEBUG Creating new enterprise transaction handler
[2024-10-03T14:56:12.233-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.233-04:00] DEBUG Enterprise transaction handler created
[2024-10-03T14:56:12.233-04:00] DEBUG Initializing enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Balance manager calcMethod setting: max role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Initialized enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG NewOrchestrator for signing address 0xa47781b615fb07aaeca3546d2f401583e181b021 created: &{pubTxManager:0x140005b7760 resubmitInterval:300000000000 stageRetryTimeout:10000000000 persistenceRetryTimeout:5000000000 ethClient:0x140005caf00 bIndexer:0x140005caf50 transactionSubmissionRetry:0x140000402e0 orchestratorBirthTime:{wall:13958863146407812936 ext:30692126 loc:0x102526700} orchestratorPollingInterval:3600000000000 signingAddress:[164 119 129 182 21 251 7 170 236 163 84 109 47 64 21 131 225 129 176 33] hasZeroGasPrice:false unavailableBalanceHandlingStrategy:wait maxInFlightTxs:500 inFlightTxs:[] inFlightTxsMux:{state:0 sema:0} orchestratorLoopDone:<nil> InFlightTxsStale:0x1400004d9d0 stopProcess:0x1400004da40 totalCompleted:0 state:new stateEntryTime:{wall:13958863146407812936 ext:30692251 loc:0x102526700} staleTimeout:300000000000 lastQueueUpdate:{wall:0 ext:0 loc:<nil>}} role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0xa47781b615fb07aaeca3546d2f401583e181b021:1 role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Transaction with ID 0xa47781b615fb07aaeca3546d2f401583e181b021:1 entering retrieve gas price as no gas price available. role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0xa47781b615fb07aaeca3546d2f401583e181b021:1 role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xa47781b615fb07aaeca3546d2f401583e181b021:1, on stage: retrieveGasPrice , current stage context lived: 20.833µs , stage lived: 20.917µs, last stage error: trigger stage error role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] ERROR Failed to trigger sate due to trigger stage error, cleaning up the context and retry role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Transaction with ID 0xa47781b615fb07aaeca3546d2f401583e181b021:1 clearing stage context for stage: retrieveGasPrice after 34.25µs, total time spent on this stage so far: 34.292µs, txHash: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Transaction with ID 0xa47781b615fb07aaeca3546d2f401583e181b021:1 entering retrieve gas price as no gas price available. role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from node eth call role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Creating new enterprise transaction handler
[2024-10-03T14:56:12.233-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.233-04:00] DEBUG Enterprise transaction handler created
[2024-10-03T14:56:12.233-04:00] DEBUG Retrieving gas price from node eth call role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] ERROR Panic error detected for transaction 0xa47781b615fb07aaeca3546d2f401583e181b021:1, when executing: retrieveGasPrice, error: runtime error: invalid memory address or nil pointer dereference role=public_tx_mgr
goroutine 148 [running]:
runtime/debug.Stack()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/debug/stack.go:16 +0x1c
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1.1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:666 +0x184
panic({0x101be84c0?, 0x102489270?})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/panic.go:770 +0x124
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*HybridGasPriceClient).getGasPriceJSON(0x140004b3560, {0x101d763c8, 0x140005cbc20})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/gas_price_client.go:111 +0x290
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*HybridGasPriceClient).GetGasPriceObject(0x140004b3560, {0x101d763c8, 0x140005cbc20})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/gas_price_client.go:86 +0xa4
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).TriggerRetrieveGasPrice.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:607 +0x88
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:677 +0x254
created by github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync in goroutine 145
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:661 +0x1c0
[2024-10-03T14:56:12.233-04:00] ERROR Panic error detected for transaction 0xa47781b615fb07aaeca3546d2f401583e181b021:1, when executing: retrieveGasPrice, error: runtime error: invalid memory address or nil pointer dereference role=public_tx_mgr
goroutine 149 [running]:
runtime/debug.Stack()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/debug/stack.go:16 +0x1c
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1.1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:666 +0x184
panic({0x101be84c0?, 0x102489270?})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/panic.go:770 +0x124
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*HybridGasPriceClient).getGasPriceJSON(0x140004b3560, {0x101d763c8, 0x140005cbc20})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/gas_price_client.go:111 +0x290
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*HybridGasPriceClient).GetGasPriceObject(0x140004b3560, {0x101d763c8, 0x140005cbc20})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/gas_price_client.go:86 +0xa4
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).TriggerRetrieveGasPrice.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:607 +0x88
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:677 +0x254
created by github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync in goroutine 145
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:661 +0x1c0
[2024-10-03T14:56:12.234-04:00] DEBUG 0xa47781b615fb07aaeca3546d2f401583e181b021:1 AddPanicOutput took 1µs to write the result role=public_tx_mgr
[2024-10-03T14:56:12.233-04:00] DEBUG Initializing enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG 0xa47781b615fb07aaeca3546d2f401583e181b021:1 AddPanicOutput took 917ns to write the result role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG Balance manager calcMethod setting: max role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG Initialized enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG NewOrchestrator for signing address 0xf47a803de32605230ac4a9c212af8a6539a32e51 created: &{pubTxManager:0x140002c6000 resubmitInterval:300000000000 stageRetryTimeout:10000000000 persistenceRetryTimeout:5000000000 ethClient:0x140001c2550 bIndexer:0x140001c2690 transactionSubmissionRetry:0x14000040640 orchestratorBirthTime:{wall:13958863146408487936 ext:31367042 loc:0x102526700} orchestratorPollingInterval:3600000000000 signingAddress:[244 122 128 61 227 38 5 35 10 196 169 194 18 175 138 101 57 163 46 81] hasZeroGasPrice:false unavailableBalanceHandlingStrategy:wait maxInFlightTxs:500 inFlightTxs:[] inFlightTxsMux:{state:0 sema:0} orchestratorLoopDone:<nil> InFlightTxsStale:0x14000454930 stopProcess:0x140004549a0 totalCompleted:0 state:new stateEntryTime:{wall:13958863146408488936 ext:31367709 loc:0x102526700} staleTimeout:300000000000 lastQueueUpdate:{wall:0 ext:0 loc:<nil>}} role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 AddPersistenceOutput took 625ns to write the result role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, on stage:  , current stage context lived: 31.666µs , stage lived: 32.625µs, last stage error: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, has 1 inputs role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, 1 inputs is carrying on role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 AddPersistenceOutput took 375ns to write the result role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, on stage:  , current stage context lived: 73.541µs , stage lived: 74.417µs, last stage error: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, has 1 inputs role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, 0 inputs is carrying on role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 AddPersistenceOutput took 125ns to write the result role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, on stage:  , current stage context lived: 102.25µs , stage lived: 103.084µs, last stage error: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, has 1 inputs role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG Transaction with ID 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 reached desired new status: suspending role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG Transaction with ID 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 clearing stage context for stage: statusUpdate after 113.125µs, total time spent on this stage so far: 113.959µs, txHash: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0xf47a803de32605230ac4a9c212af8a6539a32e51:1, 0 inputs is carrying on role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG Transaction with ID 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 is waiting for removal in status: suspending. role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] ERROR Cannot persist transaction state, no running context or stageOutputsToBePersisted role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG 0xf47a803de32605230ac4a9c212af8a6539a32e51:1 AddPersistenceOutput took 291ns to write the result role=public_tx_mgr
[2024-10-03T14:56:12.234-04:00] DEBUG Creating new enterprise transaction handler
[2024-10-03T14:56:12.234-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.234-04:00] DEBUG Enterprise transaction handler created
[2024-10-03T14:56:12.234-04:00] DEBUG Initializing enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.235-04:00] DEBUG Balance manager calcMethod setting: max role=public_tx_mgr
[2024-10-03T14:56:12.235-04:00] DEBUG Initialized enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.235-04:00] DEBUG NewOrchestrator for signing address 0x0a53d5d75348a1630010e56021b316e956bbd135 created: &{pubTxManager:0x140005b7b80 resubmitInterval:300000000000 stageRetryTimeout:10000000000 persistenceRetryTimeout:5000000000 ethClient:0x140001c3c20 bIndexer:0x140001c3cc0 transactionSubmissionRetry:0x140004e00a0 orchestratorBirthTime:{wall:13958863146409513936 ext:32393417 loc:0x102526700} orchestratorPollingInterval:3600000000000 signingAddress:[10 83 213 215 83 72 161 99 0 16 229 96 33 179 22 233 86 187 209 53] hasZeroGasPrice:false unavailableBalanceHandlingStrategy:wait maxInFlightTxs:500 inFlightTxs:[] inFlightTxsMux:{state:0 sema:0} orchestratorLoopDone:<nil> InFlightTxsStale:0x1400062a460 stopProcess:0x1400062a4d0 totalCompleted:0 state:new stateEntryTime:{wall:13958863146409513936 ext:32393542 loc:0x102526700} staleTimeout:300000000000 lastQueueUpdate:{wall:0 ext:0 loc:<nil>}} role=public_tx_mgr
[2024-10-03T14:56:12.235-04:00] ERROR Panic error detected for transaction 0x0a53d5d75348a1630010e56021b316e956bbd135:1, when executing: , error: runtime error: invalid memory address or nil pointer dereference role=public_tx_mgr
goroutine 117 [running]:
runtime/debug.Stack()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/debug/stack.go:16 +0x1c
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1.1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:666 +0x184
panic({0x101be84c0?, 0x102489270?})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/panic.go:770 +0x124
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*RunningStageContext).SetNewPersistenceUpdateOutput(...)
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/types.go:356
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).TriggerStatusUpdate.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:616 +0x100
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:677 +0x254
created by github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync in goroutine 154
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:661 +0x1c0
[2024-10-03T14:56:12.235-04:00] DEBUG 0x0a53d5d75348a1630010e56021b316e956bbd135:1 AddPanicOutput took 1µs to write the result role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG Creating new enterprise transaction handler
[2024-10-03T14:56:12.236-04:00] DEBUG Gas price cache size: 100
[2024-10-03T14:56:12.236-04:00] DEBUG Enterprise transaction handler created
[2024-10-03T14:56:12.236-04:00] DEBUG Initializing enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG Balance manager calcMethod setting: max role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG Initialized enterprise transaction handler role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG NewOrchestrator for signing address 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf created: &{pubTxManager:0x140005b6000 resubmitInterval:300000000000 stageRetryTimeout:10000000000 persistenceRetryTimeout:5000000000 ethClient:0x1400003c140 bIndexer:0x1400003c190 transactionSubmissionRetry:0x14000040340 orchestratorBirthTime:{wall:13958863146410292936 ext:33172001 loc:0x102526700} orchestratorPollingInterval:3600000000000 signingAddress:[74 160 233 195 93 97 142 57 176 73 217 209 187 1 202 69 182 29 102 207] hasZeroGasPrice:false unavailableBalanceHandlingStrategy:wait maxInFlightTxs:500 inFlightTxs:[] inFlightTxsMux:{state:0 sema:0} orchestratorLoopDone:<nil> InFlightTxsStale:0x14000454e70 stopProcess:0x14000454ee0 totalCompleted:0 state:new stateEntryTime:{wall:13958863146410292936 ext:33172167 loc:0x102526700} staleTimeout:300000000000 lastQueueUpdate:{wall:0 ext:0 loc:<nil>}} role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG Transaction with ID 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 entering status update, current status: pending, target status: suspending role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 AddPanicOutput took 166ns to write the result role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG ProduceLatestInFlightStageContext entry for tx 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1, on stage: statusUpdate , current stage context lived: 13.708µs , stage lived: 13.792µs, last stage error: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1, has 1 inputs role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] ERROR persistenceOutput should not be nil for transaction with ID: 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1, in the stage output object: &{Stage:statusUpdate PersistenceOutput:<nil> SubmitOutput:<nil> SignOutput:<nil> GasPriceOutput:<nil> ConfirmationOutput:<nil>}. role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG Transaction with ID 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 clearing stage context for stage: statusUpdate after 44.125µs, total time spent on this stage so far: 44.167µs, txHash: <nil> role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG ProduceLatestInFlightStageContext for tx 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1, 0 inputs is carrying on role=public_tx_mgr
[2024-10-03T14:56:12.236-04:00] DEBUG Transaction with ID 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 entering status update, current status: pending, target status: suspending role=public_tx_mgr
fatal error: concurrent map writes
[2024-10-03T14:56:12.236-04:00] DEBUG 0x4aa0e9c35d618e39b049d9d1bb01ca45b61d66cf:1 AddPersistenceOutput took 375ns to write the result role=public_tx_mgr

goroutine 121 [running]:
github.com/Code-Hex/go-generics-cache/policy/lru.(*Cache[...]).Set(0x101d83be0, {0x14000476480, 0x2c}, 0x140006a4020)
        /Users/pbroadhurst/go/pkg/mod/github.com/!code-!hex/go-generics-cache@v1.5.1/policy/lru/lru.go:82 +0x138
github.com/kaleido-io/paladin/toolkit/pkg/cache.(*cache[...]).Set(...)
        /Users/pbroadhurst/dev/paladin/paladin/toolkit/go/pkg/cache/cache.go:53
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*pubTxManager).addActivityRecord(0x140005b6000, {0x14000476480, 0x2c}, {0x140001c81e0, 0xa0})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/transaction_manager.go:752 +0x1d0
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*pubTxManager).UpdateSubStatus(0x140005b6000, {0x101d762e0, 0x14000349650}, {0x129497480, 0x14000538258}, {0x10183deb1, 0x8}, {0x101845e85, 0xf}, 0x140003ea020, ...)
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/transaction_manager.go:718 +0x51c
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*RunningStageContextPersistenceOutput).UpdateSubStatus.func1({0x101d69468?, 0x14000296d80?})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/types.go:380 +0xa0
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionState).PersistTxState(0x1400032c0f0, {0x101d763c8, 0x1400003cfa0})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_state_manager.go:294 +0x280
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).TriggerStatusUpdate.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:621 +0x2c4
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync.func1()
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:677 +0x254
created by github.com/kaleido-io/paladin/core/internal/publictxmgr.(*inFlightTransactionStageController).executeAsync in goroutine 118
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller.go:661 +0x1c0

goroutine 1 [chan receive]:
testing.(*T).Run(0x14000122b60, {0x101873528?, 0x14000571b38?}, 0x101d62398)
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:1750 +0x32c
testing.runTests.func1(0x14000122b60)
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:2161 +0x40
testing.tRunner(0x14000122b60, 0x14000571c58)
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:1689 +0xec
testing.runTests(0x1400033e840, {0x1024aeac0, 0x55, 0x55}, {0x14000571d18?, 0x100ea07f0?, 0x102526700?})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:2159 +0x3b0
testing.(*M).Run(0x1400045ce60)
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:2027 +0x5a4
main.main()
        _testmain.go:249 +0x18c

goroutine 123 [runnable]:
github.com/stretchr/testify/mock.(*Mock).On(0x1400003d4f0, {0x101840ebd, 0xa}, {0x0?, 0x0?, 0x0})
        /Users/pbroadhurst/go/pkg/mod/github.com/stretchr/testify@v1.9.0/mock/mock.go:355 +0x244
github.com/kaleido-io/paladin/core/internal/publictxmgr.newTestPublicTxManager(0x140001ba340, 0x0, {0x140000dbe80, 0x1, 0xe0?})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/transaction_manager_test.go:140 +0x4c0
github.com/kaleido-io/paladin/core/internal/publictxmgr.newTestOrchestrator(0x603?, {0x0?, 0x140005916b8?, 0x100d9415c?})
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/transaction_orchestrator_test.go:41 +0x60
github.com/kaleido-io/paladin/core/internal/publictxmgr.TestProduceLatestInFlightStageContextRetrieveGas(0x140001ba340)
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/in_flight_transaction_stage_controller_test_retrieve_gas_test.go:40 +0x38
testing.tRunner(0x140001ba340, 0x101d62398)
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:1689 +0xec
created by testing.(*T).Run in goroutine 1
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/testing/testing.go:1742 +0x318

goroutine 8 [select]:
database/sql.(*DB).connectionOpener(0x140000c65b0, {0x101d763c8, 0x140001c2a00})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 7
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 41 [select]:
database/sql.(*DB).connectionOpener(0x1400011c340, {0x101d763c8, 0x1400003c5f0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 40
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 51 [select]:
database/sql.(*DB).connectionOpener(0x140002f21a0, {0x101d763c8, 0x1400053a410})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 50
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 12 [select]:
database/sql.(*DB).connectionOpener(0x140000c68f0, {0x101d763c8, 0x140001c3d60})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 40
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 63 [select]:
database/sql.(*DB).connectionOpener(0x140004649c0, {0x101d763c8, 0x140003a9950})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 62
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 56 [select]:
database/sql.(*DB).connectionOpener(0x140002f2680, {0x101d763c8, 0x1400053b810})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 55
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 58 [select]:
database/sql.(*DB).connectionOpener(0x14000464340, {0x101d763c8, 0x140003a82d0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 57
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 138 [select]:
database/sql.(*DB).connectionOpener(0x140006ada00, {0x101d763c8, 0x14000567630})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 137
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 46 [select]:
database/sql.(*DB).connectionOpener(0x1400011c270, {0x101d763c8, 0x1400003c3c0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 45
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 135 [select]:
database/sql.(*DB).connectionOpener(0x140006ac8f0, {0x101d763c8, 0x140006bd7c0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 134
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 98 [select]:
database/sql.(*DB).connectionOpener(0x140001ad5f0, {0x101d763c8, 0x140001c3c70})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 33
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 29 [select]:
database/sql.(*DB).connectionOpener(0x1400011cc30, {0x101d763c8, 0x140005cb8b0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 28
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 90 [select]:
database/sql.(*DB).connectionOpener(0x140006a3040, {0x101d763c8, 0x140006bda40})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 89
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 24 [select]:
database/sql.(*DB).connectionOpener(0x140002f2340, {0x101d763c8, 0x140005ca1e0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 23
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 131 [select]:
database/sql.(*DB).connectionOpener(0x140006a32b0, {0x101d763c8, 0x140006bc4b0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 130
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 103 [select]:
database/sql.(*DB).connectionOpener(0x140000c7790, {0x101d763c8, 0x14000119400})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 102
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 108 [select]:
database/sql.(*DB).connectionOpener(0x14000465930, {0x101d763c8, 0x1400053a190})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 107
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 111 [select]:
database/sql.(*DB).connectionOpener(0x140006a21a0, {0x101d763c8, 0x1400053b180})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 110
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 114 [select]:
database/sql.(*DB).connectionOpener(0x140006a25b0, {0x101d763c8, 0x140005662d0})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 113
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 146 [select]:
database/sql.(*DB).connectionOpener(0x1400011c680, {0x101d763c8, 0x140005cb040})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 145
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 151 [select]:
database/sql.(*DB).connectionOpener(0x1400011d040, {0x101d763c8, 0x140001c2870})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 150
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 155 [select]:
database/sql.(*DB).connectionOpener(0x1400011d450, {0x101d763c8, 0x140001c3e00})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 154
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 119 [select]:
database/sql.(*DB).connectionOpener(0x140000c64e0, {0x101d763c8, 0x1400003c280})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:1246 +0x80
created by database/sql.OpenDB in goroutine 118
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140

goroutine 120 [runnable]:
github.com/kaleido-io/paladin/core/internal/publictxmgr.(*nonceCacheStruct).reapLoop(0x1400003d0e0)
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/nonces.go:81 +0x168
created by github.com/kaleido-io/paladin/core/internal/publictxmgr.newNonceCache in goroutine 118
        /Users/pbroadhurst/dev/paladin/paladin/core/go/internal/publictxmgr/nonces.go:65 +0x194

goroutine 124 [runnable]:
database/sql.OpenDB.gowrap1()
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824
runtime.goexit({})
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/runtime/asm_arm64.s:1222 +0x4
created by database/sql.OpenDB in goroutine 123
        /opt/homebrew/Cellar/go@1.22/1.22.7/libexec/src/database/sql/sql.go:824 +0x140
FAIL    github.com/kaleido-io/paladin/core/internal/publictxmgr 0.837s
ok      github.com/kaleido-io/paladin/core/internal/registrymgr 0.447s  coverage: 98.6% of statements
ok      github.com/kaleido-io/paladin/core/internal/statemgr    8.868s  coverage: 100.0% of statements
ok      github.com/kaleido-io/paladin/core/internal/tls 2.341s  coverage: 100.0% of statements
ok      github.com/kaleido-io/paladin/core/internal/transportmgr        0.910s  coverage: 95.7% of statements
ok      github.com/kaleido-io/paladin/core/internal/txmgr       5.915s  coverage: 99.8% of statements
FAIL

> Task :core:go:unitTestPostgres FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:go:unitTestPostgres'.
> Process 'command 'go'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.9/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 43s
22 actionable tasks: 9 executed, 13 up-to-date
``
peterbroadhurst commented 1 month ago

Think I've got to the bottom of this one

peterbroadhurst commented 1 month ago

We didn't have the initialization of the gas price client correct with respect to the config.

Overall there is quite a bit of tidy up and reconciliation needed of this code still after the port into Paladin.

peterbroadhurst commented 1 month ago

I believe the bug that needs to still be fixed here is two fold:

  1. We have lost Gas Oracle API support - unclear how or why
  2. The behavior of the gas client is affected by a single failure at startup, meaning that it thinks it cannot query the gas price, so goes into a mode where it thinks the gas price is zero (lost a lot of time to this in the tests)

I think both issues are a result of code port problems.

github-actions[bot] commented 3 weeks ago

This issue is stale because it has been open 30 days with no activity.