Monitor and Alert when routers not correctly filling transfers

[preethamr]

Problem

When a properly setup active router stops bidding, we don't know of it, until operator (we or router) checks.

Impact

Router operator miss on potential revenue, possibly transfers not getting boosted.

Proposed Solution

Monitor every active router, and alert when it stops sending successful bids, for more than a threshold amount of time

Acceptance Criteria

From an end user perspective, the below criteria must be met to consider this done:

• [ ] Every active router is monitored
• [ ] Alert sent to a Discord alert channel

[just-a-node]

Examples of transactions impacted by routers not bidding:

• https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5510
• https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5543
• https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5482
• https://app.zenhub.com/workspaces/core-engineering-6272a34e0368a200141cd949/issues/gh/connext/monorepo/5478

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xae6fd740c77e5d65888df0a1d1500ba751a11ca0d0a4a05c36651688783bd553?source=search Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927 Ticket: https://discord.com/channels/454734546869551114/1196852916334641302/1196853006247928041

[MinistroDolar]

Transaction:

https://connextscan.io/tx/0x544f0fff95308c4502ee9273ac2a4c3e91b1e0862b2495a7392367f3353b1a3b

Router

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket:

https://discord.com/channels/454734546869551114/1196981228457426944

[Tommy-R8]

Transactions from Optimism to Polygon:

https://connextscan.io/tx/0xc8e4c1cdcd852deb3eef1d140ce8e45eb395ddf348258b897fc690b6f2a88a50 https://connextscan.io/tx/0x64bbd9125bfbb9e46aea9bbd3047348e3fd76373587a0b03463b4320134aa826

Routers on Polygon:

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Transactions from Optimism to Arbitrum:

https://connextscan.io/tx/0xfe5d464d5b6d9c2f5c8b0108a3a9fdacf7259c74114c0a043ead731d2b90709f

Routers on Arbitrum:

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Offline router on arbitrum:

https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket on Discord: https://discord.com/channels/454734546869551114/1197943725003915364/1197943801570926664

[Tommy-R8]

Transactions from Optimism to Arbitrum:

https://connextscan.io/tx/0x31a819db847bbe0bf1c3bc32d39961c8084937043ada2ec2c8dd26f1dcfb60be https://connextscan.io/tx/0xa74cc5d97c2e181530d282c15e50bd395a2c85926c81e008f73e44114ca9e3c6

Routers on Arbitrum:

https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Offline router on arbitrum:

https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket on Discord: https://discord.com/channels/454734546869551114/1197944382502993970/1197944398047100938

[MinistroDolar]

Large txn from BNB Chain to Gnosis:

https://connextscan.io/tx/0x165434fae546628bf17b84dce493f79aa5160a0713635de89f9971af8c3fd325

Gnosis router not bidding:

https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xde18d913300013af0e92c764ca3ae8ea82c473d427873fad9883f7213f255a66

Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

Ticket: https://discord.com/channels/454734546869551114/1199301697118416993/1199301968397602886

[Tommy-R8]

Transaction: https://connextscan.io/tx/0x15100b04549b63e93be0fe62ef01ea771d4505ac899132b1bbb27dd306935c14

Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

Ticket: https://discord.com/channels/454734546869551114/1199292091004624946/1199292226568724520

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xfe54645511f482ba98532c5437ab52c773e4a9a1840224f95f30d31e7adc5f65

Router: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587

Ticket: https://discord.com/channels/454734546869551114/1199445475749789756/1199445960472924191

[Tommy-R8]

Transaction: https://connextscan.io/tx/0x29e1d009280003c59552a700e7c60419dc277b97d55798bc3f7b761bc8ea6ecf

Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1200519660160942211/1200519707770503300

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xe4f6447eda37bef24a8798f3a7b2c415ecd35e35909d573192d4da9bee9a562c

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1200609226733670462/1200614121666773002

[ValerywArz]

Route: USDC BNB to Linea Transaction: https://connextscan.io/tx/0xb647f4fb648962dfdf9a336abe3c93058d8c0a8c94cff7fbc39757917a0f301d Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927 Ticket: https://discord.com/channels/454734546869551114/1200718193581371442

[Tommy-R8]

Transaction: https://connextscan.io/tx/0x1b1c292906cf0d951e1f259c6ae20cb6c56e14119d5c8b89a4486735cf013bc9

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1200963969960132719/1200964049257644043

[alexwhte]

https://connextscan.io/tx/0xc572d6076b0ca8cf95a1f1f3819fb35e6e6a07662215ee665839cffa1c673319

from Karpatkey didnt go through

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xda93e386b259a4ef6a548815b2815135b96d67865a4e0a842201461ee86299f1

Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x33b2ad85f7dba818e719fb52095dc768e0ed93ec https://connextscan.io/router/0x7ce49752ffa7055622f444df3c69598748cb2e5f

Ticket: 120 on Intercom

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xec1d64082f907a140e5ed12de01ef55b264bdb44bb65de0838568cb8260e8200

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1202926909541978112/1202927071991697438

[preethamr]

RCA: https://www.notion.so/Gnosis-Asman-Router-Issues-b9a864625a394f06870d206fbf8aa80e Initial root cause for 3:

• Routers poll loop around retryXcalls likely freeze on connection mq.publish https://github.com/connext/monorepo/blob/4fb5d1ea4597ff61604b73fe6dc32a9912c5a97a/packages/agents/router/src/tasks/publisher/operations/retryXCalls.ts#L95-L115

Overall Cause:

Our routers don’t run rabbitMQ in a HA environment.

[MinistroDolar]

Transaction: https://connextscan.io/tx/0xfd1b7fc34491d0ada8afde841d3491e261e2e96fa519ba71557d125c2fa25c5f

Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket: https://discord.com/channels/454734546869551114/1204446970480042035

[MinistroDolar]

Transaction:

https://connextscan.io/tx/0xfd1b7fc34491d0ada8afde841d3491e261e2e96fa519ba71557d125c2fa25c5f

Routers:

https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket:

https://discord.com/channels/454734546869551114/1204446970480042035

Adding 2 more. Same destination chain and routers

Transaction: https://connextscan.io/tx/0xbf10c1f5ff68959ed45ea595d28b719a5c6d80a67cb9409005117d66172e1928

Ticket: https://discord.com/channels/454734546869551114/1205144727813619744

Transaction: https://connextscan.io/tx/0x50f3bd5d77009b420e095e3647f1839e2ffd4d3ec0b71bc3d3dcf431109b0a14

Ticket: https://discord.com/channels/454734546869551114/1205235577650413639

[MinistroDolar]

Case 1: 3.2 Metis from Andromeda to Mainnet

Transaction: https://connextscan.io/tx/0x48781fdf83b7036b46a7993b685d1c37a2341f5ee36f65410035293e7f6a4c73

Routers: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1207316640471977994

Case 2: 304 USDT from BNB to Arbitrum

Transaction: https://connextscan.io/tx/0x287a9edb884664bcc5d5d71843593f43fa852d00a04178fe2446f0a9949b3d80

Routers that could fulfill the txn: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c

Ticket: https://discord.com/channels/454734546869551114/1207336207097528360

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xdcae3368f11ee562ba51dbc57ef0b62d4eaaa3d1a8ab6d675e9bef3606863905 Ticket 152 on Intercom

Transaction: https://connextscan.io/tx/0x032a1173d7e868b3cd0376d537253bdb3401984eba0efc85e64bd6b0c8715106 Ticket 154 on Intercom

Transaction: https://connextscan.io/tx/0xad762da074eeafd097160ea9af9fd926e0571f4957d6478c46b69433ef17864 Ticket 150 on Intercom

Transaction: https://connextscan.io/tx/0x0440af350352d17d8b9468b538f62e646857139b724ed16005e492f5f8a02942 Ticket 153 on Intercom

[preethamr]

Transaction: https://connextscan.io/tx/0xdcae3368f11ee562ba51dbc57ef0b62d4eaaa3d1a8ab6d675e9bef3606863905 Ticket 152 on Intercom

Transaction: https://connextscan.io/tx/0x032a1173d7e868b3cd0376d537253bdb3401984eba0efc85e64bd6b0c8715106 Ticket 154 on Intercom

Transaction: https://connextscan.io/tx/0xad762da074eeafd097160ea9af9fd926e0571f4957d6478c46b69433ef17864 Ticket 150 on Intercom

Transaction: https://connextscan.io/tx/0x0440af350352d17d8b9468b538f62e646857139b724ed16005e492f5f8a02942 Ticket 153 on Intercom

All of these transfers received router bids, but after the gelato relayer address migration, they are awaiting retry from routers. FYI routers progressively decrease their retry frequency.

[MinistroDolar]

Transaction: (WETH from mainnet to ARB) https://connextscan.io/tx/0x0e001487c0f1adedc6d5fd99702a6e1a86bd80e7b6bd659f2c8eaeace5514ea1

Router: https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket: https://discord.com/channels/454734546869551114/1209217534373531718

Idk why but seems that there is only 1 router for WETH now on Arbitrum

[MinistroDolar]

Transaction: (WETH from mainnet to ARB) https://connextscan.io/tx/0x0e001487c0f1adedc6d5fd99702a6e1a86bd80e7b6bd659f2c8eaeace5514ea1

Router: https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa

Ticket: https://discord.com/channels/454734546869551114/1209217534373531718

Idk why but seems that there is only 1 router for WETH now on Arbitrum

Adding 1 more txn from Gnosis to Arbitrum (it's sad that a txn lower than 0.006 eth can't achieve fast path) https://connextscan.io/tx/0xe6f14c5f9e0cb4eb71ae611ac3baf1b6214284d7d6b6f9b88f3a19ab30acdf47

[MinistroDolar]

Transaction: (USDC from Linea to BNB) https://connextscan.io/tx/0xa2e053242596cdc3cfd987d03315feed9676b4d59ea2648374ceaea7b3290398

Routers: (this txn should be completed by more than 1 router bc of it size) https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x33b2ad85f7dba818e719fb52095dc768e0ed93ec https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1209255352139251792

[Tommy-R8]

Transaction (ETH from BNB to ETH): https://connextscan.io/tx/0xded162388328a7dfc5177a2346bb8843a99aba1a8e7b97299bf517e07fd1c4be

Routers: https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket 173 on Intercom

[Tommy-R8]

Transaction (USDT from BNB to Metis): https://connextscan.io/tx/0x66a9fe845c3bbc07fdce121ddf9a7496950e47cf730318649f03515bb9b1dc06

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket 185 on Intercom

[Tommy-R8]

Transaction: https://connextscan.io/tx/0x2508942ff99ea2949322cd8024db25e9d5b328124661bc8fc5949932aa77ed03 Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 Ticket: https://discord.com/channels/454734546869551114/1212515483769905194/1212516276866650112

[MinistroDolar]

Transaction (USDT from Optimism to Metis): https://connextscan.io/tx/0xff8052f6ab4259542ddf3ba9c7b352d13a9ff14014b5c22fe50d16d93177bffc

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1212731078805291109

[MinistroDolar]

Transaction (USDT from Base to Arb): https://connextscan.io/tx/0x5f8db76a9ce28053c74728146122a0eb7cbeb1701797c8725fd3727ddc4b804e

Router: https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c https://connextscan.io/router/0x49a9e7ec76bc8fdf658d09557305170d9f01d2fa https://connextscan.io/router/0xbe7bc00382a50a711d037eaecad799bb8805dfa8

Ticket: https://discord.com/channels/454734546869551114/1215046089229737994

[preethamr]

Create a notion document to check router bid status, preferably copy-pasta curl commands.

[ValerywArz]

Transaction: https://connextscan.io/tx/0x45f5abad2eb9ec1a9c19d68b5192192fe219fcfb16a1ee89e6816cd99869a9df Route: Base to Mode (WETH to ETH) Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170 Ticket: https://discord.com/channels/454734546869551114/1217387791009185842

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xba96522bfc991d96975fca5f17c4ab90ed72efe61b4d5be7a34aca7c34680712 Routers: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170 https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c Ticket: https://discord.com/channels/454734546869551114/1217975229780066444/1217975414815723530

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xd6b9558223ead55843b97edde21397c3cd2406bb0e7803fc479f8c53e3a7aea3 Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

[Tommy-R8]

Transaction: https://connextscan.io/tx/0x0971023fda9aeecf89555a8ea7e4e21eb02a295d7f0e0fb182069d0cdd8acedf Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927 Ticket: https://discord.com/channels/454734546869551114/1218575631114305556/1218577072969224222

[Tommy-R8]

Transaction: https://connextscan.io/tx/0xcce22852487b48fc97dc6060c7f6fd32774b3dbde85687af0be64b3175770865 Router: https://connextscan.io/router/0xf26c772c0ff3a6036bddabdaba22cf65eca9f97c Ticket: https://discord.com/channels/454734546869551114/1218643339415650384/1218643780471754882

[MinistroDolar]

Transaction: https://connextscan.io/tx/0x1c20e7cf31a6a7e8a1e9cc2866281093ffac6b6775482330e989f067e6a2924d Router: https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c Ticket: https://discord.com/channels/454734546869551114/1219680111628910642

[MinistroDolar]

Transactions: https://connextscan.io/tx/0xd2c4d4c167ffe589b611bd90fe1147c5e60cc8d1778034dde8278a6327b90c2d https://connextscan.io/tx/0xfd6361873f65d164d33a2524a23a9ddf295bab72d116edef6047c704b39363da

Router: https://connextscan.io/router/0xeca085906cb531bdf1f87efa85c5be46aa5c9d2c Ticket: https://discord.com/channels/454734546869551114/1219689939508265092

[MinistroDolar]

USDC from Base to Polygon

Transaction: https://connextscan.io/tx/0x2cee9c2734f961e2fd937cddfca359e2e7e8f3a2b0d80659202fa6306f9389b6

Routers: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927 https://connextscan.io/router/0x7ce49752ffa7055622f444df3c69598748cb2e5f https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0xc4ae07f276768a3b74ae8c47bc108a2af0e40eba

Ticket: https://discord.com/channels/454734546869551114/1219796089243303996

[MinistroDolar]

USDC from Arb to Polygon

Transaction: https://connextscan.io/tx/0x5ac3e720e088cf0840d8d3a57f2395351b2bdeb89d23201af13ce55d1f86a7ab

Routers: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1219812445917282346

[MinistroDolar]

USDC from Polygon to Linea

Transaction: https://connextscan.io/tx/0x48cab5944d00451a50fff36e6b30d42e271a7f304c35dd96de0f109d070fae7b

Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1219814857382170784

[MinistroDolar]

Another one; USDC from Arb to Polygon

Transaction: https://connextscan.io/tx/0x72aaece677878dda593aedd9374e3568f4b475811f899b2411c6e1675e1b556d

Routers: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 https://connextscan.io/router/0x7ce49752ffa7055622f444df3c69598748cb2e5f

Ticket: https://discord.com/channels/454734546869551114/1219815007403905178

[MinistroDolar]

WETH on Mainnet to nextWETH to Arbitrum

Transaction:https://connextscan.io/tx/0x29cd4d3d8edc278b31e07b38ee8f8fdbd97213fa2409270fe3eca2e1bae5beea

Router: https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket: https://discord.com/channels/454734546869551114/1219838885073518643

[MinistroDolar]

USDT from BNB to Linea

Tx: https://connextscan.io/tx/0x66fd68b562b7a8e23059e786bf40301fe6777afa4ad83fdcde3bc6a82901bf49

Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1219843169328431164

[MinistroDolar]

USDC from Gnosis to Base

Tx: https://connextscan.io/tx/0xed40b1d1450a93f3e8df683482d0091ae6479f959440468f5aa5cc97974f8a54

Router: https://connextscan.io/router/0x6fd84ba95525c4ccd218f2f16f646a08b4b0a598

Ticket: https://discord.com/channels/454734546869551114/1219848356919120065

[MinistroDolar]

WETH from Base to Mode

Tx: https://connextscan.io/tx/0xa42faf6e95918568aa9ff5c15079fc22e975668ba82a41fcb06be9281a861560

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1220067437647757473

[MinistroDolar]

USDC from Arbitrum to Linea

Tx: https://connextscan.io/tx/0xab7c9640ca80dfe520732383ac32e8cde0ca824ed1cf6c56614843b34ceb517e

Router: https://connextscan.io/router/0x58507fed0cb11723dfb6848c92c59cf0bbeb9927

Ticket: https://discord.com/channels/454734546869551114/1220071565610319933

[MinistroDolar]

USDC from BNB to Metis

Tx: https://connextscan.io/tx/0x4abaeed58105c2ddd9756077b91a30aae2fb45ed2d48ec37bc81bb2140a6cbab

Router: https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4

Ticket: https://discord.com/channels/454734546869551114/1220087029002403960

[MinistroDolar]

WETH from Polygon to Mode

Tx: https://connextscan.io/tx/0x2e71a209f83ecf76ea07f105efc56527b525ffaeeab0bb31d96b62cf56296887

Router: https://connextscan.io/router/0x8cb19ce8eedf740389d428879a876a3b030b9170

Ticket: https://discord.com/channels/454734546869551114/1220094862041550898

[Tommy-R8]

DAI from Arbitrum to Gnosis Transaction: https://connextscan.io/tx/0xc06fc8a855260ae31360822a112671b6f463f1fc84bcb284cf34c47a6df10e01 Routers: https://connextscan.io/router/0x048a5ecc705c280b2248aeff88fd581abbeb8587 https://connextscan.io/router/0x97b9dcb1aa34fe5f12b728d9166ae353d1e7f5c4 Ticket 321 on Discord