search

akash-network / support

Akash Support and Issue Tracking
5 stars 3 forks source link

provider closed 32 leases after one of K8s nodes experience a network error #15

Closed andy108369 closed 8 months ago

andy108369 commented 1 year ago

One of d3akash.cloud provider's K8s cluster nodes had a network error which caused it to close 32 leases.

I think the root cause is the https://github.com/akash-network/support/issues/14 ; waiting for the provider logs from the provider owner to confirm.

provider address: akash1u5cdg7k3gl43mukca4aeultuz8x2j68mgwn28e

heights 9636347 ... 9636379 (MsgCloseBid) issued to 32 leases

# akash query txs --events "akash.v1.action=lease-closed&akash.v1.provider=akash1u5cdg7k3gl43mukca4aeultuz8x2j68mgwn28e" --page 1 --limit 1000 -o json | jq -r '(["timestamp","height","code","txhash","provider/owner","type"]), (.txs[] | [.timestamp, .height, .code, .txhash, (.tx.body.messages[] | .provider+.id.owner+.bid_id.owner, ."@type")]) | @csv' | column -t -s","
"timestamp"             "height"   "code"  "txhash"                                                            "provider/owner"                                "type"                                                                                          
"2023-01-30T16:41:44Z"  "9564942"  0       "3DF507E33BF00C77DE3AAEC56533CC4EA07A36501ED57DCEE2AFA03E97AD08C4"  "akash1dte9q7nhqj58rj3dam7vppwn5zvp45wwu96545"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-01-30T21:12:13Z"  "9567607"  0       "D8924C45E2582F85D16D35DE46764AA6CCBDBC7AE534C0871850A9EC5454D6FD"  "akash1d3wzys4n4uc7yqt84p4r6t08hafjnsfsaat965"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-01-31T17:26:59Z"  "9579604"  0       "CE1F066ECE195EA2654C8D905809DBD1FB16BABEFC48EED9B9CE1932AB8BB2E2"  "akash1dte9q7nhqj58rj3dam7vppwn5zvp45wwu96545"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-01T00:20:10Z"  "9583682"  0       "047907CE61FCE804807D15C9A3DCCA6377845E54E4913A89F094C2777299DBE0"  "akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-01T10:31:55Z"  "9589735"  0       "6011704513100A367ADCF4052BD516DA58470F46906171703B0FC6AB9890AFA8"  "akash1kwdtgtzfn5kh5wf0lrshdnvgflm7urrpzpnvsw"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-01T10:35:39Z"  "9589772"  0       "AE218232BEB9DE17DC29F4F73E3A8DBCEB4B8104342F14D7537746E6F5566A4D"  "akash1kwdtgtzfn5kh5wf0lrshdnvgflm7urrpzpnvsw"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-01T12:50:18Z"  "9591100"  0       "F4F98EF4E66BBA4A54D545DF49C6BA2F560292832EC4B407DF7DF6C5C7E89866"  "akash1f53fp8kk470f7k26yr5gztd9npzpczqv4ufud7"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-01T13:59:26Z"  "9591783"  0       "5ACF87DE0EECA4145FAF06E01251C808B05AB3E33584ECA62ECD669397F56132"  "akash1mh979xxjzs54q8h732cvuhtmqhpa7wmcz9syqf"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-02T06:29:29Z"  "9601540"  0       "98E2D9A7A068A8D3BC2FB9204255231EBC28760977D90AFADAB982E72D9823F6"  "akash10a670qvazh3qa6xdvclc4jtrdjrnwm392x66we"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-02T14:02:46Z"  "9606018"  0       "86341CC6FCB87B84F94D37CC8903E5C2A018F3CE7AC5E51E9B3531DB8CC8440C"  "akash1463j4ujx57xfn2t0a3a0z78gu4hlss000fwxkr"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-02T14:07:11Z"  "9606061"  0       "2D193A767065EEA1543FFFB6D271259426FD48B7C1B47A2D04CD9FA8C493B714"  "akash1463j4ujx57xfn2t0a3a0z78gu4hlss000fwxkr"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-02T16:28:35Z"  "9607459"  0       "4E0D3498A878EB60FAEAAF95A417317BFB56EE5B306BDB2D83C030A935C866DD"  "akash1qh0f0h7jlq4x5gpxghrxvps5l09y7uuvcumcyd"  "/akash.deployment.v1beta2.MsgCloseDeployment"  "akash1qh0f0h7jlq4x5gpxghrxvps5l09y7uuvcumcyd"  "/akash.deployment.v1beta2.MsgCloseDeployment"
"2023-02-02T17:50:19Z"  "9608268"  0       "C86808D72131655F4BAFA6DCECC544D17546801C617EE2C8B0A24B78035039B3"  "akash1leryrncx03rn8t50lmsx77shuyu5nmek00ewg9"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-03T09:15:43Z"  "9617433"  0       "9F367C7876BA5D48006CCE239BFCD8AA44EEB3A471F3E0B0FBF7601AAF5BDFBB"  "akash1z5wtzgq6mwmr3cda2trfyw3nuzhwe7vf6mt78y"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-03T18:49:14Z"  "9623085"  0       "C01F90CEA7A55FE61BCE86705F41E93F8E7249CF36AAA98C9CC06186D2E61535"  "akash16mdmh3lfphnmqzn20c59x75nfhfktfz9tec4kf"  "/akash.market.v1beta2.MsgWithdrawLease"                                                        
"2023-02-04T03:40:55Z"  "9628333"  0       "DD371478B46715A309A9B3636FB557561F5EB6671B567640D4215B0573B04D8C"  "akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-04T03:42:57Z"  "9628353"  0       "400EB84F2AFCB71CCF6D639C9202EEFF58E8BE284F449D28A1BAD5C39C8C6517"  "akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-04T03:46:17Z"  "9628386"  0       "157000CC027B3A90DD0E5E8AD71A48FDAD4FB7ED1862AEE56FCC73299B3A23E6"  "akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-04T03:48:30Z"  "9628408"  0       "B48ACC58F5A14ABEB635D793FFCFF969DBD5CC479F6B0126EE7A65416A9AAD58"  "akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-04T03:50:00Z"  "9628423"  0       "4088015A6045335C8AF94435AEE62E8F8D0F350F9F9151C0C5B16753C077A2D6"  "akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-04T12:46:59Z"  "9633722"  0       "9837078583712744A6ECEA112D13D422FFF828B85478E38C2D6F6BDF81584EC8"  "akash16mdmh3lfphnmqzn20c59x75nfhfktfz9tec4kf"  "/akash.deployment.v1beta2.MsgCloseDeployment"                                                  
"2023-02-04T17:13:16Z"  "9636347"  0       "D657AEABA4E11A6E2C23ADE7D67A287B264163036E205D7BF86C5A0639E81C8C"  "akash1zrce7fke2pxmnrwlwdjxcgyfcz43vljw5tekr2"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:22Z"  "9636348"  0       "13F73AFF4EDFA053E248CC3736A77B544BAC33DE0A7674970A4CF53F54A92B75"  "akash1kcg7r9hywww5rdfhdx0wr747dm9zk6rsxq40zl"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:27Z"  "9636349"  0       "412C19E8838CA42C8CE66D26FEC7435FAB9FCEAE63D1347F68CB718ABAFADF77"  "akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:34Z"  "9636350"  0       "B497C1D8F02AB91385720BBF5913B9A1B098D93EC3221EAEDB15DDC727B0B2C6"  "akash1efqx0n93x4y3tlssulcwdgf6l8j7y8xkpw3kev"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:40Z"  "9636351"  0       "546FFD0CE6941DB68FB088E3B7B15554A43394BC5CD0BAF67D3715541D033C5D"  "akash1smpmfz6czccs4xpt6r5lcxknlsulmshfjqpj97"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:46Z"  "9636352"  0       "44A7BC6DE28B3DFC509D98DBE20D1D0884156443FEF3FFD817C83B52D2458FE6"  "akash1zrce7fke2pxmnrwlwdjxcgyfcz43vljw5tekr2"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:52Z"  "9636353"  0       "43DF122EC7349427210A97B06CE1C17AE2D7AE5CD9CBD305DAA3BE472A15DC90"  "akash1yc6av7dvcypjmemshg3z826fj9ny2xgfxwhpth"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:13:58Z"  "9636354"  0       "9A1FC2E735D9CEC41C025346E54347296B181517097693FA9ACE7DC86264538C"  "akash153vxg53pe3jr3feq8k0ddeldddjejkng8d5jp3"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:04Z"  "9636355"  0       "7822687E584542855F0B62E7ABE656719FFCCB2DEAD8C5AE9BBAA9534224B60C"  "akash16mdmh3lfphnmqzn20c59x75nfhfktfz9tec4kf"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:10Z"  "9636356"  0       "E6F1E641BED7093253C195AF021824207F7DD8FFAE1D1DB78F29C594990D53CC"  "akash1qpmg4awdexhp7yu22vknn0dlfjuu0fuchnu4fr"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:16Z"  "9636357"  0       "0EA7DA70C4CE01E3FEE16C494E30EC9847A3D71DFA1C38782BA3E18D377FCA3F"  "akash19jsyw4e2mvdk6tdpwzgghf9y9dljmfgwxg2xtd"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:23Z"  "9636358"  0       "8DFC41938F072D00566675510A183DAB1D3CFE6260DB35677E43FD1F9A773E8E"  "akash1p83uq6sr8nqrv68cce7yjndrleswy7j55e3y29"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:29Z"  "9636359"  0       "5585E37CD0E19482BE173DC09D701C5C08A288F9A70D93F1909EBC3721D831F0"  "akash1wl2a4npdcj3nxu9xd60ys2y9m53ys2rld7xkem"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:35Z"  "9636360"  0       "93E013665E6122D131B9A08EB9A92AA31E45E07834B7EF0913302511748122BF"  "akash18nn8e9709c99kckp3hqrgdntpkpnra8rw6uv9p"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:40Z"  "9636361"  0       "62FBEADD6239083A4828B324BFF48C8E806743F0C8B3D401BE5AEFED6A716369"  "akash1k8auevuhscxznfjqpncfa7kp3dssneup55d60u"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:47Z"  "9636362"  0       "16415E5FFDD3D62B74D3749628896C93BC4C407A8F225E7C303EE7099BBA4B1D"  "akash1kcg7r9hywww5rdfhdx0wr747dm9zk6rsxq40zl"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:53Z"  "9636363"  0       "14413FCA4B349627EDC8546DC40EE3E14ED8B7BAD1ED3EF4977C6ACC5981C00F"  "akash1dte9q7nhqj58rj3dam7vppwn5zvp45wwu96545"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:14:59Z"  "9636364"  0       "6AE0FFBE983F1E560051D1E46B9A881382D67CC2C7E9001048EFAE15085EEE23"  "akash1nw3akegr5duutstwaqvzffnff9szle30t2k7hj"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:05Z"  "9636365"  0       "756730889993D50E00112E3EEFFB32B76B64B5800E1D43E98463DCD68C79C1EE"  "akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:11Z"  "9636366"  0       "593D74224F2613051E34D50FF5683C0E1E6F64CBD32AD6DEB609B8C8B233C8E9"  "akash10h542ett8fl257wlrmduplf05jmaydr8lkz9t8"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:17Z"  "9636367"  0       "1A0365160B778BEF455DF3F49070F9BDB012B05A856FC04D4025AFDD115742A0"  "akash19fezsaqahsp9ysjqjklghw3cj02lsnwkukzrft"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:23Z"  "9636368"  0       "D97628CE1594C94E119066BB7EA9F5B2AB4CBAA1C2D7598B06444FA09ED41197"  "akash19jsyw4e2mvdk6tdpwzgghf9y9dljmfgwxg2xtd"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:29Z"  "9636369"  0       "8DFCBDADAE8150CBBA7A904B288E81FCDFA1D952BB0CCC95C070E2DCA7F25339"  "akash19fezsaqahsp9ysjqjklghw3cj02lsnwkukzrft"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:35Z"  "9636370"  0       "D619A3FEFC95BBAA5141968610CAB164655CBEA6155E26F4D30162C0AA536D56"  "akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:41Z"  "9636371"  0       "5A00F4B9E559190B80584B8A5683B7F9E5F35488DA90A7B53617B1E5CB0EF0FF"  "akash17agrq3zll2545mms3xvffrzfgj7tym3rehr9ut"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:47Z"  "9636372"  0       "ED8F916DA7C51302CFF1D398215E970FAFCE79D1B216EED6B35BD0F8B7481579"  "akash1a2hukwlndnz7svs08z290dm67rjmtuhkq7tq5g"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:53Z"  "9636373"  0       "72D9B21099E983C9892B009BD8763FCAA71E605EFD26761E54CD78328AB2E396"  "akash1x22y03x52p3n9qct4q4z6yl9g8en8fzqk6wudx"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:15:59Z"  "9636374"  0       "750E2EC01995CF112D4A76730EED33E1405E4CE5F5682F129B7637305B89A769"  "akash1lc7sxhpg4ge5ksmn2qwspm20araf26s4gawu6w"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:16:06Z"  "9636375"  0       "D63B4E0C7BAD59FF29F362E5DE5B8C371528BC2DA3CB513FB8D800EA5795C70D"  "akash1fmj5fxsuwhp6p7rszjs65gut2c24vt8r4y8n7k"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:16:12Z"  "9636376"  0       "B4F01D15DCFC5661F6396FB988A47477555F6FC4FB2A626755724D26E77AB71A"  "akash1dte9q7nhqj58rj3dam7vppwn5zvp45wwu96545"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:16:24Z"  "9636378"  0       "965F9E0CD9D6AF60160790A0F75F18F9FFA6FD9E64580A5B5DBD895897898837"  "akash1wznrpsdj040ua0d2gnzed89t34ze6zrpq6dahs"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T17:16:30Z"  "9636379"  0       "385553132DA48246DFB6E675201212D44E699F73A4913F9D277920A8D5F4C3C5"  "akash1ruja6tgd07slar393ee45l6j3u7vcr7w5dhyud"  "/akash.market.v1beta2.MsgCloseBid"                                                             
"2023-02-04T18:29:21Z"  "9637094"  0       "72E662CA8B3FC63F1A8997A3A773C11BCDE0C2B246C6F06A77293C9AFF1D309B"  "akash19fezsaqahsp9ysjqjklghw3cj02lsnwkukzrft"  "/akash.deployment.v1beta2.MsgCloseDeployment"

leases before it closed them (height 9636346)

This is the height before provider started closing 32 leases.

This is mainly to verify the withdrawn vs consumed to rule out the case where the provider could have been running without withdrawing the leases (e.g. due to some bug / misconfig) for some time until it was restarted.

$ PROVIDER=akash1u5cdg7k3gl43mukca4aeultuz8x2j68mgwn28e; HEIGHT=9636346; provider-services query market lease list --height $HEIGHT --provider $PROVIDER --gseq 0 --oseq 0 --page 1 --limit 10000 --state active -o json | jq --argjson h $HEIGHT -r '["owner","dseq/gseq/oseq","rate","monthly","withdrawn","consumed","days"], (.leases[] | [(.lease.lease_id | .owner, (.dseq|tostring) + "/" + (.gseq|tostring) + "/" + (.oseq|tostring)), (.escrow_payment | (.rate.amount|tonumber), (.rate.amount|tonumber), (.withdrawn.amount|tonumber)), (($h-(.lease.created_at|tonumber))*(.escrow_payment.rate.amount|tonumber)/pow(10;6)), (($h-(.lease.created_at|tonumber))/((60/6)*60*24))]) | @csv' | awk -F ',' '{if (NR==1) {$1=$1; printf $0"\n"} else {block_time=6; rate_akt=(($4*((60/block_time)*60*24*30.436875))/10^6); $4=rate_akt; withdrawn_akt=($5/10^6); $5=withdrawn_akt; $6; $7; print $0}}' | column -t ; TZ=UTC date
"owner"                                         "dseq/gseq/oseq"  "rate"  "monthly"  "withdrawn"  "consumed"  "days"
"akash1qpmg4awdexhp7yu22vknn0dlfjuu0fuchnu4fr"  "9479714/1/1"     8       3.50633    1.20382      1.252944    10.87625
"akash1qpmg4awdexhp7yu22vknn0dlfjuu0fuchnu4fr"  "9497888/1/1"     5       2.19145    0.66162      0.692265    9.614791666666667
"akash1p83uq6sr8nqrv68cce7yjndrleswy7j55e3y29"  "9218669/1/1"     23      10.0807    9.46542      9.606502    29.00513888888889
"akash1zrce7fke2pxmnrwlwdjxcgyfcz43vljw5tekr2"  "9304583/1/1"     9       3.94462    2.93059      2.985804    23.038611111111113
"akash1zrce7fke2pxmnrwlwdjxcgyfcz43vljw5tekr2"  "9304617/1/1"     9       3.94462    2.93023      2.985516    23.03638888888889
"akash1ruja6tgd07slar393ee45l6j3u7vcr7w5dhyud"  "9295158/1/1"     22      9.6424     7.37077      7.50607     23.693402777777777
"akash1yc6av7dvcypjmemshg3z826fj9ny2xgfxwhpth"  "9211200/1/1"     70      30.6804    29.3311      29.75973    29.523541666666667
"akash19fezsaqahsp9ysjqjklghw3cj02lsnwkukzrft"  "9086398/1/1"     2       0.876582   1.08764      1.099886    38.19048611111111
"akash19fezsaqahsp9ysjqjklghw3cj02lsnwkukzrft"  "9086406/1/1"     2       0.876582   1.08759      1.099872    38.19
"akash19jsyw4e2mvdk6tdpwzgghf9y9dljmfgwxg2xtd"  "9420877/1/1"     19      8.32753    3.97706      4.093455    14.961458333333333
"akash19jsyw4e2mvdk6tdpwzgghf9y9dljmfgwxg2xtd"  "9420969/1/1"     27      11.8339    5.64956      5.814909    14.956041666666666
"akash1x22y03x52p3n9qct4q4z6yl9g8en8fzqk6wudx"  "9278572/1/1"     29      12.7104    10.1967      10.37533    24.84513888888889
"akash18nn8e9709c99kckp3hqrgdntpkpnra8rw6uv9p"  "9433449/1/1"     12      5.25949    2.36096      2.434728    14.08986111111111
"akash1f0nqvjyz37g3cen4zh8xlz4jtajq3xjnjd4q4f"  "9435632/1/1"     25      10.9573    4.86418      5.017775    13.938263888888889
"akash1fmj5fxsuwhp6p7rszjs65gut2c24vt8r4y8n7k"  "9497358/1/1"     7       3.06804    0.930006     0.972881    9.651597222222222
"akash1tf5avu5d4fhkl53ppa379s2t4srjkdlamh2zh9"  "9629627/1/1"     25      10.9573    0            0.16785     0.46625
"akash1dte9q7nhqj58rj3dam7vppwn5zvp45wwu96545"  "9564944/1/1"     9       3.94462    0.587385     0.642564    4.958055555555555
"akash1dte9q7nhqj58rj3dam7vppwn5zvp45wwu96545"  "9566127/1/1"     14      6.13607    0.896224     0.98238     4.872916666666667
"akash1wznrpsdj040ua0d2gnzed89t34ze6zrpq6dahs"  "9471301/1/1"     121     53.0332    19.2252      19.970082   11.46125
"akash1wl2a4npdcj3nxu9xd60ys2y9m53ys2rld7xkem"  "9486216/1/1"     65      28.4889    9.35961      9.758255    10.42548611111111
"akash10h542ett8fl257wlrmduplf05jmaydr8lkz9t8"  "9479634/1/1"     7       3.06804    1.05394      1.096935    10.882291666666667
"akash1smpmfz6czccs4xpt6r5lcxknlsulmshfjqpj97"  "9210717/1/1"     21      9.20411    8.80876      8.938062    29.557083333333335
"akash1nw3akegr5duutstwaqvzffnff9szle30t2k7hj"  "9481413/1/1"     5       2.19145    0.74389      0.774585    10.758125
"akash153vxg53pe3jr3feq8k0ddeldddjejkng8d5jp3"  "9578643/1/1"     23      10.0807    1.18549      1.326824    4.006111111111111
"akash1k8auevuhscxznfjqpncfa7kp3dssneup55d60u"  "9441132/1/1"     13      5.69778    2.45796      2.537743    13.556319444444444
"akash1kcg7r9hywww5rdfhdx0wr747dm9zk6rsxq40zl"  "9119443/1/1"     43      18.8465    21.9619      22.226485   35.89548611111111
"akash1kcg7r9hywww5rdfhdx0wr747dm9zk6rsxq40zl"  "9119493/1/1"     43      18.8465    21.9606      22.224421   35.89215277777778
"akash1efqx0n93x4y3tlssulcwdgf6l8j7y8xkpw3kev"  "9397796/1/1"     16      7.01266    3.71837      3.816752    16.56576388888889
"akash16mdmh3lfphnmqzn20c59x75nfhfktfz9tec4kf"  "9536859/1/1"     29      12.7104    2.70721      2.884891    6.9082638888888885
"akash1ml77mzlfzkdug4gf60u9v9zrkxrqe5uhhc5ajd"  "9627413/1/1"     42      18.4082    0.29862      0.374724    0.6195833333333334
"akash1a2hukwlndnz7svs08z290dm67rjmtuhkq7tq5g"  "9534097/1/1"     3       1.31487    0.288306     0.306696    7.099444444444444
"akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "9142431/1/1"     24      10.519     11.706       11.853672   34.29881944444445
"akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "9169371/1/1"     22      9.6424     10.1381      10.27334    32.428472222222226
"akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "9583685/1/1"     26      11.3956    1.20952      1.368952    3.656388888888889
"akash17agrq3zll2545mms3xvffrzfgj7tym3rehr9ut"  "9126532/1/1"     23      10.0807    11.5841      11.725584   35.403333333333336
"akash1lc7sxhpg4ge5ksmn2qwspm20araf26s4gawu6w"  "9528400/1/1"     14      6.13607    1.42524      1.511034    7.495208333333333

leases after the drop (height 9636379)

$ PROVIDER=akash1u5cdg7k3gl43mukca4aeultuz8x2j68mgwn28e; HEIGHT=9636379; provider-services query market lease list --height $HEIGHT --provider $PROVIDER --gseq 0 --oseq 0 --page 1 --limit 10000 --state active -o json | jq --argjson h $HEIGHT -r '["owner","dseq/gseq/oseq","rate","monthly","withdrawn","consumed","days"], (.leases[] | [(.lease.lease_id | .owner, (.dseq|tostring) + "/" + (.gseq|tostring) + "/" + (.oseq|tostring)), (.escrow_payment | (.rate.amount|tonumber), (.rate.amount|tonumber), (.withdrawn.amount|tonumber)), (($h-(.lease.created_at|tonumber))*(.escrow_payment.rate.amount|tonumber)/pow(10;6)), (($h-(.lease.created_at|tonumber))/((60/6)*60*24))]) | @csv' | awk -F ',' '{if (NR==1) {$1=$1; printf $0"\n"} else {block_time=6; rate_akt=(($4*((60/block_time)*60*24*30.436875))/10^6); $4=rate_akt; withdrawn_akt=($5/10^6); $5=withdrawn_akt; $6; $7; print $0}}' | column -t ; TZ=UTC date
"owner"                                         "dseq/gseq/oseq"  "rate"  "monthly"  "withdrawn"  "consumed"  "days"
"akash1qpmg4awdexhp7yu22vknn0dlfjuu0fuchnu4fr"  "9497888/1/1"     5       2.19145    0.66162      0.69243     9.617083333333333
"akash1f0nqvjyz37g3cen4zh8xlz4jtajq3xjnjd4q4f"  "9435632/1/1"     25      10.9573    4.86418      5.0186      13.940555555555555
"akash1ml77mzlfzkdug4gf60u9v9zrkxrqe5uhhc5ajd"  "9627413/1/1"     42      18.4082    0.29862      0.37611     0.621875
"akash17fqxak4kprh2rlatjlv9w04s9ugl7mn32ckut3"  "9169371/1/1"     22      9.6424     10.1381      10.274066   32.43076388888889
andy108369 commented 1 year ago

It is the third time he is losing the leases, and the reasons are a failing k8s node (or when the worker node is down).

It looks like when the cluster does not have enough room (resources) for the apps to start on the other available nodes, which then triggers the monitorMaxRetries counter, then, leading to the provider closing the deployment as described in https://github.com/akash-network/support/issues/14 ;

andy108369 commented 1 year ago

provider logs (after stripping off the deployment hostnames sed -e 's/service-name=.*/service-name==REDACTED/' -e 's/hostname=.*/hostname=REDACTED/' ) => 0-redacted.log

These logs do have some messages such as bid in unexpected state but not for the dseq's in question (the closed ones mentioned above). They don't contain the deployment failed. closing lease (with the accumulating attempts value up to 40 ; refs), as, again, they contain the data after the incident occurred => starting on 2023-02-04|17:32:44.100. And the leases were closed between 2023-02-04T17:13:16Z ... 2023-02-04T17:16:30Z

I've asked the provider whether he can find older logs find /var/log/pods -xdev -type f -name '*.log' -ls |grep provider.

Update: unfortunately, the provider didn't save the logs before restarting the akash-provider pod.

andy108369 commented 1 year ago

It's the 4th time d3akash is losing its leases: image (source https://status.d3akash.cloud/status/status )

andy108369 commented 8 months ago

That's OK to have the provider close the leases when provider isn't charging for the lease that doesn't work for quite some time.

So I think the Alternative proposal (client-defined) would be ideal if the timeout (the amount of time when the lease is down because it cannot redeploy as the worker node is down) could be configured by the clients themselves in their SDL (say deployment_grace_period in deployment manifests).

I'll close this issue in the favor of the Alternative proposal.