the deployment got closed due to an unknown reason along with the strange address

andy108369 commented 2 years ago

I am trying to figure where this address akash14pphss726thpwws3yc458hggufynm9x77l4l2u is coming from. I remember seeing it few times already throughout long period of time.

I found it in this TX 3AD714F1AFE7487BF4108FCA9A98F6485F9FF1A77C7C21482385A2A80A591814 while investigating the reason why the user (owner) akash15jd3ye3sf47yqtf6xvfar9jffmc5czprhrz252 had his deployment 6907521/1/1 lease-create tx on this provider akash18ga02jzaq8cw52anyhzkwta5wygufgu6zsz6xc (https://provider.europlots.com:8443) closed for an unknown reason. He used Akashlytics to deploy his app.

Similar report where akash14pphss726thpwws3yc458hggufynm9x77l4l2u appeared as well from April 2021 (a year+ ago) is here + analysis where a user was trying to figure who closed the deployment (dseq 1114883).

TX 0E759586D0C161B0CB2A4196CBDAACD969137BCCAE51F422F842D567C7907FD6

I also found akash14pphss726thpwws3yc458hggufynm9x77l4l2u address here and here

tidrolpolelsef commented 2 years ago

Is that address some internal address we use for something? Like community funds?

andy108369 commented 2 years ago

Is that address some internal address we use for something? Like community funds?

Not sure.. it's got 0 txs there and yet non zero balance https://www.mintscan.io/akash/account/akash14pphss726thpwws3yc458hggufynm9x77l4l2u (if mintscan reports well)

andy108369 commented 2 years ago

The same occurred again, and again akash14pphss726thpwws3yc458hggufynm9x77l4l2u is spotted.

dseq 6922309 (oseq, gseq both 1), owner akash1lye33kdywf6dd2ddfsat8pvs9t3e46e6amjlez.

provider akash18ga02jzaq8cw52anyhzkwta5wygufgu6zsz6xc (https://provider.europlots.com:8443)

txhash FCE209EC3F3F3037DB56A233774BF68B16186A6D9EE846DDAAD9CBB63762AF5F

$ akash query tx FCE209EC3F3F3037DB56A233774BF68B16186A6D9EE846DDAAD9CBB63762AF5F | jq
{
  "height": "6954018",
  "txhash": "FCE209EC3F3F3037DB56A233774BF68B16186A6D9EE846DDAAD9CBB63762AF5F",
  "codespace": "",
  "code": 0,
  "data": "0A230A212F616B6173682E6D61726B65742E763162657461322E4D7367436C6F7365426964",
...
          "type": "akash.v1",
          "attributes": [
            {
              "key": "module",
              "value": "deployment"
            },
            {
              "key": "action",
              "value": "group-paused"
            },
            {
              "key": "owner",
              "value": "akash1lye33kdywf6dd2ddfsat8pvs9t3e46e6amjlez"
            },
            {
              "key": "dseq",
              "value": "6922309"
            },
...
        {
          "type": "message",
          "attributes": [
            {
              "key": "action",
              "value": "/akash.market.v1beta2.MsgCloseBid"
            },
            {
              "key": "sender",
              "value": "akash14pphss726thpwws3yc458hggufynm9x77l4l2u"
            },
            {
              "key": "sender",
              "value": "akash14pphss726thpwws3yc458hggufynm9x77l4l2u"
            }
          ]
        },
...
  "tx": {
    "@type": "/cosmos.tx.v1beta1.Tx",
    "body": {
      "messages": [
        {
          "@type": "/akash.market.v1beta2.MsgCloseBid",
          "bid_id": {
            "owner": "akash1lye33kdywf6dd2ddfsat8pvs9t3e46e6amjlez",
            "dseq": "6922309",
            "gseq": 1,
            "oseq": 1,
            "provider": "akash18ga02jzaq8cw52anyhzkwta5wygufgu6zsz6xc"
          }
        }
      ],
...

andy108369 commented 1 year ago

The same issue occurred again.

dseq 6940849, owner akash12hs7vay7pheanr4ze5lwnt7xg9krrtew34empl, provider akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp.

https://www.mintscan.io/akash/txs/56D783E12FDB646C93C841D5005E37770292E614B91F83E183764D0B5C13E44B

reported here https://discord.com/channels/747885925232672829/771909909335506955/1006165965391798302

andy108369 commented 1 year ago

FWIW: users are using a standard wordpress template available in Akashlytics

Which must be this https://github.com/ovrclk/awesome-akash/tree/b749184b/wordpress

andy108369 commented 1 year ago

@troian suggested disabling the withdrawal on the provider and see if that helps AKASH_WITHDRAWAL_PERIOD=0.

andy108369 commented 1 year ago

I found this address, while browsing through my logs, .. makes me wonder now if that account akash14pphss726thpwws3yc458hggufynm9x77l4l2u is somehow related to the depositor address

That account even got 20k AKT https://www.mintscan.io/akash/account/akash14pphss726thpwws3yc458hggufynm9x77l4l2u

(Click to expand) akash-testnet-3@stealth:~$ akash tx deployment create --depositor-account

``` akash-testnet-3@stealth:~$ akash tx deployment create --depositor-account akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7 --from second deploy.yaml Enter keyring passphrase: {"body":{"messages":[{"@type":"/akash.deployment.v1beta2.MsgCreateDeployment","id":{"owner":"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh","dseq":"131748"},"groups":[{"name":"westcoast","requirements":{"signed_by":{"all_of":[],"any_of":[]},"attributes":[{"key":"host","value":"akash"}]},"resources":[{"resources":{"cpu":{"units":{"val":"100"},"attributes":[]},"memory":{"quantity":{"val":"536870912"},"attributes":[]},"storage":[{"name":"default","quantity":{"val":"536870912"},"attributes":[]}],"endpoints":[{"kind":"SHARED_HTTP","sequence_number":0}]},"count":1,"price":{"denom":"uakt","amount":"1000.000000000000000000"}}]}],"version":"o8/63tCm60x0kMKl5gFR7tetyaEhtEx+h7L+p4RxJBg=","deposit":{"denom":"uakt","amount":"5000000"},"depositor":"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7"}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]} confirm transaction before signing and broadcasting [y/N]: y ``` `{"height":"131750","txhash":"8E34F8001B239DDDF9B7D3F8A1FF90AD68AD4F1438780A98A9131CC48817B21C","codespace":"","code":0,"data":"0A2F0A2D2F616B6173682E6465706C6F796D656E742E763162657461322E4D73674372656174654465706C6F796D656E74","raw_log":"[{\"events\":[{\"type\":\"akash.v1\",\"attributes\":[{\"key\":\"module\",\"value\":\"deployment\"},{\"key\":\"action\",\"value\":\"deployment-created\"},{\"key\":\"version\",\"value\":\"a3cffaded0a6eb4c7490c2a5e60151eed7adc9a121b44c7e87b2fea784712418\"},{\"key\":\"owner\",\"value\":\"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh\"},{\"key\":\"dseq\",\"value\":\"131748\"},{\"key\":\"module\",\"value\":\"market\"},{\"key\":\"action\",\"value\":\"order-created\"},{\"key\":\"owner\",\"value\":\"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh\"},{\"key\":\"dseq\",\"value\":\"131748\"},{\"key\":\"gseq\",\"value\":\"1\"},{\"key\":\"oseq\",\"value\":\"1\"}]},{\"type\":\"coin_received\",\"attributes\":[{\"key\":\"receiver\",\"value\":\"akash14pphss726thpwws3yc458hggufynm9x77l4l2u\"},{\"key\":\"amount\",\"value\":\"5000000uakt\"}]},{\"type\":\"coin_spent\",\"attributes\":[{\"key\":\"spender\",\"value\":\"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7\"},{\"key\":\"amount\",\"value\":\"5000000uakt\"}]},{\"type\":\"cosmos.authz.v1beta1.EventRevoke\",\"attributes\":[{\"key\":\"msg_type_url\",\"value\":\"\\\"/akash.deployment.v1beta2.MsgDepositDeployment\\\"\"},{\"key\":\"granter\",\"value\":\"\\\"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7\\\"\"},{\"key\":\"grantee\",\"value\":\"\\\"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh\\\"\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"/akash.deployment.v1beta2.MsgCreateDeployment\"},{\"key\":\"sender\",\"value\":\"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"akash14pphss726thpwws3yc458hggufynm9x77l4l2u\"},{\"key\":\"sender\",\"value\":\"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7\"},{\"key\":\"amount\",\"value\":\"5000000uakt\"}]}]}]","logs":[{"msg_index":0,"log":"","events":[{"type":"akash.v1","attributes":[{"key":"module","value":"deployment"},{"key":"action","value":"deployment-created"},{"key":"version","value":"a3cffaded0a6eb4c7490c2a5e60151eed7adc9a121b44c7e87b2fea784712418"},{"key":"owner","value":"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh"},{"key":"dseq","value":"131748"},{"key":"module","value":"market"},{"key":"action","value":"order-created"},{"key":"owner","value":"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh"},{"key":"dseq","value":"131748"},{"key":"gseq","value":"1"},{"key":"oseq","value":"1"}]},{"type":"coin_received","attributes":[{"key":"receiver","value":"akash14pphss726thpwws3yc458hggufynm9x77l4l2u"},{"key":"amount","value":"5000000uakt"}]},{"type":"coin_spent","attributes":[{"key":"spender","value":"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7"},{"key":"amount","value":"5000000uakt"}]},{"type":"cosmos.authz.v1beta1.EventRevoke","attributes":[{"key":"msg_type_url","value":"\"/akash.deployment.v1beta2.MsgDepositDeployment\""},{"key":"granter","value":"\"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7\""},{"key":"grantee","value":"\"akash1fedhgmq6mjv94p2lu06x2hq3el9l2x4pqvg0dh\""}]},{"type":"message","attributes":[{"key":"action","value":"/akash.deployment.v1beta2.MsgCreateDeployment"},{"key":"sender","value":"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7"}]},{"type":"transfer","attributes":[{"key":"recipient","value":"akash14pphss726thpwws3yc458hggufynm9x77l4l2u"},{"key":"sender","value":"akash1qvr9vwsae25rn8pk9m77s9fkwrxdc65r2s3tt7"},{"key":"amount","value":"5000000uakt"}]}]}],"info":"","gas_wanted":"200000","gas_used":"88242","tx":null,"timestamp":"","events":[{"type":"tx","attributes":[{"key":"ZmVl","value":"","index":true}]},{"type":"tx","attributes":[{"key":"YWNjX3NlcQ==","value":"YWthc2gxZmVkaGdtcTZtanY5NHAybHUwNngyaHEzZWw5bDJ4NHBxdmcwZGgvNA==","index":true}]},{"type":"tx","attributes":[{"key":"c2lnbmF0dXJl","value":"RWNJNVoxSVYxd2s5ZnIwdUt5MDl2V2dXTGZEQ1R3cTRQOHZkYUwzeFRFVVFXcXVsY3FvdGw5R01ubW51aTZacTFJT3hKZ0FKcHZqMm5Rb1BSbTNHTnc9PQ==","index":true}]},{"type":"message","attributes":[{"key":"YWN0aW9u","value":"L2FrYXNoLmRlcGxveW1lbnQudjFiZXRhMi5Nc2dDcmVhdGVEZXBsb3ltZW50","index":true}]},{"type":"cosmos.authz.v1beta1.EventRevoke","attributes":[{"key":"bXNnX3R5cGVfdXJs","value":"Ii9ha2FzaC5kZXBsb3ltZW50LnYxYmV0YTIuTXNnRGVwb3NpdERlcGxveW1lbnQi","index":true},{"key":"Z3JhbnRlcg==","value":"ImFrYXNoMXF2cjl2d3NhZTI1cm44cGs5bTc3czlma3dyeGRjNjVyMnMzdHQ3Ig==","index":true},{"key":"Z3JhbnRlZQ==","value":"ImFrYXNoMWZlZGhnbXE2bWp2OTRwMmx1MDZ4MmhxM2VsOWwyeDRwcXZnMGRoIg==","index":true}]},{"type":"akash.v1","attributes":[{"key":"bW9kdWxl","value":"ZGVwbG95bWVudA==","index":true},{"key":"YWN0aW9u","value":"ZGVwbG95bWVudC1jcmVhdGVk","index":true},{"key":"dmVyc2lvbg==","value":"YTNjZmZhZGVkMGE2ZWI0Yzc0OTBjMmE1ZTYwMTUxZWVkN2FkYzlhMTIxYjQ0YzdlODdiMmZlYTc4NDcxMjQxOA==","index":true},{"key":"b3duZXI=","value":"YWthc2gxZmVkaGdtcTZtanY5NHAybHUwNngyaHEzZWw5bDJ4NHBxdmcwZGg=","index":true},{"key":"ZHNlcQ==","value":"MTMxNzQ4","index":true}]},{"type":"akash.v1","attributes":[{"key":"bW9kdWxl","value":"bWFya2V0","index":true},{"key":"YWN0aW9u","value":"b3JkZXItY3JlYXRlZA==","index":true},{"key":"b3duZXI=","value":"YWthc2gxZmVkaGdtcTZtanY5NHAybHUwNngyaHEzZWw5bDJ4NHBxdmcwZGg=","index":true},{"key":"ZHNlcQ==","value":"MTMxNzQ4","index":true},{"key":"Z3NlcQ==","value":"MQ==","index":true},{"key":"b3NlcQ==","value":"MQ==","index":true}]},{"type":"coin_spent","attributes":[{"key":"c3BlbmRlcg==","value":"YWthc2gxcXZyOXZ3c2FlMjVybjhwazltNzdzOWZrd3J4ZGM2NXIyczN0dDc=","index":true},{"key":"YW1vdW50","value":"NTAwMDAwMHVha3Q=","index":true}]},{"type":"coin_received","attributes":[{"key":"cmVjZWl2ZXI=","value":"YWthc2gxNHBwaHNzNzI2dGhwd3dzM3ljNDU4aGdndWZ5bm05eDc3bDRsMnU=","index":true},{"key":"YW1vdW50","value":"NTAwMDAwMHVha3Q=","index":true}]},{"type":"transfer","attributes":[{"key":"cmVjaXBpZW50","value":"YWthc2gxNHBwaHNzNzI2dGhwd3dzM3ljNDU4aGdndWZ5bm05eDc3bDRsMnU=","index":true},{"key":"c2VuZGVy","value":"YWthc2gxcXZyOXZ3c2FlMjVybjhwazltNzdzOWZrd3J4ZGM2NXIyczN0dDc=","index":true},{"key":"YW1vdW50","value":"NTAwMDAwMHVha3Q=","index":true}]},{"type":"message","attributes":[{"key":"c2VuZGVy","value":"YWthc2gxcXZyOXZ3c2FlMjVybjhwazltNzdzOWZrd3J4ZGM2NXIyczN0dDc=","index":true}]}]}`

andy108369 commented 1 year ago

the quick reproducer for `akash14pphss726thpwws3yc458hggufynm9x77l4l2u` address

Looks like that address has something to do with how escrow is working.

deploy a multi-service SDL and make sure one of the services would never get ready / constantly cause K8s restarting it;
wait for about 7-8 minutes;
the lease will get automatically closed with MsgCloseBid TX containing the akash14pphss726thpwws3yc458hggufynm9x77l4l2u address (coin_spent message about 5 AKT deposit and few uakt being spent with that address as a spender, and MsgCloseBid message with that address as a sender) (example TX);
the deployment will switch into a paused state;

Deployment SDL: 1676-repro.yml

---
version: "2.0"

services:
  app:
    image: ubuntu:22.04
    command:
      - "sh"
      - "-c"
    args:
      - sleep infinity
  app2:
    image: ubuntu:22.04
    expose:
      - port: 80
        as: 80
        to:
          - global: true

profiles:
  compute:
    app:
      resources:
        cpu:
          units: 1
        memory:
          size: 512Mi
        storage:
          size: 512Mi
    app2:
      resources:
        cpu:
          units: 1
        memory:
          size: 512Mi
        storage:
          size: 512Mi
  placement:
    akash:
      attributes:
        host: akash
      signedBy:
        anyOf:
          - "akash1365yvmc4s7awdyj3n2sav7xfx76adc6dnmlx63"
      pricing:
        app:
          denom: uakt
          amount: 100000
        app2:
          denom: uakt
          amount: 100000

deployment:
  app:
    akash:
      profile: app
      count: 1
  app2:
    akash:
      profile: app2
      count: 1

chandadharap commented 1 year ago

Needs retesting after provider services upgrade as part of mainnet-4 @andy108369

andy108369 commented 1 year ago

Could be related to the situation when provider doesn't have enough resources and closes the deployment when reaching monitorMaxRetries => see https://github.com/akash-network/support/issues/14 for details.

andy108369 commented 1 year ago

As of akash14pphss726thpwws3yc458hggufynm9x77l4l2u address, that's the escrow address:

$ akash query account akash14pphss726thpwws3yc458hggufynm9x77l4l2u
'@type': /cosmos.auth.v1beta1.ModuleAccount
base_account:
  account_number: "6732"
  address: akash14pphss726thpwws3yc458hggufynm9x77l4l2u
  pub_key: null
  sequence: "0"
name: escrow       <<<<<<<<<<<<<<<<
permissions: []

net/mainnet$ cat genesis.json |jq -r '.app_state.escrow'
{
  "accounts": [],
  "payments": []
}

Ref. https://github.com/akash-network/node/tree/v0.20.0/x/escrow

akash-network / support