Normalnoise commented 9 months ago

We have Three new providers that need to be audited.

Provider Name: provider.provider01.nebulablock.com
- Provider Address: provider.provider01.nebulablock.com
Provider Name: provider.provider02.nebulablock.com
- Provider Address: provider.provider02.nebulablock.com
Provider Name: provider.provider04.nebulablock.com
- Provider Address: provider.provider04.nebulablock.com
Objectives
- Confirm the security of the providers
- Assess performance metrics
- Validate data integrity

andy108369 commented 9 months ago

getting no bids from the provider. asked Leoj to verify his provider logs for dseq 12859110 & 12859083

Normalnoise commented 9 months ago

Defaulted container "provider" out of: provider, init (init)
I[2023-09-18|09:59:44.216] order detected                               module=bidengine-service cmp=provider order=order/akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
I[2023-09-18|09:59:44.348] group fetched                                module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
I[2023-09-18|09:59:44.348] requesting reservation                       module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
D[2023-09-18|09:59:44.348] reservation requested                        module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1 resources[{resource:{id:1,cpu:{units:{val:1000}},memory:{size:{val:1073741824}},storage:[{name:default,size:{val:1073741824}}],gpu:{units:{val:0}},endpoints:[{kind:1,sequence_number:0},{sequence_number:0}]},count:1,price:{denom:uakt,amount:1000000.000000000000000000}}]=(MISSING)
I[2023-09-18|09:59:44.348] Reservation fulfilled                        module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
D[2023-09-18|09:59:44.771] submitting fulfillment                       module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1 price=4.602194000000000000uakt
E[2023-09-18|09:59:44.845] bid failed                                   module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1 err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: 1053108uakt is smaller than 5000000uakt: insufficient funds [cosmos/cosmos-sdk@v0.45.16/x/bank/keeper/send.go:186] With gas wanted: '0' and gas used: '77834' : unknown request"
I[2023-09-18|09:59:44.845] shutting down                                module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
D[2023-09-18|09:59:44.845] unreserving reservation                      module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
D[2023-09-18|09:59:44.846] unreserving capacity                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
I[2023-09-18|09:59:44.846] attempting to removing reservation           module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
I[2023-09-18|09:59:44.846] removing reservation                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
I[2023-09-18|09:59:44.846] unreserve capacity complete                  module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1
I[2023-09-18|09:59:50.476] order detected                               module=bidengine-service cmp=provider order=order/akash183ydydk5ndz9vu2ujld5tm3a69h7zzwts88g40/12859110/1/1
I[2023-09-18|09:59:50.547] group fetched                                module=bidengine-order cmp=provider order=akash183ydydk5ndz9vu2ujld5tm3a69h7zzwts88g40/12859110/1/1
D[2023-09-18|09:59:50.547] unable to fulfill: incompatible provider attributes module=bidengine-order cmp=provider order=akash183ydydk5ndz9vu2ujld5tm3a69h7zzwts88g40/12859110/1/1
D[2023-09-18|09:59:50.547] declined to bid                              module=bidengine-order cmp=provider order=akash183ydydk5ndz9vu2ujld5tm3a69h7zzwts88g40/12859110/1/1
I[2023-09-18|09:59:50.547] shutting down                                module=bidengine-order cmp=provider order=akash183ydydk5ndz9vu2ujld5tm3a69h7zzwts88g40/12859110/1/1

I have enough funds to create the bids, can you help send task to me again? @andy108369

Normalnoise commented 9 months ago

these providers can be tested

andy108369 commented 9 months ago

Online providers

as of 21 Sept 2023

I can see there are 3 online providers only, though 5 are registered on the chain.

By "online", I mean they respond over :8443/status

"https://provider.provider01.nebulablock.com:8443","akash1c07l4rms9aapqhwvjnk58eqtp268hvc68swwy4"   << OFFLINE
"https://provider.provider01.nebulablock.com:8443","akash1mwyy7uhu0e8vuhc6xhkcgrg6s2tcfqcanm3xhf"
"https://provider.provider02.nebulablock.com:8443","akash14dny4x397a33s6e93lyj03nykvjgtttgy252wa"
"https://provider.provider03.nebulablock.com:8443","akash1luunluzs2sgjerpneydm3nqewsh5guavdt65da"  << OFFLINE
"https://provider.provider04.nebulablock.com:8443","akash1cuk7gtj6sterew7d0dsa7e3j2ppudrhe7e82fl"

To figure which provider.provider01.nebulablock.com is alive:

$ curl -s -k https://provider.provider01.nebulablock.com:8443/status  |jq -r .address
akash1mwyy7uhu0e8vuhc6xhkcgrg6s2tcfqcanm3xhf

Bidding

I can see all online providers are bidding to dseq 12902680:

[12902680--1]$ akash_accept 
    rate    monthly usd dseq/gseq/oseq  provider                    host
0>  21.79   9.36    $7.96   12902680/1/1    akash14dny4x397a33s6e93lyj03nykvjgtttgy252wa    provider.provider02.nebulablock.com:8443    
1>  16.62   7.13    $6.06   12902680/1/1    akash1cuk7gtj6sterew7d0dsa7e3j2ppudrhe7e82fl    provider.provider04.nebulablock.com:8443    
2>  21.79   9.36    $7.96   12902680/1/1    akash1mwyy7uhu0e8vuhc6xhkcgrg6s2tcfqcanm3xhf    provider.provider01.nebulablock.com:8443    
Choose your bid from the list [2]: 
INFO: Accepting the bid offered by akash1mwyy7uhu0e8vuhc6xhkcgrg6s2tcfqcanm3xhf provider for 12902680/1/1 deployment
INFO: Broadcasting 'provider-services market lease create -y' transaction...
INFO: Waiting for the TX A5FF5F8E322F99D356CD365A328C19E80C313017772CC4E0BCBBB258F64FA55D to get processed by the Akash network

Issues

provider01.nebulablock.com

Seems like a FW issue

Can't access the nodePorts nor the http/https ports

[12902680-1-1]$ akash_status 
Detected provider for 12902680/1/1: akash1mwyy7uhu0e8vuhc6xhkcgrg6s2tcfqcanm3xhf
{
  "services": {
    "app": {
      "name": "app",
      "available": 1,
      "total": 1,
      "uris": [
        "1qfdo9t62p9cl7qfnj3n57oaes.ingress.provider01.nebulablock.com"
      ],
      "observed_generation": 1,
      "replicas": 1,
      "updated_replicas": 1,
      "ready_replicas": 1,
      "available_replicas": 1
    }
  },
  "forwarded_ports": {
    "app": [
      {
        "host": "provider.provider01.nebulablock.com",
        "port": 22,
        "externalPort": 32596,
        "proto": "TCP",
        "name": "app"
      }
    ]
  },
  "ips": null
}

accessing ingresses is taking long time and then times out:


$ curl -LI 1qfdo9t62p9cl7qfnj3n57oaes.ingress.provider01.nebulablock.com
HTTP/1.1 504 Gateway Time-out
Date: Thu, 21 Sep 2023 11:03:21 GMT
Content-Type: text/html
Content-Length: 160
Connection: keep-alive

$ curl -k -LI https://1qfdo9t62p9cl7qfnj3n57oaes.ingress.provider01.nebulablock.com HTTP/2 504 date: Thu, 21 Sep 2023 11:04:48 GMT content-type: text/html content-length: 160 strict-transport-security: max-age=15724800; includeSubDomains


### stopped bidding

provider01 stopped biding on the following dseqs `12902728`, `12902754` as I continued to test provider02 and provider04

## provider02.nebulablock.com

80,443 ports are OK
but nodePorts aren't, most likely a FW issue as well.

## provider04.nebulablock.com

Same FW issue as with provider01.nebulablock.com

[12902754-1-1-app]$ curl b9ou9v16blbh97f9eh07bcss00.ingress.provider04.nebulablock.com

504 Gateway Time-out

nginx

[12902754-1-1]$ akash_status Detected provider for 12902754/1/1: akash1cuk7gtj6sterew7d0dsa7e3j2ppudrhe7e82fl { "services": { "app": { "name": "app", "available": 1, "total": 1, "uris": [ "b9ou9v16blbh97f9eh07bcss00.ingress.provider04.nebulablock.com" ], "observed_generation": 1, "replicas": 1, "updated_replicas": 1, "ready_replicas": 1, "available_replicas": 1 } }, "forwarded_ports": { "app": [ { "host": "provider.provider04.nebulablock.com", "port": 22, "externalPort": 31183, "proto": "TCP", "name": "app" } ] }, "ips": null }

andy108369 commented 9 months ago

providers have low balances

Providers barely have any balance, hence they aren't bidding which can also be confirmed by the error in the logs seen on the provider side:

E[2023-09-18|09:59:44.845] bid failed                                   module=bidengine-order cmp=provider order=akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys/12859110/1/1 err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: 1053108uakt is smaller than 5000000uakt: insufficient funds [cosmos/cosmos-sdk@v0.45.16/x/bank/keeper/send.go:186] With gas wanted: '0' and gas used: '77834' : unknown request"

$ provider-services query bank balances akash1mwyy7uhu0e8vuhc6xhkcgrg6s2tcfqcanm3xhf -o json | jq -r '.balances[] | select(.denom == "uakt") | .amount|tonumber / pow(10;6)'
0.999379

$ provider-services query bank balances akash14dny4x397a33s6e93lyj03nykvjgtttgy252wa -o json | jq -r '.balances[] | select(.denom == "uakt") | .amount|tonumber / pow(10;6)'
5.993776

$ provider-services query bank balances akash1cuk7gtj6sterew7d0dsa7e3j2ppudrhe7e82fl -o json | jq -r '.balances[] | select(.denom == "uakt") | .amount|tonumber / pow(10;6)'
1.126194

andy108369 commented 9 months ago

based on the available CPU's, say if 1 deployment equals 1 CPU (on average), providers should have the following amount of AKTs on their balances:

434*5 = 2170 AKT for provider.provider01.nebulablock.com
158*5 = 790 AKT for provider.provider02.nebulablock.com
62*5 = 310 AKT for provider.provider04.nebulablock.com

However, for the beginning provider can top-up each provider with about 250 AKT and monitor. As it will be earning AKT automatically with more deployments to come.

Providers resource state as of 21 Sept 2023:

$ for i in 01 02 04; do echo === provider_info.sh provider.provider${i}.nebulablock.com ===; provider_info.sh provider.provider${i}.nebulablock.com; done
=== provider_info.sh provider.provider01.nebulablock.com ===
type       cpu     gpu  ram                 ephemeral           persistent
used       2       0    4                   30                  0
pending    0       0    0                   0                   0
available  432.35  12   4575.843891143799   34550.766135375015  0
node       125.9   4    751.6058158874512   812.7106885313988   N/A
node       127.7   4    755.4922904968262   842.7106885313988   N/A
node       19.9    1    188.18901824951172  1648.9856387497857  N/A
node       31.9    1    995.4250450134277   9978.01118724607    N/A
node       63.9    1    881.6513061523438   10643.850002977066  N/A
node       63.05   1    1003.4804153442383  10624.497929339297  N/A

=== provider_info.sh provider.provider02.nebulablock.com ===
type       cpu     gpu  ram                 ephemeral           persistent
used       0       0    0                   0                   0
pending    0       0    0                   0                   0
available  158.65  10   3640.367847442627   65247.58544723969   0
node       47.7    4    881.5189819335938   21749.19517883472   N/A
node       47.9    4    755.6440238952637   21749.19517883472   N/A
node       63.05   2    2003.2048416137695  21749.195089570247  N/A

=== provider_info.sh provider.provider04.nebulablock.com ===
type       cpu    gpu  ram                ephemeral          persistent
used       2      0    4                  30                 0
pending    0      0    0                  0                  0
available  60.85  1    991.0471458435059  6356.820292008109  0
node       60.85  1    991.0471458435059  6356.820292008109  N/A

andy108369 commented 9 months ago

Summary

Top-up the provider balances:

make sure to top-up your providers with enough AKTs (say 100 or 250.. the higher the better based on the above calculations);

Correct the FW rules:

review K8s firewall rules: https://docs.akash.network/providers/build-a-cloud-provider/kubernetes-cluster-for-akash-providers/step-9-review-firewall-policies
review Akash firewall rules https://docs.akash.network/providers/build-a-cloud-provider/akash-cloud-provider-build-with-helm-charts/step-9-firewall-rule-review

Normalnoise commented 9 months ago

@andy108369 DSEQ: 12912700 http://krsheaokcp95h3l2r0hk9u6je8.ingress.provider02.nebulablock.com/

it is available for me, how to check nodePorts is ok?

Normalnoise commented 9 months ago

@andy108369 after the adjust the config, I have deploy the hell0-world to the three providers:

all of them are available, can you check it again?

DSEQ
12914101
g11ub47m5lfbv76rmgo22ctees.ingress.provider01.nebulablock.com

DSEQ
12914001
ptigalses9fu7atqkhrmsktnik.ingress.provider04.nebulablock.com

DSEQ
12912700
krsheaokcp95h3l2r0hk9u6je8.ingress.provider02.nebulablock.com

Normalnoise commented 9 months ago

Because I don't enough funds to deploy more applications, if you want to let me create the bid, please let me know, I will release the current jobs

andy108369 commented 9 months ago

Discussed in Discord (DM)

ingresses are good now
GPU deployments are working!
K8s node port range needs to be allowed and then I can sign the providers ( node ports https://docs.akash.network/providers/build-a-cloud-provider/akash-cloud-provider-build-with-helm-charts/step-9-firewall-rule-review )

andy108369 commented 9 months ago

Just for the record:

andy108369 commented 8 months ago

checking with Leoj for the updates

arno01 commented 8 months ago

update:

andy108369 commented 8 months ago

Closing for now until further audit request is placed.

akash-network / support

Audit for NebulaBlock Providers #123

Online providers

Bidding

Issues

provider01.nebulablock.com

Seems like a FW issue

504 Gateway Time-out

providers have low balances

Summary