search

akash-network / support

Akash Support and Issue Tracking
5 stars 4 forks source link

service available replicas below target #33

Closed andy108369 closed 1 year ago

andy108369 commented 2 years ago

akash v0.16.4

I'm often seeing providers to hit service available replicas below target message despite the pod/svc/ing seem to be running well.

Related code path https://github.com/ovrclk/akash/blob/v0.16.4/provider/cluster/monitor.go#L185

provider logs

D[2022-06-28|13:12:52.338] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev/6194991/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=tezos cmp=deployment-monitor service=tezos available=0 target=1
root@node1:~# NS=$(akash provider show-cluster-ns --dseq 6194991 --owner akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev --provider akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0)

root@node1:~# kubectl -n lease get manifest $NS -o yaml
apiVersion: akash.network/v2beta1
kind: Manifest
metadata:
  creationTimestamp: "2022-06-06T01:14:19Z"
  generation: 2
  labels:
    akash.network: "true"
    akash.network/lease.id.dseq: "6194991"
    akash.network/lease.id.gseq: "1"
    akash.network/lease.id.oseq: "1"
    akash.network/lease.id.owner: akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev
    akash.network/lease.id.provider: akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
    akash.network/namespace: i8srn0jevt07jq3ittrgrg1q3ja3srij6t7efeftq3fi4
  name: i8srn0jevt07jq3ittrgrg1q3ja3srij6t7efeftq3fi4
  namespace: lease
  resourceVersion: "5343337"
  uid: 7b8f4027-c978-44d0-ae94-2f971a846a6e
spec:
  group:
    name: tezos
    services:
    - count: 1
      env:
      - SNAPSHOT_NETWORK=mainnet
      - SNAPSHOT_TYPE=rolling
      - SNAPSHOT_PATH=/var/run/tezos
      expose:
      - external_port: 80
        global: true
        hosts:
        - eu-central.nulldao.xyz
        http_options: {}
        port: 8732
        proto: TCP
      image: ghcr.io/sfdotstudio/tezos-quikn:latest
      name: tezos
      unit:
        cpu: 1000
        memory: "4294967296"
        storage:
        - name: default
          size: "10737418240"
  lease_id:
    dseq: "6194991"
    gseq: 1
    oseq: 1
    owner: akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev
    provider: akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0

root@node1:~# kubectl -n $NS get ing 
NAME                                           CLASS                 HOSTS                                          ADDRESS        PORTS   AGE
brgb45eveddpv99qqqbgosldf4.ingress.akash.pro   akash-ingress-class   brgb45eveddpv99qqqbgosldf4.ingress.akash.pro   65.108.6.185   80      22d
eu-central.nulldao.xyz                         akash-ingress-class   eu-central.nulldao.xyz                         65.108.6.185   80      22d

root@node1:~# kubectl -n $NS get svc
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
tezos   ClusterIP   10.233.48.214   <none>        80/TCP    22d
andy108369 commented 2 years ago

Interesting, once I bounced the akash-provider, that lease terminated due to deployment failed. closing lease., see below in the logs:

# kubectl -n akash-services logs $(kubectl -n akash-services get pods -l app=akash-provider --output jsonpath='{.items[-1].metadata.name}') | grep -Ev 'running check|check result' | grep 6489621
D[2022-06-28|14:57:56.716] found existing hostname                      module=provider-cluster cmp=service hostname=h0oufm2j6ldhj1v6hooc2nm2e0.ingress.akash.pro id=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
D[2022-06-28|14:57:56.716] found existing hostname                      module=provider-cluster cmp=service hostname=hkfdsh.fans id=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
I[2022-06-28|14:57:56.718] hostnames withheld                           module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cnt=0
D[2022-06-28|14:57:56.798] no services                                  module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=brwpdb
D[2022-06-28|14:57:56.802] no services                                  module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=brwpf
D[2022-06-28|14:57:56.818] no services                                  module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=dns
I[2022-06-28|14:57:56.820] declaring hostname                           module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service-name=cms external-port=80
I[2022-06-28|14:57:56.827] declaring hostname                           module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service-name=cms external-port=80
D[2022-06-28|14:57:56.829] deploy complete                              module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|14:58:07.136] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:18.480] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:31.328] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:49.678] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:56.744] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:11.938] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:21.488] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:33.548] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:50.719] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:01.556] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:14.627] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:30.558] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:36.478] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:50.019] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:58.803] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:12.285] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:23.590] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:28.275] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:36.042] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:54.820] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:08.357] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:17.871] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:24.385] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:35.412] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:48.164] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:04.027] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:14.691] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:24.546] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:34.115] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:44.495] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:52.101] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:03.612] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:10.769] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:28.467] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:47.203] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:56.890] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:01.079] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:08.390] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:18.091] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:23.101] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:36.299] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
E[2022-06-28|15:05:36.299] deployment failed.  closing lease.           module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
I[2022-06-28|15:05:36.319] bidding on lease closed                      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
D[2022-06-28|15:05:37.616] teardown request                             module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.616] shutting down                                module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
D[2022-06-28|15:05:37.616] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
D[2022-06-28|15:05:37.644] teardown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] shutting down                                module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] waiting on withdrawal                        module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] shutting down                                module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-withdrawal
D[2022-06-28|15:05:37.644] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-withdrawal
D[2022-06-28|15:05:37.644] waiting on dm.wg                             module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
I[2022-06-28|15:05:37.644] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] hostnames released                           module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] sending manager into channel                 module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
I[2022-06-28|15:05:37.644] manager done                                 module=provider-cluster cmp=service lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
D[2022-06-28|15:05:37.644] unreserving capacity                         module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
I[2022-06-28|15:05:37.644] attempting to removing reservation           module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
I[2022-06-28|15:05:37.644] removing reservation                         module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
I[2022-06-28|15:05:37.644] unreserve capacity complete                  module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
andy108369 commented 2 years ago

And that lease looked okay, i.e. did not get overdrawn, was running for less than 2 days:

$ PROVIDER=akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0; HEIGHT=$(akash query block | jq -r '.block.header.height'); akash query market lease list --height $HEIGHT --provider $PROVIDER --gseq 0 --oseq 0 --page 1 --limit 10000 --state active -o json | jq --argjson h $HEIGHT -r '["owner","dseq/gseq/oseq","rate","monthly","withdrawn","consumed","days"], (.leases[] | [(.lease.lease_id | .owner, (.dseq|tostring) + "/" + (.gseq|tostring) + "/" + (.oseq|tostring)), (.escrow_payment | (.rate.amount|tonumber), (.rate.amount|tonumber), (.withdrawn.amount|tonumber)), (($h-(.lease.created_at|tonumber))*(.escrow_payment.rate.amount|tonumber)/pow(10;6)), (($h-(.lease.created_at|tonumber))/((60/6)*60*24))]) | @csv' | awk -F ',' '{if (NR==1) {$1=$1; printf $0"\n"} else {block_time=6; rate_akt=(($4*((60/block_time)*60*24*30.436875))/10^6); $4=rate_akt; withdrawn_akt=($5/10^6); $5=withdrawn_akt; $6; $7; print $0}}' | column -t
"owner"                                         "dseq/gseq/oseq"  "rate"   "monthly"  "withdrawn"  "consumed"          "days"
"akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua"  "6489621/1/1"     105      46.0206    1.47473      2.583945            1.7089583333333334
...
tidrolpolelsef commented 2 years ago

This can happen when the docker container is unavailable, or has an error in it. Is it possible ghcr.io/sfdotstudio/tezos-quikn:latest crashes at some point?

andy108369 commented 2 years ago

Could be, I've entirely forgot to check the pod itself. Going to try kubectl -n $NS get pods next time. Thanks for the pointer.

andy108369 commented 1 year ago

haven't seen these messages for quite some time.

andy108369 commented 7 months ago

FWIW, the service available replicas below target followed by deployment failed. closing lease. message relates to the monitorMaxRetries counter (ref https://github.com/akash-network/support/issues/14 ) ; that means the pods in the K8s ReplicaSet were unable to get into the READY state.

More details:

In Akash deployments, the process involves converting the deployment into the Kubernetes (K8s) Deployment kind, except in the case of persistent storage deployments, which use the StatefulSet kind. Here's how the 'Deployment' kind functions:

  1. Kubernetes creates a ReplicaSet based on the Deployment (the amount of replicas is typically set to 1);
  2. If any issues arise during deployment, Kubernetes will not achieve the target replica availability. Consequently, the Akash Provider logs the status as service available replicas below target. This also implies that the deployment attempt count will continue to increase until it reaches 40. At that point, the akash-provider automatically closes the lease.

When the (monitorMaxRetries counter in the code or attempt counter in the logs) reaches 40, the next message you see is:

E[2022-12-08|16:46:15.299] deployment failed.  closing lease.           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/8816000/1/1/akash1vky0uh4wayh9npd74uqesglpaxwymynnspf6a4 manifest-group=dcloud cmp=deployment-monitor

I.e. the incrementing attempt counter along with these lines indicate there is some issue due to which the K8s cannot reach the active replicas (i.e. deploy the deployment) 1/1 (DESIRED/READY)

service available replicas below target
deployment failed.  closing lease.

image