Open andy108369 opened 1 year ago
interesting.. there is absolutely no resources related to uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche
namespace (checked all CRDs, k8s resources):
akash@akash0001:~$ provider-services show-cluster-ns --provider akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d --dseq 9562948 --owner akash17582ja9fw6k0m0gf9tm9g64w2ks6c7aqdntyst
uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche
akash@akash0001:~$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -A | grep -iE 'uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche|9562948|akash17582ja9fw6k0m0gf9tm9g64w2ks6c7aqdntyst'
and yet, the provider keeps on reporting every few seconds:
CRD manifest not found cmp=provider client=kube lease-ns=uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche
(have bounced the hostname-operator too)
interesting.. there is absolutely no resources related to
uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche
namespace (checked all CRDs, k8s resources):akash@akash0001:~$ provider-services show-cluster-ns --provider akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d --dseq 9562948 --owner akash17582ja9fw6k0m0gf9tm9g64w2ks6c7aqdntyst uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche
akash@akash0001:~$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -A | grep -iE 'uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche|9562948|akash17582ja9fw6k0m0gf9tm9g64w2ks6c7aqdntyst'
and yet, the provider keeps on reporting every few seconds:
CRD manifest not found cmp=provider client=kube lease-ns=uvtrenbt6bch737mf10geqjbl3dmhb85pml2pkugogche
(have bounced the hostname-operator too)
oh, ignore it. this like just the owner is trying to monitor it, e.g.
provider-services lease-status --provider akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d --dseq 9562948 --from akash17582ja9fw6k0m0gf9tm9g64w2ks6c7aqdntyst
I've tried running that using my key:
provider-services lease-status --provider akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d --dseq 9562948 --from default
and it produced this line, even I don't have any deployments on that provider:
I[2023-02-07|14:48:48.452] CRD manifest not found cmp=provider client=kube lease-ns=i7t9u9ie3mnj06o9ovcsoq5fubpuj5k1mmfqr9p1ts9ga
ns (namespace) can be derived manually:
# provider-services show-cluster-ns --provider akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d --dseq 9562949 --owner akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
i7t9u9ie3mnj06o9ovcsoq5fubpuj5k1mmfqr9p1ts9ga
I've tmate
-ed to his provider, have tried deploying my app there (dseq 9677551) and restarting akash-provider after that => no issues. :man_shrugging:
praetor-based provider-services:
0.1.0
provider address:akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d
A provider owner
SGC#3172
reported thej0asgbmq1a6p4s7ii0tlvuoco.ingress.dcnorse.ddns.net
ingress host resource (for the DSEQ9562948
) started to return 404 and other deployments disappeared from his k8s cluster.What's interesting is that I can see in the provider logs
providerleasedips CRD does not exist
messages, which is part of the praetor-based provider starting script check (you can see it below). Which means that there is some condition where K8s cluster does not seem to be fully initialized or so.. which is causing the provider think it's got no leases (lease query failed ... err=(MISSING)
leading to the lease removal; though, it does not close them on the blockchain [bid, lease, order, deployment staying in the active/open state]).leases are still active from the blockchain point of view
Provider logs
See more complete logs for the past
7 days
here (90Mi
) => https://transfer.sh/Fg7vTc/logs.txtAfter about
3 hours
from the provider start, the following lines started to appear in the logs:grep MISSING logs.txt
Additional information