flanksource / mission-control

Other
10 stars 2 forks source link

Mission control crashlooping after 0.1.141 chart deployment #1164

Closed dabasvibhor closed 1 month ago

dabasvibhor commented 1 month ago
{"level":"info","ts":"2024-07-15T06:53:14.557101505Z","msg":"Initialized DB: mission-control-cloud-sql-proxy.mission-control.svc.cluster.local (932 MB)"}                                                                                                                                                                                         
{"level":"info","ts":"2024-07-15T06:53:14.741050361Z","msg":"Migrating database mission_control"}                                                                                                                                                                                                                                                 
{"level":"info","ts":"2024-07-15T06:53:15.913905542Z","msg":"Applied 4 changes"}                                                                                                                                                                                                                                                                  
{"level":"error","ts":"2024-07-15T06:53:15.976492615Z","msg":"failed to set statement timeout for role \"mission-control-wrapper-sa@bcb-group-hub.iam\": ERROR: syntax error at or near \"-\" (SQLSTATE 42601)"}                                                                                                                                  
{"level":"info","ts":"2024-07-15T06:53:15.982011013Z","msg":"System user ID: 018df49c-60e7-1566-94ad-7eaab74fc968"}                                                                                                                                                                                                                               
{"level":"info","ts":"2024-07-15T06:53:15.982659876Z","msg":"Loading properties from mission-control.properties"}                                                                                                                                                                                                                                 
{"level":"info","ts":"2024-07-15T06:53:15.982707589Z","msg":"incidents.disable(local)=true"}                                                                                                                                                                                                                                                      
{"level":"info","ts":"2024-07-15T06:53:15.982712222Z","msg":"logs.disable(local)=true"}                                                                                                                                                                                                                                                           
W0715 06:53:16.001400       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:53:16.001438       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
{"level":"info","ts":"2024-07-15T06:53:16.062450416Z","msg":"Listening on :8080"}                                                                                                                                                                                                                                                                 
⇨ http server started on :8080                                                                                                                                                                                                                                                                                                                    
{"level":"info","ts":"2024-07-15T06:53:16.109827973Z","msg":"Cleaning up jobs every 1m0s"}                                                                                                                                                                                                                                                        
W0715 06:53:17.379716       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:53:17.379782       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          

2024/07/15 06:53:18 logger.go:175: /go/pkg/mod/github.com/flanksource/duty@v1.0.545/query/config_cache.go:45 SLOW SQL >= 1s                                                                                                                                                                                                                       
[1866.506ms] [rows:9104] SELECT * FROM "config_items" WHERE deleted_at is NULL AND agent_id = '00000000-0000-0000-0000-000000000000' OR agent_id IS NULL                                                                                                                                                                                          
W0715 06:53:20.265723       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:53:20.265786       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
W0715 06:53:26.175759       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:53:26.175806       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
W0715 06:53:37.297858       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:53:37.297901       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
{"time":"2024-07-15T06:53:44.924897043Z","id":"179c223c3ab89f15f903cf2fed65770d","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/kratos/sessions/whoami","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","status":200,"error":"","la
{"time":"2024-07-15T06:53:45.142501632Z","id":"d5a708586f32fde801e2abbfa87c7072","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/kratos/sessions/whoami","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","status":200,"error":"","la
{"time":"2024-07-15T06:53:45.168274787Z","id":"34b9fcd46e96638f8c2492b5da1043c9","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/properties","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","status":200,"error":"","latency":22836
{"time":"2024-07-15T06:53:45.33840574Z","id":"d8c713394f80e043367c1d92c24e5ed8","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/auth/whoami","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","status":200,"error":"","latency":19333
{"time":"2024-07-15T06:53:45.564096013Z","id":"099c8873faed2411ca219dcd1def22c4","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/db/people_roles?id=in.(766b60fc-5157-4795-8353-2f42ea1bd32c)","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safa
{"time":"2024-07-15T06:53:46.00270666Z","id":"55eb474d8fda2c185cf1c1e4170c20b3","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/canary/api/topology?sortBy=status&sortOrder=desc","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","s
{"time":"2024-07-15T06:53:46.222483427Z","id":"a6b09c3180dea473f06814609506f1a9","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/db/agents?select=id%2Cname%2Cdescription","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","status":
{"time":"2024-07-15T06:53:47.467388916Z","id":"234de62aa737c7295b81693633acd37b","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/db/component_labels","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","status":200,"error":"","laten
{"time":"2024-07-15T06:53:54.545892388Z","id":"c15327219d7cfd24f6c48b2e1cec1e6d","remote_ip":"10.192.17.4","host":"mission-control:8080","method":"GET","uri":"/db/topologies_with_status?select=*%2Ccreated_by(id%2Cname%2Cavatar)%2Cagent%3Aagent_id(id%2Cname%2Cdescription)&deleted_at=is.null&order=created_at.desc&limit=100","user_agent":"
W0715 06:54:00.163064       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:54:00.163109       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
{"time":"2024-07-15T06:54:09.681202365Z","id":"9941e9212ce4bc4af9c0a0d35eb53960","remote_ip":"10.192.16.7","host":"mission-control:8080","method":"GET","uri":"/db/people_roles?id=in.(766b60fc-5157-4795-8353-2f42ea1bd32c)","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safa
{"time":"2024-07-15T06:54:09.681512177Z","id":"f89df337f68c41792a462a067443fd41","remote_ip":"10.192.16.7","host":"mission-control:8080","method":"GET","uri":"/db/topologies_with_status?select=*%2Ccreated_by(id%2Cname%2Cavatar)%2Cagent%3Aagent_id(id%2Cname%2Cdescription)&deleted_at=is.null&order=created_at.desc&limit=100","user_agent":"
W0715 06:54:41.173237       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:54:41.173272       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
W0715 06:55:14.527236       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                                                          
E0715 06:55:14.527287       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.Connection: failed to list *v1.Connection: json: cannot unmarshal object into Go struct field HTTPConnection.items.spec.prometheus.url of type string                                                          
{"level":"panic","ts":"2024-07-15T06:55:15.987458262Z","msg":"error running manager: failed to wait for connection caches to sync: timed out waiting for cache to be synced for Kind *v1.Connection"}                                                                                                                                             
panic: error running manager: failed to wait for connection caches to sync: timed out waiting for cache to be synced for Kind *v1.Connection                                                                                                                                                                                                      

goroutine 178 [running]:                                                                                                                                                                                                                                                                                                                          
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x0?, 0x0?, {0x0?, 0x0?, 0xc00a194500?})                                                                                                                                                                                                                                                         
    /go/pkg/mod/go.uber.org/zap@v1.26.0/zapcore/entry.go:196 +0x54                                                                                                                                                                                                                                                                                
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00a0ffe10, {0x0, 0x0, 0x0})                                                                                                                                                                                                                                                                      
    /go/pkg/mod/go.uber.org/zap@v1.26.0/zapcore/entry.go:262 +0x24e                                                                                                                                                                                                                                                                               
go.uber.org/zap.(*SugaredLogger).log(0xc0005054c8, 0x4, {0x9ef7fa0?, 0xc00a19b990?}, {0xc00a19b990?, 0x0?, 0xc000ad1860?}, {0x0, 0x0, 0x0})                                                                                                                                                                                                       
    /go/pkg/mod/go.uber.org/zap@v1.26.0/sugar.go:316 +0xec                                                                                                                                                                                                                                                                                        
go.uber.org/zap.(*SugaredLogger).Panicf(...)                                                                                                                                                                                                                                                                                                      
    /go/pkg/mod/go.uber.org/zap@v1.26.0/sugar.go:202                                                                                                                                                                                                                                                                                              
github.com/flanksource/commons/logger.ZapLogger.Fatalf(...)                                                                                                                                                                                                                                                                                       
    /go/pkg/mod/github.com/flanksource/commons@v1.24.2/logger/zap.go:163