yunionio / cloudpods

A cloud-native open-source unified multi-cloud and hybrid-cloud platform. 开源、云原生的多云管理及混合云融合平台
https://www.cloudpods.org
Apache License 2.0
2.56k stars 520 forks source link

[求助/Help] 组件POD:default-vpcagent 无法启动 #21236

Open nashzosi opened 4 days ago

nashzosi commented 4 days ago

3.11.6版本arm架构服务器,使用上没感觉有什么影响,就是这个pod异常,请求协助排查。 [error 2024-09-13 15:43:46 auth.(*authManager).startRefreshRevokeTokens(auth.go:193)] refreshRevokeTokens: No valid admin token credential 经过测试正常认证

image image

[info 240913 15:43:46 options.parseOptions(options.go:336)] Use configuration file: /etc/yunion/vpcagent.conf [info 240913 15:43:46 options.parseOptions(options.go:359)] Set log level to "debug" [error 2024-09-13 15:43:46 auth.(authManager).startRefreshRevokeTokens(auth.go:193)] refreshRevokeTokens: No valid admin token credential [info 2024-09-13 15:43:50 service.StartService.func1(service.go:59)] auth finished ok [debug 2024-09-13 15:43:50 policy.(SPolicyManager).init(policy.go:148)] DefaultPolicyFetcher: 654cf0 RemotePolicyFetcher: 654cf0 [info 2024-09-13 15:43:50 policy.(SPolicyManager).init(policy.go:160)] policy fetch worker count 1 [info 2024-09-13 15:43:50 consts.SetNonDefaultDomainProjects(consts.go:109)] set non_default_domain_projects to false [debug 2024-09-13 15:43:50 syncman.(SSyncManager).SyncOnce(sync.go:80)]AuthManager: SyncOnce isFirst false isTimeout false [info 2024-09-13 15:43:50 options.StartOptionManagerWithSessionDriver(manager.go:68)] OptionManager start to fetch service configs with interval 30m0s ... [info 2024-09-13 15:43:50 watcher.(SInformerSyncManager).startWatcher(watcher.go:83)]EndpointChangeManager: Start resource informer watcher for endpoint [debug 2024-09-13 15:43:50 etcd.(SEtcdClient).Unwatch(etcd.go:369)] prefix / not watched!! [info 2024-09-13 15:43:50 informer.(EtcdBackendForClient).StartClientWatch(etcd_client.go:84)] /onecloud/informer watched [info 2024-09-13 15:43:50 informer.NewWatchManagerBySessionBg.func1(watcher.go:51)] callback with watchMan success. [info 2024-09-13 15:43:50 options.optionsEquals(manager.go:120)] Options added: {"api_server":"https://10.51.0.60","dns_domain":"cloud.onecloud.io","dns_server":"10.51.0.60"} [debug 2024-09-13 15:43:50 options.OnBaseOptionsChange(changes.go:63)] api_server changed from to https://10.51.0.60 [info 2024-09-13 15:43:50 app.InitApp(app.go:32)] RequestWorkerCount: 8 [info 2024-09-13 15:43:50 watcher.(SInformerSyncManager).startWatcher(watcher.go:83)]ServiceConfigManager: Start resource informer watcher for service [info 2024-09-13 15:43:50 appsrv.NewApplication(appsrv.go:121)] App hostId: 2uR0W2xqa5gLOALX_MZyghixZos= (vpcagent,default-vpcagent-5d56946c9f-qqflc,10.40.202.30) 2024/09/13 15:43:50 Allow hosts [] [info 2024-09-13 15:43:50 appsrv.(Application).SetDefaultTimeout(appsrv.go:137)] adjust application default timeout to 60.000000 seconds [info 2024-09-13 15:43:50 app.ServeForeverExtended(app.go:60)] Start listen on https://0.0.0.0:30932, isMaster: true [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch vpcs with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","provider":"OneCloud","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:50 etcd.(SEtcdClient).Unwatch(etcd.go:369)] prefix / not watched!! [info 2024-09-13 15:43:50 informer.(EtcdBackendForClient).StartClientWatch(etcd_client.go:84)] /onecloud/informer watched [info 2024-09-13 15:43:50 informer.NewWatchManagerBySessionBg.func1(watcher.go:51)] callback with watchMan success. [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch wires with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","provider":"OneCloud","scope":"system","show_emulated":true,"system":true} [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch networks with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","provider":"OneCloud","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch servers with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","provider":"OneCloud","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch hosts with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","provider":"OneCloud","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch secgroups with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch secgrouprules with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:50 apihelper.GetModels(reflect.go:106)] fetch guestnetworks with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch guestsecgroups with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch eips with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch networkaddresses with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch dns_zones with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch dnsrecords with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch route_tables with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch groupguests with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch groupnetworks with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch instancegroups with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch loadbalancernetworks with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch loadbalancerlisteners with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [debug 2024-09-13 15:43:51 apihelper.GetModels(reflect.go:106)] fetch loadbalanceracls with params {"admin":true,"cloud_env":"onpremise","details":false,"filter.0":"manager_id.isnullorempty()","limit":1024,"offset":0,"order":"asc","order_by.0":"updated_at","order_by.1":"created_at","order_by.2":"id","scope":"system","show_emulated":false,"system":true} [error 2024-09-13 15:43:51 models.Networks.joinNetworkAddresses(modelset.go:449)] cannot find network id 35a07c04-1f95-44ff-857a-26dcb1b1a6b9 of network address 762a9e2a-0884-4764-8062-4767f5aa9f3b [error 2024-09-13 15:43:51 models.Guestnetworks.joinNetworkAddresses(modelset.go:573)] cannot find guestnetwork row id 243 of network address 82a70d0b-7079-4fcd-89a7-0bbb931f2ad0 [info 2024-09-13 15:43:51 apihelper.(APIHelper).doSync.func1(apihelper.go:154)] sync data done, changed: false, elapsed: 541.852396ms [info 2024-09-13 15:43:51 apihelper.(*APIHelper).Start.func1(apihelper.go:88)] apihelper: bye panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x180 pc=0x1ddd4d4]

goroutine 232 [running]: yunion.io/x/onecloud/pkg/vpcagent/models.Guestnetworks.joinNetworkAddresses(0x2049520?, 0x4000918000?) /root/go/src/yunion.io/x/onecloud/pkg/vpcagent/models/modelset.go:576 +0x2b4 yunion.io/x/onecloud/pkg/vpcagent/models.(ModelSets).join(0x4000bbf0e0) /root/go/src/yunion.io/x/onecloud/pkg/vpcagent/models/modelsets.go:280 +0x101c yunion.io/x/onecloud/pkg/vpcagent/models.(ModelSets).ApplyUpdates(0x4000bbf0e0, {0x2ca7400?, 0x4000bbf180?}) /root/go/src/yunion.io/x/onecloud/pkg/vpcagent/models/modelsets.go:228 +0x28c yunion.io/x/onecloud/pkg/apihelper.SyncModelSets({0x2ca7400, 0x4000bbf0e0}, 0x40012c8390?, 0x40012d0300) /root/go/src/yunion.io/x/onecloud/pkg/apihelper/interface.go:88 +0xd0 yunion.io/x/onecloud/pkg/apihelper.(APIHelper).doSync(0x4000635fb0, {0x2ca5040, 0x40012c8390}) /root/go/src/yunion.io/x/onecloud/pkg/apihelper/apihelper.go:160 +0xbc yunion.io/x/onecloud/pkg/apihelper.(APIHelper).run(0x4000635fb0, {0x2ca5040, 0x40012c8390}) /root/go/src/yunion.io/x/onecloud/pkg/apihelper/apihelper.go:136 +0x2c yunion.io/x/onecloud/pkg/apihelper.(APIHelper).Start(0x4000635fb0, {0x2ca5040, 0x40012c8390}, 0x78ed4?, {0x4000b97040?, 0x0?}) /root/go/src/yunion.io/x/onecloud/pkg/apihelper/apihelper.go:97 +0x88 created by yunion.io/x/onecloud/pkg/vpcagent/ovn.(Worker).Start /root/go/src/yunion.io/x/onecloud/pkg/vpcagent/ovn/worker.go:77 +0x1a4

wanyaoqi commented 4 days ago

[error 2024-09-13 15:43:51 models.Networks.joinNetworkAddresses(modelset.go:449)] cannot find network id 35a07c04-1f95-44ff-857a-26dcb1b1a6b9 of network address 762a9e2a-0884-4764-8062-4767f5aa9f3b [error 2024-09-13 15:43:51 models.Guestnetworks.joinNetworkAddresses(modelset.go:573)] cannot find guestnetwork row id 243 of network address 82a70d0b-7079-4fcd-89a7-0bbb931f2ad0

看起来有残留数据, network id 35a07c04-1f95-44ff-857a-26dcb1b1a6b9 ,这个子网在吗,用climc network-show 35a07c04-1f95-44ff-857a-26dcb1b1a6b9 看下

nashzosi commented 4 hours ago

感谢老师回答,这个子网确实存在,我是vSphere环境迁移cloudpods,使用经典网络,这两个子网具有相同的vlan相同的IP地址,不知是不是这个原因导致了这个问题,目前我的两个子网网络均正常