search

oceanbase / ob-operator

Kubernetes operator for OceanBase
https://oceanbase.github.io/ob-operator/
Other
145 stars 38 forks source link

kubectl delete pod手工删除sapp开头的主节点对应的pod后，该pod无法自动恢复 #5

Closed artongu closed 2 years ago

artongu commented 2 years ago

手工删除pod后，进到容器里面查看observer.log发现是observer启动不起来。请问这个是配置不当导致的问题吗？ [2022-05-22 21:00:27.356380] INFO [SERVER] main.cpp:508 [51][0][Y0-0000000000000000] [lt=44] [dc=0] observer is exit(observer_version="OceanBase CE 3.1.3")

ERROR级别日志如下： [2022-05-22 21:00:24.009866] ERROR [SERVER] init_storage (ob_server.cpp:1692) [52][0][Y0-0000000000000000] [lt=2] [dc=0] init partition service fail(ret=-4009, storageenv={data_dir:"/home/admin/oceanbase/store/", default_block_size:2097152, disk_avail_space:25769803776, datafile_disk_percentage:90, redundancy_level:1, log_spec:{log_dir:"/home/admin/oceanbase/store//slog", max_log_size:268435456, log_sync_type:0}, clog_dir:"/home/admin/oceanbase/store//clog", ilog_dir:"/home/admin/oceanbase/store//ilog", clog_shm_path:"/home/admin/oceanbase/store//clog_shm", ilog_shm_path:"/home/admin/oceanbase/store//ilog_shm", index_cache_priority:10, user_block_cache_priority:1, user_row_cache_priority:1, fuse_row_cache_priority:1, bf_cache_priority:1, clog_cache_priority:1, index_clog_cache_priority:1, bf_cache_miss_count_threshold:100, ethernet_speed:1310720000}) BACKTRACE:0x99bf8ce 0x97535e1 0x22af46f 0x22af0bb 0x22aee82 0x8056e7b 0x93345be 0x932a007 0x2270dd5 0x7f3a6a0ff493 0x226faae [2022-05-22 21:00:25.282121] ERROR [SERVER] main (main.cpp:494) [52][0][Y0-0000000000000000] [lt=4] [dc=0] observer init fail(ret=-4009) BACKTRACE:0x99bf8ce 0x97535e1 0x2274784 0x227426b 0x2273fd1 0x2272c68 0x22710da 0x7f3a6a0ff493 0x226faae

WARN级别日志如下： $ grep WARN observer.log.20220522 [2022-05-22 21:00:23.162562] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [52][0][Y0-0000000000000000] [lt=21] [dc=0] clock generator not inited [2022-05-22 21:00:23.173137] WARN [COMMON] load_from_file (ob_io_benchmark.cpp:240) [52][0][Y0-0000000000000000] [lt=20] [dc=0] the io bench result file not exist! [2022-05-22 21:00:23.173159] WARN [COMMON] init (ob_io_benchmark.cpp:782) [52][0][Y0-0000000000000000] [lt=19] [dc=0] Fail to load io benchmark result, (ret=-4027) [2022-05-22 21:00:23.173166] WARN [SERVER] init_io (ob_server.cpp:1088) [52][0][Y0-0000000000000000] [lt=6] [dc=0] init io benchmark fail, (ret=-4027) [2022-05-22 21:00:23.173221] WARN [SERVER.OMT] init_cgroup_rootdir (ob_cgroup_ctrl.cpp:540) [52][0][Y0-0000000000000000] [lt=5] [dc=0] no cgroup directory found. disable cgroup support(cgroup_path="cgroup", ret=-4027) [2022-05-22 21:00:23.173246] WARN [SERVER.OMT] init (ob_cgroup_ctrl.cpp:52) [52][0][Y0-0000000000000000] [lt=20] [dc=0] init cgroup dir failed(ret=-4027, rootcgroup="cgroup") [2022-05-22 21:00:23.173256] WARN [COMMON] get_instance (memory_dump.cpp:91) [52][0][Y0-0000000000000000] [lt=6] [dc=0] memory dump not init [2022-05-22 21:00:23.446586] WARN [LIB] init (ob_tsc_timestamp.cpp:42) [52][0][Y0-0000000000000000] [lt=14] [dc=0] invariant TSC not support(ret=-4007) [2022-05-22 21:00:23.518720] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=14] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:23.518749] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=27] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:23.532934] WARN [SERVER] get_network_speed_from_config_file (ob_server.cpp:1795) [52][0][Y0-0000000000000000] [lt=3] [dc=0] NIC Config file doesn't exist, auto detecting(nic_rate_path="etc/nic.rate.config", ret=-4027) [2022-05-22 21:00:23.587680] WARN [STORAGE] inner_get_super_block_version (ob_local_file_system.cpp:877) [52][0][Y0-0000000000000000] [lt=19] [dc=0] read superblock error.(ret=-4009, offset=0, read_size=0, errno=2, super_block_buf_holder={buf:0x7f3a0d405000, len:65536}, fd={fd:290, disk_id:{disk_idx:0, install_seq:0}}, errmsg="No such file or directory") [2022-05-22 21:00:23.587709] WARN [STORAGE] get_super_block_version (ob_local_file_system.cpp:851) [52][0][Y0-0000000000000000] [lt=5] [dc=0] fail to get super block version from master(ret=-4009) [2022-05-22 21:00:23.587912] WARN [STORAGE] inner_get_super_block_version (ob_local_file_system.cpp:877) [52][0][Y0-0000000000000000] [lt=4] [dc=0] read superblock error.(ret=-4009, offset=2097152, read_size=0, errno=2, super_block_buf_holder={buf:0x7f3a0d405000, len:65536}, fd={fd:290, disk_id:{disk_idx:0, install_seq:0}}, errmsg="No such file or directory") [2022-05-22 21:00:23.587930] WARN [STORAGE] get_super_block_version (ob_local_file_system.cpp:853) [52][0][Y0-0000000000000000] [lt=3] [dc=0] fail to get super block version from backup(ret=-4009) [2022-05-22 21:00:23.587934] WARN [STORAGE] init (ob_local_file_system.cpp:296) [52][0][Y0-0000000000000000] [lt=3] [dc=0] fail to get super block version(ret=-4009) [2022-05-22 21:00:23.587966] WARN [STORAGE] init (ob_store_file_system.cpp:830) [52][0][Y0-0000000000000000] [lt=14] [dc=0] fail to init store file system(ret=-4009, storage_env={data_dir:"/home/admin/oceanbase/store/", default_block_size:2097152, disk_avail_space:25769803776, datafile_disk_percentage:90, redundancy_level:1, log_spec:{log_dir:"/home/admin/oceanbase/store//slog", max_log_size:268435456, log_sync_type:0}, clog_dir:"/home/admin/oceanbase/store//clog", ilog_dir:"/home/admin/oceanbase/store//ilog", clog_shm_path:"/home/admin/oceanbase/store//clog_shm", ilog_shm_path:"/home/admin/oceanbase/store//ilog_shm", index_cache_priority:10, user_block_cache_priority:1, user_row_cache_priority:1, fuse_row_cache_priority:1, bf_cache_priority:1, clog_cache_priority:1, index_clog_cache_priority:1, bf_cache_miss_count_threshold:100, ethernet_speed:1310720000}) [2022-05-22 21:00:23.587999] WARN [STORAGE] init (ob_partition_service.cpp:300) [52][0][Y0-0000000000000000] [lt=9] [dc=0] init store file system failed.(ret=-4009, env={data_dir:"/home/admin/oceanbase/store/", default_block_size:2097152, disk_avail_space:25769803776, datafile_disk_percentage:90, redundancy_level:1, log_spec:{log_dir:"/home/admin/oceanbase/store//slog", max_log_size:268435456, log_sync_type:0}, clog_dir:"/home/admin/oceanbase/store//clog", ilog_dir:"/home/admin/oceanbase/store//ilog", clog_shm_path:"/home/admin/oceanbase/store//clog_shm", ilog_shm_path:"/home/admin/oceanbase/store//ilog_shm", index_cache_priority:10, user_block_cache_priority:1, user_row_cache_priority:1, fuse_row_cache_priority:1, bf_cache_priority:1, clog_cache_priority:1, index_clog_cache_priority:1, bf_cache_miss_count_threshold:100, ethernet_speed:1310720000}) [2022-05-22 21:00:23.588272] WARN begin (ob_hashtable.h:686) [52][0][Y0-0000000000000000] [lt=9] [dc=0] hashtable not init, backtrace=0x99bf8ce 0x85488c4 0x850a488 0x8495166 0x7deea8b 0x7dde4cd 0x9334391 0x932a007 0x2270dd5 0x7f3a6a0ff493 0x226faae [2022-05-22 21:00:23.588318] WARN begin (ob_hashtable.h:686) [52][0][Y0-0000000000000000] [lt=10] [dc=0] hashtable not init, backtrace=0x99bf8ce 0x8541f1c 0x8486b95 0x8495180 0x7deea8b 0x7dde4cd 0x9334391 0x932a007 0x2270dd5 0x7f3a6a0ff493 0x226faae [2022-05-22 21:00:23.735353] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=22] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:23.735379] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=24] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:23.951567] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=4] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:23.951590] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=21] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:24.009931] WARN [SERVER] init (ob_server.cpp:296) [52][0][Y0-0000000000000000] [lt=62] [dc=0] init storage fail(ret=-4009) [2022-05-22 21:00:24.009953] WARN [SERVER] destroy (ob_server.cpp:387) [52][0][Y0-0000000000000000] [lt=4] [dc=0] destroy observer begin [2022-05-22 21:00:24.009984] WARN [SERVER] destroy (ob_server.cpp:390) [52][0][Y0-0000000000000000] [lt=11] [dc=0] backup info destroyed [2022-05-22 21:00:24.009995] WARN [SERVER] destroy (ob_server.cpp:392) [52][0][Y0-0000000000000000] [lt=3] [dc=0] ObBackupDestDetector destroyed [2022-05-22 21:00:24.010000] WARN [SERVER] destroy (ob_server.cpp:394) [52][0][Y0-0000000000000000] [lt=3] [dc=0] backup file lock mgr detroy [2022-05-22 21:00:24.011399] WARN [SERVER] destroy (ob_server.cpp:398) [52][0][Y0-0000000000000000] [lt=9] [dc=0] timer destroyed [2022-05-22 21:00:24.012528] WARN [SERVER] destroy (ob_server.cpp:400) [52][0][Y0-0000000000000000] [lt=5] [dc=0] freeze timer destroyed [2022-05-22 21:00:24.013629] WARN [SERVER] destroy (ob_server.cpp:402) [52][0][Y0-0000000000000000] [lt=8] [dc=0] sql memory manager timer destroyed [2022-05-22 21:00:24.014561] WARN [SERVER] destroy (ob_server.cpp:404) [52][0][Y0-0000000000000000] [lt=6] [dc=0] server trace timer destroyed [2022-05-22 21:00:24.094082] WARN [SHARE] add_event (ob_event_history_table_operator.h:433) [52][0][Y0-0000000000000000] [lt=3] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094103] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=3] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094111] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094116] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=1] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094121] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094127] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=1] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094132] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094137] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094180] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=4] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094202] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=4] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094205] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=5] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094211] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094212] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094218] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094220] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=3] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094224] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094228] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=3] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094229] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094235] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094236] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094240] WARN [SHARE] stop (ob_reentrant_thread.cpp:106) [165][224][Y0-0000000000000000] [lt=3] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094245] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [52][0][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094250] WARN [SHARE] wait (ob_reentrant_thread.cpp:123) [165][224][Y0-0000000000000000] [lt=2] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.094410] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [165][224][Y0-0000000000000000] [lt=3] [dc=0] clock generator not inited [2022-05-22 21:00:24.095010] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=20] [dc=0] clock generator not inited [2022-05-22 21:00:24.095033] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=20] [dc=0] clock generator not inited [2022-05-22 21:00:24.096257] WARN [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:6639) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=2] [dc=0] synonym not exist(tenant_id=1, database_id=1099511627777, table_name=all_tenant_backup_info, ret=-5019) [2022-05-22 21:00:24.096265] WARN [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:6515) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=6] [dc=0] fail to resolve table relation recursively(tenant_id=1, ret=-5019) [2022-05-22 21:00:24.096275] WARN [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:6295) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=9] [dc=0] resolve table relation factor failed(ret=-5019) [2022-05-22 21:00:24.096285] WARN [SHARE.SCHEMA] get_dblink_user (ob_schema_getter_guard.cpp:6554) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=4] [dc=0] dblink user name is empty(tenant_id=1, dblink_name=all_tenant_backup_info) [2022-05-22 21:00:24.096295] WARN [SQL.RESV] get_dblink_user (ob_schema_checker.cpp:1303) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=7] [dc=0] failed to get dblink id(ret=-4016, tenant_id=1, dblink_name=all_tenant_backup_info) [2022-05-22 21:00:24.096299] WARN [SQL.RESV] resolve_dblink_with_synonym (ob_dml_resolver.cpp:6339) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] fail to exec schemachecker->get_dblink_user(tenant_id, tmp_table_name, tmp_dbname, *allocator)(ret=-4016) [2022-05-22 21:00:24.096302] WARN [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:6300) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] try synonym with dblink failed(ret=-4016) [2022-05-22 21:00:24.096308] WARN [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:950) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] fail to resolve table(ret=-5019) [2022-05-22 21:00:24.096312] WARN [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:983) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] fail to resolve sys view(ret=-5019) [2022-05-22 21:00:24.096334] WARN resolve_basic_table (ob_dml_resolver.cpp:1076) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] Table 'oceanbase.all_tenant_backup_info' doesn't exist [2022-05-22 21:00:24.096346] WARN [SQL.RESV] resolve_table (ob_dml_resolver.cpp:1301) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=10] [dc=0] resolve basic table failed(ret=-5019) [2022-05-22 21:00:24.096351] WARN [SQL.RESV] resolve_table_list (ob_update_resolver.cpp:392) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=4] [dc=0] failed to resolve table(ret=-5019) [2022-05-22 21:00:24.096354] WARN [SQL.RESV] resolve (ob_update_resolver.cpp:63) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] resolve table failed(ret=-5019) [2022-05-22 21:00:24.096367] WARN [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:126) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=4] [dc=0] execute stmt_resolver failed(ret=-5019, parsetree.type=3036) [2022-05-22 21:00:24.096394] WARN [SQL] generate_stmt (ob_sql.cpp:1440) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=8] [dc=0] failed to resolve(ret=-5019) [2022-05-22 21:00:24.096410] WARN [SQL] generate_physical_plan (ob_sql.cpp:1528) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=13] [dc=0] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2022-05-22 21:00:24.096416] WARN [SQL] handle_physical_plan (ob_sql.cpp:3218) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=4] [dc=0] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2022-05-22 21:00:24.096420] WARN [SQL] handle_text_query (ob_sql.cpp:1209) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=4] [dc=0] fail to handle physical plan(ret=-5019) [2022-05-22 21:00:24.096427] WARN [SQL] stmt_query (ob_sql.cpp:171) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] fail to handle text query(stmt=update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882', ret=-5019) [2022-05-22 21:00:24.096431] WARN [SERVER] do_query (ob_inner_sql_connection.cpp:665) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] executor execute failed(ret=-5019) [2022-05-22 21:00:24.096439] WARN [SERVER] query (ob_inner_sql_connection.cpp:815) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] execute failed(ret=-5019, executor={ObIExecutor:, sql:"update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882'"}, retry_cnt=0) [2022-05-22 21:00:24.096452] WARN [SERVER] query (ob_inner_sql_connection.cpp:818) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=9] [dc=0] failed to process retry(tmp_ret=-5019, ret=-5019, executor={ObIExecutor:, sql:"update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882'"}, retry_cnt=0) [2022-05-22 21:00:24.096468] WARN [SERVER] inner_close (ob_inner_sql_result.cpp:152) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=4] [dc=0] result set close failed(ret=-5019, need_retry=false) [2022-05-22 21:00:24.096479] WARN [SERVER] force_close (ob_inner_sql_result.cpp:136) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=9] [dc=0] result set close failed(ret=-5019) [2022-05-22 21:00:24.096483] WARN [SERVER] query (ob_inner_sql_connection.cpp:824) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] failed to close result(close_ret=-5019, ret=-5019) [2022-05-22 21:00:24.096512] WARN [SERVER.OMT] get_tenant_ctx_with_tenant_lock (ob_multi_tenant.cpp:64) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] get tenant from omt failed(ret=-5150, tenant_id=1) [2022-05-22 21:00:24.096526] WARN [SHARE] ObTenantSpaceFetcher (ob_context.cpp:176) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=12] [dc=0] get tenant ctx failed(ret=-5150, tenant_id=1) [2022-05-22 21:00:24.096534] WARN [SERVER] query (ob_inner_sql_connection.cpp:852) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=5] [dc=0] failed to process record(executor={ObIExecutor:, sql:"update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882'"}, record_ret=-5019, ret=-5019) [2022-05-22 21:00:24.096539] WARN [SERVER] query (ob_inner_sql_connection.cpp:868) [165][224][YB42AC1A9F4A-0005DF994E790E88] [lt=3] [dc=0] failed to process final(executor={ObIExecutor:, sql:"update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882'"}, aret=-5019, ret=-5019) [2022-05-22 21:00:24.096546] WARN [SERVER] execute_write (ob_inner_sql_connection.cpp:1284) [165][224][Y0-0000000000000000] [lt=3] [dc=0] execute sql failed(ret=-5019, tenant_id=1, sql="update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882'") [2022-05-22 21:00:24.096637] WARN [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:104) [165][224][Y0-0000000000000000] [lt=4] [dc=0] execute sql failed(ret=-5019, conn=0x7f3a564d5850, start=1653224424094285, sql="update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882'") [2022-05-22 21:00:24.096699] WARN [SERVER] clean_backup_scheduler_leader (ob_backup_operator.cpp:1519) [165][224][Y0-0000000000000000] [lt=14] [dc=0] execute sql failed(ret=-5019, sql=update all_tenant_backup_info set value = '' where name = 'backup_scheduler_leader' and value='172.26.159.74:2882') [2022-05-22 21:00:24.096712] WARN [SERVER] clean_backup_scheduler_leader (ob_backup_manager.cpp:895) [165][224][Y0-0000000000000000] [lt=11] [dc=0] failed to clean backup scheduler leader(ret=-5019) [2022-05-22 21:00:24.097652] WARN [SERVER] destroy (ob_server.cpp:406) [52][0][Y0-0000000000000000] [lt=3] [dc=0] root service destroyed [2022-05-22 21:00:24.167561] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=4] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:24.167583] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=20] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:24.214599] WARN [SERVER] destroy (ob_server.cpp:408) [52][0][Y0-0000000000000000] [lt=18] [dc=0] ob service destroyed [2022-05-22 21:00:24.220559] WARN [SERVER] destroy (ob_server.cpp:410) [52][0][Y0-0000000000000000] [lt=11] [dc=0] sql engine destroyed [2022-05-22 21:00:24.247763] WARN [SERVER] destroy (ob_server.cpp:412) [52][0][Y0-0000000000000000] [lt=18] [dc=0] ob dag scheduler destroyed [2022-05-22 21:00:24.247848] WARN begin (ob_hashtable.h:686) [52][0][Y0-0000000000000000] [lt=3] [dc=0] hashtable not init, backtrace=0x99bf8ce 0x7b51f64 0x7afa378 0x7ab87d2 0x7dee9eb 0x9328d8d 0x932a6ab 0x2270dd5 0x7f3a6a0ff493 0x226faae [2022-05-22 21:00:24.247910] WARN begin (ob_hashtable.h:686) [52][0][Y0-0000000000000000] [lt=9] [dc=0] hashtable not init, backtrace=0x99bf8ce 0x85488c4 0x850a488 0x8495166 0x7deea8b 0x9328d8d 0x932a6ab 0x2270dd5 0x7f3a6a0ff493 0x226faae [2022-05-22 21:00:24.247932] WARN begin (ob_hashtable.h:686) [52][0][Y0-0000000000000000] [lt=7] [dc=0] hashtable not init, backtrace=0x99bf8ce 0x8541f1c 0x8486b95 0x8495180 0x7deea8b 0x9328d8d 0x932a6ab 0x2270dd5 0x7f3a6a0ff493 0x226faae [2022-05-22 21:00:24.248412] WARN [SERVER] destroy (ob_server.cpp:414) [52][0][Y0-0000000000000000] [lt=3] [dc=0] partition service destroyed [2022-05-22 21:00:24.248431] WARN [SERVER] destroy (ob_server.cpp:416) [52][0][Y0-0000000000000000] [lt=5] [dc=0] gc partition adapter destroyed [2022-05-22 21:00:24.248448] WARN [SERVER] destroy (ob_server.cpp:418) [52][0][Y0-0000000000000000] [lt=2] [dc=0] location cache destroyed [2022-05-22 21:00:24.248464] WARN [SERVER] destroy (ob_server.cpp:420) [52][0][Y0-0000000000000000] [lt=2] [dc=0] weak read service destroyed [2022-05-22 21:00:24.248590] WARN [SERVER] destroy (ob_server.cpp:422) [52][0][Y0-0000000000000000] [lt=2] [dc=0] net frame destroyed [2022-05-22 21:00:24.349352] WARN [COMMON] get_warning_disks (ob_io_disk.cpp:930) [71][36][Y0-0000000000000000] [lt=0] [dc=0] not init(ret=-4006) [2022-05-22 21:00:24.349374] WARN [COMMON] check_disk_error (ob_io_manager.cpp:699) [71][36][Y0-0000000000000000] [lt=19] [dc=0] fail to get warning disks(ret=-4006) [2022-05-22 21:00:24.383524] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=19] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:24.383547] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=21] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:24.537646] WARN [SHARE] blacklistloop (ob_server_blacklist.cpp:278) [181][256][Y0-0000000000000000] [lt=17] [dc=0] ObServerBlacklist is not inited [2022-05-22 21:00:24.599574] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=4] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:24.599609] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=34] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:24.815640] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=4] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:24.815675] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=33] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:25.031542] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=4] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:25.031577] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=33] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:25.198148] WARN [SERVER] destroy (ob_server.cpp:428) [52][0][Y0-0000000000000000] [lt=48] [dc=0] io manager destroyed [2022-05-22 21:00:25.247591] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:582) [101][96][Y0-0000000000000000] [lt=7] [dc=0] tenant manager not init(ret=-4006) [2022-05-22 21:00:25.247628] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [101][96][Y0-0000000000000000] [lt=34] [dc=0] Fail to get all tenant ids, (ret=-4006) [2022-05-22 21:00:25.280214] WARN [SERVER] destroy (ob_server.cpp:430) [52][0][Y0-0000000000000000] [lt=7] [dc=0] memory dump destroyed [2022-05-22 21:00:25.280250] WARN [SERVER] destroy (ob_server.cpp:432) [52][0][Y0-0000000000000000] [lt=21] [dc=0] tenant timezone manager destroyed [2022-05-22 21:00:25.282098] WARN [SERVER] destroy (ob_server.cpp:434) [52][0][Y0-0000000000000000] [lt=10] [dc=0] log compressor destroyed [2022-05-22 21:00:25.282114] WARN [SERVER] destroy (ob_server.cpp:435) [52][0][Y0-0000000000000000] [lt=15] [dc=0] destroy observer end

chris-sun-star commented 2 years ago

这个看起来是observer进程启动失败了，看日志有可能是磁盘资源不足，可以确认一下相关的配置

artongu commented 2 years ago

使用local-path-provisioner后，再次kubectl delete po sapp-ob-test2-obshdev-zone1-0删除主节点后，发现该pod一直无限重启中。（删除从节点后，pod会自动恢复，通过SELECT status FROM oceanbase.__all_server;检查状态都是active的）

==============当前配置情况如下=========================================================== 1、local-path路径大小为1TB，空间应该是充足的，因为启动3个副本时是能够正常启动的。 2、obcluster.yaml文件内容如下： apiVersion: cloud.oceanbase.com/v1 kind: OBCluster metadata: name: ob-test2 spec: version: v3.1.2-10000392021123010 clusterID: 2 topology:

================pod无限重启过程中describe该pod结果如下================================ $ kubectl describe po sapp-ob-test2-obshdev-zone1-0 Name: sapp-ob-test2-obshdev-zone1-0 Namespace: default Priority: 0 Node: 10.128.20.185/10.128.20.185 Start Time: Thu, 09 Jun 2022 17:17:46 +0800 Labels: app=sapp-ob-test2 index=0 subset=zone1 Annotations: cni.projectcalico.org/podIP: 172.26.159.99/32 Status: Terminating (lasts ) Termination Grace Period: 30s IP: 172.26.159.99 IPs: IP: 172.26.159.99 Containers: observer: Container ID: docker://3230b788df87d16626dff760e883f5763ff1e14902cb7d4e2afccea07d6d745c Image: oceanbase-ce:operator-v3.1.2-10000392021123010 Image ID: docker-pullable://oceanbase-ce@sha256:6b41b67d2ad28e24440a53ad92fde29b7a0848ce297decaf36f317a8a4fd14d9 Ports: 19001/TCP, 2881/TCP, 2882/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 09 Jun 2022 17:17:47 +0800 Finished: Thu, 09 Jun 2022 17:18:06 +0800 Ready: False Restart Count: 0 Limits: cpu: 2 memory: 10Gi Requests: cpu: 2 memory: 10Gi Readiness: http-get http://:19001/api/ob/readiness delay=0s timeout=1s period=2s #success=1 #failure=3 Environment: Mounts: /home/admin/data_file from data-file (rw) /home/admin/data_log from data-log (rw) /home/admin/log from log (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-dgdb6 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data-file: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sapp-ob-test2-obshdev-zone1-0-data-file ReadOnly: false data-log: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sapp-ob-test2-obshdev-zone1-0-data-log ReadOnly: false log: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sapp-ob-test2-obshdev-zone1-0-log ReadOnly: false default-token-dgdb6: Type: Secret (a volume populated by a Secret) SecretName: default-token-dgdb6 Optional: false QoS Class: Guaranteed Node-Selectors: topology.kubernetes.io/zone=zone1 Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s **Events: Type Reason Age From Message

Warning FailedScheduling 26s default-scheduler persistentvolumeclaim "sapp-ob-test2-obshdev-zone1-0-data-file" not found Warning FailedScheduling 26s default-scheduler persistentvolumeclaim "sapp-ob-test2-obshdev-zone1-0-data-log" not found Warning FailedScheduling 26s default-scheduler persistentvolumeclaim "sapp-ob-test2-obshdev-zone1-0-log" not found Normal Scheduled 21s default-scheduler Successfully assigned default/sapp-ob-test2-obshdev-zone1-0 to 10.128.20.185 Normal CreatedPod 27s statefulapp-controller create Podsapp-ob-test2-obshdev-zone1-0 Normal Pulling 21s kubelet Pulling image "oceanbase-ce:operator-v3.1.2-10000392021123010" Normal Pulled 21s kubelet Successfully pulled image "oceanbase-ce:operator-v3.1.2-10000392021123010" Normal Created 21s kubelet Created container observer Normal Started 21s kubelet Started container observer Warning Unhealthy 4s (x9 over 20s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 400 Normal DeletedPod 3s (x4 over 3s) obcluster-controller delete Podsapp-ob-test2-obshdev-zone1-0 Normal Killing 3s kubelet Stopping container observer**

=====检查手工删除节点上的存储目录，发现仅会生成空目录，目录下没有日志文件产生，直至目录由于pod退出被删除======= $ ll * pvc-48136f66-88c4-4e00-b177-91e7d956c0fa_default_sapp-ob-test2-obshdev-zone1-0-data-log: 总用量 0

pvc-6697751d-8fa7-489e-9f28-5d1f7a6224d4_default_sapp-ob-test2-obshdev-zone1-0-data-file: 总用量 0

pvc-f4d4e013-5e27-4b8e-9c52-615e4453846a_default_sapp-ob-test2-obshdev-zone1-0-log: 总用量 0 $ ll ls: 无法访问: 没有那个文件或目录

chris-sun-star commented 2 years ago

好的，可以帮忙提供一下 ob-operator 的日志 kubectl logs ob-operator-controller-manager-xxx-xxx manager -n oceanbase-system 另外，删除了主节点之后，OB 集群过一段时间是否还能过访问 我们也尝试复现下这种场景

artongu commented 2 years ago

1、删除主节点后过一会集群可以通过service成功连接，切换没有问题，但原1节点状态是inactive的。 mysql> SELECT id,status FROM oceanbase.__all_server; +----+----------+ | id | status | +----+----------+ | 3 | active | | 1 | inactive | | 2 | active | +----+----------+ 3 rows in set (0.01 sec)

2、日志如下（包含创建集群、删除主节点pod），集群名字是ob-test2。

I0610 07:21:35.027521 1 deleg.go:130] controller-runtime/metrics "msg"="metrics server is starting to listen" "addr"="127.0.0.1:8080" I0610 07:21:35.028061 1 controllers.go:37] load github.com/oceanbase/ob-operator/pkg/controllers/statefulapp.Add I0610 07:21:35.028318 1 controllers.go:37] load github.com/oceanbase/ob-operator/pkg/controllers/observer.Add I0610 07:21:35.028573 1 main.go:140] starting manager. I0610 07:21:35.028999 1 leaderelection.go:248] attempting to acquire leader lease oceanbase-system/oceanbase-system... I0610 07:21:35.029105 1 internal.go:383] "msg"="starting metrics server" "path"="/metrics" I0610 07:21:51.720173 1 leaderelection.go:258] successfully acquired lease oceanbase-system/oceanbase-system I0610 07:21:51.721070 1 controller.go:178] controller/obcluster-controller "msg"="Starting EventSource" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"version":"","clusterID":0,"topology":null,"resources":{"cpu":"0","memory":"0","storage":null}},"status":{"status":"","topology":null}}} I0610 07:21:51.723104 1 controller.go:178] controller/statefulapp-controller "msg"="Starting EventSource" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"cluster":"","subsets":null,"podTemplate":{"containers":null},"storageTemplates":null},"status":{"cluster":"","clusterStatus":"","subsets":null}}} I0610 07:21:51.723222 1 controller.go:178] controller/obcluster-controller "msg"="Starting EventSource" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"cluster":"","subsets":null,"podTemplate":{"containers":null},"storageTemplates":null},"status":{"cluster":"","clusterStatus":"","subsets":null}}} I0610 07:21:51.723278 1 controller.go:186] controller/obcluster-controller "msg"="Starting Controller"
I0610 07:21:51.723323 1 controller.go:178] controller/statefulapp-controller "msg"="Starting EventSource" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"containers":null},"status":{}}} I0610 07:21:51.723444 1 controller.go:178] controller/statefulapp-controller "msg"="Starting EventSource" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{}}} I0610 07:21:51.723471 1 controller.go:186] controller/statefulapp-controller "msg"="Starting Controller"
I0610 07:21:51.824406 1 controller.go:220] controller/statefulapp-controller "msg"="Starting workers" "worker count"=3 I0610 07:21:51.824509 1 controller.go:220] controller/obcluster-controller "msg"="Starting workers" "worker count"=1 I0610 07:21:51.852922 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Pending"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:21:51.865084 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.119","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} E0610 07:21:51.928486 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:51.930767 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.034629 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.036690 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.140210 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.148545 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.148676 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:21:52.158968 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.119 E0610 07:21:52.161608 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:21:52.162592 1 cable_executer.go:39] start observer 172.26.159.119 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:21:52.194822 1 log.go:28] core.(*OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer E0610 07:21:52.197432 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.199526 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.201774 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.203555 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.205106 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.212317 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.212402 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:52.225850 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.227830 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.230013 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.231569 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.233407 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.240807 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.240916 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:52.263890 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.265790 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.267263 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.268676 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.270070 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.277108 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.277197 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:52.320307 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.322320 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.324553 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.326428 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.327961 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.336083 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.336144 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:52.418708 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.420698 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.422752 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.424461 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.427046 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.434449 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.434519 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:52.597529 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.599739 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.601277 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.602690 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.604317 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.612390 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.612560 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:52.935377 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.937309 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.940054 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.943412 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.946505 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:52.952601 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:52.952715 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:53.595916 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:53.598168 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:53.600395 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:53.601868 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:53.603935 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:53.611769 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:53.611862 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:54.895274 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:54.897767 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:54.900123 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:54.901916 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:54.903998 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:54.912168 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:54.912301 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:21:57.164516 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:21:57.475872 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:57.483449 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:57.507364 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:57.508843 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:57.510220 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:21:57.518746 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:21:57.518834 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:01.430392 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:01.432535 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:01.433885 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:01.435375 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:01.436771 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:01.443634 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:01.443724 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:02.164593 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:02.641909 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:02.643447 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:02.646347 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:02.648503 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:02.650063 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:02.659740 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:02.659827 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:06.271138 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:06.273287 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:06.275811 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:06.277341 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:06.279187 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:06.287269 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:06.287413 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:07.163899 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:11.122907 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:11.124452 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:11.126290 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:11.128252 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:11.129697 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:11.136865 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:11.136972 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:12.164093 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:15.971309 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:15.978971 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:15.980799 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:15.982173 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:15.983538 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:15.991396 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:15.991485 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:17.163882 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:20.802871 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:20.806799 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:20.808266 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:20.809536 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:20.810827 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:20.818168 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:20.818273 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:22.163648 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:23.143378 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:23.145614 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:23.147603 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:23.149080 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:23.150519 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:23.161988 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:23.162088 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:25.647778 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:25.650644 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:25.652728 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:25.654597 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:25.656208 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:25.663330 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:25.663416 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:27.164561 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:30.492748 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:30.494688 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:30.508118 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:30.509705 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:30.511047 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:30.520448 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:30.520575 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:22:32.171762 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:22:32.192700 1 log.go:28] core.(*OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:22:32.192743 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.119:2882' ZONE 'zone1'; I0610 07:22:32.197609 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.119 2022/06/10 07:22:32 Get "http://172.26.159.119:19001/api/system/info": dial tcp 172.26.159.119:19001: connect: connection refused E0610 07:22:32.199315 1 sql_executer.go:30] 8001 Server is initializing

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:22:32]  Error 8001: Server is initializing  I0610 07:22:32.203425 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:22:32.204095 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:22:32 Get "http://172.26.159.119:19001/api/ob/status": dial tcp 172.26.159.119:19001: connect: connection refused E0610 07:22:32.208189 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:22:32 Post "http://172.26.159.119:19001/api/ob/start": dial tcp 172.26.159.119:19001: connect: connection refused E0610 07:22:32.208612 1 cable_executer.go:39] start observer 172.26.159.119 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:22:32.209681 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:22:32 Post "http://172.26.159.119:19001/api/ob/readinessUpdate": dial tcp 172.26.159.119:19001: connect: connection refused I0610 07:22:32.215037 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:22:32.228236 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:22:32.228294 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.119 I0610 07:22:32.231531 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.119 2022/06/10 07:22:32 Get "http://172.26.159.119:19001/api/system/info": dial tcp 172.26.159.119:19001: connect: connection refused I0610 07:22:32.236591 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:22:32 Get "http://172.26.159.119:19001/api/ob/status": dial tcp 172.26.159.119:19001: connect: connection refused E0610 07:22:32.243117 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:22:32 Post "http://172.26.159.119:19001/api/ob/start": dial tcp 172.26.159.119:19001: connect: connection refused E0610 07:22:32.243495 1 cable_executer.go:39] start observer 172.26.159.119 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:22:32.247464 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer E0610 07:22:35.338073 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:35.339925 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:35.341815 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:35.343278 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:35.344756 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:35.352495 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:35.352584 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:22:38 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:38.210091 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:22:38 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:38.245410 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready I0610 07:22:38.547463 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:22:38.561932 1 pvc.go:41] persistentvolumeclaims "sapp-ob-test2-obshdev-zone1-0-data-file" already exists E0610 07:22:38.561971 1 pod_controller.go:98] pvc is already exist, need recreate I0610 07:22:38.570992 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:22:38.588709 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:22:38.595002 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:22:38.600509 1 log.go:26] delete sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:22:40.183802 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:40.207154 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:40.208910 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:40.210359 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:40.211808 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:40.220913 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:40.221023 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:22:43 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:43.209710 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready I0610 07:22:43.212052 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.119","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:22:43.226767 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:22:43 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:43.244718 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:45.030098 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:45.031587 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:45.036033 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:45.040242 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:45.041848 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:45.050806 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.050898 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:22:45.085291 1 log.go:28] core.(OBClusterCtrl).FixStatus update OBCluster ob-test2 to Ready Zone zone1 Status Ready E0610 07:22:45.093843 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.093881 1 conn.go:29] 0 E0610 07:22:45.094064 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.094084 1 conn.go:29] 0 E0610 07:22:45.094276 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.094304 1 conn.go:29] 0 E0610 07:22:45.101497 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.101578 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.114715 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.114749 1 conn.go:29] 0 E0610 07:22:45.114868 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.114896 1 conn.go:29] 0 E0610 07:22:45.115172 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.115200 1 conn.go:29] 0 E0610 07:22:45.124570 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.124649 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.146024 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.146049 1 conn.go:29] 0 E0610 07:22:45.146157 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.146181 1 conn.go:29] 0 E0610 07:22:45.146321 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.146354 1 conn.go:29] 0 E0610 07:22:45.154797 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.154867 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.187235 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.187275 1 conn.go:29] 0 E0610 07:22:45.187420 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.187442 1 conn.go:29] 0 E0610 07:22:45.187588 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.187609 1 conn.go:29] 0 E0610 07:22:45.196296 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.196400 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.246785 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.246810 1 conn.go:29] 0 E0610 07:22:45.246931 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.246974 1 conn.go:29] 0 E0610 07:22:45.247172 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.247190 1 conn.go:29] 0 E0610 07:22:45.254651 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.254722 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.345666 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.345690 1 conn.go:29] 0 E0610 07:22:45.345804 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.345823 1 conn.go:29] 0 E0610 07:22:45.345990 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.346013 1 conn.go:29] 0 E0610 07:22:45.352972 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.353038 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.524622 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.524660 1 conn.go:29] 0 E0610 07:22:45.524801 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.524833 1 conn.go:29] 0 E0610 07:22:45.525052 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.525073 1 conn.go:29] 0 E0610 07:22:45.532171 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:45.532347 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:45.998179 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.998216 1 conn.go:29] 0 E0610 07:22:45.998346 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.998359 1 conn.go:29] 0 E0610 07:22:45.998592 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:45.998635 1 conn.go:29] 0 E0610 07:22:46.006525 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:46.006633 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:46.655973 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:46.656008 1 conn.go:29] 0 E0610 07:22:46.656154 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:46.656177 1 conn.go:29] 0 E0610 07:22:46.656401 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:46.656421 1 conn.go:29] 0 E0610 07:22:46.663635 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:46.663727 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:47.955293 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:47.955329 1 conn.go:29] 0 E0610 07:22:47.955463 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:47.955493 1 conn.go:29] 0 E0610 07:22:47.955679 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:47.955696 1 conn.go:29] 0 E0610 07:22:47.963587 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:47.963668 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:22:48 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:48.209759 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:22:48 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:48.244473 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:49.875422 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:49.877380 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:49.879394 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:49.880838 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:49.882227 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:49.889797 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:49.889909 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:49.899195 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:49.899220 1 conn.go:29] 0 E0610 07:22:49.899334 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:49.899357 1 conn.go:29] 0 E0610 07:22:49.899499 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:49.899519 1 conn.go:29] 0 E0610 07:22:49.906164 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:49.906228 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:22:50.536548 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:50.536591 1 conn.go:29] 0 E0610 07:22:50.536730 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:50.536747 1 conn.go:29] 0 E0610 07:22:50.536912 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:50.536932 1 conn.go:29] 0 E0610 07:22:50.543420 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:50.543507 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:22:52.563788 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:22:52.569836 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:22:52.586318 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:22:52.593295 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:22:52.609098 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:22:53 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:53.210401 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:22:53 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:53.244712 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:54.720834 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:54.722996 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:54.724934 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:54.726933 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:54.728792 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:54.736816 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:54.736936 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:54.997331 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:54.997374 1 conn.go:29] 0 E0610 07:22:54.997501 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:54.997534 1 conn.go:29] 0 E0610 07:22:54.997744 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:54.997770 1 conn.go:29] 0 E0610 07:22:55.005602 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:55.005735 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:22:57.246644 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Pending"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:22:58 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:58.209800 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:22:58 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:22:58.245198 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:22:59.566858 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:59.569440 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:59.571059 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:59.572448 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:59.573857 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:22:59.582464 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:59.582574 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:22:59.593255 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:59.593287 1 conn.go:29] 0 E0610 07:22:59.593433 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:59.593462 1 conn.go:29] 0 E0610 07:22:59.593671 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:22:59.593701 1 conn.go:29] 0 E0610 07:22:59.600808 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:22:59.600928 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:00.794395 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:00.794430 1 conn.go:29] 0 E0610 07:23:00.794578 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:00.794606 1 conn.go:29] 0 E0610 07:23:00.794799 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:00.794823 1 conn.go:29] 0 E0610 07:23:00.801244 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:00.801365 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:23:01.924678 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:23:01.936986 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.122","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:23:03 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:03.210059 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:23:03 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:03.244336 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:23:04.411738 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:04.418105 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:04.419986 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:04.423438 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:04.426192 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:04.433257 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:04.433343 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:23:04.438962 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.122 E0610 07:23:04.447345 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:04.448282 1 cable_executer.go:39] start observer 172.26.159.122 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:23:04.452285 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:23:08 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:08.210073 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready 2022/06/10 07:23:08 Get "http://172.26.159.119:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:08.244542 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.119 starting not ready E0610 07:23:09.256765 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:09.258599 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:09.260393 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:09.261728 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:09.263292 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:09.272917 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:09.273042 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:23:09.449592 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready I0610 07:23:12.229451 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:12.254002 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:12.257508 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:23:12.257545 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.119:2882' ZONE 'zone1'; I0610 07:23:12.264750 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.122 E0610 07:23:12.265205 1 sql_executer.go:30] 8001 Server is initializing 2022/06/10 07:23:12 Get "http://172.26.159.122:19001/api/system/info": dial tcp 172.26.159.122:19001: connect: connection refused

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:23:12]  Error 8001: Server is initializing  E0610 07:23:12.266650 1 obcluster.go:60] Operation cannot be fulfilled on obclusters.cloud.oceanbase.com "ob-test2": the object has been modified; please apply your changes to the latest version and try again I0610 07:23:12.268266 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:12.270505 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:23:12 Get "http://172.26.159.122:19001/api/ob/status": dial tcp 172.26.159.122:19001: connect: connection refused E0610 07:23:12.274364 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:12 Post "http://172.26.159.122:19001/api/ob/start": dial tcp 172.26.159.122:19001: connect: connection refused E0610 07:23:12.274774 1 cable_executer.go:39] start observer 172.26.159.122 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:23:12.278693 1 log.go:28] core.(*OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer I0610 07:23:12.282032 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.119:2882' ZONE 'zone1';

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:23:12]  Error 8001: Server is initializing  E0610 07:23:12.289051 1 sql_executer.go:30] 8001 Server is initializing I0610 07:23:12.292902 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:12.305740 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:23:12.314262 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.122 2022/06/10 07:23:12 Get "http://172.26.159.122:19001/api/system/info": dial tcp 172.26.159.122:19001: connect: connection refused I0610 07:23:12.318867 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:23:12 Get "http://172.26.159.122:19001/api/ob/status": dial tcp 172.26.159.122:19001: connect: connection refused E0610 07:23:12.324209 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:12 Post "http://172.26.159.122:19001/api/ob/start": dial tcp 172.26.159.122:19001: connect: connection refused E0610 07:23:12.324610 1 cable_executer.go:39] start observer 172.26.159.122 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:23:12.340480 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer E0610 07:23:14.107125 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:14.109003 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:14.110515 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:14.112010 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:14.113402 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:14.121047 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:14.121171 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:23:15 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:15.449311 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready I0610 07:23:15.962286 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.122","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:23:15.972532 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:23:18 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:18.277867 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:18 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:18.325054 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:18.947910 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:18.950163 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:18.952230 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:18.953724 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:18.955028 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:18.961241 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:18.961335 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:23:18.988738 1 log.go:28] core.(*OBClusterCtrl).FixStatus update OBCluster ob-test2 to Ready Zone zone1 Status Ready E0610 07:23:18.996800 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:18.996826 1 conn.go:29] 0 E0610 07:23:18.996977 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:18.997004 1 conn.go:29] 0 E0610 07:23:18.997154 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:18.997185 1 conn.go:29] 0 E0610 07:23:19.003411 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.003511 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.017707 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.017755 1 conn.go:29] 0 E0610 07:23:19.017895 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.017921 1 conn.go:29] 0 E0610 07:23:19.018134 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.018151 1 conn.go:29] 0 E0610 07:23:19.024449 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.024522 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.043166 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.043193 1 conn.go:29] 0 E0610 07:23:19.043351 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.043372 1 conn.go:29] 0 E0610 07:23:19.043526 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.043551 1 conn.go:29] 0 E0610 07:23:19.052010 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.052131 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.083454 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.083479 1 conn.go:29] 0 E0610 07:23:19.083635 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.083659 1 conn.go:29] 0 E0610 07:23:19.083968 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.083991 1 conn.go:29] 0 E0610 07:23:19.093311 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.093380 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.142800 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.142836 1 conn.go:29] 0 E0610 07:23:19.142994 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.143022 1 conn.go:29] 0 E0610 07:23:19.143189 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.143227 1 conn.go:29] 0 E0610 07:23:19.153456 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.153530 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.241190 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.241223 1 conn.go:29] 0 E0610 07:23:19.241351 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.241372 1 conn.go:29] 0 E0610 07:23:19.241560 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.241581 1 conn.go:29] 0 E0610 07:23:19.248586 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.248658 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.419848 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.419876 1 conn.go:29] 0 E0610 07:23:19.420030 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.420111 1 conn.go:29] 0 E0610 07:23:19.420291 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.420311 1 conn.go:29] 0 E0610 07:23:19.428260 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.428336 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:19.757666 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.757686 1 conn.go:29] 0 E0610 07:23:19.757815 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.757846 1 conn.go:29] 0 E0610 07:23:19.758031 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:19.758052 1 conn.go:29] 0 E0610 07:23:19.820937 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:19.821051 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:23:20 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:20.449926 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:20.472856 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:20.472886 1 conn.go:29] 0 E0610 07:23:20.472990 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:20.473017 1 conn.go:29] 0 E0610 07:23:20.473169 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:20.473192 1 conn.go:29] 0 E0610 07:23:20.480508 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:20.480596 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:23:20.640411 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:23:20.656747 1 pvc.go:41] persistentvolumeclaims "sapp-ob-test2-obshdev-zone1-0-data-file" already exists E0610 07:23:20.656773 1 pod_controller.go:98] pvc is already exist, need recreate I0610 07:23:20.675222 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:23:20.688113 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:23:20.696352 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:23:20.702568 1 log.go:26] delete sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:23:21.782975 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:21.783009 1 conn.go:29] 0 E0610 07:23:21.783158 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:21.783181 1 conn.go:29] 0 E0610 07:23:21.783368 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:21.783387 1 conn.go:29] 0 E0610 07:23:21.789937 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:21.790038 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:23:22 Post "http://172.26.159.119:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:23:22.278112 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:22.284464 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.119 2022/06/10 07:23:22 Post "http://172.26.159.119:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:23:22.309467 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:22.314149 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.119 2022/06/10 07:23:23 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:23.276176 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:23 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:23.325413 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:23.814904 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:23.817531 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:23.819426 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:23.820833 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:23.822294 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:23.829647 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:23.829751 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:23:24.071502 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:24.071529 1 conn.go:29] 0 E0610 07:23:24.071721 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:24.071753 1 conn.go:29] 0 E0610 07:23:24.071915 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:24.071938 1 conn.go:29] 0 E0610 07:23:24.080069 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:24.080158 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:24.359797 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:24.359826 1 conn.go:29] 0 E0610 07:23:24.359953 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:24.359966 1 conn.go:29] 0 E0610 07:23:24.360228 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:24.360260 1 conn.go:29] 0 E0610 07:23:24.367852 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:24.367937 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:23:25 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:25.450176 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:28 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:28.276151 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:28 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:28.326316 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:28.638196 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:28.640335 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:28.642377 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:28.644226 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:28.645889 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:28.652629 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:28.652736 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:23:28.660914 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:28.660957 1 conn.go:29] 0 E0610 07:23:28.661100 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:28.661121 1 conn.go:29] 0 E0610 07:23:28.661297 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:28.661320 1 conn.go:29] 0 E0610 07:23:28.667608 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:28.667690 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:23:29.990757 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:30.000750 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:23:30.011160 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:23:30.017462 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:23:30.028693 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:23:30 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:30.449624 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:33 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:33.276141 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:33 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:33.325369 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:33.484336 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:33.486318 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:33.487885 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:33.489326 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:33.491158 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:33.498283 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:33.498373 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:23:33.517013 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:33.517053 1 conn.go:29] 0 E0610 07:23:33.517210 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:33.517236 1 conn.go:29] 0 E0610 07:23:33.517403 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:33.517422 1 conn.go:29] 0 E0610 07:23:33.538793 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:33.538879 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:23:34.631753 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:34.631786 1 conn.go:29] 0 E0610 07:23:34.632092 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:34.632134 1 conn.go:29] 0 E0610 07:23:34.632415 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:34.632435 1 conn.go:29] 0 E0610 07:23:34.644873 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:34.644979 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:23:34.681544 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Pending"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:23:35 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:35.450466 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:38 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:38.275810 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:38 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:38.325970 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:38.329539 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:38.331116 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:38.332485 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:38.333805 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:38.335098 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:38.343027 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:38.343116 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:23:38.354236 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:38.354261 1 conn.go:29] 0 E0610 07:23:38.354472 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:38.354515 1 conn.go:29] 0 E0610 07:23:38.354840 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:23:38.354861 1 conn.go:29] 0 E0610 07:23:38.361191 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:38.361288 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:23:39.370456 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:23:39.383560 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.75","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:23:40 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:40.448962 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready E0610 07:23:43.175228 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:43.177113 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:43.181600 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:43.182984 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:43.185196 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:43.192127 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:43.192220 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:23:43 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:43.276070 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready I0610 07:23:43.282011 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.75 E0610 07:23:43.284161 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready E0610 07:23:43.284938 1 cable_executer.go:39] start observer 172.26.159.75 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:23:43.296479 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:23:43 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:43.325580 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready I0610 07:23:44.456813 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:44.481290 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:23:44.481340 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.122:2882' ZONE 'zone1'; I0610 07:23:44.486594 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.75 2022/06/10 07:23:44 Get "http://172.26.159.75:19001/api/system/info": dial tcp 172.26.159.75:19001: connect: connection refused E0610 07:23:44.488696 1 sql_executer.go:30] 8001 Server is initializing

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:23:44]  Error 8001: Server is initializing  I0610 07:23:44.490744 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:44.492261 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:23:44 Get "http://172.26.159.75:19001/api/ob/status": dial tcp 172.26.159.75:19001: connect: connection refused E0610 07:23:44.495386 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:23:44 Post "http://172.26.159.75:19001/api/ob/start": dial tcp 172.26.159.75:19001: connect: connection refused E0610 07:23:44.499374 1 cable_executer.go:39] start observer 172.26.159.75 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:23:44.500661 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer E0610 07:23:48.020056 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:48.021726 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:48.023167 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:48.024438 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:48.026563 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:23:48.034005 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:23:48.034093 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:23:48 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:48.276408 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready 2022/06/10 07:23:48 Get "http://172.26.159.122:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:48.326752 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.122 starting not ready I0610 07:23:48.707757 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.75","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:23:48.720024 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"172.26.159.75","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:23:49 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:49.286245 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:23:50 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:50.500319 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready I0610 07:23:52.281907 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:52.296994 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:23:52.297032 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.122:2882' ZONE 'zone1'; I0610 07:23:52.340800 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:52.345423 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.122:2882' ZONE 'zone1'; 2022/06/10 07:23:54 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:54.286680 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:23:54 Post "http://172.26.159.122:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:23:54.501114 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:23:54.507179 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.122 2022/06/10 07:23:55 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:55.500384 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready E0610 07:23:57.323666 1 error.go:25] dial tcp 172.26.159.75:2881: i/o timeout E0610 07:23:57.323711 1 conn.go:29] 0 2022/06/10 07:23:59 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:23:59.286684 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:24:00 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:00.500681 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready E0610 07:24:01.315040 1 sql_executer.go:30] 4012 Timeout E0610 07:24:01.315246 1 pod.go:51] Pod "sapp-ob-test2-obshdev-zone1-0" not found

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:24:01]  Error 4012: Timeout  E0610 07:24:01.350706 1 sql_executer.go:30] 4012 Timeout

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:24:01]  Error 4012: Timeout  E0610 07:24:01.350884 1 pod.go:51] Pod "sapp-ob-test2-obshdev-zone1-0" not found E0610 07:24:02.324811 1 error.go:25] dial tcp 172.26.159.75:2881: i/o timeout E0610 07:24:02.324858 1 conn.go:29] 0 I0610 07:24:02.745487 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:24:02.754250 1 pvc.go:41] persistentvolumeclaims "sapp-ob-test2-obshdev-zone1-0-data-file" already exists E0610 07:24:02.754272 1 pod_controller.go:98] pvc is already exist, need recreate I0610 07:24:02.766116 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:24:02.774385 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:24:02.802214 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:24:02.809433 1 log.go:26] delete sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:24:04 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:04.286686 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:24:05 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:05.500486 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready E0610 07:24:07.325701 1 error.go:25] dial tcp 172.26.159.75:2881: i/o timeout E0610 07:24:07.325750 1 conn.go:29] 0 E0610 07:24:07.334088 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:07.334184 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:07.336552 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:07.338621 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:07.340370 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:07.341661 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:07.342889 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:07.351219 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:07.351324 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:24:07.425157 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"172.26.159.75","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:24:09 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:09.285827 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:24:10 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:10.501739 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:24:11 Post "http://172.26.159.122:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:11.321489 1 pod.go:51] Pod "sapp-ob-test2-obshdev-zone1-0" not found I0610 07:24:11.327725 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.122 2022/06/10 07:24:11 Post "http://172.26.159.122:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:11.356176 1 pod.go:51] Pod "sapp-ob-test2-obshdev-zone1-0" not found I0610 07:24:11.360886 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.122 I0610 07:24:12.094856 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:12.099691 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:24:12.104509 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:24:12.115649 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:24:12.128311 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} E0610 07:24:12.359446 1 error.go:25] dial tcp 172.26.159.75:2881: i/o timeout E0610 07:24:12.359480 1 conn.go:29] 0 2022/06/10 07:24:14 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:14.286799 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:24:15 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:15.499908 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready I0610 07:24:16.799692 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Pending"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} E0610 07:24:17.360541 1 error.go:25] dial tcp 172.26.159.75:2881: i/o timeout E0610 07:24:17.360591 1 conn.go:29] 0 2022/06/10 07:24:19 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:19.286512 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready 2022/06/10 07:24:20 Get "http://172.26.159.75:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:20.500858 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.75 starting not ready I0610 07:24:21.458799 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:24:21.470724 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.80","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} E0610 07:24:22.361542 1 error.go:25] dial tcp 172.26.159.75:2881: i/o timeout E0610 07:24:22.361604 1 conn.go:29] 0 E0610 07:24:22.369985 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:22.370081 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:22.374292 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:22.376244 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:22.377725 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:22.379105 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:22.380432 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:22.388099 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:22.388211 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:24:22.393961 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 E0610 07:24:22.396259 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:22.397087 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:24:22.408109 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer I0610 07:24:23.295242 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:23.311920 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:23.311981 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.75:2882' ZONE 'zone1'; I0610 07:24:23.322673 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 2022/06/10 07:24:23 Get "http://172.26.159.80:19001/api/system/info": dial tcp 172.26.159.80:19001: connect: connection refused I0610 07:24:23.328512 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:24:23 Get "http://172.26.159.80:19001/api/ob/status": dial tcp 172.26.159.80:19001: connect: connection refused E0610 07:24:23.332875 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:23 Post "http://172.26.159.80:19001/api/ob/start": dial tcp 172.26.159.80:19001: connect: connection refused E0610 07:24:23.333362 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:24:23.336360 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer I0610 07:24:24.504930 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:24.526957 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:24.527010 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.75:2882' ZONE 'zone1'; I0610 07:24:24.533926 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 2022/06/10 07:24:24 Get "http://172.26.159.80:19001/api/system/info": dial tcp 172.26.159.80:19001: connect: connection refused I0610 07:24:24.541017 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:24.556613 1 log.go:28] core.(*OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:24:25 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:25.551703 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:26.783964 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:26.785796 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:26.787716 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:26.789354 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:26.790867 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:26.799069 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:26.799187 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:24:28 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:28.397659 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:29 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:29.333744 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:24:30.810316 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:24:30.818319 1 pvc.go:41] persistentvolumeclaims "sapp-ob-test2-obshdev-zone1-0-data-file" already exists E0610 07:24:30.818354 1 pod_controller.go:98] pvc is already exist, need recreate I0610 07:24:30.825864 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:24:30.835382 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:24:30.843054 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:24:30.849660 1 log.go:26] delete sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:24:31.629582 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:31.631981 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:31.633824 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:31.635466 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:31.636809 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:31.643051 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:31.643142 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:24:32.317107 1 sql_executer.go:30] 4012 Timeout

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:24:32]  Error 4012: Timeout  I0610 07:24:32.328730 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:32.341736 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:32.352249 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 I0610 07:24:32.366073 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:24:33 Get "http://172.26.159.80:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:33.358189 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:33.372828 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:33.377483 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 I0610 07:24:33.389035 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:24:33 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:33.398684 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:33.534316 1 sql_executer.go:30] 4012 Timeout

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:24:33]  Error 4012: Timeout  I0610 07:24:33.548218 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:33.563412 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:33.567777 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 I0610 07:24:33.580549 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:24:34 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:34.335229 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:34 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:34.373352 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:34 Get "http://172.26.159.80:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:34.383883 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:34.405347 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:34.414669 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 I0610 07:24:34.430003 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:24:34 Get "http://172.26.159.80:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:34.574348 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:34.600834 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:34.605683 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 I0610 07:24:34.620568 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:24:35 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:35.406050 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:35 Get "http://172.26.159.80:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:35.420589 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:35.439346 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:24:35.444645 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.80 I0610 07:24:35.466547 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer I0610 07:24:35.493302 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.80","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:24:35.504193 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:24:35 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:35.552254 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} 2022/06/10 07:24:35 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:35.601472 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:35 Get "http://172.26.159.80:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:35.610986 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:35.628592 1 log.go:28] core.(*OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready E0610 07:24:35.638018 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.638047 1 conn.go:29] 0 E0610 07:24:35.638202 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.638242 1 conn.go:29] 0 E0610 07:24:35.638540 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.638560 1 conn.go:29] 0 E0610 07:24:35.647315 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:35.647393 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:35.662687 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.662714 1 conn.go:29] 0 E0610 07:24:35.662859 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.662883 1 conn.go:29] 0 E0610 07:24:35.663093 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.663118 1 conn.go:29] 0 E0610 07:24:35.669238 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:35.669358 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:35.689145 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.689178 1 conn.go:29] 0 E0610 07:24:35.689290 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.689308 1 conn.go:29] 0 E0610 07:24:35.689475 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.689503 1 conn.go:29] 0 E0610 07:24:35.695775 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:35.695850 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:35.725255 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.725282 1 conn.go:29] 0 E0610 07:24:35.725406 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.725425 1 conn.go:29] 0 E0610 07:24:35.725613 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.725636 1 conn.go:29] 0 E0610 07:24:35.733085 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:35.733206 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:35.784125 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.784158 1 conn.go:29] 0 E0610 07:24:35.784324 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.784351 1 conn.go:29] 0 E0610 07:24:35.784535 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.784553 1 conn.go:29] 0 E0610 07:24:35.791586 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:35.791686 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:35.882676 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.882712 1 conn.go:29] 0 E0610 07:24:35.882839 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.882866 1 conn.go:29] 0 E0610 07:24:35.883092 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:35.883115 1 conn.go:29] 0 E0610 07:24:35.890585 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:35.890684 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:36.061513 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.061539 1 conn.go:29] 0 E0610 07:24:36.061643 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.061663 1 conn.go:29] 0 E0610 07:24:36.061875 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.061918 1 conn.go:29] 0 E0610 07:24:36.078588 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:36.078686 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:36.421895 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.421925 1 conn.go:29] 0 E0610 07:24:36.422082 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.422102 1 conn.go:29] 0 E0610 07:24:36.422276 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.422301 1 conn.go:29] 0 E0610 07:24:36.430176 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:36.430272 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:24:36 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:36.440283 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:36 Get "http://172.26.159.80:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:36.449887 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:24:36.475243 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:36.477076 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:36.478593 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:36.479931 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:36.481285 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:36.492355 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:36.492427 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:24:36.520786 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.520810 1 conn.go:29] 0 E0610 07:24:36.520998 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.521025 1 conn.go:29] 0 E0610 07:24:36.521232 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:36.521256 1 conn.go:29] 0 E0610 07:24:36.540622 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:36.540726 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:24:36 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:36.629736 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:37.082494 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:37.082533 1 conn.go:29] 0 E0610 07:24:37.082714 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:37.082736 1 conn.go:29] 0 E0610 07:24:37.082887 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:37.082905 1 conn.go:29] 0 E0610 07:24:37.089613 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:37.089702 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:24:37 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:37.457090 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:38 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:38.398269 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:39 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:39.334368 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:39.660697 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:39.660729 1 conn.go:29] 0 E0610 07:24:39.660853 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:39.660872 1 conn.go:29] 0 E0610 07:24:39.661111 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:39.661152 1 conn.go:29] 0 E0610 07:24:39.667759 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:39.667861 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:24:40.170752 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:40.176609 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:24:40.183176 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:24:40.194600 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:24:40.204581 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} E0610 07:24:41.320538 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:41.322897 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:41.325555 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:41.327178 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:41.329357 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:41.336438 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:41.336533 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:24:41.346786 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:41.346817 1 conn.go:29] 0 E0610 07:24:41.347009 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:41.347034 1 conn.go:29] 0 E0610 07:24:41.347216 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:41.347236 1 conn.go:29] 0 E0610 07:24:41.354015 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:41.354114 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:24:41 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:41.554322 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:42 Post "http://172.26.159.75:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:24:42.359152 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:42.363610 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.75 2022/06/10 07:24:43 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:43.398133 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:43 Post "http://172.26.159.75:19001/api/ob/readinessUpdate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:43.564525 1 pod.go:51] Pod "sapp-ob-test2-obshdev-zone1-0" not found I0610 07:24:43.582918 1 task_observer_add.go:115] add OBServer finish zone1 172.26.159.75 2022/06/10 07:24:44 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:44.334053 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:44 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:44.374209 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} E0610 07:24:44.798967 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:44.799004 1 conn.go:29] 0 E0610 07:24:44.799182 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:44.799220 1 conn.go:29] 0 E0610 07:24:44.799398 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:44.799420 1 conn.go:29] 0 E0610 07:24:44.818536 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:44.818621 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" I0610 07:24:44.844865 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:24:44.854082 1 pvc.go:41] persistentvolumeclaims "sapp-ob-test2-obshdev-zone1-0-data-file" already exists E0610 07:24:44.854118 1 pod_controller.go:98] pvc is already exist, need recreate I0610 07:24:44.859205 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:24:44.868252 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:24:44.875516 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:24:44.892957 1 log.go:26] delete sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:24:45 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:45.406798 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} 2022/06/10 07:24:45 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:45.602548 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} E0610 07:24:46.165796 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:46.167420 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:46.170121 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:46.171409 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:46.172926 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:46.182262 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:46.182359 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:24:46.190746 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:46.190773 1 conn.go:29] 0 E0610 07:24:46.190866 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:46.190886 1 conn.go:29] 0 E0610 07:24:46.191041 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:46.191065 1 conn.go:29] 0 E0610 07:24:46.197840 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:46.197929 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:24:46 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:46.441191 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} 2022/06/10 07:24:46 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:46.553408 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:46 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:46.630697 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} 2022/06/10 07:24:47 Post "http://172.26.159.80:19001/api/ob/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:47.458171 1 cable_executer.go:39] start observer 172.26.159.80 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} 2022/06/10 07:24:48 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:48.399043 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:49 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:49.334131 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:24:49.520762 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:24:49.530592 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:24:49.553744 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:24:49.563320 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:24:49.577041 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Pending"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:24:50 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:50.376434 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:51.010625 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:51.012623 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:51.014004 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:51.015258 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:51.016683 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:51.024102 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:51.024187 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:24:51.033144 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:51.033180 1 conn.go:29] 0 E0610 07:24:51.033300 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:51.033328 1 conn.go:29] 0 E0610 07:24:51.033499 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:51.033521 1 conn.go:29] 0 E0610 07:24:51.039700 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:51.039766 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:24:51 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:51.408236 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:51 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:51.553687 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:51 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:51.603377 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:52 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:52.442357 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:52 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:52.632053 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:53 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:53.399058 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:53 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:53.459020 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:24:54.206333 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Pending"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:24:54 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:54.334835 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:55 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:55.376027 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:24:55.861782 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:55.861816 1 conn.go:29] 0 E0610 07:24:55.861928 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:55.861939 1 conn.go:29] 0 E0610 07:24:55.862102 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:24:55.862124 1 conn.go:29] 0 E0610 07:24:55.868348 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:55.868435 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:24:55.870962 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:55.872795 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:55.874519 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:55.876103 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:55.877656 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:24:55.883589 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:24:55.883715 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:24:56 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:56.408136 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:56 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:56.553538 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:56 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:56.603793 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:57 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:57.442738 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:57 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:57.631075 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:58 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:58.398609 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:24:58 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:58.459924 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:24:58.888968 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Pending"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:24:58.900327 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Prepareing","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.96","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:24:59 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:24:59.334843 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:00 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:00.375959 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:25:00.701557 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:00.703383 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:00.710242 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:00.711776 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:00.713197 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:00.720507 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:00.720616 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" I0610 07:25:00.728119 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.96 E0610 07:25:00.730219 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready E0610 07:25:00.731014 1 cable_executer.go:39] start observer 172.26.159.96 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:25:00.741240 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:25:01 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:01.408031 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:01 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:01.554437 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:01 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:01.604724 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:25:02.404925 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:02.420685 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:25:02.420736 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.80:2882' ZONE 'zone1'; I0610 07:25:02.425260 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.96 2022/06/10 07:25:02 Get "http://172.26.159.96:19001/api/system/info": dial tcp 172.26.159.96:19001: connect: connection refused I0610 07:25:02.430042 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:25:02 Get "http://172.26.159.96:19001/api/ob/status": dial tcp 172.26.159.96:19001: connect: connection refused E0610 07:25:02.434571 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:02 Post "http://172.26.159.96:19001/api/ob/start": dial tcp 172.26.159.96:19001: connect: connection refused E0610 07:25:02.434984 1 cable_executer.go:39] start observer 172.26.159.96 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:25:02.440734 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:25:02 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:02.442481 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:02 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:02.631706 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:25:03.338905 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:03.350818 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:25:03.350859 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.80:2882' ZONE 'zone1'; I0610 07:25:03.361440 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.96 2022/06/10 07:25:03 Get "http://172.26.159.96:19001/api/system/info": dial tcp 172.26.159.96:19001: connect: connection refused I0610 07:25:03.365723 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:25:03 Get "http://172.26.159.96:19001/api/ob/status": dial tcp 172.26.159.96:19001: connect: connection refused E0610 07:25:03.369726 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:03 Post "http://172.26.159.96:19001/api/ob/start": dial tcp 172.26.159.96:19001: connect: connection refused E0610 07:25:03.370152 1 cable_executer.go:39] start observer 172.26.159.96 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"} I0610 07:25:03.374049 1 log.go:28] core.(*OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:25:03 Get "http://172.26.159.80:19001/api/ob/status": dial tcp 172.26.159.80:19001: i/o timeout (Client.Timeout exceeded while awaiting headers) E0610 07:25:03.458981 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:05 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:05.375679 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:25:05.549236 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:05.551203 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:05.552913 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:05.554273 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:05.555648 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:05.564443 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:05.564538 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:25:06 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:06.408222 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:06 Get "http://172.26.159.80:19001/api/ob/status": dial tcp 172.26.159.80:19001: i/o timeout (Client.Timeout exceeded while awaiting headers) E0610 07:25:06.553653 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:06 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:06.603873 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:06 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:06.732571 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:07 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:07.442094 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:07 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:07.631370 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:25:08.245161 1 log.go:26] create sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:25:08.257585 1 pvc.go:41] persistentvolumeclaims "sapp-ob-test2-obshdev-zone1-0-data-file" already exists E0610 07:25:08.257617 1 pod_controller.go:98] pvc is already exist, need recreate I0610 07:25:08.267910 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-file "succeed" I0610 07:25:08.276056 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-data-log "succeed" I0610 07:25:08.287982 1 log.go:26] delete PersistentVolumeClaim sapp-ob-test2-obshdev-zone1-0-log "succeed" I0610 07:25:08.294317 1 log.go:26] delete sapp-ob-test2-obshdev-zone1-0 "succeed" 2022/06/10 07:25:08 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:08.436306 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:08 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:08.459808 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:09 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:09.371718 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:10 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:10.375539 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:25:10.420212 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:10.422822 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:10.424836 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:10.426355 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:10.427683 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:10.433603 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:10.433695 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" 2022/06/10 07:25:11 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:11.408521 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:25:11.425528 1 sql_executer.go:30] 4012 Timeout

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:25:11]  Error 4012: Timeout  I0610 07:25:11.429620 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:11.446318 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:25:11.448468 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.96 I0610 07:25:11.461249 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:25:11 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:11.553455 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:11 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:11.603651 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:11 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:11.732346 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready E0610 07:25:12.358593 1 sql_executer.go:30] 4012 Timeout

(/workspace/pkg/controllers/observer/sql/sql_executer.go:27) [2022-06-10 07:25:12]  Error 4012: Timeout  I0610 07:25:12.363854 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:12.377272 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:25:12.381700 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.96 I0610 07:25:12.392786 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:25:12 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:12.441561 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready 2022/06/10 07:25:12 Get "http://172.26.159.96:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:25:12.454520 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:12.469085 1 log.go:28] core.(OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready I0610 07:25:12.473415 1 task_observer_add.go:60] begin add OBServer zone1 172.26.159.96 I0610 07:25:12.484786 1 log.go:28] core.(OBClusterCtrl).AddOBServer update OBCluster ob-test2 to Scale UP Zone zone1 Status Add OBServer 2022/06/10 07:25:12 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:12.631089 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:25:12.920287 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Running","podIP":"172.26.159.96","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} I0610 07:25:12.929134 1 log.go:26] update status StatefulApp sapp-ob-test2 {"cluster":"obshdev","clusterStatus":"Ready","subsets":[{"name":"zone1","region":"region1","expectedReplicas":1,"availableReplicas":0,"pods":[{"name":"sapp-ob-test2-obshdev-zone1-0","index":0,"podPhase":"Pending","podIP":"","nodeIP":"10.128.20.185","pvcs":[{"name":"sapp-ob-test2-obshdev-zone1-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone1-0-log","phase":"Bound"}]}]},{"name":"zone2","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone2-0","index":0,"podPhase":"Running","podIP":"172.26.210.132","nodeIP":"10.128.20.186","pvcs":[{"name":"sapp-ob-test2-obshdev-zone2-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone2-0-log","phase":"Bound"}]}]},{"name":"zone3","region":"region1","expectedReplicas":1,"availableReplicas":1,"pods":[{"name":"sapp-ob-test2-obshdev-zone3-0","index":0,"podPhase":"Running","podIP":"172.26.146.19","nodeIP":"10.128.20.187","pvcs":[{"name":"sapp-ob-test2-obshdev-zone3-0-data-file","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-data-log","phase":"Bound"},{"name":"sapp-ob-test2-obshdev-zone3-0-log","phase":"Bound"}]}]}]} 2022/06/10 07:25:13 Get "http://172.26.159.96:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:25:13.386826 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:13.401824 1 log.go:28] core.(*OBClusterCtrl).AddOBServerExecuter update OBCluster ob-test2 to Ready Zone zone1 Status Ready E0610 07:25:13.410381 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.410409 1 conn.go:29] 0 E0610 07:25:13.410511 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.410533 1 conn.go:29] 0 E0610 07:25:13.410712 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.410730 1 conn.go:29] 0 E0610 07:25:13.416385 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.416464 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:25:13.430373 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.430399 1 conn.go:29] 0 E0610 07:25:13.430541 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.430561 1 conn.go:29] 0 E0610 07:25:13.430685 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.430701 1 conn.go:29] 0 2022/06/10 07:25:13 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:13.436338 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready E0610 07:25:13.437933 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.438023 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:25:13 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:13.458876 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready E0610 07:25:13.461310 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.461333 1 conn.go:29] 0 E0610 07:25:13.461472 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.461493 1 conn.go:29] 0 E0610 07:25:13.461661 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.461683 1 conn.go:29] 0 E0610 07:25:13.467458 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.467525 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:25:13 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:13.469568 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:13 Get "http://172.26.159.96:19001/api/system/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers) I0610 07:25:13.476727 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" E0610 07:25:13.496642 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.496663 1 conn.go:29] 0 E0610 07:25:13.496750 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.496768 1 conn.go:29] 0 E0610 07:25:13.496904 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.496926 1 conn.go:29] 0 E0610 07:25:13.502635 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.502709 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:25:13.550994 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.551017 1 conn.go:29] 0 E0610 07:25:13.551130 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.551151 1 conn.go:29] 0 E0610 07:25:13.551307 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.551328 1 conn.go:29] 0 E0610 07:25:13.557990 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.558060 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:25:13.648329 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.648355 1 conn.go:29] 0 E0610 07:25:13.648494 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.648515 1 conn.go:29] 0 E0610 07:25:13.648664 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.648685 1 conn.go:29] 0 E0610 07:25:13.654988 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.655104 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:25:13.822542 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.822570 1 conn.go:29] 0 E0610 07:25:13.822718 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.822739 1 conn.go:29] 0 E0610 07:25:13.822904 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:13.822922 1 conn.go:29] 0 E0610 07:25:13.829144 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:13.829214 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:25:14.160415 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:14.160451 1 conn.go:29] 0 E0610 07:25:14.160567 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:14.160582 1 conn.go:29] 0 E0610 07:25:14.160776 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:14.160797 1 conn.go:29] 0 E0610 07:25:14.168046 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:14.168130 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:25:14 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:14.370365 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:14 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:14.402709 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready 2022/06/10 07:25:14 Get "http://172.26.159.96:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:14.483093 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.96 starting not ready E0610 07:25:14.821182 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:14.821226 1 conn.go:29] 0 E0610 07:25:14.821425 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:14.821457 1 conn.go:29] 0 E0610 07:25:14.821659 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:14.821679 1 conn.go:29] 0 E0610 07:25:14.829140 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:14.829222 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" E0610 07:25:15.237995 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:15.241610 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:15.243432 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:15.245197 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:15.246865 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0610 07:25:15.255126 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:15.255215 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0610 07:25:15.264726 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:15.264752 1 conn.go:29] 0 E0610 07:25:15.264906 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:15.264937 1 conn.go:29] 0 E0610 07:25:15.265272 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:15.265295 1 conn.go:29] 0 E0610 07:25:15.274499 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:15.274567 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default" 2022/06/10 07:25:15 Get "http://172.26.159.80:19001/api/ob/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) E0610 07:25:15.374573 1 cable_executer.go:49] OBCluster ob-test2 observer 172.26.159.80 starting not ready I0610 07:25:15.557605 1 log.go:26] delete Pod sapp-ob-test2-obshdev-zone1-0 "succeed" I0610 07:25:15.563386 1 sql_executer.go:23] ALTER SYSTEM ADD SERVER '172.26.159.80:2882' ZONE 'zone1'; E0610 07:25:16.120091 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:16.120120 1 conn.go:29] 0 E0610 07:25:16.120225 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:16.120248 1 conn.go:29] 0 E0610 07:25:16.120408 1 error.go:25] dial tcp :2881: connect: connection refused E0610 07:25:16.120429 1 conn.go:29] 0 E0610 07:25:16.126083 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test2" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null" E0610 07:25:16.126170 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test2\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test2" "namespace"="default"

chris-sun-star commented 2 years ago

好的，感谢提供

chris-sun-star commented 2 years ago

这个问题主要的原因在于更新rs_list的时候是通过连接第一个pod来进行查询的，当第一个pod不可用的时候，rs_list查询失败，是空值 可以看到这样的日志 E0610 07:21:52.162592 1 cable_executer.go:39] start observer 172.26.159.119 failed {"clusterId":2,"clusterName":"ob-test2","cpuLimit":2,"memoryLimit":10,"rsList":"","zoneName":"zone1"}

chris-sun-star commented 2 years ago

ob-operator 的镜像可以替换成 hongxinsun/ob-operator:1.0.5 先测试 observer 的镜像这样配置(配置文件的格式有了些修改) imageRepo: hongxinsun/oceanbase-cn tag: v3.1.3-10100042022051821

有问题欢迎随时沟通

artongu commented 2 years ago

升级之后observer还是无法自动启动，没有日志文件生成，相关日志见下：

1、oboperator日志 ob-operator.log 2、kubectl logs sapp-ob-test-cn-zone1-0================================================================ 2022-06-21T10:31:19.63333+08:00 INFO [6,] caller=initialization/app.go:31:InitApp: init directory for oceanbase 2022-06-21T10:31:20.44197+08:00 INFO [6,] caller=initialization/app.go:35:InitApp: init status variables 2022-06-21T10:31:20.44209+08:00 INFO [6,] caller=initialization/app.go:40:InitApp: init http server 2022-06-21T10:31:21.16323+08:00 INFO [6,] caller=server/http.go:95:func1: request: from 10.128.20.185, method GET, path /api/ob/readiness, response: status code %!s(int=500), l atency 168.238µs, comment 2022-06-21T10:31:23.16278+08:00 INFO [6,] caller=server/http.go:95:func1: request: from 10.128.20.185, method GET, path /api/ob/readiness, response: status code %!s(int=500), l atency 104.797µs, comment

......

2022-06-21T11:17:05.1632+08:00 INFO [6,] caller=server/http.go:95:func1: request: from 10.128.20.185, method GET, path /api/ob/readiness, response: status code %!s(int=500), la tency 128.265µs, comment

3、kubectl describe po sapp-ob-test-cn-zone1-0==================================================== Name: sapp-ob-test-cn-zone1-0 Namespace: ccse-ns-ob Priority: 0 Node: 10.128.20.185/10.128.20.185 Start Time: Tue, 21 Jun 2022 10:31:18 +0800 Labels: app=sapp-ob-test index=0 subset=zone1 Annotations: cni.projectcalico.org/podIP: 172.26.159.72/32 Status: Running IP: 172.26.159.72 IPs: IP: 172.26.159.72 Containers: observer: Container ID: docker://0d4449be6f2560e0baf621883e3a2643e781e5aad7b70dd7c0f21275aa1cc3b1 Image: hongxinsun/oceanbase-cn:v3.1.3-10100042022051821 Image ID: docker-pullable://hongxinsun/oceanbase-cn@sha256:69124a72b87a64cc31122ce2a72a65cd528666da6b209baf4b3312c800dd8420 Ports: 19001/TCP, 2881/TCP, 2882/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP State: Running Started: Tue, 21 Jun 2022 10:31:19 +0800 Ready: False Restart Count: 0 Limits: cpu: 2 memory: 10Gi Requests: cpu: 2 memory: 10Gi Readiness: http-get http://:19001/api/ob/readiness delay=0s timeout=1s period=2s #success=1 #failure=3 Environment: Mounts: /home/admin/data_file from data-file (rw) /home/admin/data_log from data-log (rw) /home/admin/log from log (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-tfrcs (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data-file: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sapp-ob-test-cn-zone1-0-data-file ReadOnly: false data-log: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sapp-ob-test-cn-zone1-0-data-log ReadOnly: false log: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sapp-ob-test-cn-zone1-0-log ReadOnly: false default-token-tfrcs: Type: Secret (a volume populated by a Secret) SecretName: default-token-tfrcs Optional: false QoS Class: Guaranteed Node-Selectors: topology.kubernetes.io/zone=zone1 Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Warning FailedScheduling 47m default-scheduler persistentvolumeclaim "sapp-ob-test-cn-zone1-0-data-file" not found Warning FailedScheduling 47m default-scheduler persistentvolumeclaim "sapp-ob-test-cn-zone1-0-data-log" not found Normal Scheduled 47m default-scheduler Successfully assigned ccse-ns-ob/sapp-ob-test-cn-zone1-0 to 10.128.20.185 Normal CreatedPod 47m statefulapp-controller create Podsapp-ob-test-cn-zone1-0 Normal Pulled 47m kubelet Container image "10.128.126.35:8021/ibocdev_pub_repo/oceanbase-cn:v3.1.3-10100042022051821" already present on machine Normal Created 47m kubelet Created container observer Normal Started 47m kubelet Started container observer Warning Unhealthy 2m18s (x1351 over 47m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500

4、容器内文件目录，日志文件未生成====================================================== [root@sapp-ob-test-cn-zone1-0 admin]# ls -a data_/ log data_file/sort_dir: . ..

data_file/sstable: . ..

data_log/clog: . ..

data_log/ilog: . ..

data_log/slog: . ..

log: . ..

chris-sun-star commented 2 years ago

I0621 10:26:27.661441 1 log.go:28] core.(*OBClusterCtrl).OBClusterReadyForStep update OBCluster ob-test to Ready Zone Status E0621 10:29:19.107697 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) 看 ob-operator 日志中，部署成功之后在10:29:19 开始一直报 Access denied ， 是有修改过root密码吗

artongu commented 2 years ago

是的，修改了root口令。是需要在什么地方配置一下再重启operator吗？

chris-sun-star commented 2 years ago

现在还没有支持修改密码，有些信息的更新是依靠root用户去查询的，如果密码变了会更新失败，这个之后得再完善一下，目前是还不能改root的密码

artongu commented 2 years ago

不修改密码的情况下测试删除第一个pod已经可以自动恢复了。

不过operator里面还是有一直循环报如下的错：

E0621 16:42:26.963004 1 controller.go:317] controller/obcluster-controller "msg"="Reconciler error" "error"="OBZone.cloud.oceanbase.com \"obzone-ob-test\" is invalid: status.topology.zone.nodes: Invalid value: \"null\": status.topology.zone.nodes in body must be of type array: \"null\"" "name"="ob-test" "namespace"="ccse-ns-ob" E0621 16:42:32.204362 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0621 16:42:32.206504 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0621 16:42:32.206930 1 error.go:25] dial tcp 172.26.159.90:2881: connect: connection refused E0621 16:42:32.206955 1 conn.go:29] 0 E0621 16:42:32.207274 1 error.go:25] dial tcp 172.26.159.90:2881: connect: connection refused E0621 16:42:32.207297 1 conn.go:29] 0 E0621 16:42:32.208854 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0621 16:42:32.210254 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0621 16:42:32.211816 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0621 16:42:32.213424 1 conn.go:29] 1045 Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO) E0621 16:42:32.213838 1 error.go:25] dial tcp 172.26.159.90:2881: connect: connection refused E0621 16:42:32.213879 1 conn.go:29] 0 E0621 16:42:32.220641 1 obzone.go:75] OBZone.cloud.oceanbase.com "obzone-ob-test" is invalid: status.topology.zone.nodes: Invalid value: "null": status.topology.zone.nodes in body must be of type array: "null"

chris-sun-star commented 2 years ago

日志看起来还是和用户密码相关的问题，连不上 OB 集群，更新zone的status信息失败，目前还是要保持root的密码先不要修改

密码的问题开了另一个issue，可以在那里讨论 https://github.com/oceanbase/ob-operator/issues/9