[Bug]: docker run with errors ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)

YuriyGavrilov commented 12 months ago

Describe the bug

There is a docker run error such as:

2023-10-09 22:18:32 Check before start obagent ok
2023-10-09 22:18:32 Start observer ok
2023-10-09 22:18:35 observer program health check x
2023-10-09 22:18:35 [WARN] OBD-2002: Failed to start 127.0.0.1 observer
2023-10-09 22:18:35 [ERROR] oceanbase-ce start failed
2023-10-09 22:18:35 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-09 22:18:35 Trace ID: 95948eb8-66d8-11ee-887b-0242ac110002
2023-10-09 22:18:35 If you want to view detailed obd logs, please run: obd display-trace 95948eb8-66d8-11ee-887b-0242ac110002
Deploy "obcluster" is deployed
2023-10-09 22:18:36 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-09 22:18:36 Trace ID: a4f4bbe4-66d8-11ee-8469-0242ac110002
2023-10-09 22:18:36 If you want to view detailed obd logs, please run: obd display-trace a4f4bbe4-66d8-11ee-8469-0242ac110002
2023-10-09 22:18:36 ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)
2023-10-09 22:18:36 deploy success!
2023-10-09 22:18:36 boot success!

Environment Docker on Mac OS

OS Version and CPU Arch(23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:42:42 PDT 2023; root:xnu-10002.1.13~1/RELEASE_X86_64 x86_64):
OB Version(LD_LIBRARY_PATH=../lib:$LD_LIBRARY_PATH ./observer -V):

Fast Reproduce Steps(Required)

docker run -p 2881:2881 --name oceanbase-ce -e MINI_MODE=1 -d oceanbase/oceanbase-ce docker exec -it oceanbase-ce ob-mysql sys # Connect to the root user of the sys tenant.

Steps to reproduce the behavior:

Expected behavior Docker run successfully Second docker command run interactively without this error: docker exec -it oceanbase-ce ob-mysql sys login as root@sys Command is: obclient -h127.1 -uroot@sys -A -Doceanbase -P2881 ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)

Actual Behavior

Additional context just tried fresh run in docker

hnwyllmm commented 12 months ago

Please tell me some information that allocate to docker engine:

memory
cpu

YuriyGavrilov commented 12 months ago

@hnwyllmm I actually didn't specify any resources for this container but I have a lot of overall free resources in local docker which could be used. there were no others run containers.

hnwyllmm commented 12 months ago

@LINxiansheng Please take a look. It's difficult to debug the scenario, could you please provide some solutions in the feature?

LINxiansheng commented 12 months ago

The observer started failed. Please provide the observer log in ~/ob/log.

YuriyGavrilov commented 11 months ago

@LINxiansheng This is docker log and full logs ocean_logs.zip


2023-10-09 22:18:05 generate boot.yaml ...
2023-10-09 22:18:05 oceanbase-ce docker in mini mode
2023-10-09 22:18:06 create boot dirs and deploy ob cluster ...
2023-10-09 22:18:07 name: obagent
2023-10-09 22:18:07 version: 4.2.0
2023-10-09 22:18:07 release:100000062023080210.el7
2023-10-09 22:18:07 arch: x86_64
2023-10-09 22:18:07 md5: 5401a2aba62669742bda2c8309fab96520aa528b
2023-10-09 22:18:07 add /root/pkg/obagent-4.2.0-100000062023080210.el7.x86_64.rpm to local mirror
2023-10-09 22:18:07 name: oceanbase-ce
2023-10-09 22:18:07 version: 4.2.0.0
2023-10-09 22:18:07 release:100010032023083021.el7
2023-10-09 22:18:07 arch: x86_64
2023-10-09 22:18:07 md5: e77b40f1634befc1d8cb411beafef876e33e31f1
2023-10-09 22:18:07 add /root/pkg/oceanbase-ce-4.2.0.0-100010032023083021.el7.x86_64.rpm to local mirror
2023-10-09 22:18:07 name: oceanbase-ce-libs
2023-10-09 22:18:07 version: 4.2.0.0
2023-10-09 22:18:07 release:100010032023083021.el7
2023-10-09 22:18:07 arch: x86_64
2023-10-09 22:18:07 md5: e3222298d5b50aedf067323b542d290c5f913afb
2023-10-09 22:18:07 add /root/pkg/oceanbase-ce-libs-4.2.0.0-100010032023083021.el7.x86_64.rpm to local mirror
2023-10-09 22:18:07 Trace ID: 93d5448c-66d8-11ee-8b0e-0242ac110002
2023-10-09 22:18:07 If you want to view detailed obd logs, please run: obd display-trace 93d5448c-66d8-11ee-8b0e-0242ac110002
2023-10-09 22:18:08 +----------------------------------------------------------------------------------------------------------+
2023-10-09 22:18:08 |                                            local Package List                                            |
2023-10-09 22:18:08 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-09 22:18:08 | name              | version | release                | arch   | md5                                      |
2023-10-09 22:18:08 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-09 22:18:08 | obagent           | 4.2.0   | 100000062023080210.el7 | x86_64 | 5401a2aba62669742bda2c8309fab96520aa528b |
2023-10-09 22:18:08 | oceanbase-ce      | 4.2.0.0 | 100010032023083021.el7 | x86_64 | e77b40f1634befc1d8cb411beafef876e33e31f1 |
2023-10-09 22:18:08 | oceanbase-ce-libs | 4.2.0.0 | 100010032023083021.el7 | x86_64 | e3222298d5b50aedf067323b542d290c5f913afb |
2023-10-09 22:18:08 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-09 22:18:08 Trace ID: 945bdd4e-66d8-11ee-a662-0242ac110002
2023-10-09 22:18:08 If you want to view detailed obd logs, please run: obd display-trace 945bdd4e-66d8-11ee-a662-0242ac110002
2023-10-09 22:18:08 Local deploy is empty
2023-10-09 22:18:08 Trace ID: 94c3b464-66d8-11ee-8111-0242ac110002
2023-10-09 22:18:08 If you want to view detailed obd logs, please run: obd display-trace 94c3b464-66d8-11ee-8111-0242ac110002
2023-10-09 22:18:09 Dev Mode: ON
2023-10-09 22:18:09 Trace ID: 952d2bba-66d8-11ee-a81c-0242ac110002
2023-10-09 22:18:09 If you want to view detailed obd logs, please run: obd display-trace 952d2bba-66d8-11ee-a81c-0242ac110002
2023-10-09 22:18:10 Package oceanbase-ce-4.2.0.0-100010032023083021.el7 is available.
2023-10-09 22:18:10 Package obagent-4.2.0-100000062023080210.el7 is available.
2023-10-09 22:18:25 install oceanbase-ce-4.2.0.0 for local ok
2023-10-09 22:18:30 install obagent-4.2.0 for local ok
2023-10-09 22:18:30 Cluster param config check ok
2023-10-09 22:18:30 Open ssh connection ok
2023-10-09 22:18:30 Generate observer configuration ok
2023-10-09 22:18:30 Generate obagent configuration ok
2023-10-09 22:18:30 +--------------------------------------------------------------------------------------------+
2023-10-09 22:18:30 |                                          Packages                                          |
2023-10-09 22:18:30 +--------------+---------+------------------------+------------------------------------------+
2023-10-09 22:18:30 | Repository   | Version | Release                | Md5                                      |
2023-10-09 22:18:30 +--------------+---------+------------------------+------------------------------------------+
2023-10-09 22:18:30 | oceanbase-ce | 4.2.0.0 | 100010032023083021.el7 | e77b40f1634befc1d8cb411beafef876e33e31f1 |
2023-10-09 22:18:30 | obagent      | 4.2.0   | 100000062023080210.el7 | 5401a2aba62669742bda2c8309fab96520aa528b |
2023-10-09 22:18:30 +--------------+---------+------------------------+------------------------------------------+
2023-10-09 22:18:30 Repository integrity check ok
2023-10-09 22:18:30 Parameter check ok
2023-10-09 22:18:30 Cluster status check ok
2023-10-09 22:18:31 Initializes observer work home ok
2023-10-09 22:18:31 Initializes obagent work home ok
2023-10-09 22:18:31 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository install ok
2023-10-09 22:18:31 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository lib check !!
2023-10-09 22:18:31 Remote obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b repository install ok
2023-10-09 22:18:32 Remote obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b repository lib check ok
2023-10-09 22:18:32 Try to get lib-repository
2023-10-09 22:18:32 Package oceanbase-ce-libs-4.2.0.0-100010032023083021.el7 is available.
2023-10-09 22:18:32 install oceanbase-ce-libs-4.2.0.0 for local ok
2023-10-09 22:18:32 Remote oceanbase-ce-libs-4.2.0.0-100010032023083021.el7-e3222298d5b50aedf067323b542d290c5f913afb repository install ok
2023-10-09 22:18:32 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository lib check ok
2023-10-09 22:18:32 obcluster deployed
2023-10-09 22:18:32 Get local repositories ok
2023-10-09 22:18:32 Search plugins ok
2023-10-09 22:18:32 Open ssh connection ok
2023-10-09 22:18:32 Load cluster param plugin ok
2023-10-09 22:18:32 Check before start observer ok
2023-10-09 22:18:32 [WARN] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
2023-10-09 22:18:32 
2023-10-09 22:18:32 Check before start obagent ok
2023-10-09 22:18:32 Start observer ok
2023-10-09 22:18:35 observer program health check x
2023-10-09 22:18:35 [WARN] OBD-2002: Failed to start 127.0.0.1 observer
2023-10-09 22:18:35 [ERROR] oceanbase-ce start failed
2023-10-09 22:18:35 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-09 22:18:35 Trace ID: 95948eb8-66d8-11ee-887b-0242ac110002
2023-10-09 22:18:35 If you want to view detailed obd logs, please run: obd display-trace 95948eb8-66d8-11ee-887b-0242ac110002
Deploy "obcluster" is deployed
2023-10-09 22:18:36 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-09 22:18:36 Trace ID: a4f4bbe4-66d8-11ee-8469-0242ac110002
2023-10-09 22:18:36 If you want to view detailed obd logs, please run: obd display-trace a4f4bbe4-66d8-11ee-8469-0242ac110002
2023-10-09 22:18:36 deploy success!
2023-10-09 22:18:36 boot success!
2023-10-09 22:18:36 ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)

caifeizhi commented 11 months ago

[2023-10-12 19:43:30.702434] INFO  [SERVER] init (ob_server.cpp:233) [238][observer][T0][Y0-0000000000000000-0-0] [lt=16] [OBSERVER_NOTICE] start to init observer
feizhi.cfz@feizhicfzdeMacBook-Pro ocean_logs % vim observer.log
feizhi.cfz@feizhicfzdeMacBook-Pro ocean_logs % tail -f observer.log
[2023-10-12 19:43:32.384846] INFO  [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [280][DiagnoseQueueTh][T0][Y0-0000000000000000-0-0] [lt=5] Init thread local success

i check the observer.log, and but the log timestamp(last timestamp is 2023-10-12 19:43:32) is not match docker.log(2023-10-09 22:18:36);

@LINxiansheng This is docker log and full logs ocean_logs.zip

@YuriyGavrilov is this ocean_logs and docker log are in the same run?

YuriyGavrilov commented 11 months ago

@caifeizhi sorry for disappointing it is different run

caifeizhi commented 11 months ago

@caifeizhi sorry for disappointing it is different run

Never mind. Could you please reproduce it and send the full log (both docker logs and oceanbase log)?

YuriyGavrilov commented 11 months ago

@caifeizhi fresh logs:

Docker

2023-10-14 19:50:45 generate boot.yaml ...
2023-10-14 19:50:45 oceanbase-ce docker in mini mode
2023-10-14 19:50:45 create boot dirs and deploy ob cluster ...
2023-10-14 19:50:46 name: obagent
2023-10-14 19:50:46 version: 4.2.0
2023-10-14 19:50:46 release:100000062023080210.el7
2023-10-14 19:50:46 arch: x86_64
2023-10-14 19:50:46 md5: 5401a2aba62669742bda2c8309fab96520aa528b
2023-10-14 19:50:46 add /root/pkg/obagent-4.2.0-100000062023080210.el7.x86_64.rpm to local mirror
2023-10-14 19:50:46 name: oceanbase-ce
2023-10-14 19:50:46 version: 4.2.0.0
2023-10-14 19:50:46 release:100010032023083021.el7
2023-10-14 19:50:46 arch: x86_64
2023-10-14 19:50:46 md5: e77b40f1634befc1d8cb411beafef876e33e31f1
2023-10-14 19:50:47 add /root/pkg/oceanbase-ce-4.2.0.0-100010032023083021.el7.x86_64.rpm to local mirror
2023-10-14 19:50:47 name: oceanbase-ce-libs
2023-10-14 19:50:47 version: 4.2.0.0
2023-10-14 19:50:47 release:100010032023083021.el7
2023-10-14 19:50:47 arch: x86_64
2023-10-14 19:50:47 md5: e3222298d5b50aedf067323b542d290c5f913afb
2023-10-14 19:50:47 add /root/pkg/oceanbase-ce-libs-4.2.0.0-100010032023083021.el7.x86_64.rpm to local mirror
2023-10-14 19:50:47 Trace ID: d277beb8-6ab1-11ee-8c89-0242ac110002
2023-10-14 19:50:47 If you want to view detailed obd logs, please run: obd display-trace d277beb8-6ab1-11ee-8c89-0242ac110002
2023-10-14 19:50:48 +----------------------------------------------------------------------------------------------------------+
2023-10-14 19:50:48 |                                            local Package List                                            |
2023-10-14 19:50:48 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-14 19:50:48 | name              | version | release                | arch   | md5                                      |
2023-10-14 19:50:48 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-14 19:50:48 | obagent           | 4.2.0   | 100000062023080210.el7 | x86_64 | 5401a2aba62669742bda2c8309fab96520aa528b |
2023-10-14 19:50:48 | oceanbase-ce      | 4.2.0.0 | 100010032023083021.el7 | x86_64 | e77b40f1634befc1d8cb411beafef876e33e31f1 |
2023-10-14 19:50:48 | oceanbase-ce-libs | 4.2.0.0 | 100010032023083021.el7 | x86_64 | e3222298d5b50aedf067323b542d290c5f913afb |
2023-10-14 19:50:48 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-14 19:50:48 Trace ID: d36a8418-6ab1-11ee-8937-0242ac110002
2023-10-14 19:50:48 If you want to view detailed obd logs, please run: obd display-trace d36a8418-6ab1-11ee-8937-0242ac110002
2023-10-14 19:50:49 Local deploy is empty
2023-10-14 19:50:49 Trace ID: d3e3a41a-6ab1-11ee-8cdd-0242ac110002
2023-10-14 19:50:49 If you want to view detailed obd logs, please run: obd display-trace d3e3a41a-6ab1-11ee-8cdd-0242ac110002
2023-10-14 19:50:50 Dev Mode: ON
2023-10-14 19:50:50 Trace ID: d46bcffc-6ab1-11ee-8859-0242ac110002
2023-10-14 19:50:50 If you want to view detailed obd logs, please run: obd display-trace d46bcffc-6ab1-11ee-8859-0242ac110002
2023-10-14 19:50:50 Package oceanbase-ce-4.2.0.0-100010032023083021.el7 is available.
2023-10-14 19:50:50 Package obagent-4.2.0-100000062023080210.el7 is available.
2023-10-14 19:51:08 install oceanbase-ce-4.2.0.0 for local ok
2023-10-14 19:51:14 install obagent-4.2.0 for local ok
2023-10-14 19:51:14 Cluster param config check ok
2023-10-14 19:51:14 Open ssh connection ok
2023-10-14 19:51:14 Generate observer configuration ok
2023-10-14 19:51:14 Generate obagent configuration ok
2023-10-14 19:51:14 +--------------------------------------------------------------------------------------------+
2023-10-14 19:51:14 |                                          Packages                                          |
2023-10-14 19:51:14 +--------------+---------+------------------------+------------------------------------------+
2023-10-14 19:51:14 | Repository   | Version | Release                | Md5                                      |
2023-10-14 19:51:14 +--------------+---------+------------------------+------------------------------------------+
2023-10-14 19:51:14 | oceanbase-ce | 4.2.0.0 | 100010032023083021.el7 | e77b40f1634befc1d8cb411beafef876e33e31f1 |
2023-10-14 19:51:14 | obagent      | 4.2.0   | 100000062023080210.el7 | 5401a2aba62669742bda2c8309fab96520aa528b |
2023-10-14 19:51:14 +--------------+---------+------------------------+------------------------------------------+
2023-10-14 19:51:14 Repository integrity check ok
2023-10-14 19:51:14 Parameter check ok
2023-10-14 19:51:14 Cluster status check ok
2023-10-14 19:51:14 Initializes observer work home ok
2023-10-14 19:51:14 Initializes obagent work home ok
2023-10-14 19:51:17 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository install ok
2023-10-14 19:51:17 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository lib check !!
2023-10-14 19:51:18 Remote obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b repository install ok
2023-10-14 19:51:18 Remote obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b repository lib check ok
2023-10-14 19:51:18 Try to get lib-repository
2023-10-14 19:51:18 Package oceanbase-ce-libs-4.2.0.0-100010032023083021.el7 is available.
2023-10-14 19:51:18 install oceanbase-ce-libs-4.2.0.0 for local ok
2023-10-14 19:51:18 Remote oceanbase-ce-libs-4.2.0.0-100010032023083021.el7-e3222298d5b50aedf067323b542d290c5f913afb repository install ok
2023-10-14 19:51:18 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository lib check ok
2023-10-14 19:51:18 obcluster deployed
2023-10-14 19:51:18 Get local repositories ok
2023-10-14 19:51:18 Search plugins ok
2023-10-14 19:51:18 Open ssh connection ok
2023-10-14 19:51:18 Load cluster param plugin ok
2023-10-14 19:51:18 Check before start observer ok
2023-10-14 19:51:18 [WARN] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
2023-10-14 19:51:18 
2023-10-14 19:51:18 Check before start obagent ok
2023-10-14 19:51:18 Start observer ok
2023-10-14 19:51:21 observer program health check x
2023-10-14 19:51:21 [WARN] OBD-2002: Failed to start 127.0.0.1 observer
2023-10-14 19:51:21 [ERROR] oceanbase-ce start failed
2023-10-14 19:51:21 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-14 19:51:21 Trace ID: d4f33582-6ab1-11ee-85ca-0242ac110002
2023-10-14 19:51:21 If you want to view detailed obd logs, please run: obd display-trace d4f33582-6ab1-11ee-85ca-0242ac110002
Deploy "obcluster" is deployed
2023-10-14 19:51:22 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-14 19:51:22 Trace ID: e7dc3fcc-6ab1-11ee-ad2a-0242ac110002
2023-10-14 19:51:22 If you want to view detailed obd logs, please run: obd display-trace e7dc3fcc-6ab1-11ee-ad2a-0242ac110002
2023-10-14 19:51:22 ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)
2023-10-14 19:51:22 deploy success!
2023-10-14 19:51:22 boot success!

files: ocean_logs.zip

try to open interactively

(base) yuriygavrilov@MBP-Yuriy Downloads % docker exec -it oceanbase-ce ob-mysql sys 
login as root@sys
Command is: obclient -h127.1 -uroot@sys -A -Doceanbase -P2881 
ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)
(base) yuriygavrilov@MBP-Yuriy Downloads %

caifeizhi commented 11 months ago

sorry for the late reply. i found:

oceanbase logs end in '2023-10-14 16:51:19.871289' , and lack of clear error logs indicating the cause of the exit.
obd log end in '2023-10-14 19:51:21' (+3 hour), indicating that it failed to connect to OceanBase as the observer has exited.

Therefore, I suspect that OceanBase was terminated (possibly with a -9 signal) by the operating system. You may want to check the logs in dmesg for more information.

YuriyGavrilov commented 11 months ago

@caifeizhi here is fresh logs https://github.com/oceanbase/docker-images/issues/24. 16:51 and 19:51 difference it is just UTC+3 timezone difference I think. May be I will make update it soon again.

YuriyGavrilov commented 11 months ago

some fresh logs:

2023-10-31 23:43:41 generate boot.yaml ...
2023-10-31 23:43:41 oceanbase-ce docker in mini mode
2023-10-31 23:43:41 create boot dirs and deploy ob cluster ...
2023-10-31 23:43:43 name: obagent
2023-10-31 23:43:43 version: 4.2.0
2023-10-31 23:43:43 release:100000062023080210.el7
2023-10-31 23:43:43 arch: x86_64
2023-10-31 23:43:43 md5: 5401a2aba62669742bda2c8309fab96520aa528b
2023-10-31 23:43:43 add /root/pkg/obagent-4.2.0-100000062023080210.el7.x86_64.rpm to local mirror
2023-10-31 23:43:43 name: oceanbase-ce
2023-10-31 23:43:43 version: 4.2.0.0
2023-10-31 23:43:43 release:100010032023083021.el7
2023-10-31 23:43:43 arch: x86_64
2023-10-31 23:43:43 md5: e77b40f1634befc1d8cb411beafef876e33e31f1
2023-10-31 23:43:43 add /root/pkg/oceanbase-ce-4.2.0.0-100010032023083021.el7.x86_64.rpm to local mirror
2023-10-31 23:43:43 name: oceanbase-ce-libs
2023-10-31 23:43:43 version: 4.2.0.0
2023-10-31 23:43:43 release:100010032023083021.el7
2023-10-31 23:43:43 arch: x86_64
2023-10-31 23:43:43 md5: e3222298d5b50aedf067323b542d290c5f913afb
2023-10-31 23:43:43 add /root/pkg/oceanbase-ce-libs-4.2.0.0-100010032023083021.el7.x86_64.rpm to local mirror
2023-10-31 23:43:43 Trace ID: 2e02e378-782e-11ee-9106-0242ac110002
2023-10-31 23:43:43 If you want to view detailed obd logs, please run: obd display-trace 2e02e378-782e-11ee-9106-0242ac110002
2023-10-31 23:43:44 +----------------------------------------------------------------------------------------------------------+
2023-10-31 23:43:44 |                                            local Package List                                            |
2023-10-31 23:43:44 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-31 23:43:44 | name              | version | release                | arch   | md5                                      |
2023-10-31 23:43:44 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-31 23:43:44 | obagent           | 4.2.0   | 100000062023080210.el7 | x86_64 | 5401a2aba62669742bda2c8309fab96520aa528b |
2023-10-31 23:43:44 | oceanbase-ce      | 4.2.0.0 | 100010032023083021.el7 | x86_64 | e77b40f1634befc1d8cb411beafef876e33e31f1 |
2023-10-31 23:43:44 | oceanbase-ce-libs | 4.2.0.0 | 100010032023083021.el7 | x86_64 | e3222298d5b50aedf067323b542d290c5f913afb |
2023-10-31 23:43:44 +-------------------+---------+------------------------+--------+------------------------------------------+
2023-10-31 23:43:44 Trace ID: 2eb0d19a-782e-11ee-9a02-0242ac110002
2023-10-31 23:43:44 If you want to view detailed obd logs, please run: obd display-trace 2eb0d19a-782e-11ee-9a02-0242ac110002
2023-10-31 23:43:44 Local deploy is empty
2023-10-31 23:43:44 Trace ID: 2f23939c-782e-11ee-b9c8-0242ac110002
2023-10-31 23:43:44 If you want to view detailed obd logs, please run: obd display-trace 2f23939c-782e-11ee-b9c8-0242ac110002
2023-10-31 23:43:45 Dev Mode: ON
2023-10-31 23:43:45 Trace ID: 2f8e2720-782e-11ee-b75e-0242ac110002
2023-10-31 23:43:45 If you want to view detailed obd logs, please run: obd display-trace 2f8e2720-782e-11ee-b75e-0242ac110002
2023-10-31 23:43:46 Package oceanbase-ce-4.2.0.0-100010032023083021.el7 is available.
2023-10-31 23:43:46 Package obagent-4.2.0-100000062023080210.el7 is available.
2023-10-31 23:44:03 install oceanbase-ce-4.2.0.0 for local ok
2023-10-31 23:44:08 install obagent-4.2.0 for local ok
2023-10-31 23:44:09 Cluster param config check ok
2023-10-31 23:44:09 Open ssh connection ok
2023-10-31 23:44:09 Generate observer configuration ok
2023-10-31 23:44:09 Generate obagent configuration ok
2023-10-31 23:44:09 +--------------------------------------------------------------------------------------------+
2023-10-31 23:44:09 |                                          Packages                                          |
2023-10-31 23:44:09 +--------------+---------+------------------------+------------------------------------------+
2023-10-31 23:44:09 | Repository   | Version | Release                | Md5                                      |
2023-10-31 23:44:09 +--------------+---------+------------------------+------------------------------------------+
2023-10-31 23:44:09 | oceanbase-ce | 4.2.0.0 | 100010032023083021.el7 | e77b40f1634befc1d8cb411beafef876e33e31f1 |
2023-10-31 23:44:09 | obagent      | 4.2.0   | 100000062023080210.el7 | 5401a2aba62669742bda2c8309fab96520aa528b |
2023-10-31 23:44:09 +--------------+---------+------------------------+------------------------------------------+
2023-10-31 23:44:09 Repository integrity check ok
2023-10-31 23:44:09 Parameter check ok
2023-10-31 23:44:09 Cluster status check ok
2023-10-31 23:44:09 Initializes observer work home ok
2023-10-31 23:44:09 Initializes obagent work home ok
2023-10-31 23:44:11 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository install ok
2023-10-31 23:44:11 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository lib check !!
2023-10-31 23:44:11 Remote obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b repository install ok
2023-10-31 23:44:11 Remote obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b repository lib check ok
2023-10-31 23:44:11 Try to get lib-repository
2023-10-31 23:44:11 Package oceanbase-ce-libs-4.2.0.0-100010032023083021.el7 is available.
2023-10-31 23:44:11 install oceanbase-ce-libs-4.2.0.0 for local ok
2023-10-31 23:44:11 Remote oceanbase-ce-libs-4.2.0.0-100010032023083021.el7-e3222298d5b50aedf067323b542d290c5f913afb repository install ok
2023-10-31 23:44:11 Remote oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1 repository lib check ok
2023-10-31 23:44:11 obcluster deployed
2023-10-31 23:44:11 Get local repositories ok
2023-10-31 23:44:11 Search plugins ok
2023-10-31 23:44:11 Open ssh connection ok
2023-10-31 23:44:11 Load cluster param plugin ok
2023-10-31 23:44:11 Check before start observer ok
2023-10-31 23:44:11 [WARN] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
2023-10-31 23:44:11 
2023-10-31 23:44:11 Check before start obagent ok
2023-10-31 23:44:12 Start observer ok
2023-10-31 23:44:15 observer program health check x
2023-10-31 23:44:15 [WARN] OBD-2002: Failed to start 127.0.0.1 observer
2023-10-31 23:44:15 [ERROR] oceanbase-ce start failed
2023-10-31 23:44:15 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-31 23:44:15 Trace ID: 2fffb7be-782e-11ee-973f-0242ac110002
2023-10-31 23:44:15 If you want to view detailed obd logs, please run: obd display-trace 2fffb7be-782e-11ee-973f-0242ac110002
Deploy "obcluster" is deployed
2023-10-31 23:44:15 See https://www.oceanbase.com/product/ob-deployer/error-codes .
2023-10-31 23:44:15 Trace ID: 4188da10-782e-11ee-a0f6-0242ac110002
2023-10-31 23:44:15 If you want to view detailed obd logs, please run: obd display-trace 4188da10-782e-11ee-a0f6-0242ac110002
2023-10-31 23:44:15 deploy success!
2023-10-31 23:44:15 boot success!
2023-10-31 23:44:15 ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115)

LINxiansheng commented 11 months ago

Sorry for reply late.

The obd log shows that the observer process started failed.
The observer log show no ERROR log. There are many WDIAG warning log such as clock, cgroup and sql failed but they may not cause the observer process down.
It seems that the observer core with no core file or killed by some other process.

Solutions:

Check why no core file. Maybe the env ulimit is no config. Maybe the file has no space.
Maybe the docker images Out of Memory, and then observer core. We can find the log in dmesg
Maybe some process kill the observer. The observer log has no receive kill -15(soft term) signal. So maybe the observer killed by received kill -9 signal. We can find some infomation by some tools such as trace, gdb or python signal.
Maybe observer killed by observer self and the log will not show because the log is async. So we can run the observer ./bin/observer -o enable_async_syslog=False or use gdb.

This is difficult for new developer to check them and I see you cpu and memory configuration. So maybe you can check your file space firstly.

YuriyGavrilov commented 10 months ago

@LINxiansheng thx for helping. I will check!

oceanbase / docker-images

[Bug]: docker run with errors ERROR 2002 (HY000): Can't connect to OceanBase server on '127.1' (115) #24