OpenBMB / XAgent

An Autonomous LLM Agent for Complex Task Solving
https://blog.x-agent.net/blog/xagent/
Apache License 2.0
7.82k stars 794 forks source link

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> #383

Open sheng-di opened 5 months ago

sheng-di commented 5 months ago

Issue Description / 问题描述

Please provide a detailed description of the error or issue you encountered. / 请详细描述您遇到的错误或问题。

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

Steps to Reproduce / 复现步骤

Please provide the specific steps to reproduce the error. / 请提供复现错误的具体步骤。

docker compose up

Expected Behavior / 预期行为

Describe the behavior you expected to see. / 请描述您期望的正确行为。 No Error.

Environment / 环境信息

Error Screenshots or Logs / 错误截图或日志

If possible, please provide relevant screenshots or logs of the error. / 如果可能,请提供相关的错误截图或日志文件。

Additional Notes / 其他备注

If you have any additional information or notes, please add them here. / 如果有其他补充信息,请在此处添加。

➜  XAgent git:(main) ✗ docker compose up
[+] Running 5/0
 ⠿ Container xagent-ToolServerNode-1  Created                                                                                                                                  0.0s
 ⠿ Container xagent-xagent-mysql-1    Running                                                                                                                                  0.0s
 ⠿ Container xagent-db-1              Running                                                                                                                                  0.0s
 ⠿ Container xagent-xagent-redis-1    Running                                                                                                                                  0.0s
 ⠿ Container XAgent-Server            Running                                                                                                                                  0.0s
Attaching to XAgent-Server, xagent-ToolServerManager-1, xagent-ToolServerNode-1, xagent-db-1, xagent-xagent-mysql-1, xagent-xagent-redis-1
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Starting gunicorn 21.2.0
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [7] [INFO] Booting worker with pid: 7
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [8] [INFO] Booting worker with pid: 8
xagent-ToolServerNode-1     | Traceback (most recent call last):
xagent-ToolServerNode-1     |   File "/usr/local/bin/uvicorn", line 8, in <module>
xagent-ToolServerNode-1     |     sys.exit(main())
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
xagent-ToolServerNode-1     |     return self.main(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
xagent-ToolServerNode-1     |     rv = self.invoke(ctx)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
xagent-ToolServerNode-1     |     return ctx.invoke(self.callback, **ctx.params)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
xagent-ToolServerNode-1     |     return __callback(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
xagent-ToolServerNode-1     |     run(
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
xagent-ToolServerNode-1     |     server.run()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
xagent-ToolServerNode-1     |     return asyncio.run(self.serve(sockets=sockets))
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
xagent-ToolServerNode-1     |     return loop.run_until_complete(main)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
xagent-ToolServerNode-1     |     return future.result()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
xagent-ToolServerNode-1     |     config.load()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
xagent-ToolServerNode-1     |     self.loaded_app = import_from_string(self.app)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
xagent-ToolServerNode-1     |     module = importlib.import_module(module_str)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerNode-1     |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerNode-1     |   File "/app/main.py", line 13, in <module>
xagent-ToolServerNode-1     |     from config import CONFIG,logger
xagent-ToolServerNode-1     |   File "/app/config.py", line 73, in <module>
xagent-ToolServerNode-1     |     CONFIG = NodeConfig()
xagent-ToolServerNode-1     |   File "/app/config.py", line 31, in __init__
xagent-ToolServerNode-1     |     self.cfg:Dict = yaml.load(open(config_file_path, "r", encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerNode-1     | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/node.yml'
xagent-ToolServerNode-1 exited with code 1
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [7] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [8] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [8] [INFO] Worker exiting (pid: 8)
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [7] [INFO] Worker exiting (pid: 7)
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [1] [ERROR] Worker (pid:7) exited with code 3
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [1] [ERROR] Worker (pid:8) exited with code 3
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 209, in run
xagent-ToolServerManager-1  |     self.sleep()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 360, in sleep
xagent-ToolServerManager-1  |     ready = select.select([self.PIPE[0]], [], [], 1.0)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
xagent-ToolServerManager-1  |     self.reap_workers()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
xagent-ToolServerManager-1  |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
xagent-ToolServerManager-1  | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | During handling of the above exception, another exception occurred:
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 662, in kill_worker
xagent-ToolServerManager-1  |     os.kill(pid, sig)
xagent-ToolServerManager-1  | ProcessLookupError: [Errno 3] No such process
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | During handling of the above exception, another exception occurred:
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/bin/gunicorn", line 8, in <module>
xagent-ToolServerManager-1  |     sys.exit(run())
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
xagent-ToolServerManager-1  |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 236, in run
xagent-ToolServerManager-1  |     super().run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run
xagent-ToolServerManager-1  |     Arbiter(self).run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 229, in run
xagent-ToolServerManager-1  |     self.halt(reason=inst.reason, exit_status=inst.exit_status)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 342, in halt
xagent-ToolServerManager-1  |     self.stop()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 393, in stop
xagent-ToolServerManager-1  |     self.kill_workers(sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 652, in kill_workers
xagent-ToolServerManager-1  |     self.kill_worker(pid, sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 662, in kill_worker
xagent-ToolServerManager-1  |     os.kill(pid, sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
xagent-ToolServerManager-1  |     self.reap_workers()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
xagent-ToolServerManager-1  |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
xagent-ToolServerManager-1  | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
xagent-ToolServerManager-1 exited with code 1
sheng-di commented 5 months ago
docker compose up
[+] Running 6/0
 ⠿ Container xagent-db-1                 Created                                                                             0.0s
 ⠿ Container xagent-xagent-redis-1       Created                                                                             0.0s
 ⠿ Container xagent-ToolServerManager-1  Created                                                                             0.0s
 ⠿ Container xagent-xagent-mysql-1       Created                                                                             0.0s
 ⠿ Container xagent-ToolServerNode-1     Created                                                                             0.0s
 ⠿ Container XAgent-Server               Created                                                                             0.0s
Attaching to XAgent-Server, xagent-ToolServerManager-1, xagent-ToolServerNode-1, xagent-db-1, xagent-xagent-mysql-1, xagent-xagent-redis-1
^Bxagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.542+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.544+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":21},"outgoing":{"minWireVersion":6,"maxWireVersion":21},"isInternalClient":true}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.546+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrationDonors"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMigrationRecipients"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.553+00:00"},"s":"I",  "c":"TENANT_M", "id":7091600, "ctx":"main","msg":"Starting TenantMigrationAccessBlockerRegistry"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"daf36b8df185"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"7.0.5","gitVersion":"7809d71e84e314b497f282ea8aa06d7ded3eb205","openSSLVersion":"OpenSSL 3.0.2 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2204","distarch":"x86_64","target_arch":"x86_64"}}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.554+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"security":{"authorization":"enabled"}}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.557+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.557+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:17.557+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3299M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.710 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.712 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.712 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
xagent-xagent-redis-1       | 1:C 28 Jan 2024 10:18:17.712 * Configuration loaded
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.712 * monotonic clock: POSIX clock_gettime
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.714 * Running mode=standalone, port=6379.
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.716 * Server initialized
xagent-xagent-redis-1       | 1:M 28 Jan 2024 10:18:17.716 * Ready to accept connections tcp
xagent-xagent-mysql-1       | 2024-01-28 10:18:17+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
xagent-xagent-mysql-1       | 2024-01-28 10:18:18+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
xagent-xagent-mysql-1       | 2024-01-28 10:18:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
xagent-xagent-mysql-1       | '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [1] [INFO] Starting gunicorn 21.2.0
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [7] [INFO] Booting worker with pid: 7
xagent-ToolServerManager-1  | [2024-01-28 10:18:18 +0000] [8] [INFO] Booting worker with pid: 8
xagent-ToolServerNode-1     | Traceback (most recent call last):
xagent-ToolServerNode-1     |   File "/usr/local/bin/uvicorn", line 8, in <module>
xagent-ToolServerNode-1     |     sys.exit(main())
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
xagent-ToolServerNode-1     |     return self.main(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
xagent-ToolServerNode-1     |     rv = self.invoke(ctx)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
xagent-ToolServerNode-1     |     return ctx.invoke(self.callback, **ctx.params)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
xagent-ToolServerNode-1     |     return __callback(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
xagent-ToolServerNode-1     |     run(
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
xagent-ToolServerNode-1     |     server.run()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
xagent-ToolServerNode-1     |     return asyncio.run(self.serve(sockets=sockets))
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
xagent-ToolServerNode-1     |     return loop.run_until_complete(main)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
xagent-ToolServerNode-1     |     return future.result()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
xagent-ToolServerNode-1     |     config.load()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
xagent-ToolServerNode-1     |     self.loaded_app = import_from_string(self.app)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
xagent-ToolServerNode-1     |     module = importlib.import_module(module_str)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerNode-1     |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerNode-1     |   File "/app/main.py", line 13, in <module>
xagent-ToolServerNode-1     |     from config import CONFIG,logger
xagent-ToolServerNode-1     |   File "/app/config.py", line 73, in <module>
xagent-ToolServerNode-1     |     CONFIG = NodeConfig()
xagent-ToolServerNode-1     |   File "/app/config.py", line 31, in __init__
xagent-ToolServerNode-1     |     self.cfg:Dict = yaml.load(open(config_file_path, "r", encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerNode-1     | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/node.yml'
xagent-xagent-mysql-1       | 2024-01-28T10:18:18.253736Z 0 [System] [MY-015015] [Server] MySQL Server - start.
xagent-xagent-mysql-1       | 2024-01-28T10:18:18.873413Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1
xagent-xagent-mysql-1       | 2024-01-28T10:18:18.903021Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.408+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":1850}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.408+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.412621Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [8] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.422+00:00"},"s":"W",  "c":"CONTROL",  "id":22178,   "ctx":"initandlisten","msg":"/sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'","tags":["startupWarnings"]}
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.423+00:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [8] [INFO] Worker exiting (pid: 8)
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.429+00:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":21},"outgoing":{"minWireVersion":6,"maxWireVersion":21},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":21,"maxWireVersion":21},"outgoing":{"minWireVersion":21,"maxWireVersion":21},"isInternalClient":true}}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.429+00:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"7.0","context":"startup"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.430+00:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.432+00:00"},"s":"I",  "c":"CONTROL",  "id":6608200, "ctx":"initandlisten","msg":"Initializing cluster server parameters from disk"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.432+00:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.439+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [7] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [7] [INFO] Worker exiting (pid: 7)
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.452+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.452+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
xagent-db-1                 | {"t":{"$date":"2024-01-28T10:18:19.456+00:00"},"s":"I",  "c":"CONTROL",  "id":8423403, "ctx":"initandlisten","msg":"mongod startup complete","attr":{"Summary of time elapsed":{"Startup from clean shutdown?":true,"Statistics":{"Transport layer setup":"0 ms","Run initial syncer crash recovery":"0 ms","Create storage engine lock file in the data directory":"0 ms","Get metadata describing storage engine":"0 ms","Validate options in metadata against current startup options":"0 ms","Create storage engine":"1853 ms","Write current PID to file":"10 ms","Initialize FCV before rebuilding indexes":"6 ms","Drop abandoned idents and get back indexes that need to be rebuilt or builds that need to be restarted":"1 ms","Rebuild indexes for collections":"0 ms","Load cluster parameters from disk for a standalone":"1 ms","Build user and roles graph":"0 ms","Verify indexes for admin.system.users collection":"0 ms","Set up the background thread pool responsible for waiting for opTimes to be majority committed":"1 ms","Initialize information needed to make a mongod instance shard aware":"0 ms","Start up the replication coordinator":"2 ms","Start transport layer":"1 ms","_initAndListen total elapsed time":"1902 ms"}}}}
xagent-ToolServerNode-1 exited with code 1
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Worker (pid:8) exited with code 3
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Worker (pid:7) was sent SIGTERM!
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.616525Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.620672Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.657981Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.658054Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.661305Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Shutting down: Master
xagent-ToolServerManager-1  | [2024-01-28 10:18:19 +0000] [1] [ERROR] Reason: Worker failed to boot.
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.690017Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
xagent-xagent-mysql-1       | 2024-01-28T10:18:19.690152Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.3.0'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
xagent-ToolServerManager-1 exited with code 3
sailaoda commented 3 months ago

hello, please check your file "./assets/config/node.yml and manager.yml" like this

image