search

meolu / walle-web

walle - 瓦力 Devops开源项目代码部署平台
https://walle-web.io
Apache License 2.0
11.85k stars 2.76k forks source link

BUG 创建发布单时候，如果分支某种原因没有列出来，则整个进程都hang住了 #627

Closed wu0407 closed 5 years ago

wu0407 commented 5 years ago

问题描述 在owner账号下，创建上线单，执行[ -d ".git" ] && git status卡住，则任何请求都会卡主。

python版本：3.4.9 (default, Aug 14 2018, 21:28:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] walle版本： 0d96099886960760d580fe49f072eeac8a8a5f86

问题描述 strace 发现卡在

strace: Process 17123 attached with 2 threads
[pid 17123] 23:04:49 futex(0x2a84b70, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17130] 23:04:49 futex(0x7fe2e0004de0, FUTEX_WAIT_PRIVATE, 0, NULL

gdb的py-bt

Traceback (most recent call first):
  File "/usr/lib64/python3.4/threading.py", line 1067, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
  File "/usr/lib64/python3.4/threading.py", line 1051, in join
    self._wait_for_tstate_lock()
  File "/data/venv/lib/python3.4/site-packages/invoke/runners.py", line 344, in _run_body
    thread.join(self._thread_timeout(target))
  File "/data/venv/lib/python3.4/site-packages/invoke/runners.py", line 271, in run
    return self._run_body(command, **kwargs)
  File "/data/venv/lib/python3.4/site-packages/invoke/context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "/data/venv/lib/python3.4/site-packages/invoke/context.py", line 94, in run
    return self._run(runner, command, **kwargs)
  File "/data/venv/lib/python3.4/site-packages/fabric2/connection.py", line 655, in local
    return super(Connection, self).run(*args, **kwargs)
  File "/mnt/hgfs/walle-web/walle/service/waller.py", line 51, in run
    result = super(Waller, self).local(command, pty=pty, warn=True, watchers=[say_yes()], env=self.custom_global_env, **kwargs)
  File "/mnt/hgfs/walle-web/walle/service/waller.py", line 136, in local
    return self.run(command, wenv=wenv, run_mode=self.run_mode_local, **kwargs)
  File "/mnt/hgfs/walle-web/walle/service/deployer.py", line 473, in init_repo
    is_git_dir = self.localhost.local('[ -d ".git" ] && git status', exception=False, wenv=self.config())
  File "/mnt/hgfs/walle-web/walle/service/deployer.py", line 407, in list_branch
    self.init_repo()
  File "/mnt/hgfs/walle-web/walle/service/websocket.py", line 62, in on_branches
    branches = wi.list_branch()
  File "/data/venv/lib/python3.4/site-packages/flask_socketio/__init__.py", line 634, in _handle_event
    ret = handler(*args)
  File "/data/venv/lib/python3.4/site-packages/flask_socketio/namespace.py", line 26, in trigger_event
    *args)
  File "/data/venv/lib/python3.4/site-packages/socketio/server.py", line 495, in _trigger_event
    event, *args)
  File "/data/venv/lib/python3.4/site-packages/socketio/server.py", line 461, in _handle_event_internal
    r = server._trigger_event(data[0], namespace, sid, *data[1:])
  File "/data/venv/lib/python3.4/site-packages/socketio/server.py", line 458, in _handle_event
    self._handle_event_internal(self, sid, data, namespace, id)
  File "/data/venv/lib/python3.4/site-packages/socketio/server.py", line 522, in _handle_eio_message
---Type <return> to continue, or q <return> to quit---
    self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
  File "/data/venv/lib/python3.4/site-packages/engineio/server.py", line 423, in _trigger_event
    return self.handlers[event](*args)
  File "/data/venv/lib/python3.4/site-packages/engineio/socket.py", line 58, in receive
    run_async=self.server.async_handlers)
  File "/data/venv/lib/python3.4/site-packages/engineio/socket.py", line 224, in _websocket_handler
    self.receive(pkt)
  File "/data/venv/lib/python3.4/site-packages/engineio/async_gevent.py", line 43, in __call__
    return self.app(self)
  File "/data/venv/lib/python3.4/site-packages/engineio/socket.py", line 145, in _upgrade_websocket
    return ws(environ, start_response)
  File "/data/venv/lib/python3.4/site-packages/engineio/socket.py", line 103, in handle_get_request
    start_response)
  File "/data/venv/lib/python3.4/site-packages/engineio/server.py", line 282, in handle_request
    environ, start_response)
  File "/data/venv/lib/python3.4/site-packages/socketio/server.py", line 360, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "/data/venv/lib/python3.4/site-packages/engineio/middleware.py", line 59, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "/data/venv/lib/python3.4/site-packages/flask_socketio/__init__.py", line 43, in __call__
    start_response)
  File "/data/venv/lib/python3.4/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/data/venv/lib/python3.4/site-packages/geventwebsocket/handler.py", line 52, in run_websocket
    list(self.application(self.environ, lambda s, h, e=None: []))
  File "/data/venv/lib/python3.4/site-packages/geventwebsocket/handler.py", line 75, in run_application
    self.run_websocket()
  File "/data/venv/lib/python3.4/site-packages/gevent/pywsgi.py", line 976, in handle_one_response
    self.run_application()
  File "/data/venv/lib/python3.4/site-packages/gevent/pywsgi.py", line 674, in handle_one_request
    self.handle_one_response()
  File "/data/venv/lib/python3.4/site-packages/gevent/pywsgi.py", line 457, in handle
---Type <return> to continue, or q <return> to quit---
    result = self.handle_one_request()
  File "/data/venv/lib/python3.4/site-packages/gevent/pywsgi.py", line 1519, in handle
    handler.handle()
  File "/data/venv/lib/python3.4/site-packages/gevent/baseserver.py", line 26, in _handle_and_close_when_done
    return handle(*args_tuple)
  File "/data/venv/lib/python3.4/site-packages/gevent/hub.py", line 565, in run
    loop.run()

日志堆栈贴上来。

logs/runtime.log

Server initialized for gevent.
 * Restarting with stat
walle.config.settings_prod_3
Server initialized for gevent.
 * Debugger is active!
 * Debugger PIN: 304-690-871
[2019-01-13 20:59:39,491] INFO in app: 2
[2019-01-13 20:59:41,192] INFO in app: 2
[2019-01-13 20:59:46,040] INFO in app: 2
[2019-01-13 20:59:46,061] INFO in access: <bound method UserModel.is_authenticated of {'avatar': '',
 'created_at': datetime.datetime(2017, 3, 20, 19, 5, 44),
 'email': 'owner@walle-web.io',
 'id': 2,
 'is_email_verified': 1,
 'last_space': 1,
 'password': 'pbkdf2:sha256:50000$AyRSJVSn$448c69b93158b30b9e3625d340b48dbdbce1186fcf30fc72663a9361ffec339b',
 'role': '',
 'status': 1,
 'updated_at': datetime.datetime(2018, 11, 24, 17, 1, 23),
 'username': 'Owner'}>
[2019-01-13 20:59:48,584] INFO in app: 2
[2019-01-13 20:59:48,622] INFO in access: <bound method UserModel.is_authenticated of {'avatar': '',
 'created_at': datetime.datetime(2017, 3, 20, 19, 5, 44),
 'email': 'owner@walle-web.io',
 'id': 2,
 'is_email_verified': 1,
 'last_space': 1,
 'password': 'pbkdf2:sha256:50000$AyRSJVSn$448c69b93158b30b9e3625d340b48dbdbce1186fcf30fc72663a9361ffec339b',
 'role': '',
 'status': 1,
 'updated_at': datetime.datetime(2018, 11, 24, 17, 1, 23),
 'username': 'Owner'}>
[2019-01-13 20:59:50,831] INFO in app: 2
[2019-01-13 20:59:50,849] INFO in access: <bound method UserModel.is_authenticated of {'avatar': '',
 'created_at': datetime.datetime(2017, 3, 20, 19, 5, 44),
 'email': 'owner@walle-web.io',
 'id': 2,
 'is_email_verified': 1,
 'last_space': 1,
 'password': 'pbkdf2:sha256:50000$AyRSJVSn$448c69b93158b30b9e3625d340b48dbdbce1186fcf30fc72663a9361ffec339b',
 'role': '',
 'status': 1,
 'updated_at': datetime.datetime(2018, 11, 24, 17, 1, 23),
 'username': 'Owner'}>
[2019-01-13 20:59:53,176] INFO in app: 2
[2019-01-13 20:59:53,241] INFO in access: <bound method UserModel.is_authenticated of {'avatar': '',
 'created_at': datetime.datetime(2017, 3, 20, 19, 5, 44),
 'email': 'owner@walle-web.io',
 'id': 2,
 'is_email_verified': 1,
 'last_space': 1,
 'password': 'pbkdf2:sha256:50000$AyRSJVSn$448c69b93158b30b9e3625d340b48dbdbce1186fcf30fc72663a9361ffec339b',
 'role': '',
 'status': 1,
 'updated_at': datetime.datetime(2018, 11, 24, 17, 1, 23),
 'username': 'Owner'}>
[2019-01-13 20:59:53,287] INFO in project: 1
[2019-01-13 20:59:53,362] INFO in project: {'enable_online': False, 'excludes': '', 'post_deploy': '', 'user_id': 2, 'status': 1, 'name': 'test', 'id': 1, 'space_id': 1, 'task_audit': 0, 'servers_info': [{'enable_online': False, 'enable_block': False, 'port': 22, 'name': 'localhost', 'id': 10, 'enable_audit': False, 'host': '127.0.0.1', 'enable_delete': True, 'enable_update': True, 'user': '', 'updated_at': '2018-12-19 20:23:37', 'enable_view': True, 'created_at': '2018-12-19 20:23:37', 'enable_create': False}], 'is_include': 0, 'enable_view': True, 'target_releases': '/home/xiaoqing/codes', 'post_release': '', 'repo_mode': 'branch', 'environment_id': 1, 'repo_url': 'git@git.bilibili.co:wuzunqing/live-uat-cache.git', 'repo_username': '', 'repo_password': '', 'enable_create': False, 'keep_version_num': 5, 'enable_block': False, 'task_vars': '', 'repo_type': None, 'prev_release': '', 'server_ids': '10', 'target_root': '/home/xiaoqing/deploy', 'enable_delete': True, 'enable_update': True, 'notice_type': '', 'enable_audit': False, 'prev_deploy': '', 'created_at': '2019-01-12 01:42:49', 'updated_at': '2019-01-12 01:42:49', 'version': None, 'master': '', 'notice_hook': ''}
[2019-01-13 20:59:53,443] INFO in member: []
[2019-01-13 20:59:53,444] INFO in member: []
fc5028ccaa5f40daa8138fa06a49dd33: Sending packet OPEN data {'sid': 'fc5028ccaa5f40daa8138fa06a49dd33', 'pingInterval': 25000, 'pingTimeout': 60000, 'upgrades': ['websocket']}
fc5028ccaa5f40daa8138fa06a49dd33: Sending packet MESSAGE data 0
fc5028ccaa5f40daa8138fa06a49dd33: Received packet MESSAGE data 0/walle,
fc5028ccaa5f40daa8138fa06a49dd33: Sending packet MESSAGE data 0/walle
fc5028ccaa5f40daa8138fa06a49dd33: Received request to upgrade to websocket
fc5028ccaa5f40daa8138fa06a49dd33: Received packet MESSAGE data 2/walle,["open",{"project_id":1}]
received event "open" from fc5028ccaa5f40daa8138fa06a49dd33 [/walle]
[2019-01-13 20:59:54,003] INFO in websocket: {'project_id': 1}
[2019-01-13 20:59:54,005] INFO in app: 2
fc5028ccaa5f40daa8138fa06a49dd33 is entering room 1 [/walle]
emitting event "construct" to 1 [/walle]
fc5028ccaa5f40daa8138fa06a49dd33: Sending packet MESSAGE data 2/walle,["construct",{"event":"connect","data":[]}]
fc5028ccaa5f40daa8138fa06a49dd33: Sending packet NOOP data None
fc5028ccaa5f40daa8138fa06a49dd33: Upgrade to websocket successful
fc5028ccaa5f40daa8138fa06a49dd33: Received packet MESSAGE data 2/walle,["branches"]
received event "branches" from fc5028ccaa5f40daa8138fa06a49dd33 [/walle]
[2019-01-13 21:00:10,738] INFO in app: 2
[2019-01-13 21:00:10,760] INFO in waller: deploying task_id=0 [root@127.0.0.1]$ [ -d ".git" ] && git status 
[2019-01-13 21:00:10,771] INFO in waller: {}

运行环境

zhouyouth commented 5 years ago

我的也是 ，这个问题，

meolu commented 5 years ago

收到

meolu commented 5 years ago

@wu0407 会用strace debug已经很不错了

wu0407 commented 5 years ago

基本查到问题了，调invoke时，不能使用默认的in_stream、err_stream.还不知道改成什么