优雅地调试OpenStack

Yikun commented 8 years ago

恩，题目首先要起的高逼格一些。2333。

在前面学习代码的过程中，主要通过源码来学习，开始学起来确实有点费劲，因为欠缺对OpenStack的整体的意识，于是搭建OpenStack开发环境对OpenStack的运行环境和使用有了初步认知。也看到了启动OpenStack后的一些相关进程，那么这些进程是如何与源码对应起来的呢？如何去调试OpenStack呢？本篇文章就讲下我的探索。

1. 初识Python调试

在Python 代码调试技巧一文中提到了pdb、PyCharm、PyDev、日志等几种常见的调试方法。具体可以看看这篇文章，写的很详细，不赘述。

因为PyCharm出色的用户体验（那写代码就是要开心嘛），所以决定使用PyCharm进行调试，但是问题来了，在远端（如虚拟机或者服务器）服务已经启动起来了，我们如何进行调试呢？答案就是Remote Debug。

2. 启动OpenStack服务

在搭建OpenStack开发环境一文中，我们介绍了使用devstack启动开发环境，我们通过DevStack启动各个服务后：

# ... ...
This is your host IP address: 192.168.56.101
This is your host IPv6 address: ::1
Horizon is now available at http://192.168.56.101/dashboard
Keystone is serving at http://192.168.56.101:5000/
The default users are: admin and demo
The password: 1

使用screen来查看：

Note(2018.01.18): 目前社区已经废弃了screen这种方式来启动进程了（参考Remove screen support from devstack completely），原因是OpenStack需要管理的进程太多了，screen已没有“能力”去管理。

社区已经切换到systemd来进行进程管理，可以参考官方提供的Using Systemd in DevStack一文来入门。

screen -x stack

不得不说screen真是神器，虚拟了多个Terminal的Tab，使用"Ctrl+A+P"和"Ctrl+A+N"可以切换tab，使用"Ctrl+A+D"可以断开连接，在每个tab中可以使用“Ctrl+C”来中断进程： qq20160223-0 2x 我们看到在图中，有Nova和Cinder相关的进程，并且停在了cinder-api的进程上，每个tab中的进程都在运行着。

3. 编辑代码

因为代码大部分都在远端的运行（比如虚拟机），而开发环境则在近端（比如宿主机），如果在远端和近端都维护一套代码，不可避免的会拷来拷去，有时拷错了还得找半天原因。所以得想办法把远端的代码“共享”到近端。因此，我们使用sshfs共享文件：

➜  ~  sshfs stack@192.168.56.101:/opt/stack /Users/jiangyikun/development/openstack/code
➜  ~  ls /Users/jiangyikun/development/openstack/code
cinder            heat              logs              noVNC             requirements
data              heat-cfntools     neutron           nova              status
devstack.subunit  heat-templates    neutron-fwaas     os-apply-config   swift
dib-utils         horizon           neutron-lbaas     os-collect-config tempest
glance            keystone          neutron-vpnaas    os-refresh-config

达到的目的就是，我们编辑/Users/jiangyikun/development/openstack/code的时候，就相当于在远端编辑/opt/stack。在Windows下，也有win-sshfs工具。

4. 启动调试服务器

当我们修改好代码后，就可以进行调试了。调试的原理大致是在近端启动一个debug server，然后，在代码中添加连接服务器的动作，这样，当代码运行到那段调试代码时，便会和调试服务器建立连接。在我的实验环境中，调试环境是这样的：可以看到在宿主机和虚拟机有2条通路，一条是NAT，作用是让虚拟机通过宿主机的公网IP上网，从而保证Devstack能够顺利启动OpenStack，第二条是Host Only，保证在宿主机内可以对虚拟机进行SSH访问、sshfs文件挂载以及调试。

因此我们先配置一下远程调试的配置： qq20160223-1 2x

然后，我们就可以把由于几个调试的服务都启动起来了，例如，我们要调试跟踪Cinder的创建过程，我们就首先建立三个远程调试，其次将调试代码添加到入口处并保存，最后增加断点： qq20160223-2 2x 使用Ctrl+c把修改过代码的进程都结束，然后按“上”重新执行指令： qq20160223-3 2x 重启服务后，代码就生效了，当代码运行到我们需要连接到调试服务的代码后，就会进入断点了： qq20160223-4 2x 接下来就随心所欲的进行调试吧！

参考资料

使用DEVSTACK搭建OPENSTACK可remote debug的开发测试环境 DevStack-install-in-China

bibapple commented 6 years ago

请问你的screen -x stack命令执行了，出现的截图是怎么搞出来的？我的虚拟机安装了devstack的openstack之后，输入命令screen -x stack，没出现这些个进程呢？还报下面的错： stack@xh-virtual-machine:~/devstack$ screen -x stack There is no screen to be attached matching stack.

Yikun commented 6 years ago

@bibapple 目前社区已经废弃了screen这种方式来启动进程了（参考Remove screen support from devstack completely），原因是OpenStack需要管理的进程太多了，screen已没有“能力”去管理。

社区已经切换到systemd来进行进程管理，可以参考官方提供的Using Systemd in DevStack一文来入门。

bibapple commented 6 years ago

@Yikun多谢你的指点，我去研究研究官方文档。再请问下API的调试还是上面你写的方法吗？还是说也有些变化呢？

Yikun commented 6 years ago

@bibapple 参考systemd的debugging一节，这个调试更方便，目前我是使用pdb的方式多一些。

bibapple commented 6 years ago

@Yikun谢谢你的指点，我去研究研究pdb调试。

bibapple commented 6 years ago

@Yikun我这两天都在研究devstack的pdb调试，但是下面这个问题差了好多资料都没搞定，请教一下你是怎么弄得呢？

1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: (Pdb) 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi [None req-9510783d-ef6a-4615-8e21-e4b5925fdec9 None None] : BdbQuit 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi Traceback (most recent call last): 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 352, in _inner 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi return method(self, request) 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/middleware/auth.py", line 166, in process_request 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi self.fill_context(request) 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/middleware/auth.py", line 191, in fill_context 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi if request.environ.get(core.CONTEXT_ENV, {}).get('is_admin', False): 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/middleware/auth.py", line 191, in fill_context 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi if request.environ.get(core.CONTEXT_ENV, {}).get('is_admin', False): 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi return self.dispatch_line(frame) 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi File "/usr/lib/python2.7/bdb.py", line 68, in dispatch_line 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi if self.quitting: raise BdbQuit 1月 23 10:00:56 xh-virtual-machine devstack@keystone.service[5543]: ERROR keystone.common.wsgi BdbQuit

Yikun commented 6 years ago

@bibapple

systemctl show devstack@keystone.service -p ExecStart --no-pager

ExecStart={ path=/usr/local/bin/uwsgi ; argv[]=/usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }

可以通过systemctl show去看执行的命令，WSGI比较特殊些，需要加一个--honour-stdin进行调试。即

/usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini --honour-stdin

bibapple commented 6 years ago

@Yikun多谢大神，现在貌似可以调试起来了，希望以后可以多和大神交流交流^-^

Yikun commented 5 years ago

[[install]]

[[local|localrc]]
GIT_BASE=https://github.com

#SSL
#USE_SSL=True
#SSL_ENABLED_SERVICES=glance

# Define the devstack install directory
#DEST=/opt/stack/
#Data directory
#DATA_DIR=/opt/stack/data/
#Service dirctory
#SERVICE_DIR=$DEST/service

# No internet
#OFFLINE=True
#Reclone
RECLONE=True

ADMIN_PASSWORD=1
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

# Using milestone-proposed branches
#CINDER_BRANCH=stable/juno
#GLANCE_BRANCH=stable/juno
#KEYSTONE_BRANCH=stable/juno
#NOVA_BRANCH=stable/juno
#NEUTRON_BRANCH=stable/juno
#CEILOMETER_BRANCH=stable/juno
#HORIZON_BRANCH=stable/juno
#REQUIREMENTS_BRANCH=stable/juno

# ``HOST_IP`` should be set manually for best results if the NIC configuration
# of the host is unusual, i.e. ``eth1`` has the default route but ``eth0`` is the
# public interface.  It is auto-detected in ``stack.sh`` but often is indeterminate
# on later runs due to the IP moving from an Ethernet interface to a bridge on
# the host. Setting it here also makes it available for ``openrc`` to include
# when setting ``OS_AUTH_URL``.
# ``HOST_IP`` is not set by default.
#HOST_IP=w.x.y.z

# Logging
# -------
# By default ``stack.sh`` output only goes to the terminal where it runs.  It can
# be configured to additionally log to a file by setting ``LOGFILE`` to the full
# path of the destination log file.  A timestamp will be appended to the given name.
LOGFILE=$DEST/logs/stack.sh.log
# Old log files are automatically removed after 7 days to keep things neat.  Change
# the number of days by setting ``LOGDAYS``.
LOGDAYS=2
# Nova logs will be colorized if ``SYSLOG`` is not set; turn this off by setting
# ``LOG_COLOR`` false.
#LOG_COLOR=False
#------------------------------------------------------------------------------
# disable all services first if you don't want to install the default services.
# disable_all_services
#------------------------------------------------------------------------------
# Pre-requisite
enable_service rabbit mysql key
#enable_service -mysql postgresql
#------------------------------------------------------------------------------
#Libraries from Git
LIBS_FROM_GIT=python-openstackclient,python-keystoneclient,python-glanceclient,python-novaclient,python-cinderclient
#------------------------------------------------------------------------------
# Keystone
KEYSTONE_TOKEN_FORMAT=fernet
#KEYSTONE_DEPLOY=uwsgi

# OpenLdap, this will create a domain in keystone using openldap as the identity driver.
# enable_service ldap
# Federation, this will set the Keystone as Service Provider, and use testshib.org
# as Identity Provider
# enable_service keystone-saml2-federation
# Placement
enable_service placement,placement-client
#------------------------------------------------------------------------------
# Nova
enable_service n-api n-obj n-cpu n-cond n-sch n-schi n-novnc n-cauth
#------------------------------------------------------------------------------
# Cinder
enable_service cinder c-api c-vol c-sch c-bak
#------------------------------------------------------------------------------
# Neutron
enable_service q-svc q-agt q-dhcp q-l3 q-meta neutron 

# Dragonflow
#Q_ENABLE_DRAGONFLOW_LOCAL_CONTROLLER=True
#enable_plugin dragonflow https://github.com/openstack/dragonflow.git
#enable_service df-controller
#enable_service db-ext-services
#enable_service q-svc
#enable_service q-l3
#disable_service q-agt
#disable_service n-net

# BGP dynamic routing
#enable_service q-bgp q-bgp-agt
#------------------------------------------------------------------------------
# Glance
enable_service g-api g-reg
#------------------------------------------------------------------------------
# Horizon
enable_service horizon
#------------------------------------------------------------------------------
# Tempest(enabled by default)
#enable_service -tempest
#------------------------------------------------------------------------------
# Heat
#enable_service tempest heat
# Heat - Orchestration Service
#enable_service heat h-api h-api-cfn h-api-cw h-eng
#------------------------------------------------------------------------------
# Swift
# Swift is now used as the back-end for the S3-like object store. If Nova's
# objectstore (``n-obj`` in ``ENABLED_SERVICES``) is enabled, it will NOT
# run if Swift is enabled. Setting the hash value is required and you will
# be prompted for it if Swift is enabled so just set it to something already:
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
# For development purposes the default of 3 replicas is usually not required.
# Set this to 1 to save some resources:
SWIFT_REPLICAS=1
# The data for Swift is stored by default in (``$DEST/data/swift``),
# or (``$DATA_DIR/swift``) if ``DATA_DIR`` has been set, and can be
# moved by setting ``SWIFT_DATA_DIR``. The directory will be created
# if it does not exist.
SWIFT_DATA_DIR=$DEST/data
#Enable Swift
enable_service s-proxy s-object s-container s-account
#Cyborg
enable_plugin cyborg https://opendev.org/openstack/cyborg
OPAE_INSTALL_ENABLE=false

jin544642965 commented 4 years ago

@Yikun pycharm 远程调试devstack，发现进入不了断点，求解。操作步骤：在nova/compute/api.py ,get_all加入断点，systemctl restart devstack@n-api.service服务，在页面点击获取实例，发现进入不了断点，监听信息一次性全打印出来。devstack是当前最新版

Yikun / yikun.github.com