Issue with libvirt breaks ORE

[lkatalin]

Trying to ./stack.sh in the devstack directory using the current single_node_devstack.yml file on s-large flavor Centos created Dec. 18 produces the attached error (the text file includes the last part of output, not all the output). ORE_error.txt

The solutions mentioned here (i.e. upgrading pip or issuing virtualenv ../requirements/.venv/) do not work: https://askubuntu.com/questions/1040238/openstack-devstack-installation-ubuntu-16-04

This part of the ORE readme looks relevant:

after cloning requirements repo, change python-libvirt to 3.9 in upper-constraints.txt (this is because CentOS 7 ships with python-libvirt 3.9 but devstack pike sets upper limit of python-libvirt 3.5)

However, manually changing this also did not work. upper-constraints.txt appears to be 3 different files:

./requirements/openstack_requirements/tests/files/upper-constraints.txt
./requirements/.venv/lib/python2.7/site-packages/openstack_requirements/tests/files/upper-constraints.txt
./requirements/upper-constraints.txt

These files do not have a python-libvirt field, only a libvirt-python field. I did update that field to libvirt-python===3.9.0 and ran ./stack.sh in /devstack again. But I got the same error as previously.

[lkatalin]

May also be relevant: https://stackoverflow.com/questions/51609205/devstack-fails-with-error-markers-python-version-3-4-dont-match-your-env

[ghost]

Can you post your local.conf?

[lkatalin]

Yeah, sure. Attaching as a file also. I didn't edit it at all. local.conf.txt

[[local|localrc]]
ADMIN_PASSWORD=Devstack1
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=1
SYSLOG=True

# offline will be automatically turned on after a successful
# install of devstack. this is done in order to avoid
# installing new, incompatible versions of software.
#
# to install new versions of everything, comment the following
# line. warning: this may break things
#OFFLINE=True

###################
#
#  Changing Openstack Service Source
#
###################

# to use custom repos or branches for openstack service
# uncomment relevant line(s) and insert the git url / exact branch name
# 
# to clone a specific commit, uncomment the relevant line
# and insert the SHA of the desired commit
# e.g. 7638417db6d59f3c431d3e1f261cc637155684cd
#
# ensure that the commit specified falls in the branch/repo indicated
# or in the stable redhat repo if branch/repo not specified
#
# git repos must be manually retrieved by entering /opt/stack/$SERVICE
# and executing `git pull`
#
# ORE provides a set of known, stable commits that you can use
# to ensure(?) a more stable/consistent environment

#CINDER_REPO=git://git.openstack.org/openstack/cinder.git
#CINDER_BRANCH=stable/pike
#CINDER_COMMIT=00f9a753ac4ed9a224d32e5614c5d57192f45a22
#
#GLANCE_REPO=git://git.openstack.org/openstack/glance.git
#GLANCE_BRANCH=stable/pike
#GLANCE_COMMIT=a4562abeb13b47f8bc765f792794f6d214df96cd
#
#HORIZON_REPO=git://git.openstack.org/openstack/horizon.git
#HORIZON_BRANCH=stable/pike
#HORIZON_COMMIT=0de6c6a06f87573392f4beac1b41d07401548ba5
#
#KEYSTONE_REPO=git://git.openstack.org/openstack/keystone.git
#KEYSTONE_BRANCH=stable/pike
#KEYSTONE_COMMIT=22af1d9f35c86e9c5bca288c2996be5c19e3cd61
#
#NEUTRON_REPO=git://git.openstack.org/openstack/neutron.git
#NEUTRON_BRANCH=stable/pike
#NEUTRON_COMMIT=b0ca86f6b27069c6cc5726c950008c636aa9aad9
#
#NOVA_REPO=git://git.openstack.org/openstack/nova.git
#NOVA_BRANCH=stable/pike
#NOVA_COMMIT=b1ab231836f2622a4fcc38a4062740f87f058cee
#
#SWIFT_REPO=
#SWIFT_BRANCH=
#SWIFT_COMMIT=
#
#TEMPEST_REPO=git://git.openstack.org/openstack/tempest.git
#TEMPEST_BRANCH=master
#TEMPEST_COMMIT=83b154a28369a062ab7a4c014531a86abf650133

###################

enable_plugin rally https://github.com/hgibson1/rally.git profile
enable_plugin panko https://git.openstack.org/openstack/panko stable/pike
enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/pike
enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler stable/pike
OSPROFILER_BRANCH=stable/queens
OSPROFILER_COLLECTOR=redis
OSPROFILER_HMAC_KEYS=Devstack1
CEILOMETER_NOTIFICATION_TOPICS=notifications,profiler

[[post-config|/etc/nova/nova.conf]]
[profiler]
enabled = True
connection_string = redis://localhost:6379
hmac_keys = Devstack1
trace_wsgi_transport = True
trace_message_store = True
trace_management_store = True
trace_sqlalchemy = False

[[post-config|/etc/keystone/keystone.conf]]
[profiler]
enabled = True
connection_string = redis://localhost:6379
hmac_keys = Devstack1
trace_wsgi_transport = True
trace_message_store = True
trace_management_store = True
trace_sqlalchemy = False

[[post-config|/etc/cinder/cinder.conf]]
[profiler]
enabled = True
connection_string = redis://localhost:6379
hmac_keys = Devstack1
trace_wsgi_transport = True
trace_message_store = True
trace_management_store = True
trace_sqlalchemy = False

[[post-config|/etc/glance/glance-api.conf]]
[profiler]
enabled = True
connection_string = redis://localhost:6379
hmac_keys = Devstack1
trace_wsgi_transport = True
trace_message_store = True
trace_management_store = True
trace_sqlalchemy = False

[[post-config|/etc/ceilometer/ceilometer.conf]]
[event]
store_raw=info

[ghost]

./requirements/openstack_requirements/tests/files/upper-constraints.txt are the requirements for tox testing: this shouldn't affect your deploy

./requirements/.venv/lib/python2.7/site-packages/openstack_requirements/tests/files/upper-constraints.txt: stack.sh clones various openstack repos to your home directory then runs python setup.py to install them in python site packages. That's what this file is. This file is identical to ./requirements/upper-constraints.txt Re-runing stack.sh will overwrite this file with whatever is in ./requirements/upper-constraints.txt

./requirements/upper-constraints.txt: This is where the upper constraint actual needs to be changed.

I'll try to replicate the issue. What version of Devstack are you using? Is it still using devstack's stable/pike branch?

[lkatalin]

Thanks for the explanation!

I'm using exactly what is provided by the ORE - I haven't modified anything at all, so I think that should be Pike? It worked for my last VMs (set up 2 months ago Oct. 18), so something must have changed either in the ORE or in the images available on Kaizen (maybe some dependency got updated). It's Centos 7.5 s-large. Let me know if you need more details. Appreciate it a lot.

[lkatalin]

Hmm, yeah, in my new VM it's version CentOS Linux release 7.6.1810 (Core). In my old VMs it's version CentOS Linux release 7.5.1804 (Core). I suspect this has to do with it.

Can also dig around Kaizen to see if I find a Centos 7.5.1804 now that I know this.

[ghost]

I don't have the project resources to spin up more VMs and probably will not have the time to test this before the break. If you can't find an older centos it should be possible to upload your own centos image (Images -> upload image). It might be worth it to test ORE with the queens branch of Devstack as well.

[lkatalin]

Okay, thanks - I'll try with the old Centos and I'll leave an update after the break. I made a snapshot of one of my other VMs and am trying to see if I can use the ORE with that (since that has the old Centos), but right now I can't create any new instances for some reason, so I'm waiting on that to be resolved.

[EmreAtes]

I created a new VM today with the Centos 7.5 image from MOC (after updates it's 7.6). I didn't encounter any problems. I used the docc-lab fork.

[lkatalin]

Seems to work now for me as well with Centos 7.5 m1.xlarge, current single-node-devstack.yml as the config file. Guess the earlier problem was just an extended glitch. I will go ahead and close this.