search

F5Networks / f5-openstack-lbaasv2-driver

F5 LBaaSv2 service provider driver for OpenStack Liberty and beyond
http://clouddocs.f5.com/products/openstack/lbaasv2-driver/master/
Apache License 2.0
21 stars 37 forks source link

Driver tempest tests do not run in nightly -- permission denied #728

Closed pjbreaux closed 6 years ago

pjbreaux commented 7 years ago

Description

The tempest tests that are located in the driver are not running in the nightly test runs. During the tox installation of dependencies we get a failure permission denied error:

02:12:34 WARNING:Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
02:12:35 ERROR: invocation failed (exit code 2), logfile: /home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/.tox/tempest/log/tempest-3.log
02:12:35 ERROR: actionid: tempest
02:12:35 msg: installpkg
02:12:35 cmdargs: ['/usr/local/bin/pip', 'install', '-U', '--no-deps', '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/.tox/dist/f5-openstack-lbaasv2-driver-9.3.3b2.zip']
02:12:35 env: {'EXCLUDE_FILE': 'tempest_12.1.1_undercloud_vxlan.yaml', 'BUILD_URL': 'http://jenkins.pdbld.f5net.com/job/openstack/job/driver/job/mitaka/job/12.1.1-undercloud-vxlan/106/', 'BUILD_TAG': 'jenkins-openstack-driver-mitaka-12.1.1-undercloud-vxlan-106', 'TLC_FILE': '/home/jenkins/dev-test/traffic/undercloud/tempest_ve_undercloud.tlc', 'VIRTUAL_ENV': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/.tox/tempest', 'UPPER_CONSTRAINTS_FILE': 'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/mitaka', 'TEST_COMMON_LIB': '/home/jenkins/dev-test/common', 'FROM_AGENT_SESSION': 'from.agent_3bfbcfc_20170805-014318', 'TEMPEST_VENV_ACTIVATE': '/home/jenkins/virtualenvs/tempest/bin/activate', 'MFLAGS': '-w', 'BUILD_NUMBER': '106', 'JOB_URL': 'http://jenkins.pdbld.f5net.com/job/openstack/job/driver/job/mitaka/job/12.1.1-undercloud-vxlan/', 'NEUTRON_LBAAS_REPO': 'https://github.com/F5Networks/neutron-lbaas.git', 'TEST_SESSION': 'tempest_12.1.1_undercloud_vxlan_mitaka_3bfbcfc_20170805-014318', 'OS_TENANT_NAME': 'openstack-test', 'SCENARIO_SESSION': 'scenario_3bfbcfc_20170805-014318', 'JOB_BASE_NAME': '12.1.1-undercloud-vxlan', 'TEST_OPENSTACK_NODE_COUNT': '3', 'JOB_DISPLAY_URL': 'http://jenkins.pdbld.f5net.com/job/openstack/job/driver/job/mitaka/job/12.1.1-undercloud-vxlan/display/redirect', 'TEMPEST_CONFIG_DIR': '/home/jenkins/virtualenvs/tempest/etc/tempest', 'HOSTNAME': '0069cce0060c', 'EXECUTOR_NUMBER': '2', 'VENVDIR': '/home/jenkins/virtualenvs', 'JENKINS_HOME': '/var/jenkins_home', 'MAKEFILE_DIR': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/systest/', 'HUDSON_SERVER_COOKIE': 'c7f851ca6b0adc68', 'USER': 'jenkins', 'CI_PROJECT': 'driver', 'API_SESSION': 'api_3bfbcfc_20170805-014318', 'RUN_DISPLAY_URL': 'http://jenkins.pdbld.f5net.com/job/openstack/job/driver/job/mitaka/job/12.1.1-undercloud-vxlan/106/display/redirect', 'TEST_OPENSTACK_DISTRO': 'mitaka', 'MAKELEVEL': '3', 'SHLVL': '2', 'RESULTS_DIR': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/systest//test_results/f5-openstack-lbaasv2-driver_mitaka-tempest_12.1.1_undercloud_vxlan', 'CI_BUILD_SUMMARY': '/home/jenkins/results/openstack.driver.mitaka.12.1.1-undercloud-vxlan/12.1.1-undercloud-vxlan-106/ci-build.yaml', 'EXCLUDE_DIR': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/systest//exclude/', 'DOCKER_REGISTRY_URL': 'https://docker-registry.pdbld.f5net.com', 'TEMPEST_REPO': 'http://git.openstack.org/openstack/tempest', 'CLASSPATH': '', 'CI_BRANCH': 'mitaka', 'RUN_CHANGES_DISPLAY_URL': 'http://jenkins.pdbld.f5net.com/job/openstack/job/driver/job/mitaka/job/12.1.1-undercloud-vxlan/106/display/redirect?page=changes', 'OS_PASSWORD': 'openstack-test', 'DRIVER_TEMPEST_SESSION': 'driver.tempest_3bfbcfc_20170805-014318', 'TEST_OPENSTACK_DEPLOY': 'multinode', '_': '/usr/local/bin/tox', 'TIMESTAMP': '20170805-014318', 'TLC_FILE_DIR': '/home/jenkins/dev-test/traffic', 'CI_RESULTS_DIR': '/home/jenkins/results/openstack.driver.mitaka.12.1.1-undercloud-vxlan/12.1.1-undercloud-vxlan-106', 'OS_TEST_PATH': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/f5lbaasdriver/test/tempest/tests', 'GLANCE_COMPUTE_STORAGE': 'NFS', 'TEMPEST_VENV_DIR': '/home/jenkins/virtualenvs/tempest', 'HUDSON_URL': 'http://jenkins.pdbld.f5net.com/', 'NEUTRON_LBAAS_DIR': '/home/jenkins/neutron-lbaas', 'WORKSPACE': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan', 'HOME': '/home/jenkins', 'JOB_NAME': 'openstack/driver/mitaka/12.1.1-undercloud-vxlan', 'TEST_CIRROS_IMAGE': 'cirros-0.3.4-x86_64-disk.qcow2', 'NEUTRON_LBAAS_BRANCH': 'stable/mitaka', 'HUDSON_HOME': '/var/jenkins_home', 'PYTHONHASHSEED': '4182253217', 'TEST_DIR': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/systest../test', 'TEST_VE_IMAGE': 'os_ready-BIGIP-12.1.1.2.0.204.qcow2', 'DEVTEST_REPO': 'git@gitlab.pdbld.f5net.com:openstack/dev-test.git', 'NODE_NAME': 'jenkins-docker-host-01', 'PROJROOTDIR': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/systest/../', 'OS_AUTH_URL': 'http://10.190.22.22:5000/v2.0', 'BRANCH': 'mitaka', 'TEST_OPENSTACK_CLOUD': 'undercloud', 'CI_ENV_FILE': 'systest/12.1.1-undercloud-vxlan.env', 'OS_REGION_NAME': 'RegionOne', 'JENKINS_URL': 'http://jenkins.pdbld.f5net.com/', 'OS_PROJECT_NAME': 'openstack-test', 'NODE_LABELS': 'docker jenkins-docker-host-01', 'GUMBALLS_PROJECT': 'f5-openstack-lbaasv2-driver_mitaka-tempest_12.1.1_undercloud_vxlan', 'MAKEFLAGS': 'w', 'SSH_AGENT_PID': '31', 'CI_PROGRAM': 'openstack', 'PWD': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan', 'OS_TENANT_ID': 'd32adfee36a54e899c110f5c5070ef05', 'BUILD_ID': '106', 'OS_USERNAME': 'openstack-test', 'SSH_AUTH_SOCK': '/tmp/ssh-bNlJyihNcAww/agent.30', 'PATH': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/.tox/tempest/bin:/tools/bin:/tools/bin:/tools/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'TEST_TENANT_NETWORK_TYPE': 'vxlan', 'DEVTEST_DIR': '/home/jenkins/dev-test', 'JENKINS_SERVER_COOKIE': 'durable-45a9d0a132aea51bfb98ffdb6f3f16fe', 'BUILD_DISPLAY_NAME': '#106', 'TEMPEST_DIR': '/home-local/jenkins/workspace/openstack/driver/mitaka/12.1.1-undercloud-vxlan/systest/../tempest'}
02:12:35 
02:12:35 Processing ./.tox/dist/f5-openstack-lbaasv2-driver-9.3.3b2.zip
02:12:35 Installing collected packages: f5-openstack-lbaasv2-driver
02:12:35   Found existing installation: f5-openstack-lbaasv2-driver 9.3.3b2
02:12:35     Uninstalling f5-openstack-lbaasv2-driver-9.3.3b2:
02:12:35 Exception:
02:12:35 Traceback (most recent call last):
02:12:35   File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
02:12:35     status = self.run(options, args)
02:12:35   File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
02:12:35     prefix=options.prefix_path,
02:12:35   File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 778, in install
02:12:35     requirement.uninstall(auto_confirm=True)
02:12:35   File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 754, in uninstall
02:12:35     paths_to_remove.remove(auto_confirm)
02:12:35   File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
02:12:35     renames(path, new_path)
02:12:35   File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 267, in renames
02:12:35     shutil.move(old, new)
02:12:35   File "/usr/lib/python2.7/shutil.py", line 303, in move
02:12:35     os.unlink(src)
02:12:35 OSError: [Errno 13] Permission denied: '/usr/local/bin/add_f5agent_environment'
02:12:35
pjbreaux commented 7 years ago

Here's a link to one of the latest runs from today: http://jenkins.pdbld.f5net.com/job/openstack/job/driver/job/mitaka/job/12.1.1-undercloud-vxlan/106/consoleFull

pjbreaux commented 7 years ago

@zancas: Note that this is not consistent. Last night, the mitaka vlan tests actually ran the driver tempest tests properly, but the mitaka vxlan tests got the permission denied error.

zancas commented 7 years ago

Interesting. The fix that made these tests run in the happy path was this one:

https://github.com/F5Networks/f5-openstack-lbaasv2-driver/pull/719/commits/87b356fbfc415d8ad591a5d1cba052e3f1119ed9

Examining log file...

ssorenso commented 7 years ago

Two additional instances in yesterday's traige:

richbrowne commented 6 years ago

Tests are moved to systest repo. This is no longer a valid issue.