search

GoogleCloudPlatform / PerfKitBenchmarker

PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see. PerfKit Benchmarker is licensed under the Apache 2 license terms. Please make sure to read, understand and agree to the terms of the LICENSE and CONTRIBUTING files before proceeding.
https://googlecloudplatform.github.io/PerfKitBenchmarker/
Apache License 2.0
1.91k stars 517 forks source link

OpenStack nova error with multi region deployment. #769

Open stoor opened 8 years ago

stoor commented 8 years ago

Hi,

First Thanks you very much for PrefKit!

I am having some problem on my multi-region based cloud setup. NOVA client works fine with multiple regions:

--------- salman ----------- export OS_USERNAME= export OS_PASSWORD= export OS_TENANT_NAME= export OS_PROJECT_NAME= export OS_REGION_NAME=

export OS_AUTH_URL=http://keystoneURL:5000/v2.0

ubuntu@prefkit-test:~/perfkit$ source salman ubuntu@prefkit-test:~/perfkit$ nova list +--------------------------------------+-----------+---------+------------+-------------+--------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------+---------+------------+-------------+--------------------------------------+ | 9df963e8-d0ef-44e4-993c-f3e06244961f | smurf-001 | SHUTOFF | - | Shutdown | vmnet=192.168.10.139, 130.239.46.227 | +--------------------------------------+-----------+---------+------------+-------------+--------------------------------------+ ubuntu@prefkit-test:~/perfkit$ vi salman ubuntu@prefkit-test:~/perfkit$ source salman < Basically I have changed the region parameter > ubuntu@prefkit-test:~/perfkit$ nova list +--------------------------------------+--------------+--------+------------+-------------+------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+------------------------------------+ | a9a6a1cd-1ffe-473a-9f27-1cc865937cbb | PerfKit-Test | ACTIVE | - | Running | smog-net=10.0.10.19, 130.238.29.68 | +--------------------------------------+--------------+--------+------------+-------------+------------------------------------+

So with the Nova Client everything is working fine with multiple regions. But here what happen when I use pkb.py:

ubuntu@prefkit-test:~/perfkit$ cd PerfKitBenchmarker/ ubuntu@prefkit-test:~/perfkit/PerfKitBenchmarker$ ./pkb.py --cloud=OpenStack --benchmarks=ping 2015-12-22 15:43:17,635 6a699ed7 MainThread INFO Verbose logging to: /tmp/perfkitbenchmarker/run_6a699ed7/pkb.log 2015-12-22 15:43:17,635 6a699ed7 MainThread INFO PerfKitBenchmarker version: v1.0.1-103-g5a2c738 2015-12-22 15:43:17,804 6a699ed7 MainThread INFO Flag values: --cloud=OpenStack --benchmarks=ping 2015-12-22 15:43:18,091 6a699ed7 MainThread ping(1/1) INFO Starting new HTTP connection (1): smog.uppmax.uu.se 2015-12-22 15:43:25,441 6a699ed7 MainThread ping(1/1) INFO Starting new HTTP connection (1): smog.uppmax.uu.se 2015-12-22 15:43:30,489 6a699ed7 MainThread ping(1/1) INFO Starting new HTTP connection (1): smog.uppmax.uu.se Found more than one valid endpoint. Use a more restrictive filter 2015-12-22 15:43:42,856 6a699ed7 MainThread ping(1/1) ERROR Error during benchmark ping Traceback (most recent call last): File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 338, in RunBenchmark spec.ConstructVirtualMachines() File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 262, in ConstructVirtualMachines vm = self._CreateVirtualMachine(vm_spec, os_type, cloud) File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 344, in _CreateVirtualMachine return vm_class(vm_spec) File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/providers/openstack/os_virtual_machine.py", line 48, in init self.firewall = os_network.OpenStackFirewall.GetFirewall() File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/network.py", line 51, in GetFirewall benchmark_spec.firewalls[key] = cls() File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/providers/openstack/os_network.py", line 38, in init name='perfkit_sc_group')): File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 257, in findall listing = self.list(_list_kwargs) File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/security_groups.py", line 96, in list 'security_groups') File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 74, in _list _resp, body = self.api.client.get(url) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 448, in get return self._cs_request(url, 'GET', _kwargs) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 404, in _cs_request self.authenticate() File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate auth_url = self._v2_auth(auth_url) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 652, in _v2_auth return self._authenticate(url, body) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 667, in _authenticate return self._extract_service_catalog(url, resp, respbody) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 490, in _extract_service_catalog self.management_url = self.get_service_url(self.service_type) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 467, in get_service_url volume_service_name=self.volume_service_name,) File "/usr/local/lib/python2.7/dist-packages/novaclient/service_catalog.py", line 87, in url_for endpoints=matching_endpoints) AmbiguousEndpoints: AmbiguousEndpoints: 2015-12-22 15:43:42,861 6a699ed7 MainThread ping(1/1) ERROR Benchmark 1/1 ping (UID: ping0) failed. Execution will continue.

2015-12-22 15:43:42,862 6a699ed7 MainThread ping(1/1) INFO Benchmark run statuses:

Name UID Status

ping ping0 FAILED

Success rate: 0.00% (0/1) 2015-12-22 15:43:42,862 6a699ed7 MainThread ping(1/1) INFO Complete logs can be found at: /tmp/perfkitbenchmarker/run_6a699ed7/pkb.log

Can someone please confirm? Or point out the problem.

Regards.. Salman.

cmccoy commented 8 years ago

cc @kivio, @meteorfox

meteorfox commented 8 years ago

@cmccoy I'll take a look.

meteorfox commented 8 years ago

@stoor Please try this PR, I don't have a multi-region deployment to test it, but I believe it should address your issue. You can either use the env variable OS_REGION_NAME, or pass it as a flag --openstack-region

Also, a few tips:

For example in my deployment I have this:

$ nova net-list
+--------------------------------------+---------+------+
| ID                                   | Label   | CIDR |
+--------------------------------------+---------+------+
| d82fbbfd-8209-4b60-8334-7dfd9c6da392 | private | None |
| 11d429bb-0f11-42a0-bc24-1d0486e84ec5 | public  | None |
| 7907665c-1ba9-4f8a-b2a0-d24848186293 | service | None |
+--------------------------------------+---------+------+
nova image-list
+--------------------------------------+-----------------------+--------+--------------------------------------+
| ID                                   | Name                  | Status | Server                               |
+--------------------------------------+-----------------------+--------+--------------------------------------+
| bf58d063-d28b-4c56-b2a2-f20f11a097ba | Ubuntu 14.04          | ACTIVE |                                      |
| da9145ea-edc4-4a04-950a-64dadebf6c78 | Ubuntu 14.04 (large)  | ACTIVE | 469bc375-5ddb-4652-ad9d-ccb8a67a0c7f |
| e5ab6e3f-6875-4145-987a-dfc358d911ea | Ubuntu 14.04 (medium) | ACTIVE | 469bc375-5ddb-4652-ad9d-ccb8a67a0c7f |
+--------------------------------------+-----------------------+--------+--------------------------------------+
nova endpoints

...
+-------------+-----------------------------------------------------------+
| novav21     | Value                                                     |
+-------------+-----------------------------------------------------------+
| adminURL    | http://172.29.236.10:8774/v2.1                            |
| id          | 417cb049043a4ee68250c8322f210181                          |
| internalURL | http://172.29.236.10:8774/v2.1                            |
| publicURL   | http://##########:8774/v2.1 |
| region      | RegionOne                                                 |
+-------------+-----------------------------------------------------------+
...

To run the ping benchmark in my case I use the following command:

./pkb.py --cloud=OpenStack --machine_type=m1.medium --image='Ubuntu 14.04' \ 
         --benchmarks=ping  \
         --openstack_public_network=public --openstack_private_network=service

To execute a benchmark (copy_throughput, fio, etc) that makes use of data disks use something like this:

/pkb.py --cloud=OpenStack --machine_type=m1.medium --image='Ubuntu 14.04' \
         --openstack_public_network=public --openstack_private_network=service \ 
         --benchmarks=fio \
         --data_disk_size=40 --num_striped_disks=2

This will create an instance with the m1.medium flavor using the image Ubuntu 14.04, attach it to private network named service, and use floating IPs from the public network named public, it will create 2 Cinder volumes of 40GB each and attach it to the instance, which then PKB will create an RAID0 array, mount them and use it as the path for the fio workload.

As of now, it always uses Cinder volumes, I have another PR #759 in progress which allows you to specify if either use locally attached disks, its boot disk, or remote volumes.

stoor commented 8 years ago

Hi,

Thanks for your response. But I think the problem is not related to image or instance creation or even passing correct parameters to pkb.py . The problem is pkb.py somehow couldn't pass the information of regions to nova. Since keystone has multiple endpoints for nova, I am getting response of AmbiguousEndpoints. In principle the response is correct. Can you please check that together with username, password, tenant and auth_url, is pkb.py also handling OS_REGION_NAME parameter?

ubuntu@prefkit-test:~/perfkit/PerfKitBenchmarker$ ./pkb.py --cloud=OpenStack --machine_type=m1.medium --image='99f4625e-f425-472d-8e21-7aa9c3db1c3e' --benchmarks=ping --openstack_public_network=ext-net --openstack_private_network=smog-net 2015-12-23 10:34:41,653 1798f9fc MainThread INFO Verbose logging to: /tmp/perfkitbenchmarker/run_1798f9fc/pkb.log 2015-12-23 10:34:41,653 1798f9fc MainThread INFO PerfKitBenchmarker version: v1.0.1-103-g5a2c738 2015-12-23 10:34:41,727 1798f9fc MainThread INFO Flag values: --machine_type=m1.medium --image=99f4625e-f425-472d-8e21-7aa9c3db1c3e --cloud=OpenStack --benchmarks=ping --openstack_private_network=smog-net --openstack_public_network=ext-net 2015-12-23 10:34:41,923 1798f9fc MainThread ping(1/1) INFO Starting new HTTP connection (1): publicURL 2015-12-23 10:34:49,041 1798f9fc MainThread ping(1/1) INFO Starting new HTTP connection (1): publicURL 2015-12-23 10:34:54,097 1798f9fc MainThread ping(1/1) INFO Starting new HTTP connection (1): publicURL Found more than one valid endpoint. Use a more restrictive filter 2015-12-23 10:35:07,012 1798f9fc MainThread ping(1/1) ERROR Error during benchmark ping Traceback (most recent call last): File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 338, in RunBenchmark spec.ConstructVirtualMachines() File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 262, in ConstructVirtualMachines vm = self._CreateVirtualMachine(vm_spec, os_type, cloud) File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 344, in _CreateVirtualMachine return vm_class(vm_spec) File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/providers/openstack/os_virtual_machine.py", line 48, in init self.firewall = os_network.OpenStackFirewall.GetFirewall() File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/network.py", line 51, in GetFirewall benchmark_spec.firewalls[key] = cls() File "/home/ubuntu/perfkit/PerfKitBenchmarker/perfkitbenchmarker/providers/openstack/os_network.py", line 38, in init name='perfkit_sc_group')): File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 257, in findall listing = self.list(_list_kwargs) File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/security_groups.py", line 96, in list 'security_groups') File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 74, in _list _resp, body = self.api.client.get(url) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 448, in get return self._cs_request(url, 'GET', _kwargs) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 404, in _cs_request self.authenticate() File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate auth_url = self._v2_auth(auth_url) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 652, in _v2_auth return self._authenticate(url, body) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 667, in _authenticate return self._extract_service_catalog(url, resp, respbody) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 490, in _extract_service_catalog self.management_url = self.get_service_url(self.service_type) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 467, in get_service_url volume_service_name=self.volume_service_name,) File "/usr/local/lib/python2.7/dist-packages/novaclient/service_catalog.py", line 87, in url_for endpoints=matching_endpoints) AmbiguousEndpoints: AmbiguousEndpoints: [{u'adminURL': u'http://controller:8774/v2/db8ac8cc448f48a8a9b1b0d7c302a79d', u'region': u'regionOne', u'id': u'1a66d72c4ee24ea5ad786803225970fa', 'serviceName': u'nova', u'internalURL': u'http://controller:8774/v2/db8ac8cc448f48a8a9b1b0d7c302a79d', u'publicURL': u'http://publicURL:8774/v2/db8ac8cc448f48a8a9b1b0d7c302a79d'}, {u'adminURL': u'http://controller:8774/v2/db8ac8cc448f48a8a9b1b0d7c302a79d', u'region': u'HPC2N', u'id': u'488343185e3d4fc489bef538d4329ef7', 'serviceName': u'nova', u'internalURL': u'http://controller:8774/v2/db8ac8cc448f48a8a9b1b0d7c302a79d', u'publicURL': u'http://publicURL:8774/v2/db8ac8cc448f48a8a9b1b0d7c302a79d'}] 2015-12-23 10:35:07,047 1798f9fc MainThread ping(1/1) ERROR Benchmark 1/1 ping (UID: ping0) failed. Execution will continue.

2015-12-23 10:35:07,048 1798f9fc MainThread ping(1/1) INFO Benchmark run statuses:

Name UID Status

ping ping0 FAILED

Success rate: 0.00% (0/1) 2015-12-23 10:35:07,048 1798f9fc MainThread ping(1/1) INFO Complete logs can be found at: /tmp/perfkitbenchmarker/run_1798f9fc/pkb.log

Regards. Salman.

meteorfox commented 8 years ago

@stoor Thanks for your fast response. Right, according to your logs the OS_REGION_NAME value is not being passed to the nova client.

Ah.. I forgot to update the arguments for the Nova client that's used for creating the Cinder volumes. I'll post another commit with the changes necessary.

meteorfox commented 8 years ago

@stoor OK, so I pushed another commit to PR #770 . Please feel free to test it again. Thanks for taking the time to try it out.

stoor commented 8 years ago

HI again,

No worries I will be happy to participate even more. We are in the process of building a Community Science Cloud in Sweden. All these tests are very much of our interest. Benchmark will make things easy for us, especially when its not our own :-) . Here you can find more information about SNIC Science Cloud: https://cloud.snic.se/

Looking forwards to your patch.

Regards.. Salman.