danmagorian commented 3 years ago

Hello!

I just set up perfkitbenchmarker current version for the first time on 4 Ubuntu 20.04 freshly installed local machines (no cloud instances) at home on a flat subnet initially to test, before moving it to work in the right places. Setup went ok, all can be SSHed into, and I created the yaml file:

root@nuc4:/home/itc-eng/PerfKitBenchmarker# cat perf*/configs/iperf.yaml static_vms:

&vm1 ip_address: 192.168.1.168 user_name: itc-eng ssh_private_key: /home/itc-eng/.ssh/id_rsa zone: home disk_specs: = mount_point: perfkitbenchmarker/data
&vm2 ip_address: 192.168.1.169 user_name: itc-eng ssh_private_key: /home/itc-eng/.ssh/id_rsa zone: home disk_specs: = mount_point: perfkitbenchmarker/data
&vm3 ip_address: 192.168.1.170 user_name: itc-eng ssh_private_key: /home/itc-eng/.ssh/id_rsa zone: home disk_specs: = mount_point: perfkitbenchmarker/data
&vm4 ip_address: 192.168.1.171 user_name: itc-eng ssh_private_key: /home/itc-eng/.ssh/id_rsa zone: home disk_specs: = mount_point: perfkitbenchmarker/data iperf: vm_groups: vm_1: static_vms:
- *vm2 vm_2: static_vms:
- *vm1 vm_3: static_vms:
- *vm4 vm_4: static_vms:
- *vm3

Then I invoked it from its dir on node 1 following the github readme with:

./pkb.py --benchmarks=iperf --benchmark_config_file=iperf.yaml --zones=home

but it errors off with a TypeError at the end:

2021-06-08 20:32:28,876 1c68a287 MainThread INFO Verbose logging to: /tmp/perfkitbenchmarker/runs/1c68a287/pkb.log 2021-06-08 20:32:28,877 1c68a287 MainThread INFO PerfKitBenchmarker version: v1.12.0-2629-gcaac1a88 2021-06-08 20:32:28,877 1c68a287 MainThread INFO Flag values: --benchmark_config_file=iperf.yaml --benchmarks=iperf --zones=home 2021-06-08 20:32:28,949 1c68a287 MainThread INFO Running: ssh-keygen -t rsa -N -m PEM -q -f /tmp/perfkitbenchmarker/runs/1c68a287/perfkitbenchmarker_keyfile 2021-06-08 20:32:28,984 1c68a287 MainThread INFO Setting --max_concurrent_threads=200. 2021-06-08 20:32:29,004 1c68a287 MainThread WARNING The key "static_vms" was not in the default config, but was in user overrides. This may indicate a typo. 2021-06-08 20:32:29,004 1c68a287 MainThread WARNING The key "static_vms" was not in the default config, but was in user overrides. This may indicate a typo. 2021-06-08 20:32:29,004 1c68a287 MainThread WARNING The key "vm_3" was not in the default config, but was in user overrides. This may indicate a typo. 2021-06-08 20:32:29,004 1c68a287 MainThread WARNING The key "vm_4" was not in the default config, but was in user overrides. This may indicate a typo. 2021-06-08 20:32:29,004 1c68a287 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo. Traceback (most recent call last): File "./pkb.py", line 21, in sys.exit(Main()) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 1496, in Main return RunBenchmarks() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 1245, in RunBenchmarks benchmark_specs = _CreateBenchmarkSpecs() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 577, in _CreateBenchmarkSpecs config = config_spec_class(name, expected_os_types=expected_os_types, File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/benchmark_config_spec.py", line 1618, in init super(BenchmarkConfigSpec, self).init(component_full_name, kwargs) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/spec.py", line 111, in init self._DecodeAndInit(component_full_name, kwargs, self._decoders, File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/benchmark_config_spec.py", line 1730, in _DecodeAndInit super(BenchmarkConfigSpec, self)._DecodeAndInit( File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/spec.py", line 177, in _DecodeAndInit value = decoder.Decode(config[option], component_full_name, flag_values) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/benchmark_config_spec.py", line 904, in Decode result[vm_group_name] = _VmGroupSpec( File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/benchmark_config_spec.py", line 786, in init super(_VmGroupSpec, self).init( File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/spec.py", line 111, in init self._DecodeAndInit(component_full_name, kwargs, self._decoders, File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/spec.py", line 177, in _DecodeAndInit value = decoder.Decode(config[option], component_full_name, flag_values) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/option_decoders.py", line 323, in Decode result.append(self._item_decoder.Decode(input_item, item_full_name, File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/configs/benchmark_config_spec.py", line 422, in Decode return static_virtual_machine.StaticVmSpec( File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/static_virtual_machine.py", line 88, in init self.disk_specs = [ File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/static_virtual_machine.py", line 89, in disk.BaseDiskSpec( TypeError: BaseSpecMetaClass object argument after must be a mapping, not str

No idea where that's happening. Help would be greatly appreciated. It looks pretty good if I can get it working. Thanks, Dan

cwilkes commented 3 years ago

Thanks for reporting, I'm looking into it.

cwilkes commented 3 years ago

Can you post your iperf.yaml file again but enclose in triple backticks so that there's no formatting? I tried with this and it appears to work:

base_vm: &base_vm
        user_name: itc-eng
        ssh_private_key: /home/itc-eng/.ssh/id_rsa
        disk_specs:
        - mount_point: perfkitbenchmarker/data

base_vm_spec: &base_vm_spec
        GCP:
                machine_type: n1-standard-16
                zone: us-central1-c

static_vms:
        - &vm1 
          <<: *base_vm
          ip_address: 192.168.1.168
        - &vm2
          <<: *base_vm
          ip_address: 192.168.1.169
        - &vm3
          <<: *base_vm
          ip_address: 192.168.1.170
        - &vm4
          <<: *base_vm
          ip_address: 192.168.1.171
iperf:
        vm_groups:
                vm_1:
                        vm_spec: *base_vm_spec
                        static_vms:
                        - *vm2
                vm_2:
                        vm_spec: *base_vm_spec
                        static_vms:
                        - *vm1
                vm_3:
                        vm_spec: *base_vm_spec
                        static_vms:
                        - *vm4
                vm_4:
                        vm_spec: *base_vm_spec
                        static_vms:
                        - *vm3

danmagorian commented 3 years ago

thanks

⁣Get BlueMail for Android

On Jun 8, 2021, 8:47 PM, at 8:47 PM, Chris Wilkes @.***> wrote:

Thanks for reporting, I'm looking into it.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-857287824

danmagorian commented 3 years ago

Here it is backquoted. I see how you pulled all the commons out. Actually I want all 4 VMs to iperf each other, not just the pairs. So it needs to have GCP and machine type and cloud zone even tho it doesn't touch any cloud machines?

  - &vm1
    ip_address: 192.168.1.168
    user_name: itc-eng
    ssh_private_key: /home/itc-eng/.ssh/id_rsa
    zone: home
    disk_specs:
       = mount_point: perfkitbenchmarker/data
  - &vm2
    ip_address: 192.168.1.169
    user_name: itc-eng
    ssh_private_key: /home/itc-eng/.ssh/id_rsa
    zone: home
    disk_specs:
       = mount_point: perfkitbenchmarker/data
  - &vm3
    ip_address: 192.168.1.170
    user_name: itc-eng
    ssh_private_key: /home/itc-eng/.ssh/id_rsa
    zone: home
    disk_specs:
       = mount_point: perfkitbenchmarker/data
  - &vm4
    ip_address: 192.168.1.171
    user_name: itc-eng
    ssh_private_key: /home/itc-eng/.ssh/id_rsa
    zone: home
    disk_specs:
       = mount_point: perfkitbenchmarker/data
iperf:
  vm_groups:
    vm_1:
      static_vms:
        - *vm2
    vm_2:
      static_vms:
        - *vm1
    vm_3:
      static_vms:
        - *vm4
    vm_4:
      static_vms:
        - *vm3

danmagorian commented 3 years ago

And it runs with your iperf.yaml,!! so I guess it needs the GCP machine type and zone. Had to crash out of it, didn't specify how many iterations with below, after which it showed failed.. Thanks, Dan

./pkb.py --benchmarks=iperf --benchmark_config_file=iperf.yaml

danmagorian commented 3 years ago

2021-06-09 04:02:35,693 129984b8 Thread-4 iperf(1/1) INFO Running: ssh -A -p 22 itc-eng@192.168.1.168 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /home/itc-eng/.ssh/id_rsa -o ControlPath="/tmp/perfkitbenchmarker/runs/129984b8/ssh/%h" -o ControlMaster=auto -o ControlPersist=30m hostname 2021-06-09 04:02:36,099 129984b8 Thread-3 iperf(1/1) INFO Running: ssh -A -p 22 itc-eng@192.168.1.171 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /home/itc-eng/.ssh/id_rsa -o ControlPath="/tmp/perfkitbenchmarker/runs/129984b8/ssh/%h" -o ControlMaster=auto -o ControlPersist=30m hostname 2021-06-09 04:02:36,207 129984b8 Thread-2 iperf(1/1) INFO Running: ssh -A -p 22 itc-eng@192.168.1.170 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /home/itc-eng/.ssh/id_rsa -o ControlPath="/tmp/perfkitbenchmarker/runs/129984b8/ssh/%h" -o ControlMaster=auto -o ControlPersist=30m hostname 2021-06-09 04:02:36,324 129984b8 Thread-5 iperf(1/1) INFO Running: ssh -A -p 22 itc-eng@192.168.1.169 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /home/itc-eng/.ssh/id_rsa -o ControlPath="/tmp/perfkitbenchmarker/runs/129984b8/ssh/%h" -o ControlMaster=auto -o ControlPersist=30m hostname 2

danmagorian commented 3 years ago

added to iperf.yaml:

benchmarks:

iperf: flags: iperf_sending_thread_count: 1

but that isn't the iterations just the threads and presumably the default anyway, so it still keeps doing it over and over again until Ctrl-C. Looked thru the docs and didn't see a way to control iterations.

danmagorian commented 3 years ago

hmmm, should be a way to pass the flags thru that iperf normally wants, eg:

The default test duration of Iperf is 10 seconds. You can override the default with the -t switch followed by the time in seconds the test should last.

iperf -s -t 60

cwilkes commented 3 years ago

The flag --iperf_runtime_in_seconds controls the runtime and it is set to 60 seconds by default. Is that what you want?

iperf_benchmark.py

you should be able to specify that on the command line and not have to put it into the iperf.yaml file.

You can also override the --cloud and --machine_type settings, but in order to do that there has to be a "GCP"/"Azure"/"AWS" vm_spec section in the yaml file.

Looking at the iperf_benchmark.py file:

if len(vms) != 2:   
 raise ValueError(       
    f'iperf benchmark requires exactly two machines, found {len(vms)}')

it doesn't look like we support running iperf between more than 2 VMs (netperf_benchmark.py also has that limit).

Have you run iperf between more than 2 VMs? If so we're open for a PR to get it in here! :)

danmagorian commented 3 years ago

Thanks Chris.

./pkb.py --benchmarks=iperf --benchmark_config_file=iperf.yaml --iperf_runtime_in_seconds=20

doesn't stop after 20 seconds, nor does the default without run for 60. It runs indefinitely until Ctrl-C, after which each node logs below. It looks like it's not actually managing to log in with the key, and so keeps trtying indefinitely although they seem to have been created right with ssa-keygen. Dan

Exception occurred while calling DeleteVm(192.168.1.169): Traceback (most recent call last): File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 252, in Run self.return_value = self.target(*self.args, self.kwargs) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 897, in DeleteVm vm.PackageCleanup() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 767, in PackageCleanup self.RestorePackages() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 1778, in RestorePackages self.RemoteCommand('sudo dpkg --clear-selections') File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 911, in RemoteCommand return self.RemoteCommandWithReturnCode(*args, *kwargs)[:2] File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 927, in RemoteCommandWithReturnCode return self.RemoteHostCommandWithReturnCode(args, kwargs) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 999, in RemoteHostCommandWithReturnCode raise errors.VirtualMachine.RemoteCommandError(error_text) perfkitbenchmarker.errors.VirtualMachine.RemoteCommandError: Got non-zero return code (255) executing sudo dpkg --clear-selections Full command: ssh -A -p 22 @. -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /home/itc-eng/.ssh/id_rsa -o ControlPath="/tmp/perfkitbenchmarker/runs/802d7b50/ssh/%h" -o ControlMaster=auto -o ControlPersist=30m sudo dpkg --clear-selections STDOUT: STDERR: Warning: Permanently added '192.168.1.169' (ECDSA) to the list of known hosts. @.: Permission denied (publickey,password).

2021-06-09 15:22:52,985 802d7b50 MainThread iperf(1/1) pkb.py:1134 ERROR Exception running benchmark Traceback (most recent call last): File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 1132, in RunBenchmarkTask RunBenchmark(spec, collector) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 960, in RunBenchmark DoProvisionPhase(spec, detailed_timer) File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 681, in DoProvisionPhase spec.Provision() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 651, in Provision vm_util.RunThreaded( File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 647, in RunThreaded return RunParallelThreads(target_arg_tuples, File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 582, in RunParallelThreads return _RunParallelTasks( File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 532, in _RunParallelTasks task_id = task_manager.AwaitAnyTask() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 392, in AwaitAnyTask worker_id, task_id = self._response_queue.Get() File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 163, in Get if not _WaitForCondition(lambda: self._deque, timeout): File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/background_tasks.py", line 146, in _WaitForCondition time.sleep(delay) KeyboardInterrupt 2021-06-09 15:22:52,986 802d7b50 MainThread iperf(1/1) pkb.py:1138 ERROR Benchmark 1/1 iperf (UID: iperf0) failed. Execution will not continue. 2021-06-09 15:22:52,986 802d7b50 MainThread iperf(1/1) pkb.py:1273 INFO Benchmark run statuses:

Name UID Status Failed Substatus

iperf iperf0 FAILED

Success rate: 0.00% (0/1)

From: Chris Wilkes @.> Sent: Wednesday, June 9, 2021 3:31 AM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

The flag --iperf_runtime_in_seconds controls the runtime and it is set to 60 seconds by default. Is that what you want?

iperf_benchmark.pyhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/linux_benchmarks/iperf_benchmark.py#L35

you should be able to specify that on the command line and not have to put it into the iperf.yaml file.

You can also override the --cloud and --machine_type settings, but in order to do that there has to be a "GCP"/"Azure"/"AWS" vm_spec section in the yaml file.

Looking at the iperf_benchmark.py file:

if len(vms) != 2: raise ValueError( f'iperf benchmark requires exactly two machines, found {len(vms)}')

it doesn't look like we support running iperf between more than 2 VMs (netperf_benchmark.py also has that limit).

Have you run iperf between more than 2 VMs? If so we're open for a PR to get it in here! :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-857459887, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6T33VHU7EELO3VGFKDTR4KGLANCNFSM46KXVL2A.

danmagorian commented 3 years ago

for testing purposes at home, can I give the password in the yaml file without the public/private key exchange?

cwilkes commented 3 years ago

I would try and get the SSH private key working, it is a lot better to use that in the long run.

Can you try this?

ssh -vvv -i /home/itc-eng/.ssh/id_rsa itc-eng itc-eng@192.168.168

the -vvv will show you debug information, most likely there's a file permission error. You can also use the ssh-copy-id to copy up the right id if that is the problem.

danmagorian commented 3 years ago

Thanks Chris. I verified that the private key is readable by itc-eng (not that 644 made any difference):

@.***:~/.ssh$ ls -al -rw------- 1 itc-eng itc-eng 2546 Jun 8 21:03 id_rsa -rw-r--r-- 1 itc-eng itc-eng 566 Jun 8 21:03 id_rsa.pub -rw-r--r-- 1 itc-eng itc-eng 222 Jun 10 18:23 known_hosts

ssh -vvv -i /home/itc-eng/.ssh/id_rsa @.***

-vvv produces a lot of cruft below, but the gist seems to be that the private key fails and so it reverts to keyboard-interactive password prompt. But I don't see where it says what it didn't like when trying the private key. Which I generated logged in as itc-eng with

ssh-keygen -t rsa -m PEM

and put in the normal login password for that user.

Dan

@.:/etc/ssh$ ssh -vvv -i /home/itc-eng/.ssh/id_rsa @. OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for debug2: resolve_canonicalize: hostname 192.168.1.168 is address debug2: ssh_connect_direct debug1: Connecting to 192.168.1.168 [192.168.1.168] port 22. debug1: Connection established. debug1: identity file /home/itc-eng/.ssh/id_rsa type 0 debug1: identity file /home/itc-eng/.ssh/id_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 192.168.1.168:22 as 'itc-eng' debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug3: order_hostkeyalgs: prefer hostkeyalgs: @.**@*.**@*.,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c debug2: host key algorithms: @.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: @*.**@*.**@. debug2: ciphers stoc: @.**@*.**@*. debug2: MACs ctos: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @.,zlib debug2: compression stoc: @.,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: @.**@*.**@*. debug2: ciphers stoc: @*.**@*.**@. debug2: MACs ctos: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @. debug2: compression stoc: @. debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: @. MAC: compression: none debug1: kex: client->server cipher: @. MAC: compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:vjzief74ZJMUf5GaVCR+Rrortry2N91hfLARRviwyXw debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug1: Host '192.168.1.168' is known and matches the ECDSA host key. debug1: Found key in /home/itc-eng/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: @.**@.> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password @.***'s password:

From: Chris Wilkes @.> Sent: Wednesday, June 9, 2021 2:25 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

I would try and get the SSH private key working, it is a lot better to use that in the long run.

Can you try this?

ssh -vvv -i /home/itc-eng/.ssh/id_rsa itc-eng @.***

the -vvv will show you debug information, most likely there's a file permission error. You can also use the ssh-copy-id to copy up the right id if that is the problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-857949137, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6W2PDAPW5HMSIQLAMLTR6W2TANCNFSM46KXVL2A.

danmagorian commented 3 years ago

Also uncommented this and restarted sshd, didn't help:

@.***:~/.ssh$ more /etc/ssh/sshd_config | grep Pubkey PubkeyAuthentication yes

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 2:53 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Thanks Chris. I verified that the private key is readable by itc-eng (not that 644 made any difference):

@.***:~/.ssh$ ls -al -rw------- 1 itc-eng itc-eng 2546 Jun 8 21:03 id_rsa -rw-r--r-- 1 itc-eng itc-eng 566 Jun 8 21:03 id_rsa.pub -rw-r--r-- 1 itc-eng itc-eng 222 Jun 10 18:23 known_hosts

ssh -vvv -i /home/itc-eng/.ssh/id_rsa @.***

-vvv produces a lot of cruft below, but the gist seems to be that the private key fails and so it reverts to keyboard-interactive password prompt. But I don't see where it says what it didn't like when trying the private key. Which I generated logged in as itc-eng with

ssh-keygen -t rsa -m PEM

and put in the normal login password for that user.

Dan

@.:/etc/ssh$ ssh -vvv -i /home/itc-eng/.ssh/id_rsa @. OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for debug2: resolve_canonicalize: hostname 192.168.1.168 is address debug2: ssh_connect_direct debug1: Connecting to 192.168.1.168 [192.168.1.168] port 22. debug1: Connection established. debug1: identity file /home/itc-eng/.ssh/id_rsa type 0 debug1: identity file /home/itc-eng/.ssh/id_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 192.168.1.168:22 as 'itc-eng' debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug3: order_hostkeyalgs: prefer hostkeyalgs: @.**@*.**@*.,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c debug2: host key algorithms: @.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: @*.**@*.**@. debug2: ciphers stoc: @.**@*.**@*. debug2: MACs ctos: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @.,zlib debug2: compression stoc: @.,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: @.**@*.**@*. debug2: ciphers stoc: @*.**@*.**@. debug2: MACs ctos: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @. debug2: compression stoc: @. debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: @. MAC: compression: none debug1: kex: client->server cipher: @. MAC: compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:vjzief74ZJMUf5GaVCR+Rrortry2N91hfLARRviwyXw debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug1: Host '192.168.1.168' is known and matches the ECDSA host key. debug1: Found key in /home/itc-eng/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: @.**@.> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password @.***'s password:

From: Chris Wilkes @.> Sent: Wednesday, June 9, 2021 2:25 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

I would try and get the SSH private key working, it is a lot better to use that in the long run.

Can you try this?

ssh -vvv -i /home/itc-eng/.ssh/id_rsa itc-eng @.***

the -vvv will show you debug information, most likely there's a file permission error. You can also use the ssh-copy-id to copy up the right id if that is the problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-857949137, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6W2PDAPW5HMSIQLAMLTR6W2TANCNFSM46KXVL2A.

danmagorian commented 3 years ago

Ah hah, found it! I had been assuming that the normal ssh login passphrase needed to be provided by the key exchange, but in fact that caused the key exchange to prompt for the key exchange passphrase, which is unrelated to the normal login passphrase. Also .ssh/authorized_keys is deprecated these days and .ssh/known_hosts created automatically.

ssh-keygen -t rsa #If you want to use this key for a non-interactive connection, do not provide a pass-phrase.

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 3:22 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Also uncommented this and restarted sshd, didn't help:

@.***:~/.ssh$ more /etc/ssh/sshd_config | grep Pubkey PubkeyAuthentication yes

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 2:53 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Thanks Chris. I verified that the private key is readable by itc-eng (not that 644 made any difference):

@.***:~/.ssh$ ls -al -rw------- 1 itc-eng itc-eng 2546 Jun 8 21:03 id_rsa -rw-r--r-- 1 itc-eng itc-eng 566 Jun 8 21:03 id_rsa.pub -rw-r--r-- 1 itc-eng itc-eng 222 Jun 10 18:23 known_hosts

ssh -vvv -i /home/itc-eng/.ssh/id_rsa @.***

-vvv produces a lot of cruft below, but the gist seems to be that the private key fails and so it reverts to keyboard-interactive password prompt. But I don't see where it says what it didn't like when trying the private key. Which I generated logged in as itc-eng with

ssh-keygen -t rsa -m PEM

and put in the normal login password for that user.

Dan

@.:/etc/ssh$ ssh -vvv -i /home/itc-eng/.ssh/id_rsa @. OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for debug2: resolve_canonicalize: hostname 192.168.1.168 is address debug2: ssh_connect_direct debug1: Connecting to 192.168.1.168 [192.168.1.168] port 22. debug1: Connection established. debug1: identity file /home/itc-eng/.ssh/id_rsa type 0 debug1: identity file /home/itc-eng/.ssh/id_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 192.168.1.168:22 as 'itc-eng' debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug3: order_hostkeyalgs: prefer hostkeyalgs: @.**@*.**@*.,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c debug2: host key algorithms: @.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: @*.**@*.**@. debug2: ciphers stoc: @.**@*.**@*. debug2: MACs ctos: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @.,zlib debug2: compression stoc: @.,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: @.**@*.**@*. debug2: ciphers stoc: @*.**@*.**@. debug2: MACs ctos: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @. debug2: compression stoc: @. debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: @. MAC: compression: none debug1: kex: client->server cipher: @. MAC: compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:vjzief74ZJMUf5GaVCR+Rrortry2N91hfLARRviwyXw debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug1: Host '192.168.1.168' is known and matches the ECDSA host key. debug1: Found key in /home/itc-eng/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: @.**@.> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password @.***'s password:

From: Chris Wilkes @.> Sent: Wednesday, June 9, 2021 2:25 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

I would try and get the SSH private key working, it is a lot better to use that in the long run.

Can you try this?

ssh -vvv -i /home/itc-eng/.ssh/id_rsa itc-eng @.***

the -vvv will show you debug information, most likely there's a file permission error. You can also use the ssh-copy-id to copy up the right id if that is the problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-857949137, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6W2PDAPW5HMSIQLAMLTR6W2TANCNFSM46KXVL2A.

danmagorian commented 3 years ago

Spoke too soon: it's back to prompting for the passphrase again and I'm not sure what changed from when it briefly worked. Frustrating, esp since there's a lot outdatd info out there on debugging key exchange issues.

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 3:52 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Ah hah, found it! I had been assuming that the normal ssh login passphrase needed to be provided by the key exchange, but in fact that caused the key exchange to prompt for the key exchange passphrase, which is unrelated to the normal login passphrase. Also .ssh/authorized_keys is deprecated these days and .ssh/known_hosts created automatically.

ssh-keygen -t rsa #If you want to use this key for a non-interactive connection, do not provide a pass-phrase.

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 3:22 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Also uncommented this and restarted sshd, didn't help:

@.***:~/.ssh$ more /etc/ssh/sshd_config | grep Pubkey PubkeyAuthentication yes

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 2:53 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Thanks Chris. I verified that the private key is readable by itc-eng (not that 644 made any difference):

@.***:~/.ssh$ ls -al -rw------- 1 itc-eng itc-eng 2546 Jun 8 21:03 id_rsa -rw-r--r-- 1 itc-eng itc-eng 566 Jun 8 21:03 id_rsa.pub -rw-r--r-- 1 itc-eng itc-eng 222 Jun 10 18:23 known_hosts

ssh -vvv -i /home/itc-eng/.ssh/id_rsa @.***

-vvv produces a lot of cruft below, but the gist seems to be that the private key fails and so it reverts to keyboard-interactive password prompt. But I don't see where it says what it didn't like when trying the private key. Which I generated logged in as itc-eng with

ssh-keygen -t rsa -m PEM

and put in the normal login password for that user.

Dan

@.:/etc/ssh$ ssh -vvv -i /home/itc-eng/.ssh/id_rsa @. OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for debug2: resolve_canonicalize: hostname 192.168.1.168 is address debug2: ssh_connect_direct debug1: Connecting to 192.168.1.168 [192.168.1.168] port 22. debug1: Connection established. debug1: identity file /home/itc-eng/.ssh/id_rsa type 0 debug1: identity file /home/itc-eng/.ssh/id_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 192.168.1.168:22 as 'itc-eng' debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug3: order_hostkeyalgs: prefer hostkeyalgs: @.**@*.**@*.,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c debug2: host key algorithms: @.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.**@*.,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: @*.**@*.**@. debug2: ciphers stoc: @.**@*.**@*. debug2: MACs ctos: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @.,zlib debug2: compression stoc: @.,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: @.,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: @.**@*.**@*. debug2: ciphers stoc: @*.**@*.**@. debug2: MACs ctos: @.**@*.**@*.**@*.**@*.**@*.**@*.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: @*.**@*.**@*.**@*.**@*.**@*.**@.,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: @. debug2: compression stoc: @. debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: @. MAC: compression: none debug1: kex: client->server cipher: @. MAC: compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:vjzief74ZJMUf5GaVCR+Rrortry2N91hfLARRviwyXw debug3: hostkeys_foreach: reading file "/home/itc-eng/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/itc-eng/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 192.168.1.168 debug1: Host '192.168.1.168' is known and matches the ECDSA host key. debug1: Found key in /home/itc-eng/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: @.**@.> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/itc-eng/.ssh/id_rsa RSA SHA256:c0ygr/HYb1oVIZJVBgCqiYYbYYw7ZHiu1L7TtGjerWw explicit debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password @.***'s password:

From: Chris Wilkes @.> Sent: Wednesday, June 9, 2021 2:25 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

I would try and get the SSH private key working, it is a lot better to use that in the long run.

Can you try this?

ssh -vvv -i /home/itc-eng/.ssh/id_rsa itc-eng @.***

the -vvv will show you debug information, most likely there's a file permission error. You can also use the ssh-copy-id to copy up the right id if that is the problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-857949137, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6W2PDAPW5HMSIQLAMLTR6W2TANCNFSM46KXVL2A.

cwilkes commented 3 years ago

Is /home/itc-eng/.ssh/id_rsa on an NFS mount? If not you could be on a different VM that has an older version of that file. Or maybe it gets overwritten by a cronjob.

I would create one just for this use: ssh-keygen -t rsa -f ~/.ssh/id_rsa_pkb then be sure and copy it to the remove VMs with ssh-copy-id https://www.ssh.com/academy/ssh/copy-id (it automates appending the ~/.ssh/id_rsa_pkb.pub to ~/.ssh/authorized_keys which is easy to screw up).

Feel free to email me directly to handle it (chriswilkes at google dot com), we can then talk over IM.

danmagorian commented 3 years ago

No NFS mount.

I think I actually got it. On each I had to use:

ssh-keygen -t rsa -q -P ""

and THEN push them out to each of the others with:

ssh-copy-id @.***

Apparently authorized_keys IS still needed which ssh-copy-id writes to, but manually pasting the other one-liner public keys into it didn't work for some reason even tho if you cat the files they look the same and there were no extra cr/lfs.

Bottom line is they can finally ssh into each other w/o any prompts. Back to testing the original issue to see if it actually completes successfully after 20 secs.

Dan

From: Chris Wilkes @.> Sent: Thursday, June 10, 2021 4:44 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Is /home/itc-eng/.ssh/id_rsa on an NFS mount? If not you could be on a different VM that has an older version of that file. Or maybe it gets overwritten by a cronjob.

I would create one just for this use: ssh-keygen -t rsa -f ~/.ssh/id_rsa_pkb then be sure and copy it to the remove VMs with ssh-copy-id https://www.ssh.com/academy/ssh/copy-id (it automates appending the ~/.ssh/id_rsa_pkb.pub to ~/.ssh/authorized_keys which is easy to screw up).

Feel free to email me directly to handle it (chriswilkes at google dot com), we can then talk over IM.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-859048969, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6SZTSXDM6HRNVYLPG3TSEP23ANCNFSM46KXVL2A.

danmagorian commented 3 years ago

Chris:

Ok, so now instead of complaining about failed logins endlessly, they now just error off immediately with a complaint about perfbenchmarker/init..py:

@.***:~/PerfKitBenchmarker$ ./pkb.py --benchmarks=iperf --benchmark_config_file=iperf.yaml --iperf_runtime_in_seconds=20 Traceback (most recent call last): File "./pkb.py", line 19, in from perfkitbenchmarker.pkb import Main File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/init.py", line 15, in from absl import app ModuleNotFoundError: No module named 'absl'

@.**:~/PerfKitBenchmarker/perfkitbenchmarker$ ls _.py init.py

@.**:~/PerfKitBenchmarker/perfkitbenchmarker$ cat _.py

Copyright 2014 PerfKitBenchmarker Authors. All rights reserved.

#

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#

http://www.apache.org/licenses/LICENSE-2.0

#

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

"""Import absl.app.""" from absl import app

@.**:~/PerfKitBenchmarker/perfkitbenchmarker$ ls ab ls: cannot access 'ab*': No such file or directory

Did that not get installed properly initially?

Thanks, Dan

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 5:16 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

No NFS mount.

I think I actually got it. On each I had to use:

ssh-keygen -t rsa -q -P ""

and THEN push them out to each of the others with:

ssh-copy-id @.***

Apparently authorized_keys IS still needed which ssh-copy-id writes to, but manually pasting the other one-liner public keys into it didn't work for some reason even tho if you cat the files they look the same and there were no extra cr/lfs.

Bottom line is they can finally ssh into each other w/o any prompts. Back to testing the original issue to see if it actually completes successfully after 20 secs.

Dan

From: Chris Wilkes @.> Sent: Thursday, June 10, 2021 4:44 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Is /home/itc-eng/.ssh/id_rsa on an NFS mount? If not you could be on a different VM that has an older version of that file. Or maybe it gets overwritten by a cronjob.

I would create one just for this use: ssh-keygen -t rsa -f ~/.ssh/id_rsa_pkb then be sure and copy it to the remove VMs with ssh-copy-id https://www.ssh.com/academy/ssh/copy-id (it automates appending the ~/.ssh/id_rsa_pkb.pub to ~/.ssh/authorized_keys which is easy to screw up).

Feel free to email me directly to handle it (chriswilkes at google dot com), we can then talk over IM.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-859048969, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6SZTSXDM6HRNVYLPG3TSEP23ANCNFSM46KXVL2A.

danmagorian commented 3 years ago

@.***:~$ pip install absl-py Requirement already satisfied: absl-py in /usr/local/lib/python3.8/dist-packages (0.12.0) Requirement already satisfied: six in /usr/lib/python3/dist-packages (from absl-py) (1.14.0)

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 6:00 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Chris:

Ok, so now instead of complaining about failed logins endlessly, they now just error off immediately with a complaint about perfbenchmarker/init..py:

@.***:~/PerfKitBenchmarker$ ./pkb.py --benchmarks=iperf --benchmark_config_file=iperf.yaml --iperf_runtime_in_seconds=20 Traceback (most recent call last): File "./pkb.py", line 19, in from perfkitbenchmarker.pkb import Main File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/init.py", line 15, in from absl import app ModuleNotFoundError: No module named 'absl'

@.**:~/PerfKitBenchmarker/perfkitbenchmarker$ ls _.py init.py

@.**:~/PerfKitBenchmarker/perfkitbenchmarker$ cat _.py

#

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#

http://www.apache.org/licenses/LICENSE-2.0

#

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

"""Import absl.app.""" from absl import app

@.**:~/PerfKitBenchmarker/perfkitbenchmarker$ ls ab ls: cannot access 'ab*': No such file or directory

Did that not get installed properly initially?

Thanks, Dan

From: dan magorian.net @.> Sent: Thursday, June 10, 2021 5:16 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.>; GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

No NFS mount.

I think I actually got it. On each I had to use:

ssh-keygen -t rsa -q -P ""

and THEN push them out to each of the others with:

ssh-copy-id @.***

Apparently authorized_keys IS still needed which ssh-copy-id writes to, but manually pasting the other one-liner public keys into it didn't work for some reason even tho if you cat the files they look the same and there were no extra cr/lfs.

Bottom line is they can finally ssh into each other w/o any prompts. Back to testing the original issue to see if it actually completes successfully after 20 secs.

Dan

From: Chris Wilkes @.> Sent: Thursday, June 10, 2021 4:44 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Is /home/itc-eng/.ssh/id_rsa on an NFS mount? If not you could be on a different VM that has an older version of that file. Or maybe it gets overwritten by a cronjob.

I would create one just for this use: ssh-keygen -t rsa -f ~/.ssh/id_rsa_pkb then be sure and copy it to the remove VMs with ssh-copy-id https://www.ssh.com/academy/ssh/copy-id (it automates appending the ~/.ssh/id_rsa_pkb.pub to ~/.ssh/authorized_keys which is easy to screw up).

Feel free to email me directly to handle it (chriswilkes at google dot com), we can then talk over IM.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-859048969, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6SZTSXDM6HRNVYLPG3TSEP23ANCNFSM46KXVL2A.

cwilkes commented 3 years ago

Did you go through the instructions?

Mainly:

Create a python virtualenv
pip install the requirements.txt

Also do a which python and which pip to confirm that you're using one in the same location. It is safer to do it in an python virturalenv so that you don't run into issues with libraries not being installed or having the wrong version.

danmagorian commented 3 years ago

Yes, I followed those exactly.

Kludges R'US solved that one:

pip uninstall absl-py pip install absl-py

followed by complaints needing:

sudo python3 -m pip install numpy sudo pip3 install contextlib2 sudo pip3 install pint sudo pip3 install google

but still ModuleNotFoundError: No module named 'google' so that wasn't what it wanted.

I think the issue here is that they have python3.8 installed and not python2, so I installed python 2.7 also. And then that brought me right back to what I had just fixed

Thanks, Dan:

@.:~/PerfKitBenchmarker$ ./pkb.py --benchmarks=iperf --benchmark_config_file=iperf.yaml Traceback (most recent call last): File "./pkb.py", line 19, in from perfkitbenchmarker.pkb import Main File "/home/itc-eng/PerfKitBenchmarker/perfkitbenchmarker/init.py", line 15, in from absl import app ImportError: No module named absl @.:~/PerfKitBenchmarker$ sudo pip install absl-py Requirement already satisfied: absl-py in /usr/local/lib/python3.8/dist-packages (0.12.0) Requirement already satisfied: six in /usr/lib/python3/dist-packages (from absl-py) (1.14.0)

Which python does it want? I'm tempted to start from scratch with the right one. No need for a virtualenv since these wont be doing anything else.

From: Chris Wilkes @.> Sent: Thursday, June 10, 2021 6:42 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Author @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

Did you go through the instructionshttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker#installation-and-setup?

Mainly:

Create a python virtualenv
pip install the requirements.txt

Also do a which python and which pip to confirm that you're using one in the same location. It is safer to do it in an python virturalenvhttps://docs.python.org/3/library/venv.html so that you don't run into issues with libraries not being installed or having the wrong version.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-859130890, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6Q6RWT5AQSRSITRNQDTSE5UJANCNFSM46KXVL2A.

yuyantingzero commented 3 years ago

@danmagorian any update, are you still having issues with iperf?

danmagorian commented 3 years ago

I gave up on perfkitbenchmarker, it was one problem after another. Dan

From: yuyanting @.> Sent: Monday, July 12, 2021 3:58 PM To: GoogleCloudPlatform/PerfKitBenchmarker @.> Cc: dan magorian.net @.>; Mention @.> Subject: Re: [GoogleCloudPlatform/PerfKitBenchmarker] TypeError trying to run perfkitbenchmarker first time (#2976)

@danmagorianhttps://github.com/danmagorian any update, are you still having issues with iperf?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/2976#issuecomment-878553508, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUETH6T4V3QDKO4LQ4HPFG3TXNCPPANCNFSM46KXVL2A.

yuyantingzero commented 3 years ago

sry to hear that

HariChepyala commented 2 years ago

Hello Chris, I am having similar problem too. Could you help me with that

nurSaadat commented 1 year ago

@danmagorian solved ModuleNotFoundError: No module named 'absl'. you can find full code in #4178 Reference for installation here

GoogleCloudPlatform / PerfKitBenchmarker

TypeError trying to run perfkitbenchmarker first time #2976

Name UID Status Failed Substatus

iperf iperf0 FAILED

Copyright 2014 PerfKitBenchmarker Authors. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

Copyright 2014 PerfKitBenchmarker Authors. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.