search

GoogleCloudPlatform / PerfKitBenchmarker

PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see. PerfKit Benchmarker is licensed under the Apache 2 license terms. Please make sure to read, understand and agree to the terms of the LICENSE and CONTRIBUTING files before proceeding.
https://googlecloudplatform.github.io/PerfKitBenchmarker/
Apache License 2.0
1.89k stars 480 forks source link

object_storage_service fails during cleanup - tries to delete instance, firewall and network multiple times, after successful STDERR of it being deleted #1239

Open jdwall opened 7 years ago

jdwall commented 7 years ago

Hi,

I am unable to get the object_storage_service benchmark to work besides when using the --object_storage_scenario=api_multistream flag . All others,cli, api_data, api_namespace don't work and fail during cleanup.

What seems to be happening is that pkb is deleting the instance, STDERR writes that it is successfully deleted, then pkb tries to delete the now terminated instance, thus failing. I have run into this at least a half dozen times the last two days, and it just clicked that all the log files all show it failing at the instance deletion, and then the loop continues with firewall and network resources too. Since I don't know python at all, I wouldn't even know where to start digging into vm_util.py to see where things might be going awry. If, even, vm_util.py is where the bug lies. 

2016-12-08 00:42:09,113 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:42:09,113 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute instances delete pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a
2016-12-08 00:42:56,859 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute instances delete pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/zones/us-central1-a/instances/pkb-a6869ecd-0].

2016-12-08 00:42:56,860 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:42:56,860 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute instances describe pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a
2016-12-08 00:42:57,473 a6869ecd Thread-469 object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute instances describe pkb-a6869ecd-0 --format json --quiet --project REDACTED --zone us-central1-a. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.instances.describe) Could not fetch resource:
 - The resource 'projects/REDACTED/zones/us-central1-a/instances/pkb-a6869ecd-0' was not found

2016-12-08 00:42:57,499 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:42:57,500 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute firewall-rules delete perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED
2016-12-08 00:43:08,886 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute firewall-rules delete perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/global/firewalls/perfkit-firewall-a6869ecd-22-22].

2016-12-08 00:43:08,886 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:43:08,887 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute firewall-rules describe perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED
2016-12-08 00:43:09,464 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute firewall-rules describe perfkit-firewall-a6869ecd-22-22 --format json --quiet --project REDACTED. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
 - The resource 'projects/REDACTED/global/firewalls/perfkit-firewall-a6869ecd-22-22' was not found

2016-12-08 00:43:09,464 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:43:09,465 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute firewall-rules delete default-internal-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:43:25,992 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute firewall-rules delete default-internal-a6869ecd --format json --quiet --project REDACTED. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/global/firewalls/default-internal-a6869ecd].

2016-12-08 00:43:25,993 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:43:25,994 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute firewall-rules describe default-internal-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:43:26,594 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute firewall-rules describe default-internal-a6869ecd --format json --quiet --project REDACTED. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
 - The resource 'projects/REDACTED/global/firewalls/default-internal-a6869ecd' was not found

2016-12-08 00:43:26,595 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:43:26,595 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute networks delete pkb-network-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:44:13,927 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute networks delete pkb-network-a6869ecd --format json --quiet --project REDACTED. Got return code (0).
STDOUT: []

STDERR: Deleted [https://www.googleapis.com/compute/v1/projects/REDACTED/global/networks/pkb-network-a6869ecd].

2016-12-08 00:44:13,928 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
2016-12-08 00:44:13,928 a6869ecd MainThread object_storage_service(1/1) vm_util.py:287 INFO     Running: gcloud compute networks describe pkb-network-a6869ecd --format json --quiet --project REDACTED
2016-12-08 00:44:14,522 a6869ecd MainThread object_storage_service(1/1) vm_util.py:318 DEBUG    Ran gcloud compute networks describe pkb-network-a6869ecd --format json --quiet --project REDACTED. Got return code (1).
STDOUT:
STDERR: ERROR: (gcloud.compute.networks.describe) Could not fetch resource:
 - The resource 'projects/REDACTED/global/networks/pkb-network-a6869ecd' was not found

2016-12-08 00:44:14,526 a6869ecd MainThread object_storage_service(1/1) pkb.py:551 ERROR    Benchmark 1/1 object_storage_service (UID: object_storage_service0) failed. Execution will continue.
2016-12-08 00:44:14,540 a6869ecd MainThread pkb.py:642 INFO     Benchmark run statuses:
-------------------------------------------------------
Name                    UID                      Status
-------------------------------------------------------
object_storage_service  object_storage_service0  FAILED
-------------------------------------------------------
Success rate: 0.00% (0/1)
2016-12-08 00:44:14,542 a6869ecd MainThread pkb.py:645 INFO     Complete logs can be found at: /tmp/perfkitbenchmarker/runs/a6869ecd/pkb.log

I am currently testing the following to see if it works on AWS and confirm if this is only happening on GCP. I will update this issue with my findings after it finishes running. 

screen ./pkb.py \
--cloud=AWS \
--storage=AWS \
--zones=us-west-2 \
--machine_type=c3.xlarge \
--benchmarks=object_storage_service
ehankland commented 7 years ago

The above log looks fine to me (except for the fact that object storage failed - but it looks like it failed before reaching this point in the log). I don't see the retries that you are talking about. PKB is checking to see if the resources exist after it has already issued the delete command for that resource so the order is basically

delete instance
describe instance (to check that it was really deleted)

delete firewall rule for port 22
describe firewall for port 22 (to check that it was deleted)

delete firewall for internal ips
describe  " "

delete network
describe " "

If you can provide the section of the log with a stack trace or where it first mentions an exception, I can probably help you figure out why the benchmark is failing.

jdwall commented 7 years ago

Here is the full pkb.log. I probably shouldn't comment, since my logic hasn't fared well today, but I think that it is failing because it can't copy a file (although, it showed up 25 times by ctrl +f does not exist? NotFoundException: 404 gs://pkba6869ecd/file-15.dat does not exist.

Can you set a failure threshold? Would I then use the --run_stage_retries flag?

Thanks again @ehankland - you're so on top of this! 

STDERR: Warning: Permanently added '35.184.52.85' (ECDSA) to the list of known hosts.
Copying gs://pkba6869ecd/file-13.dat...
Copying gs://pkba6869ecd/file-14.dat...
/ [0 files][    0.0 B/ 31.3 KiB]                                                
/ [0 files][    0.0 B/ 62.5 KiB]                                                
Copying gs://pkba6869ecd/file-0.dat...
/ [0 files][    0.0 B/ 78.1 KiB]                                                
Copying gs://pkba6869ecd/file-15.dat...
Copying gs://pkba6869ecd/file-1.dat...
/ [0 files][    0.0 B/109.4 KiB]                                                
/ [0 files][    0.0 B/125.0 KiB]                                                
Copying gs://pkba6869ecd/file-10.dat...
Copying gs://pkba6869ecd/file-11.dat...
Copying gs://pkba6869ecd/file-17.dat...
/ [0 files][    0.0 B/140.6 KiB]                                                
/ [0 files][    0.0 B/171.9 KiB]                                                
/ [0 files][    0.0 B/187.5 KiB]                                                
Copying gs://pkba6869ecd/file-12.dat...
Copying gs://pkba6869ecd/file-16.dat...
/ [0 files][    0.0 B/203.1 KiB]                                                
/ [0 files][    0.0 B/234.4 KiB]                                                
Copying gs://pkba6869ecd/file-18.dat...
/ [1 files][ 15.6 KiB/234.4 KiB]                                                
/ [1 files][ 15.6 KiB/265.6 KiB]                                                
/ [2/100 files][ 46.9 KiB/256.1 MiB]   0% Done                                  
/ [3/100 files][ 78.1 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-19.dat...
/ [3/100 files][ 78.1 KiB/256.1 MiB]   0% Done                                  
/ [4/100 files][ 93.8 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-2.dat...
/ [4/100 files][ 93.8 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-20.dat...
/ [4/100 files][ 93.8 KiB/256.1 MiB]   0% Done                                  
/ [5/100 files][156.3 KiB/256.1 MiB]   0% Done                                  
/ [6/100 files][156.3 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-21.dat...
Copying gs://pkba6869ecd/file-22.dat...
/ [6/100 files][156.3 KiB/256.1 MiB]   0% Done                                  
/ [6/100 files][171.9 KiB/256.1 MiB]   0% Done                                  
/ [7/100 files][187.5 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-23.dat...
/ [7/100 files][187.5 KiB/256.1 MiB]   0% Done                                  
/ [8/100 files][187.5 KiB/256.1 MiB]   0% Done                                  
NotFoundException: 404 gs://pkba6869ecd/file-15.dat does not exist.
Copying gs://pkba6869ecd/file-24.dat...
/ [8/100 files][218.8 KiB/256.1 MiB]   0% Done                                  
/ [9/100 files][218.8 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-25.dat...
/ [9/100 files][218.8 KiB/256.1 MiB]   0% Done                                  
Copying gs://pkba6869ecd/file-26.dat...
/ [9/100 files][234.4 KiB/256.1 MiB]   0% Done                                  
/ [10/100 files][234.4 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-27.dat...
/ [10/100 files][234.4 KiB/256.1 MiB]   0% Done                                 
/ [11/100 files][250.0 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-28.dat...
/ [11/100 files][250.0 KiB/256.1 MiB]   0% Done                                 
/ [12/100 files][281.3 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-29.dat...
/ [12/100 files][281.3 KiB/256.1 MiB]   0% Done                                 
/ [13/100 files][312.5 KiB/256.1 MiB]   0% Done                                 
/ [14/100 files][343.8 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-30.dat...
Copying gs://pkba6869ecd/file-3.dat...
/ [15/100 files][375.0 KiB/256.1 MiB]   0% Done                                 
/ [15/100 files][375.0 KiB/256.1 MiB]   0% Done                                 
/ [15/100 files][375.0 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-31.dat...
/ [15/100 files][375.0 KiB/256.1 MiB]   0% Done                                 
/ [16/100 files][406.3 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-32.dat...
/ [16/100 files][406.3 KiB/256.1 MiB]   0% Done                                 
/ [17/100 files][468.8 KiB/256.1 MiB]   0% Done                                 
/ [18/100 files][500.0 KiB/256.1 MiB]   0% Done                                 
/ [19/100 files][531.3 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-33.dat...
/ [19/100 files][531.3 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-34.dat...
/ [19/100 files][531.3 KiB/256.1 MiB]   0% Done                                 
/ [20/100 files][593.8 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-35.dat...
/ [20/100 files][656.3 KiB/256.1 MiB]   0% Done                                 
/ [21/100 files][656.3 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-36.dat...
/ [21/100 files][656.3 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-37.dat...
/ [21/100 files][656.3 KiB/256.1 MiB]   0% Done                                 
/ [22/100 files][687.5 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-38.dat...
/ [23/100 files][750.0 KiB/256.1 MiB]   0% Done                                 
/ [23/100 files][750.0 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-39.dat...
/ [23/100 files][750.0 KiB/256.1 MiB]   0% Done                                 
/ [24/100 files][765.6 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-4.dat...
/ [24/100 files][765.6 KiB/256.1 MiB]   0% Done                                 
/ [25/100 files][828.1 KiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-40.dat...
/ [25/100 files][953.1 KiB/256.1 MiB]   0% Done                                 
/ [26/100 files][  1.1 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-41.dat...
/ [26/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [27/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [28/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [29/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [30/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [31/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-42.dat...
/ [32/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [32/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-43.dat...
Copying gs://pkba6869ecd/file-44.dat...
/ [32/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
/ [32/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-45.dat...
/ [32/100 files][  1.2 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-46.dat...
/ [32/100 files][  1.3 MiB/256.1 MiB]   0% Done                                 
/ [33/100 files][  1.3 MiB/256.1 MiB]   0% Done                                 
/ [34/100 files][  1.3 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-47.dat...
Copying gs://pkba6869ecd/file-49.dat...
Copying gs://pkba6869ecd/file-48.dat...
/ [34/100 files][  1.3 MiB/256.1 MiB]   0% Done                                 
/ [34/100 files][  1.3 MiB/256.1 MiB]   0% Done                                 
/ [34/100 files][  1.3 MiB/256.1 MiB]   0% Done                                 
-
- [35/100 files][  1.4 MiB/256.1 MiB]   0% Done                                 
- [36/100 files][  1.4 MiB/256.1 MiB]   0% Done                                 
- [37/100 files][  1.6 MiB/256.1 MiB]   0% Done                                 
- [38/100 files][  1.7 MiB/256.1 MiB]   0% Done                                 
- [39/100 files][  1.7 MiB/256.1 MiB]   0% Done                                 
- [40/100 files][  1.7 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-5.dat...
- [40/100 files][  1.7 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-52.dat...
- [40/100 files][  1.7 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-53.dat...
- [41/100 files][  1.7 MiB/256.1 MiB]   0% Done                                 
- [42/100 files][  1.8 MiB/256.1 MiB]   0% Done                                 
- [42/100 files][  1.8 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-50.dat...
- [42/100 files][  1.8 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-54.dat...
- [42/100 files][  1.8 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-51.dat...
- [42/100 files][  1.8 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-55.dat...
- [42/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-56.dat...
- [42/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
- [43/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
- [44/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-57.dat...
Copying gs://pkba6869ecd/file-58.dat...
- [44/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
- [44/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
- [45/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-59.dat...
- [45/100 files][  1.9 MiB/256.1 MiB]   0% Done                                 
- [46/100 files][  2.1 MiB/256.1 MiB]   0% Done                                 
- [47/100 files][  2.2 MiB/256.1 MiB]   0% Done                                 
- [48/100 files][  2.2 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-6.dat...
- [49/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-60.dat...
- [49/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
- [49/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
- [50/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-61.dat...
- [50/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-62.dat...
- [50/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
- [51/100 files][  2.4 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-63.dat...
- [51/100 files][  2.6 MiB/256.1 MiB]   0% Done                                 
- [52/100 files][  2.6 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-64.dat...
- [52/100 files][  2.6 MiB/256.1 MiB]   0% Done                                 
Copying gs://pkba6869ecd/file-65.dat...
- [53/100 files][  2.8 MiB/256.1 MiB]   1% Done                                 
- [53/100 files][  2.9 MiB/256.1 MiB]   1% Done                                 
Copying gs://pkba6869ecd/file-66.dat...
- [53/100 files][  2.9 MiB/256.1 MiB]   1% Done                                 
- [54/100 files][  2.9 MiB/256.1 MiB]   1% Done                                 
- [55/100 files][  2.9 MiB/256.1 MiB]   1% Done                                 
Copying gs://pkba6869ecd/file-67.dat...
- [55/100 files][  2.9 MiB/256.1 MiB]   1% Done                                 
Copying gs://pkba6869ecd/file-68.dat...
- [55/100 files][  2.9 MiB/256.1 MiB]   1% Done                                 
- [56/100 files][  3.4 MiB/256.1 MiB]   1% Done                                 
- [57/100 files][  3.8 MiB/256.1 MiB]   1% Done                                 
- [58/100 files][  3.8 MiB/256.1 MiB]   1% Done                                 
Copying gs://pkba6869ecd/file-7.dat...
Copying gs://pkba6869ecd/file-69.dat...
- [58/100 files][  4.1 MiB/256.1 MiB]   1% Done                                 
- [58/100 files][  4.3 MiB/256.1 MiB]   1% Done                                 
- [59/100 files][  4.3 MiB/256.1 MiB]   1% Done                                 
Copying gs://pkba6869ecd/file-70.dat...
- [59/100 files][  4.8 MiB/256.1 MiB]   1% Done                                 
- [60/100 files][  5.3 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-71.dat...
- [61/100 files][  5.3 MiB/256.1 MiB]   2% Done                                 
- [61/100 files][  5.8 MiB/256.1 MiB]   2% Done                                 
- [62/100 files][  5.8 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-72.dat...
- [63/100 files][  5.8 MiB/256.1 MiB]   2% Done                                 
- [63/100 files][  5.8 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-73.dat...
- [63/100 files][  6.0 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-75.dat...
Copying gs://pkba6869ecd/file-74.dat...
- [63/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
- [63/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
- [64/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-76.dat...
Copying gs://pkba6869ecd/file-77.dat...
- [64/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
- [65/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
- [65/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
- [66/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-78.dat...
- [66/100 files][  6.2 MiB/256.1 MiB]   2% Done                                 
- [67/100 files][  7.2 MiB/256.1 MiB]   2% Done                                 
Copying gs://pkba6869ecd/file-79.dat...
- [67/100 files][  7.5 MiB/256.1 MiB]   2% Done                                 
- [68/100 files][  8.7 MiB/256.1 MiB]   3% Done                                 
Copying gs://pkba6869ecd/file-8.dat...
- [68/100 files][  9.0 MiB/256.1 MiB]   3% Done                                 
- [69/100 files][  9.7 MiB/256.1 MiB]   3% Done                                 
Copying gs://pkba6869ecd/file-80.dat...
- [69/100 files][ 10.8 MiB/256.1 MiB]   4% Done                                 
- [70/100 files][ 12.5 MiB/256.1 MiB]   4% Done                                 
Copying gs://pkba6869ecd/file-81.dat...
- [71/100 files][ 12.9 MiB/256.1 MiB]   5% Done                                 
- [72/100 files][ 12.9 MiB/256.1 MiB]   5% Done                                 
- [72/100 files][ 13.4 MiB/256.1 MiB]   5% Done                                 
- [73/100 files][ 13.6 MiB/256.1 MiB]   5% Done                                 
- [74/100 files][ 13.6 MiB/256.1 MiB]   5% Done                                 
- [75/100 files][ 13.6 MiB/256.1 MiB]   5% Done                                 
Copying gs://pkba6869ecd/file-82.dat...
- [75/100 files][ 13.6 MiB/256.1 MiB]   5% Done                                 
Copying gs://pkba6869ecd/file-84.dat...
\
\ [75/100 files][ 13.8 MiB/256.1 MiB]   5% Done                                 
Copying gs://pkba6869ecd/file-85.dat...
Copying gs://pkba6869ecd/file-83.dat...
\ [75/100 files][ 13.8 MiB/256.1 MiB]   5% Done                                 
\ [75/100 files][ 13.8 MiB/256.1 MiB]   5% Done                                 
Copying gs://pkba6869ecd/file-86.dat...
\ [75/100 files][ 13.8 MiB/256.1 MiB]   5% Done                                 
\ [76/100 files][ 14.1 MiB/256.1 MiB]   5% Done                                 
Copying gs://pkba6869ecd/file-87.dat...
\ [76/100 files][ 14.6 MiB/256.1 MiB]   5% Done                                 
\ [77/100 files][ 15.9 MiB/256.1 MiB]   6% Done                                 
\ [78/100 files][ 16.1 MiB/256.1 MiB]   6% Done                                 
Copying gs://pkba6869ecd/file-89.dat...
\ [78/100 files][ 16.5 MiB/256.1 MiB]   6% Done                                 
Copying gs://pkba6869ecd/file-88.dat...
\ [78/100 files][ 16.5 MiB/256.1 MiB]   6% Done                                 
Copying gs://pkba6869ecd/file-9.dat...
\ [79/100 files][ 16.5 MiB/256.1 MiB]   6% Done                                 
\ [79/100 files][ 16.5 MiB/256.1 MiB]   6% Done                                 
\ [80/100 files][ 17.5 MiB/256.1 MiB]   6% Done                                 
Copying gs://pkba6869ecd/file-90.dat...
\ [80/100 files][ 18.3 MiB/256.1 MiB]   7% Done                                 
\ [81/100 files][ 20.5 MiB/256.1 MiB]   7% Done                                 
\ [82/100 files][ 20.7 MiB/256.1 MiB]   8% Done                                 
Copying gs://pkba6869ecd/file-92.dat...
Copying gs://pkba6869ecd/file-91.dat...
\ [82/100 files][ 21.5 MiB/256.1 MiB]   8% Done                                 
\ [82/100 files][ 21.7 MiB/256.1 MiB]   8% Done                                 
\ [83/100 files][ 21.7 MiB/256.1 MiB]   8% Done                                 
\ [84/100 files][ 21.7 MiB/256.1 MiB]   8% Done                                 
Copying gs://pkba6869ecd/file-93.dat...
\ [84/100 files][ 21.9 MiB/256.1 MiB]   8% Done                                 
Copying gs://pkba6869ecd/file-94.dat...
\ [84/100 files][ 22.4 MiB/256.1 MiB]   8% Done                                 
\ [85/100 files][ 23.4 MiB/256.1 MiB]   9% Done                                 
\ [86/100 files][ 23.4 MiB/256.1 MiB]   9% Done                                 
\ [87/100 files][ 23.4 MiB/256.1 MiB]   9% Done                                 
Copying gs://pkba6869ecd/file-95.dat...
\ [87/100 files][ 23.4 MiB/256.1 MiB]   9% Done                                 
Copying gs://pkba6869ecd/file-96.dat...
Copying gs://pkba6869ecd/file-97.dat...
\ [87/100 files][ 23.4 MiB/256.1 MiB]   9% Done                                 
\ [87/100 files][ 23.4 MiB/256.1 MiB]   9% Done                                 
\ [88/100 files][ 35.2 MiB/256.1 MiB]  13% Done                                 
Copying gs://pkba6869ecd/file-98.dat...
\ [88/100 files][ 36.2 MiB/256.1 MiB]  14% Done                                 
|
| [89/100 files][ 91.3 MiB/256.1 MiB]  35% Done                                 
Copying gs://pkba6869ecd/file-99.dat...
| [89/100 files][ 93.4 MiB/256.1 MiB]  36% Done                                 
| [90/100 files][131.7 MiB/256.1 MiB]  51% Done                                 
| [91/100 files][141.8 MiB/256.1 MiB]  55% Done                                 
| [92/100 files][159.0 MiB/256.1 MiB]  62% Done                                 
/
/ [93/100 files][200.4 MiB/256.1 MiB]  78% Done                                 
/ [94/100 files][203.2 MiB/256.1 MiB]  79% Done                                 
/ [95/100 files][231.5 MiB/256.1 MiB]  90% Done                                 
/ [96/100 files][256.1 MiB/256.1 MiB]  99% Done                                 
-
- [97/100 files][256.1 MiB/256.1 MiB]  99% Done                                 
- [98/100 files][256.1 MiB/256.1 MiB]  99% Done                                 
- [99/100 files][256.1 MiB/256.1 MiB]  99% Done                                 
CommandException: 1 file/object could not be transferred.

real    0m4.570s
user    0m4.460s
sys 0m1.280s

2016-12-08 00:41:58,654 a6869ecd MainThread object_storage_service(1/1) object_storage_service_benchmark.py:1051 INFO     failed to download, skip this iteration.
2016-12-08 00:41:58,655 a6869ecd MainThread object_storage_service(1/1) pkb.py:506 ERROR    Error during benchmark object_storage_service
Traceback (most recent call last):
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/pkb.py", line 486, in RunBenchmark
    DoRunPhase(spec, collector, detailed_timer)
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/pkb.py", line 402, in DoRunPhase
    samples = spec.BenchmarkRun(spec)
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/linux_benchmarks/object_storage_service_benchmark.py", line 1213, in Run
    service, buckets[0], regional_bucket_name)
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/linux_benchmarks/object_storage_service_benchmark.py", line 1062, in CLIThroughputBenchmark
    raise NotEnoughResultsError('Failed to complete the required number of '
NotEnoughResultsError: Failed to complete the required number of iterations.
2016-12-08 00:41:58,657 a6869ecd MainThread object_storage_service(1/1) pkb.py:437 INFO     Cleaning up benchmark object_storage_service
2016-12-08 00:41:58,662 a6869ecd MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None
jdwall commented 7 years ago

It looks like everything is working fine when using AWS but not GCP. The AWS benchmark is still running but at least I have some results. The other tests, which were eclusively GCP, never provided any results.

screen ./pkb.py \
--cloud=AWS \
--storage=AWS \
--zones=us-west-2 \
--machine_type=c3.xlarge \
--benchmarks=object_storage_service
root@perfkitbenchmarker-2gb-nyc3-01:/tmp/perfkitbenchmarker/runs/g2gcli# cat /tmp/perfkitbenchmarker/runs/6239117a/pkb.log|grep "average"
INFO:root:One byte upload - {"average": 0.20837792897224428, "p0.1": 0.13980984687805176, "p1": 0.14522194862365723, "p10": 0.15801000595092773, "p5": 0.14963579177856445, "p50": 0.1917409896850586, "p90": 0.25295591354370117, "p95": 0.3169529438018799, "p99": 0.7566258907318115, "p99.9": 1.8729870319366455, "stddev": 0.10892065461484816}
INFO:root:One byte download - {"average": 0.1137332820892334, "p0.1": 0.07432794570922852, "p1": 0.07557487487792969, "p10": 0.08397698402404785, "p5": 0.08257412910461426, "p50": 0.09937787055969238, "p90": 0.15339994430541992, "p95": 0.19535088539123535, "p99": 0.41347312927246094, "p99.9": 1.1359481811523438, "stddev": 0.07620782573389398}
INFO:root:One byte upload - {"average": 0.20837792897224428, "p0.1": 0.13980984687805176, "p1": 0.14522194862365723, "p10": 0.15801000595092773, "p5": 0.14963579177856445, "p50": 0.1917409896850586, "p90": 0.25295591354370117, "p95": 0.3169529438018799, "p99": 0.7566258907318115, "p99.9": 1.8729870319366455, "stddev": 0.10892065461484816}
INFO:root:One byte download - {"average": 0.1137332820892334, "p0.1": 0.07432794570922852, "p1": 0.07557487487792969, "p10": 0.08397698402404785, "p5": 0.08257412910461426, "p50": 0.09937787055969238, "p90": 0.15339994430541992, "p95": 0.19535088539123535, "p99": 0.41347312927246094, "p99.9": 1.1359481811523438, "stddev": 0.07620782573389398}
INFO:root:Single stream upload throughput in Bps: {"average": 20068619.40917157, "p0.1": 1950472.9989378683, "p1": 10488691.56936807, "p10": 13769043.644852094, "p5": 12612660.68634804, "p50": 20832255.982538603, "p90": 25493116.82172014, "p95": 26986954.977141406, "p99": 31968652.960056614, "p99.9": 31968652.960056614, "stddev": 4750721.317252338}
INFO:root:Single stream download throughput in Bps: {"average": 4707167.96072251, "p0.1": 1725730.283665073, "p1": 3525880.9076201315, "p10": 4319442.0391801465, "p5": 4246016.77645854, "p50": 4791234.107325886, "p90": 5150544.877622392, "p95": 5208147.378430123, "p99": 5593050.527760669, "p99.9": 5593050.527760669, "stddev": 448875.24832001637}
INFO:root:Single stream upload throughput in Bps: {"average": 20068619.40917157, "p0.1": 1950472.9989378683, "p1": 10488691.56936807, "p10": 13769043.644852094, "p5": 12612660.68634804, "p50": 20832255.982538603, "p90": 25493116.82172014, "p95": 26986954.977141406, "p99": 31968652.960056614, "p99.9": 31968652.960056614, "stddev": 4750721.317252338}
INFO:root:Single stream download throughput in Bps: {"average": 4707167.96072251, "p0.1": 1725730.283665073, "p1": 3525880.9076201315, "p10": 4319442.0391801465, "p5": 4246016.77645854, "p50": 4791234.107325886, "p90": 5150544.877622392, "p95": 5208147.378430123, "p99": 5593050.527760669, "p99.9": 5593050.527760669, "stddev": 448875.24832001637}
jdwall commented 7 years ago

I spoke too soon, the S3 benchmark also failed. Thanks in advance for your teams help! 

Final stats:
INFO:root:list-after-write consistency percentage: 100.000000
INFO:root:list-after-write latency: {"average": 4.124196364879608, "p0.1": 3.3857500553131104, "p1": 3.4418790340423584, "p10": 3.5945241451263428, "p5": 3.5485169887542725, "p50": 3.918897867202759, "p90": 4.874804973602295, "p95": 5.913867950439453, "p99": 7.811212062835693, "p99.9": 11.186218023300171, "stddev": 0.8223924942565812}
INFO:root:list-after-update consistency percentage: 100.000000
INFO:root:list-after-update latency: {"average": 7.369844853878021, "p0.1": 4.909317970275879, "p1": 5.24261999130249, "p10": 6.204862117767334, "p5": 5.7447350025177, "p50": 7.414448976516724, "p90": 8.556472063064575, "p95": 8.838152885437012, "p99": 9.663112163543701, "p99.9": 9.87239694595337, "stddev": 0.9358969889592512}

2016-12-08 06:05:36,359 6239117a MainThread object_storage_service(1/1) pkb.py:506 ERROR    Error during benchmark object_storage_service
Traceback (most recent call last):
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/pkb.py", line 486, in RunBenchmark
    DoRunPhase(spec, collector, detailed_timer)
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/pkb.py", line 402, in DoRunPhase
    samples = spec.BenchmarkRun(spec)
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/linux_benchmarks/object_storage_service_benchmark.py", line 1213, in Run
    service, buckets[0], regional_bucket_name)
  File "/home/PerfKitBenchmarker-1.9.0/perfkitbenchmarker/linux_benchmarks/object_storage_service_benchmark.py", line 714, in ListConsistencyBenchmark
    result_string[0],
IndexError: list index out of range
2016-12-08 06:05:36,361 6239117a MainThread object_storage_service(1/1) pkb.py:437 INFO     Cleaning up benchmark object_storage_service
2016-12-08 06:05:36,365 6239117a MainThread object_storage_service(1/1) vm_util.py:284 DEBUG    Environment variables: None

Full logs available, pkb_s3_logs.zip.

Probably a dirty way of doing this, but, I am going to try changing PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/object_storage_service_benchmark.py line 714.

result_string[0], → (result_string[0] if result_string else None),
tedsta commented 7 years ago

@jdwall sorry it's been a while since anyone has commented. I intend to get to this, just have been busy with another project.

It turns out that api_data and api_namespace can both be achieved with special configurations of api_multistream, so I will re-implement them that way. After that I'll look into what's up with the list consistency scenario.

jdwall commented 7 years ago

Thanks @tedsta for the help! I wanted to see if you were aware if this has been looked at, or if there is a temporary workaround? Thanks again for everything! Happy Friday, too!

tedsta commented 7 years ago

My bad, the api_namespace scenario is the list consistency benchmark. It is pretty low priority since list consistency is a yes/no answer, we achieved list consistency a long time ago (and S3 has always had it), and so we no longer run this benchmark.

api_data just runs the a one-byte-read-write benchmark and a single stream throughput benchmark, both of which can be achieved using api_multistream and api_multistream_writes.

For example, if you want to do a one byte RW benchmark, just run api_multistream with --object_storage_object_sizes=1B.

If you want to do single stream throughput, just run api_multistream (or api_multistream_writes if you only want upload throughput) with --num_vms=1 and --object_storage_streams_per_vm=1 (which is the default)

Here's an example run of api_multistream with all sorts of flags filled out (runs 100 threads (streams) spread across 10 VMs on GCP, all uploading and downloading 1KB objects to and from GCS in parallel):

./pkb.py --gce_network_name=default --machine_type=n1-standard-4 --benchmarks=object_storage_service --object_storage_scenario=api_multistream --object_storage_object_sizes=1KB --object_storage_multistream_objects_per_stream=1000 --num_vms=10 --object_storage_streams_per_vm=10 --cloud=GCP --storage=GCP --zones=us-west-1 --object_storage_region=us-west-1
lyoungblood commented 6 years ago

I hit the IndexError: list index out of range issue when benchmarking S3 storage as well. Is there a place where all of the object_storage_scenarios are documented?

s-deitz commented 6 years ago

I usually only run with --object_storage_scenario=api_mutlistream, and ran into a similar 'index out of range' issue. Make sure you are using the following flags: --cloud=AWS --storage=AWS --machine_type=m5.2xlarge (or another recent AWS machine type) --zones= --object_storage_region= --object_storage_streams_per_vm=1 # For a single stream --object_storage_object_sizes=4KB --object_storage_multistream_objects_per_stream=1000 --object_storage_latency_histogram_interval=0.01

This isn't the easiest benchmark to configure and I hope we'll be able to simplify it in the future.

Let me know if you still run into problems, and if so, if you can attach a log, that would be helpful.

I'd also be happy to chat about a scenario that you are hoping to test too.

lyoungblood commented 6 years ago

Is there an option to use a pre-existing S3 bucket? I want to run the benchmark with access logging enabled. Thanks in advance for your help.

edit: Figured this out: --object_storage_bucket_name

Now the only annoying thing is that it wants to delete my bucket every time the test completes. Is there a way to keep the bucket?

s-deitz commented 6 years ago

It's also an option in the long list of flags in object_storage_service_benchmark.py: --object_storage_dont_delete_bucket