Open kencafe opened 2 years ago
Hi, can you post the output of kubectl get pods -n cnsbench-system
, kubectl logs -n cnsbench-system cnsbench-output-collector
, and kubectl logs -n cnsbench-system deployment/cnsbench-controller-manager -c manager
?
Hi pzoxiuv,
I also send you sc,pv, and benchmark.yaml
This is logs:
NAME READY STATUS RESTARTS AGE
cnsbench-controller-manager-65f54cf7f9-nfz49 2/2 Running 0 26h
cnsbench-output-collector 1/1 Running 0 26h```
```$ kubectl logs -n cnsbench-system cnsbench-output-collector
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853273,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853274,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853275,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853278,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853283,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853293,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853314,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853355,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853437,"remoteAddr":"10.131.0.19:46596","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853600,"remoteAddr":"10.131.0.19:46744","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647853928,"remoteAddr":"10.131.0.19:46884","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647854583,"remoteAddr":"10.131.0.19:47158","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647855583,"remoteAddr":"10.131.0.19:47576","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
{"timestamp":1647856583,"remoteAddr":"10.131.0.19:47998","endpoint":"/metrics/quickstart-benchmark","params":{},"data":"{\"kind\":\"ConfigMap\",\"name\":\"basic-parser\",\"type\":\"createObj\"}\n"}
$ kubectl logs -n cnsbench-system deployment/cnsbench-controller-manager -c manager
I0321 08:54:48.225936 1 request.go:655] Throttling request took 1.039905033s, request: GET:https://172.30.0.1:443/apis/policy/v1?timeout=32s
2022-03-21T08:54:50.492Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2022-03-21T08:54:50.492Z INFO setup starting manager
I0321 08:54:50.493189 1 leaderelection.go:243] attempting to acquire leader lease cnsbench-system/d01eb01b.example.com...
2022-03-21T08:54:50.493Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0321 08:55:07.911801 1 leaderelection.go:253] successfully acquired lease cnsbench-system/d01eb01b.example.com
2022-03-21T08:55:07.911Z INFO controller-runtime.manager.controller.benchmark Starting EventSource {"reconciler group": "cnsbench.example.com", "reconciler kind": "Benchmark", "source": "kind source: /, Kind="}
2022-03-21T08:55:07.911Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"cnsbench-system","name":"d01eb01b.example.com","uid":"9c651d23-dc9a-4a7f-9572-9b5fbebaf111","apiVersion":"v1","resourceVersion":"17270948"}, "reason": "LeaderElection", "message": "cnsbench-controller-manager-65f54cf7f9-nfz49_025244f5-5b41-431e-b309-dd22abe64c29 became leader"}
2022-03-21T08:55:07.912Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Lease","namespace":"cnsbench-system","name":"d01eb01b.example.com","uid":"4b3cf041-da74-40d2-a748-0a4c7e3d540b","apiVersion":"coordination.k8s.io/v1","resourceVersion":"17270949"}, "reason": "LeaderElection", "message": "cnsbench-controller-manager-65f54cf7f9-nfz49_025244f5-5b41-431e-b309-dd22abe64c29 became leader"}
2022-03-21T08:55:08.012Z INFO controller-runtime.manager.controller.benchmark Starting Controller {"reconciler group": "cnsbench.example.com", "reconciler kind": "Benchmark"}
2022-03-21T08:55:08.012Z INFO controller-runtime.manager.controller.benchmark Starting workers {"reconciler group": "cnsbench.example.com", "reconciler kind": "Benchmark", "worker count": 1}
2022-03-21T08:55:08.012Z INFO controllers.Benchmark Reconciling Benchmark
2022-03-21T08:55:08.012Z INFO controllers.Benchmark {"": {"runtime":"","volumes":null,"workloads":[{"name":"quickstart","workload":"fio","vars":{"fiosize":"500Mi","pvcsize":"1Gi","storageClass":"local-storage"},"count":1,"syncGroup":"","outputFiles":[{"filename":"/output/output.json","parser":"basic-parser","target":"workload","sink":"es"}],"rateName":""}],"controlOperations":null,"rates":null,"workloadsOutput":"defaultWorkloadsOutput","metadataOutput":"defaultMetadataOutput","metricsOutput":"defaultMetricsOutput","outputs":[{"name":"es","httpPostSpec":{"url":"http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/ex1"}},{"name":"defaultWorkloadsOutput","httpPostSpec":{"url":"http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/workloads"}},{"name":"defaultMetadataOutput","httpPostSpec":{"url":"http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/metadata"}},{"name":"defaultMetricsOutput","httpPostSpec":{"url":"http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/metrics"}}]}}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "pvcsize", "replacement": "1Gi"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "storageClass", "replacement": "local-storage"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "fiosize", "replacement": "500Mi"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "config", "default replacement": "fio-rw-config"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "fiosize", "default replacement": "6g"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "initConfig", "default replacement": "fio-rw-initconfig"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "pctread", "default replacement": "50"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "pvcsize", "default replacement": "7Gi"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "rw", "default replacement": "rw"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark Searching for var {"var": "volname", "default replacement": "vol-{{ACTION_NAME}}-{{INSTANCE_NUM}}"}
2022-03-21T08:55:08.413Z INFO controllers.Benchmark labels {"labels": {"role":"helper","type":"parser","workloadname":"quickstart"}}
apiVersion: v1
data:
parser.py: |
#!/usr/bin/env python
import json
import sys
with open(sys.argv[1]) as f:
r = json.load(f)
res = []
for job in r['jobs']:
job_res = {}
job_res['read_bw'] = job['read']['bw']
job_res['write_bw'] = job['write']['bw']
job_res['read_lat'] = job['read']['lat_ns']['mean']
job_res['write_lat'] = job['write']['lat_ns']['mean']
res.append(job_res)
print(json.dumps(res))
kind: ConfigMap
metadata:
annotations:
container: cnsbench/utility:latest
creationTimestamp: null
labels:
role: helper
type: parser
workloadname: quickstart
name: basic-parser
namespace: cnsbench-library
2022-03-21T08:55:08.423Z INFO controllers.Benchmark Object already exists, proceeding {"name": "basic-parser"}
Doing output http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/metrics/quickstart-benchmark
Post "http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/metrics/quickstart-benchmark": dial tcp 172.30.97.2:8888: connect: no route to host
2022-03-21T08:55:11.478Z INFO controller-runtime.manager.controller.benchmark Starting EventSource {"reconciler group": "cnsbench.example.com", "reconciler kind": "Benchmark", "source": "kind source: /v1, Kind=ConfigMap"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "pvcsize", "replacement": "1Gi"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "storageClass", "replacement": "local-storage"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "fiosize", "replacement": "500Mi"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "rw", "default replacement": "rw"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "volname", "default replacement": "vol-{{ACTION_NAME}}-{{INSTANCE_NUM}}"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "config", "default replacement": "fio-rw-config"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "fiosize", "default replacement": "6g"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "initConfig", "default replacement": "fio-rw-initconfig"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "pctread", "default replacement": "50"}
2022-03-21T08:55:11.479Z INFO controllers.Benchmark Searching for var {"var": "pvcsize", "default replacement": "7Gi"}
2022-03-21T08:55:11.483Z ERROR controllers.Benchmark Error making object child of Benchmark {"name": "helper-zddl4", "error": "cross-namespace owner references are disallowed, owner's namespace cnsbench-system, obj's namespace default"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).createObj
/workspace/controllers/actions.go:40
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).createTmpConfigMapFromDisk
/workspace/controllers/workload.go:116
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).addCountdoneScript
/workspace/controllers/workload.go:325
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).setupForOutput
/workspace/controllers/workload.go:424
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).addContainers
/workspace/controllers/workload.go:440
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).prepareAndRun
/workspace/controllers/workload.go:533
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).RunWorkload
/workspace/controllers/actions.go:132
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).startWorkloads
/workspace/controllers/benchmark_controller.go:94
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).Reconcile
/workspace/controllers/benchmark_controller.go:369
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:99
2022-03-21T08:55:11.483Z ERROR controllers.Benchmark Creating temp ConfigMap for countdown.sh {"error": "cross-namespace owner references are disallowed, owner's namespace cnsbench-system, obj's namespace default"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).createTmpConfigMapFromDisk
/workspace/controllers/workload.go:117
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).addCountdoneScript
/workspace/controllers/workload.go:325
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).setupForOutput
/workspace/controllers/workload.go:424
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).addContainers
/workspace/controllers/workload.go:440
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).prepareAndRun
/workspace/controllers/workload.go:533
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).RunWorkload
/workspace/controllers/actions.go:132
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).startWorkloads
/workspace/controllers/benchmark_controller.go:94
github.com/cnsbench/cnsbench/controllers.(*BenchmarkReconciler).Reconcile
/workspace/controllers/benchmark_controller.go:369
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:99
$ oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv 5Gi RWO Delete Available
$ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 25h
$ cat benchmark.yaml
apiVersion: cnsbench.example.com/v1alpha1
kind: Benchmark
metadata:
name: quickstart-benchmark
spec:
workloads:
- name: quickstart
workload: fio
outputFiles:
- filename: /output/output.json
parser: basic-parser
target: workload
sink: es
vars:
storageClass: local-storage
fiosize: 500Mi
pvcsize: 1Gi
outputs:
- name: es
httpPostSpec:
url: http://cnsbench-output-collector.cnsbench-system.svc.cluster.local:8888/ex1
Hi, the Benchmark object should be in the default namespace but based on your first post it looks like it's in the cnsbench-system namespace. Could you try adding namespace: default
to the metadata section of benchmark.yaml and try again?
Hi,
I have created a new benchmark in the default namespace as you mentioned then wait for 15 minutes but have no logs in the "cnsbench-output-collector" pod.
Ok, let's try again and make sure we're starting from a clean slate. Can you delete the benchmark, restart the cnsbench controller (kubectl rollout restart -n cnsbench-system deployment/cnsbench-controller-manager
), and then create the benchmark again? If that doesn't work, please attach the output from kubectl logs -n cnsbench-system deployment/cnsbench-controller-manager -c manager
again.
I have tried to restart deployment but the benchmark didn't generate output. I ran CNSBench on Openshift 4.9
''' $oc rollout restart -n cnsbench-system deployment/cnsbench-controller-manager
NAME READY STATUS RESTARTS AGE cnsbench-controller-manager-67c9d956dc-7qrfd 2/2 Running 0 61s cnsbench-output-collector 1/1 Running 0 7m17s
'''
''' $ oc logs -f cnsbench-output-collector
'''
Just to make sure - you deleted the benchmark, restarted the controller, then created the benchmark again, right? Can you post the logs from the controller (kubectl logs -n cnsbench-system deployment/cnsbench-controller-manager -c manager
)?
The instance in YAML below is a fio benchmark: