search

kubewharf / kelemetry

Global control plane tracing for Kubernetes
Apache License 2.0
252 stars 28 forks source link

Audit log is not missed in the tracing data #133

Closed calvinxu closed 1 year ago

calvinxu commented 1 year ago

Steps to reproduce

1. check out 0.2.2 tag from kelemetry repo
2. helm install kelemetry oci://ghcr.io/kubewharf/kelemetry-chart --values values.yaml

Expected behavior

on the Jaeger UI, when browsing the tracing data for a deployment/pod, it will event/audit data in the tracing data,

Actual behavior

it only displays the event data

image

it is quite odd, that the cluster is "foo", is it from a test value? my audit-log setting is as below, enabled in the apiserver setting, not sure whether the setting for webhook is correct??? I followed somehow from the quickstart demo setting.

#cat audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
# cat webhook-config.yaml
apiVersion: v1
clusters:
- cluster:
    server: http://10.200.112.184:8080/audit
  name: audit
contexts:
- context:
    cluster: audit
  name: main
current-context: main
kind: Config
preferences: {}

in quickstart deployment, event and audit data both displayed

image

Kelemetry version

0.2.2

Environment

k8s:1.23.17 Jaeger:1.4.2

SOF3 commented 1 year ago

Please show k log deploy/kelemetry-consumer

calvinxu commented 1 year ago 
# kubectl logs deploy/kelemetry-consumer
Found 3 pods, using pod/kelemetry-consumer-66cb79568c-rhqhf
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-event-span-global-tags value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-pseudo-span-global-tags value="[cluster=tu021]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-reserve-ttl value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-span-extra-ttl value=1m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-span-follow-ttl value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-span-ttl value=30m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-sub-object-primary-poll-interval value=5s
time="2023-07-20T09:27:41Z" level=info msg=Option name=aggregator-sub-object-primary-poll-timeout value=5s
time="2023-07-20T09:27:41Z" level=info msg=Option name=annotation-linker-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-consumer-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-consumer-filter-cluster-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-consumer-group value=kelemetry
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-consumer-group-failures value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-consumer-ignore-impersonate-username value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-consumer-partition value="[0,1,2,3,4,5,6,7]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-dump-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-forward-timeout value=30s
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-forward-url value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-producer-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-producer-partition-key-type value=audit-id
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-producer-worker-count value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=audit-webhook-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=cluster-name-resolver value=address
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-api-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache value=local
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache-etcd-dial-timeout value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache-etcd-endpoints value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache-etcd-prefix value=/diff/
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache-patch-ttl value=10m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache-snapshot-ttl value=10m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-cache-wrapper-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-deletion-snapshot value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-lease-duration value=15s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-name value=kelemetry-diff-controller-leader-election
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-namespace value=default
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-num-leaders value=2
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-renew-deadline value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-leader-election-retry-period value=2s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-redact-pattern value="$this matches nothing^"
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-store-timeout value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-controller-worker-count value=8
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-decorator-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-decorator-fetch-backoff value=1s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-decorator-fetch-event-timeout value=15s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-decorator-fetch-total-timeout value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-lease-duration value=15s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-name value=kelemetry-diff-writer-leader-election
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-namespace value=default
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-num-leaders value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-renew-deadline value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=diff-writer-leader-election-retry-period value=2s
time="2023-07-20T09:27:41Z" level=info msg=Option name=discovery-resync-interval value=1h0m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=dot value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-configmap-name value=kelemetry-event-controller
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-configmap-namespace value=default
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-configmap-sync-interval value=5s
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-lease-duration value=15s
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-name value=kelemetry-event-informer-leader-election
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-namespace value=default
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-num-leaders value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-renew-deadline value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-leader-election-retry-period value=2s
time="2023-07-20T09:27:41Z" level=info msg=Option name=event-informer-worker-count value=8
time="2023-07-20T09:27:41Z" level=info msg=Option name=filter-exclude-types value="[events,events.k8s.io/events,coordination.k8s.io/leases]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=filter-exclude-user-agent value="[/leader-election,local-path-provisioner/]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=http-address value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=http-port value=8080
time="2023-07-20T09:27:41Z" level=info msg=Option name=http-read-header-timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=http-read-timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=http-tls-cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=http-tls-key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-backend value=jaeger-storage
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-cluster-list value=options
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-cluster-names value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-extension-cache value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-redirect-server-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage-plugin-address value=":17271"
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage-plugin-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.consistency value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.directory-key value=/usr/local/bin/data/keys
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.directory-value value=/usr/local/bin/data/values
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.ephemeral value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.maintenance-interval value=5m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.metrics-update-interval value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.read-only value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.badger.span-store-ttl value=72h0m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.connect-timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.connections-per-host value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.consistency value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.disable-compression value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.keyspace value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.local-dc value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.max-retry-attempts value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.password value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.port value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.proto-version value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.reconnect-interval value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.servers value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.socket-keep-alive value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.tls.ca value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.tls.cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.tls.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.tls.key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.tls.server-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.tls.skip-host-verify value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra-archive.username value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.connect-timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.connections-per-host value=2
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.consistency value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.disable-compression value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.index.logs value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.index.process-tags value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.index.tag-blacklist value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.index.tag-whitelist value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.index.tags value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.keyspace value=jaeger_v1_test
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.local-dc value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.max-retry-attempts value=3
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.password value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.port value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.proto-version value=4
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.reconnect-interval value=1m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.servers value=127.0.0.1
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.socket-keep-alive value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.span-store-write-cache-ttl value=12h0m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.tls.ca value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.tls.cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.tls.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.tls.key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.tls.server-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.tls.skip-host-verify value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.cassandra.username value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.bulk.actions value=1000
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.bulk.flush-interval value=200ms
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.bulk.size value=5000000
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.bulk.workers value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.create-index-templates value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.index-date-separator value=-
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.index-prefix value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.index-rollover-frequency-services value=day
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.index-rollover-frequency-spans value=day
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.log-level value=error
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.max-doc-count value=10000
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.num-replicas value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.num-shards value=5
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.password value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.remote-read-clusters value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.send-get-body-as value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.server-urls value="http://127.0.0.1:9200"
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.sniffer value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.sniffer-tls-enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tags-as-fields.all value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tags-as-fields.config-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tags-as-fields.dot-replacement value=@
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tags-as-fields.include value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tls.ca value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tls.cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tls.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tls.key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tls.server-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.tls.skip-host-verify value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.token-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.use-aliases value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.use-ilm value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.username value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es-archive.version value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.bulk.actions value=1000
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.bulk.flush-interval value=200ms
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.bulk.size value=5000000
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.bulk.workers value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.create-index-templates value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.index-date-separator value=-
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.index-prefix value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.index-rollover-frequency-services value=day
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.index-rollover-frequency-spans value=day
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.log-level value=error
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.max-doc-count value=10000
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.max-span-age value=72h0m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.num-replicas value=1
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.num-shards value=5
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.password value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.remote-read-clusters value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.send-get-body-as value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.server-urls value="http://127.0.0.1:9200"
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.sniffer value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.sniffer-tls-enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tags-as-fields.all value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tags-as-fields.config-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tags-as-fields.dot-replacement value=@
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tags-as-fields.include value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tls.ca value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tls.cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tls.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tls.key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tls.server-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.tls.skip-host-verify value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.token-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.use-aliases value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.use-ilm value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.username value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.es.version value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage-plugin.binary value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage-plugin.configuration-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage-plugin.log-level value=warn
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.connection-timeout value=5s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.server value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.tls.ca value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.tls.cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.tls.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.tls.key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.tls.server-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.grpc-storage.tls.skip-host-verify value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.authentication value=none
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.batch-linger value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.batch-max-messages value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.batch-min-messages value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.batch-size value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.brokers value="127.0.0.1:9092"
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.compression value=none
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.compression-level value=0
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.encoding value=protobuf
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.config-file value=/etc/krb5.conf
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.disable-fast-negotiation value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.keytab-file value=/etc/security/kafka.keytab
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.password value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.realm value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.service-name value=kafka
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.use-keytab value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.kerberos.username value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.plaintext.mechanism value=PLAIN
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.plaintext.password value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.plaintext.username value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.protocol-version value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.required-acks value=local
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.tls.ca value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.tls.cert value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.tls.enabled value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.tls.key value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.tls.server-name value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.tls.skip-host-verify value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.kafka.producer.topic value=jaeger-spans
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-storage.span-storage.type value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-tf-exclusive-modifier-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-tf-extension-modifier-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-trace-cache value=local
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-trace-cache-etcd-dial-timeout value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-trace-cache-etcd-endpoints value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-trace-cache-etcd-prefix value=/trace/
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-transform-config value=default
time="2023-07-20T09:27:41Z" level=info msg=Option name=jaeger-transform-config-file value=hack/tfconfig.yaml
time="2023-07-20T09:27:41Z" level=info msg=Option name=kelemetrix-config-path value=hack/kelemetrix.toml
time="2023-07-20T09:27:41Z" level=info msg=Option name=kelemetrix-consumer-group value=kelemetrix
time="2023-07-20T09:27:41Z" level=info msg=Option name=kelemetrix-consumer-partition value="[0,1,2,3,4]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=kelemetrix-debug-log-quantities value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=kelemetrix-debug-log-tags value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=kelemetrix-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-add-dir-header value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-alsologtostderr value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-log-backtrace-at value=":0"
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-log-dir value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-log-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-log-file-max-size value=1800
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-logtostderr value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-one-output value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-skip-headers value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-skip-log-headers value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-stderrthreshold value=2
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-v value=4
time="2023-07-20T09:27:41Z" level=info msg=Option name=klog-vmodule value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-apiservers value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-config value=mapoption
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-config-paths value="[cluster1=]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-other-rest-burst value=100
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-other-rest-qps value=100
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-request-timeout value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-target-cluster value=cluster1
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-target-rest-burst value=100
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-target-rest-qps value=100
time="2023-07-20T09:27:41Z" level=info msg=Option name=kube-use-old-resource-version-clusters value="[]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=log-file value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=log-format value=text
time="2023-07-20T09:27:41Z" level=info msg=Option name=log-level value=info
time="2023-07-20T09:27:41Z" level=info msg=Option name=metrics value=noop
time="2023-07-20T09:27:41Z" level=info msg=Option name=metrics-address value=
time="2023-07-20T09:27:41Z" level=info msg=Option name=metrics-port value=9090
time="2023-07-20T09:27:41Z" level=info msg=Option name=metrics-read-header-timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=metrics-read-timeout value=0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=mq value=local
time="2023-07-20T09:27:41Z" level=info msg=Option name=mq-local-partition-by-object value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=object-cache-fetch-timeout value=5s
time="2023-07-20T09:27:41Z" level=info msg=Option name=object-cache-size value=67108864
time="2023-07-20T09:27:41Z" level=info msg=Option name=object-cache-store-ttl value=1m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=owner-linker-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=pprof-addr value=":6060"
time="2023-07-20T09:27:41Z" level=info msg=Option name=pprof-enable value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=resource-event-tag-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=resource-event-tag-filter-verbs value="[create]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=resource-object-tag-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=resource-tag-mapping value="[pods#nodes:spec.nodeName]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=shutdown-timeout value=15s
time="2023-07-20T09:27:41Z" level=info msg=Option name=span-cache value=etcd
time="2023-07-20T09:27:41Z" level=info msg=Option name=span-cache-etcd-dial-timeout value=10s
time="2023-07-20T09:27:41Z" level=info msg=Option name=span-cache-etcd-endpoints value="[kelemetry-etcd.default.svc:2379]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=span-cache-etcd-prefix value=/span/
time="2023-07-20T09:27:41Z" level=info msg=Option name=span-cache-local-cleanup-frequency value=30m0s
time="2023-07-20T09:27:41Z" level=info msg=Option name=trace-server-enable value=false
time="2023-07-20T09:27:41Z" level=info msg=Option name=tracer value=otel
time="2023-07-20T09:27:41Z" level=info msg=Option name=tracer-otel-endpoint value="kelemetry-collector.default.svc:4317"
time="2023-07-20T09:27:41Z" level=info msg=Option name=tracer-otel-insecure value=true
time="2023-07-20T09:27:41Z" level=info msg=Option name=tracer-otel-resource-attributes value="[service.version=dev]"
time="2023-07-20T09:27:41Z" level=info msg=Option name=usage value=
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=trace-server
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=event-informer
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=diff-decorator
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=diff-controller
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=resource-event-tag
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=diff-api
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=resource-object-tag
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=audit-dump
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-redirect-server
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=audit-forward
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-storage-plugin
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-consumer
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-config
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-span-reader
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-backend
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-cluster-list
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-registry
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=resource-tagger
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-transform
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-trace-cache
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix.Quantifier-list
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix.TagProviderFactory-list
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tfconfig.ModifierFactory-list
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=jaeger-transform-config
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tfconfig.RegisteredStep-list
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-quantity-request-count
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/cluster-name-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/collapse-nesting-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/object-tags-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/compact-duration-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/extract-nesting-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-default-tags
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/prune-childless-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-modifier/extension
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/prune-tags-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-modifier/exclusive
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/group-by-trace-source-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/replace-name-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-quantity-request-latency
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-quantity-mq-latency
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=tf-step/service-operation-replace-visitor
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=kelemetrix-quantity-request-latency-ratio
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=extension.ProviderFactory-list
time="2023-07-20T09:27:41Z" level=info msg="Component disabled" mod=frontend-extension/jaeger-storage
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=kube-config/mapoption
W0720 09:27:41.769007       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=kube-config
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=kube
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=metrics/noop
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=metrics
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=diff-cache/local
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=diff-cache
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=kube-object-cache
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=discovery
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=owner-linker
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=cluster-name/address
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=cluster-name
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=http
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=audit-webhook
[GIN-debug] POST   /audit                    --> github.com/kubewharf/kelemetry/pkg/audit/webhook.(*webhook).handleRequest-fm (1 handlers)
[GIN-debug] POST   /audit/:cluster           --> github.com/kubewharf/kelemetry/pkg/audit/webhook.(*webhook).handleRequest-fm (1 handlers)
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=annotation-linker
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=linker.Linker-list
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=audit.Decorator-list
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=filter
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=eventdecorator.Decorator-list
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=objectspandecorator.Decorator-list
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=spancache/etcd
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=spancache
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=tracer/otel
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=tracer
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=aggregator
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=mq/local
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=mq
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=audit-consumer
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=pprof
time="2023-07-20T09:27:41Z" level=info msg=Initializing mod=audit-producer
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=kube-config/mapoption
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=kube-config
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=kube
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=metrics/noop
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=metrics
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=diff-cache/local
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=diff-cache
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=kube-object-cache
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=discovery
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=owner-linker
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=cluster-name/address
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=cluster-name
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=http
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=audit-webhook
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=annotation-linker
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=linker.Linker-list
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=audit.Decorator-list
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=filter
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=eventdecorator.Decorator-list
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=objectspandecorator.Decorator-list
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=spancache/etcd
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=spancache
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=tracer/otel
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=tracer
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=aggregator
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=mq/local
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=mq
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=audit-consumer
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=pprof
time="2023-07-20T09:27:41Z" level=info msg=Starting mod=audit-producer
time="2023-07-20T09:27:41Z" level=info msg="Startup complete"
calvinxu commented 1 year ago

panic: metric "diff_decorator" was not initialized showed after a while:

time="2023-07-20T09:36:42Z" level=info msg="Startup complete"
panic: metric "diff_decorator_retry_count" was not initialized
        panic: metric "diff_decorator" was not initialized

goroutine 87 [running]:
github.com/kubewharf/kelemetry/pkg/metrics.(*Metric[...]).With(0x4069d9?, 0x4ff15a?)
        /src/pkg/metrics/interface.go:158 +0xf4
github.com/kubewharf/kelemetry/pkg/metrics.(*Metric[...]).DeferCount(0xc00509c2f8?, {0x51e68e?, 0x2bde0d0?, 0x3fd9a20?}, 0x2bc0320?)
        /src/pkg/metrics/interface.go:169 +0x49
panic({0x226f460, 0xc005480590})
        /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/kubewharf/kelemetry/pkg/metrics.(*Metric[...]).With(0xc005080c00?, 0x3fd9a20?)
        /src/pkg/metrics/interface.go:158 +0xf4
github.com/kubewharf/kelemetry/pkg/diff/decorator.(*decorator).tryDecorate(0xc00001c380, {0x2bde0d0, 0xc000338730}, {0x2bfccf0, 0xc0006b84d0}, 0xc0049786e0, 0xc00001ed80)
        /src/pkg/diff/decorator/decorator.go:282 +0x1325
github.com/kubewharf/kelemetry/pkg/diff/decorator.(*decorator).Decorate(0xc00001c380, {0x2bde0d0, 0xc000338730}, 0xc0049786e0, 0xc00001ed80)
        /src/pkg/diff/decorator/decorator.go:144 +0x465
github.com/kubewharf/kelemetry/pkg/audit/consumer.(*receiver).handleItem(0xc0001ac0e0, {0x2bde0d0, 0xc000338730}, {0x2bfccf0?, 0xc0006f1f80?}, 0xc0049786e0, 0xc0050802a0)
        /src/pkg/audit/consumer/consumer.go:278 +0x12d3
github.com/kubewharf/kelemetry/pkg/audit/consumer.(*receiver).handleMessage(0xc0001ac0e0, {0x2bde0d0, 0xc000338730}, {0x2bfccf0?, 0xc0006f1ea0?}, {0xc0006b4980, 0x33, 0x0?}, {0xc0049b4000, 0xbf4, ...}, ...)
        /src/pkg/audit/consumer/consumer.go:175 +0x4de
github.com/kubewharf/kelemetry/pkg/audit/consumer.(*receiver).Init.func1({0x2bde0d0?, 0xc000338730?}, {0x2bfccf0?, 0xc0006f1ea0?}, {0xc0006b4980?, 0x16?, 0xc0004f63a0?}, {0xc0049b4000, 0xbf4, 0xc00})
        /src/pkg/audit/consumer/consumer.go:130 +0x9d
github.com/kubewharf/kelemetry/pkg/audit/mq/local.(*localConsumer).start.func1()
        /src/pkg/audit/mq/local/local.go:203 +0x14c
created by github.com/kubewharf/kelemetry/pkg/audit/mq/local.(*localConsumer).start
        /src/pkg/audit/mq/local/local.go:193 +0x95
SOF3 commented 1 year ago

duplicate of #127