search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.49k stars 2.44k forks source link

[Bug]: mkdir /badger/key: permission denied #5208

Closed ivanallen closed 8 months ago

ivanallen commented 9 months ago

What happened?

I got the same problem in this issue: https://github.com/jaegertracing/jaeger/issues/4906

With the release of 1.54, jaeger report message:

Failed to init storage factory","error":"Error Creating Dir: \"/badger/key\" error: mkdir /badger/key: permission denied"

Steps to reproduce

docker run \
  -e SPAN_STORAGE_TYPE=badger \
  -e BADGER_EPHEMERAL=false \
  -e BADGER_DIRECTORY_VALUE=/badger/data \
  -e BADGER_DIRECTORY_KEY=/badger/key \
  -v /mnt/badger:/badger \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.54
2024/02/18 05:04:40 maxprocs: Leaving GOMAXPROCS=96: CPU quota undefined
2024/02/18 05:04:40 application version: git-commit=a614bb9ab161020b17917452e7d9680819622480, git-version=v1.54.0, build-date=2024-02-06T23:10:30Z
{"level":"info","ts":1708232680.1607265,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1708232680.160783,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1708232680.1609085,"caller":"flags/admin.go:130","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1708232680.160957,"caller":"flags/admin.go:144","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1708232680.1609719,"caller":"flags/admin.go:122","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"info","ts":1708232680.161055,"caller":"grpc@v1.61.0/clientconn.go:429","msg":"[core][Channel #1] Channel created","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1610985,"caller":"grpc@v1.61.0/clientconn.go:1724","msg":"[core][Channel #1] original dial target is: \"localhost:4317\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1611161,"caller":"grpc@v1.61.0/clientconn.go:1731","msg":"[core][Channel #1] parsed dial target is: resolver.Target{URL:url.URL{Scheme:\"localhost\", Opaque:\"4317\", User:(*url.Userinfo)(nil), Host:\"\", Path:\"\", RawPath:\"\", OmitHost:false, ForceQuery:false, RawQuery:\"\", Fragment:\"\", RawFragment:\"\"}}","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.161124,"caller":"grpc@v1.61.0/clientconn.go:1745","msg":"[core][Channel #1] fallback to scheme \"passthrough\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1611333,"caller":"grpc@v1.61.0/clientconn.go:1753","msg":"[core][Channel #1] parsed dial target is: passthrough:///localhost:4317","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1611414,"caller":"grpc@v1.61.0/clientconn.go:1874","msg":"[core][Channel #1] Channel authority set to \"localhost:4317\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1613002,"caller":"grpc@v1.61.0/resolver_wrapper.go:196","msg":"[core][Channel #1] Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"localhost:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"localhost:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1613472,"caller":"grpc@v1.61.0/balancer_wrapper.go:161","msg":"[core][Channel #1] Channel switches to new LB policy \"pick_first\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1613917,"caller":"grpc@v1.61.0/balancer_wrapper.go:213","msg":"[core][Channel #1 SubChannel #2] Subchannel created","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1614041,"caller":"grpc@v1.61.0/clientconn.go:532","msg":"[core][Channel #1] Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1614177,"caller":"grpc@v1.61.0/clientconn.go:335","msg":"[core][Channel #1] Channel exiting idle mode","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1614337,"caller":"grpc@v1.61.0/clientconn.go:1223","msg":"[core][Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1708232680.1614451,"caller":"grpc@v1.61.0/clientconn.go:1338","msg":"[core][Channel #1 SubChannel #2] Subchannel picks a new address \"localhost:4317\" to connect","system":"grpc","grpc_log":true}
{"level":"fatal","ts":1708232680.1617432,"caller":"all-in-one/main.go:113","msg":"Failed to init storage factory","error":"Error Creating Dir: \"/badger/key\" error: mkdir /badger/key: permission denied","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/all-in-one/main.go:113\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/all-in-one/main.go:247\nruntime.main\n\truntime/proc.go:267"}

Expected behavior

Jaeger to start up

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

v1.54

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

https://www.jaegertracing.io/docs/1.54/deployment/#badger---local-storage

docker run \
  -e SPAN_STORAGE_TYPE=badger \
  -e BADGER_EPHEMERAL=false \
  -e BADGER_DIRECTORY_VALUE=/badger/data \
  -e BADGER_DIRECTORY_KEY=/badger/key \
  -v /mnt/badger:/badger \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.54
yurishkuro commented 8 months ago

refer to #4906, there's no new information in this ticket

genfemme commented 6 months ago

@yurishkuro what is the workaround for this error? I am trying to do a helm install using this chart - https://raw.githubusercontent.com/hansehe/jaeger-all-in-one/master/helm/charts version - 0.1.12

and getting the same error -

`{"level":"fatal","ts":1715094840.969322,"caller":"all-in-one/main.go:114","msg":"Failed to init storage factory","error":"Error Creating Dir: \"/badger/key\" error: mkdir /badger/key: permission denied","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/all-in-one/main.go:114\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/all-in-one/main.go:249\nruntime.main\n\truntime/proc.go:271"} `

I have tried adding this to the values.yaml file - 

`initContainers:
          name: init-jaeger
          image: jaegertracing/all-in-one:latest
          command: ["/bin/sh", "-c"]
          args: ["mkdir -p /badger/data && touch /badger/data/.initialized && chown -R 10001:10001 /badger/data"]
          volumeMounts:
            - name: jaeger-badger-data
              mountPath: /badger
          securityContext:
            fsGroup: 10001
      volumes:
        - name: jaeger-badger-data
          emptyDir: {}`