fileexporter produces `Permissions Denied` when executed no k8s

[sfc-gh-aivanou]

Describe the bug When running fileexporter on k8s, I am getting the following error:

Error: cannot start pipelines: open /data/metrics.json: permission denied
2022/08/25 18:08:26 collector server run finished with error: cannot start pipelines: open /data/metrics.json: permission denied

The /data is mounted from the host fs, with the following configuration:

drwxrwxrwx  2 ubuntu ubuntu 4096 Aug 25 17:57 .
drwxr-x--- 15 ubuntu ubuntu 4096 Aug 25 18:06 ..
-rw-rw-rw-  1 ubuntu ubuntu 3323 Aug 16 19:21 d1
-rw-rw-rw-  1 ubuntu ubuntu    0 Aug 25 17:57 metrics.json

Container is executed in privileged mode.

Steps to reproduce

My k8s resources:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  namespace: monitoring
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:      
    processors:
      batch:
      memory_limiter:
        # 80% of maximum memory up to 2G
        limit_mib: 1500
        # 25% of limit up to 2G
        spike_limit_mib: 512
        check_interval: 5s
    extensions:
      zpages: {}
      memory_ballast:
        # Memory Ballast size should be max 1/3 to 1/2 of memory.
        size_mib: 683
    exporters:
      prometheus:
        endpoint: "0.0.0.0:9001"
      file:
        path: /data/metrics.json
    service:
      extensions: [zpages, memory_ballast]
      pipelines:
        metrics/1:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheus, file]
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: monitoring
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: otel-collector
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 1.0.0
spec:
  ports:
  - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
    port: 4318
    protocol: TCP
    targetPort: 4318
  - name: metrics # Default endpoint for querying metrics.
    port: 8888
    targetPort: 8888
  - name: metricsprom # Default endpoint for querying metrics.
    port: 9001
    targetPort: 9001
  selector:
    app.kubernetes.io/name: otel-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: otel-collector
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 1.0.0
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: otel-collector
      app.kubernetes.io/part-of: kube-prometheus
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1 #TODO - adjust this to your own requirements
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: otel-collector
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 1.0.0
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000
      containers:
      - command:
          - "/otelcol"
          - "--config=/conf/otel-collector-config.yaml"
        securityContext:
          privileged: true
        image: otel/opentelemetry-collector:0.58.0
        name: otel-collector
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        ports:
        - containerPort: 55679 # Default endpoint for ZPages.
        - containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
        - containerPort: 14250 # Default endpoint for Jaeger gRPC receiver.
        - containerPort: 14268 # Default endpoint for Jaeger HTTP receiver.
        - containerPort: 9411 # Default endpoint for Zipkin receiver.
        - containerPort: 8888  # Default endpoint for querying metrics.
        - containerPort: 9001
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /conf
        - name: data
          mountPath: /data/metrics.json
      volumes:
        - configMap:
            name: otel-collector-conf
            items:
              - key: otel-collector-config
                path: otel-collector-config.yaml
          name: otel-collector-config-vol
        - name: data
          hostPath:
            path: /home/ubuntu/data/metrics.json
            type: FileOrCreate

What did you expect to see? The pod should have access to metrics.json file

What did you see instead? The pod does not have access to metrics.json

[evan-bradley]

@sfc-gh-aivanou Can you run this same pod configuration with an image like busybox and do some interactive testing to ensure the container has the correct permissions to access the file? Nothing jumps out at me as obviously wrong, but the setup here is slightly complex.

[atingchen]

Hi @sfc-gh-aivanou . Did you solve this problem?

[sfc-gh-aivanou]

oh, I actually stopped using it, I am just using logging instead. Will try to debug it further tomorrow, and report back.

[ghost]

I ran into a similar permission issue as well. One issue worth noting is that the official otel collector docker container is built using FROM scratch, which means that there's not a writable directory by default. This means that you either need to create a custom container or you need to create a writable volume to store the file(s) that are exported. @atingchen it might make sense to add a warning to the README about that. If you're interested, I can put up a PR tomorrow

[atingchen]

I ran into a similar permission issue as well. One issue worth noting is that the official otel collector docker container is built using FROM scratch, which means that there's not a writable directory by default. This means that you either need to create a custom container or you need to create a writable volume to store the file(s) that are exported. @atingchen it might make sense to add a warning to the README about that. If you're interested, I can put up a PR tomorrow

I am glad to see a PR.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[fatsheep9146]

@ghost Any progress?

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.