[pkg/stanza] Issue with star pattern with mixed files and folders

[atoulme]

Component(s)

pkg/stanza

What happened?

Description

In a set up where we have a folder containing folders and files, the filelog receiver will fail to start scanning for files if told to scan for any content in the folder (/var/log/*).

Steps to Reproduce

Attached example as zip file.

Set up a docker-compose with the contents:

version: "3"
services:
  splunk-otel-collector:
    image: otel/opentelemetry-collector-contrib:0.67.0
    volumes:
    - ./config.yaml:/etc/otelcol-contrib/config.yaml
    - ./foo:/var/foo

Set up config.yaml file:

receivers:
    filelog:
      include: [ /var/foo/* ]

exporters:
  logging:

processors:
  batch:

extensions:
    health_check:
      endpoint: 0.0.0.0:13133
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679

service:
    extensions: [pprof, zpages, health_check]
    pipelines:
      logs:
        receivers: [filelog]
        processors: [batch]
        exporters: [logging]

Create a foo folder, containing a log file and a bar folder:

mkdir -p foo/bar
touch foo/my.log
touch foo/bar/other.log

Run the collector.

Expected Result

The collector starts.

Actual Result

The collector continuously outputs an error:

splunk-otel-collector_1  | 2022-12-13T01:28:20.189Z error   fileconsumer/file.go:178    Failed creating fingerprint {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "error": "reading fingerprint bytes: read /var/foo/bar: is a directory"}
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeReaders
splunk-otel-collector_1  |  github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:178
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume
splunk-otel-collector_1  |  github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:125
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).poll
splunk-otel-collector_1  |  github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:120
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).startPoller.func1
splunk-otel-collector_1  |  github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:101
splunk-otel-collector_1  | 2022-12-13T01:28:20.189Z error   fileconsumer/reader.go:78   Failed during scan  {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/foo/bar", "error": "scanner error: read /var/foo/bar: is a directory"}
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Reader).ReadToEnd
splunk-otel-collector_1  |  github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/reader.go:78
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
splunk-otel-collector_1  |  github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:137
splunk-otel-collector_1  | 2022-12-13T01:28:21.388Z error   fileconsumer/file.go:178    Failed creating fingerprint {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "error": "reading fingerprint bytes: read /var/foo/bar: is a directory"}

Collector version

0.67.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

See above

Log output

No response

Additional context

No response

[atoulme]

Pinging @djaglowski

[github-actions[bot]]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

If I'm understanding correctly, the doublestar library does not provide a way to match only files in such a case. We may have to post process the results to filter out directories. However, I've opened https://github.com/bmatcuk/doublestar/issues/79 in hopes that this can be supported natively in the library.