Bug/crowdsec: file datasource doesn't support multiple glob

[FinweVI]

Describe the bug I have a central syslog server with one folder per host. To read the log files with crowdsec for every host, with the file datasource, I used the following configuration:

---
source: file
filenames:
 - /data/logs/hosts/*/*.log
 - /data/logs/nginx/*/access.log
labels:
 type: syslog

However the multi-glob pattern doesn't seem to work:

time="11-01-2022 12:48:54" level=info msg="Force add watch on /data/logs/hosts/*" type=file
time="11-01-2022 12:48:54" level=error msg="Could not create watch on directory /data/logs/hosts/* : no such file or directory" type=file
time="11-01-2022 12:48:54" level=info msg="Force add watch on /data/logs/nginx/*" type=file
time="11-01-2022 12:48:54" level=error msg="Could not create watch on directory /data/logs/nginx/* : no such file or directory" type=file

Expected behavior It would be nice to either have support for multi-glob pattern or for crowdsec to reject the configuration file.

Technical Information (please complete the following information):

• OS: Ubuntu Bionic
• Version: 1.2.3

[mmetc]

Multiple glob works for me here, version 1.2.3 under freebsd: https://github.com/crowdsecurity/opnsense-plugin-crowdsec/blob/main/src/etc/crowdsec/acquis.d/opnsense.yaml

filenames:
 - /var/log/*/*.log
labels:
  type: syslog

time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/configd/configd_20220127.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/configd/configd_20220128.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/configd/configd_20220129.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/configd/configd_20220130.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/configd/configd_20220131.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/configd/configd_20220201.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/crowdsec/crowdsec-firewall-bouncer.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/crowdsec/crowdsec.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/crowdsec/crowdsec_api.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220121.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220122.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220123.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220124.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220125.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220126.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220127.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220128.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220129.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220130.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220131.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/filter/filter_20220201.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/lighttpd/lighttpd_20220127.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/pkg/pkg_20220123.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/pkg/pkg_20220127.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/pkg/pkg_20220131.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/portalauth/portalauth_20220127.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/system/system_20220121.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/system/system_20220127.log to datasources" type=file
time="01-02-2022 13:00:04" level=info msg="Adding file /var/log/system/system_20220201.log to datasources" type=file

and I just tried under Ubuntu focal, I could not replicate the issue. I notice that your nginx line does not have a double glob. Could it be something else?

[FinweVI]

Hello,

thanks for looking into it. It seems like when I disable force_inotify, the glob pattern is working.

Any way to make both of them working together?

[mmetc]

Ok, I think I see it. Do you have messages like "Could not create watch on directory" in /var/log/crowdsec.log ?

[FinweVI]

With force_inotify enabled, yep

time="11-01-2022 12:48:54" level=info msg="Force add watch on /data/logs/hosts/*" type=file
time="11-01-2022 12:48:54" level=error msg="Could not create watch on directory /data/logs/hosts/* : no such file or directory" type=file
time="11-01-2022 12:48:54" level=info msg="Force add watch on /data/logs/nginx/*" type=file
time="11-01-2022 12:48:54" level=error msg="Could not create watch on directory /data/logs/nginx/* : no such file or directory" type=file

[mmetc]

Thanks. I have no fix yet but it should land in the next version.

[LaurenceJJones]

Link to sub issue I raised in #2117