having a built-in conf.d-style out of the box (container)

[lastsky]

Is your feature request related to a problem? Please describe. I'm always frustrated when config file doesn't support conf.d or directory.d style (example: https://github.com/axkibe/lsyncd/issues/441) and i need to code some 💩for automate deploy for 200+ servers

Describe the solution you'd like I want prometheus but for logs

/opt/promtail/promtail.conf:

<...>
scrape_configs:
  - job_name: stage
    entry_parser: raw
    env: stage
    file_sd_configs:
        - files:
            - '/opt/promtail/conf.d/stage/*.yml'

  - job_name: prod
    entry_parser: raw
    env: prod
    file_sd_configs:
        - files:
            - '/opt/promtail/conf.d/prod/*.yml'

/opt/promtail/conf.d/stage/db1.example.org.yml:

  - targets:
      - localhost
    labels:
      service: database
      job: postgres
      server: db1
      __path__: /var/log/remote/stage/db1.example.org.log

/opt/promtail/conf.d/stage/db2.example.org.yml:

  - targets:
      - localhost
    labels:
      service: database
      job: postgres
      server: db2
      __path__: /var/log/remote/stage/db2.example.org.log

/opt/promtail/conf.d/prod/web1.example.org.yml:

  - targets:
      - localhost
    labels:
      service: frontend
      job: nginx
      server: web1
      __path__: /var/log/remote/prod/web1.example.org.log

/opt/promtail/conf.d/prod/web2.example.org.yml:

  - targets:
      - localhost
    labels:
      service: frontend
      job: nginx
      server: web2
      __path__: /var/log/remote/prod/web2.example.org.log

Describe alternatives you've considered

• run multiple loki/promtail services with separate config one per env 💩
• run multiple loki/promtail servers one per env 💩

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[lasczki-da]

any updates?

[cyriltovena]

Definitely interesting but we haven’t got the time to get started on this

[bnewbold]

A related use-case is configuring multiple services on the same host, independently. That is, deploy promtail with generic configuration, and then have each service/application install it's own promtail configuration file specific to that service.

For example, if installing haproxy on a host (using ansible, puppet, whatever), that install process would create an /etc/promtail/conf.d/haproxy.conf file (which would point to the haproxy log files on disk) and possibly trigger a config reload of promtail. In this way, the configuration for each individual service can be handled by the service itself, instead of having to merge all the configs into a single config that gets deployed when promtail does.

[osandum]

@bnewbold installing with Ansible, could you possibly make use of the assemble module to achieve the same thing?

[frittentheke]

@cyriltovena is there still a chance you might work on this feature? It would be awesome to be able to assemble a larger Promtail pipeline out of multiple files.

Prometheus did add this for their scraping config recently: https://github.com/prometheus/prometheus/issues/8543

[cstyan]

We're currently reevaluating promtails position as a project within Grafana Labs. Internally we're actually using the Agent for both metrics and logs collection at this point. While we haven't made a formal decision yet, we expect in the near future that all new feature work will be done in the Agent's log collection pipelines rather than in Promtail.

Additionally, the Agent's Flow configuration model is component-based that supports combining multiple config files for the single agent binary.

[chaudum]

Closing this issue as we don't develop Promtail any further. It is superseded by Grafana Alloy. Please report the issue there.