sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint
https://sematext.com/logagent
Apache License 2.0
389 stars 79 forks source link

Log routing to multiple destinations is not working as expected on Linux #230

Closed adnanrahic closed 4 years ago

adnanrahic commented 4 years ago

The problem is that even though the indices property is there and the token is defined and the names of the files are there, all the data is still sent to the main index. I don't see anything wrong here, could you look at this please?

# Global options
options:
  # print stats every 60 seconds
  printStats: 60
  # don't write parsed logs to stdout
  suppress: true
  # Enable/disable GeoIP lookups
  # Startup of logagent might be slower, when downloading the GeoIP database
  geoipEnabled: false
  # Directory to store Logagent status and temporary files
  # this is equals to LOGS_TMP_DIR env variable
  diskBufferDir: /tmp/sematext-logagent

input:
  # a list of glob patterns to watch files to tail
  files:
    - '/var/log/**/*.log'
    - '/var/www/web/foobar/storage/logs/*'

output:
  # index logs in Elasticsearch or Logsene
  elasticsearch:
    module: elasticsearch
    url: https://logsene-receiver.eu.sematext.com
    # default index (Logs token) to use:
    index: <main_token>

    indices:
        <secondary_token>:
          - updateWS\.log
          - updateWSBinance\.log
          - pusher_events.*\.log
          - laravel\.log
          - laravel-.*\.log
          - horizon\.log
          - event_listener_job.*\.log
          - price_updated-.*\.log
adnanrahic commented 4 years ago

This issue ended up being a misconfiguration.

The log routing works if you add all the tokens under the indices field.

output:
  elasticsearch:
    module: elasticsearch
    url: https://logsene-receiver.eu.sematext.com

    indices:
        <primary_token>:
          - auth.log
          - access.log
        <secondary_token>:
          - updateWS\.log
          - updateWSBinance\.log
          - pusher_events.*\.log
          - laravel\.log
          - laravel-.*\.log
          - horizon\.log
          - event_listener_job.*\.log
          - price_updated-.*\.log