Unable to create index without LOGSENE_TOKEN value

[ishu52]

Hi,

I am using ES and Kibana on IBM cloud and using only sematext daemon set. Using this I could see index is getting created with name same as the value of LOGSENE_TOKEN which is passed in daemon set.

Now, we wanted to have index based on envrionment name and getting created daily. In order to implement this, I had created a config map and pattern.yml file where-in I had mentioned the output type as

`apiVersion: v1 kind: ConfigMap metadata: name: sematext-agent-patterns data: patterns.yml: |-

Put your custom patterns below.

# Watch out on indentation to follow this comment!
# Global options

options:
  # print stats every 60 seconds 
  printStats: 60
  # don't write parsed logs to stdout
  suppress: false
  # Enable/disable GeoIP lookups
  # Startup of logagent might be slower, when downloading the GeoIP database
  geoipEnabled: false
  # Directory to store Logagent status and temporary files
  diskBufferDir: ./tmp

input:
  # a list of glob patterns to watch files to tail
  files:
      - '/var/log/**/*.log'
  # listen to udp syslog protocol  
  #syslog: 
  #  port: 514
  # listen to http to receive data from Heroku  log drains  
  #heroku: 
  #  port: 9999
  # listen to http to receive data from Cloud Foundry drains  
  #cloudFoundry:
  #  port: 8888

output:
  # index logs in Elasticsearch or Sematext Cloud
  elasticsearch: 
    module: elasticsearch
    # URL to Elasticearch server, defaults to Sematext Cloud logs receiver if not set
    url: http://52.116.16.156:31405

    # Proxy settings behind firewalls
    # httpProxy:  http://localProxy:port
    # httpsProxy: https://localHttpsProxy:port

    # default index to use, for all logs that don't match later in indices section
    # for Sematext Cloud use the Logs App Token here
    index: test123

    # specific index to use per logSource field of parsed logs
    # logSource is by default the file name of the log file
    # but it can be modified by JS transforms functions in the patterns.yml file

  # print parsed logs in YAML format to stdout (only if options.supress is set to false)    
stdout: pretty # use 'pretty' for pretty json and 'ldjson' for line delimited json (default)`

I had removed LOGSENE_TOKEN env variable from daemonset. I get error "No logs loaded since and LOGSENE_TOKEN is empty"

When I mention LOGSENE_TOKEN then it takes that value and create the index in ES.

How do i make ES pointing to patterns.yaml index.

[otisg]

Btw. there is also a https://groups.google.com/forum/#!forum/logagent which may be more suitable for questions.

[ishu52]

Btw. there is also a https://groups.google.com/forum/#!forum/logagent which may be more suitable for questions.

Thank you

[megastef]

As mentioned in the forum:

LOGS_TOKEN (or in the past we used LOGSENE_TOKEN) is a mondatory parameter for sematext/logagent. The LOGS_TOKEN is used as a default index name in Sematext Cloud in case your LOGS don’t go to any custom index by log routing. Note Logagent is a docker image optimized to work with Sematext Cloud.

You can define in YML file any function to generate the index name - see some examples inthe JS function below:

output: 
  elasticsearch: 
    module: elasticsearch
    myCustomIndexName: myIndexName
    url:  http://IP-ADDRESS:PORT
  index: !js/function function doSomeCustomeIndexNameCreation (context, conf, data) { 
     return conf.myCustomIndexName || data.logSource || context.source || anyThingElse 
  }