Issue with missing dates stamps (logs arrive in logstash-1970-01-01 index)

[leehambley]

I'm running a simple config:

<source>
  @type http
  port 8686
  bind 127.0.0.1
  body_size_limit 32m
  keepalive_timeout 10s
</source>

<source>
  @type systemd
  tag systemd
  path /var/log/journal
  read_from_head true
  <storage>
    @type local
    persistent true
    path systemd.pos
  </storage>
  <entry>
    field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<filter **>
  @type parser
  key_name log
  reserve_data true
  emit_invalid_record_to_error false
  <parse>
    @type json
  </parse>
</filter>

<match **>
  @type stdout
</match>

<match **>
  @type elasticsearch

  host elasticsearch.metrics-01.prod.harrow.io
  password for-side-experiment-capital
  port 443
  scheme https
  user harrow

  logstash_format true
  include_tag_key true
  reconnect_on_error true
  request_timeout 60s
  type_name fluentd
  @log_level debug
</match>

I'm on Ubuntu, so /var/log/journal didn't exist, and I had to create it (when it was immediately picked up)

And seeing issues that the elasticsearch index being created is named logstash-1970-01-01 not hugely surprising given the records according to the out_stdout plugin:

1970-01-01 00:33:37.000000000 +0000 systemd: {"log":"{\"time\":\"2017-11-03T19:00:48Z\",\"level\":\"info\",\"harrow\":\"projector\",\"messag......e":"alcohol","systemd_invocation_id":"b813077e938347a990eba809f6439f7c","level":"info","harrow":"projector","message":"seen=315000"}
1970-01-01 00:33:37.000000000 +0000 systemd: {"log":"{\"time\":\"2017-11-03T19:00:48Z\",\"leve.......09f6439f7c","level":"info","harrow":"projector","message":"operation.started=261086"}

I may be missing a piece of the mental model, but I'd have expected this plugin to have taken one of the __TIMESTAMP_REALTIME or __TIMESTAMP_MONOTONIC.

[errm]

Firstly it seems the clock on your system needs to be set correctly...

If for some reason you wanted to map __TIMESTAMP_REALTIME or __TIMESTAMP_MONOTONIC to something different you would need to configure that with <entry> field_map

This plugin doesn't do anything clever to alter the structure of the entries being read from systemd, you need to configure that correctly if that is what your use-case requires.