resuming harvester at end of log but doen not show up in Kibana.

[mr-oz-09]

I have log-courier shipping logs from 4 different servers. I have them set up like so

{
  "general": {
    "admin enabled": true,
    "admin listen address": "tcp:127.0.0.1:1234",
    "log level": "debug",
    "log file": "/var/log/log-courier.log"

  },

  "network": {
    "servers": [ "logstashserver:9023" ],
    "timeout": 10,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
  "files": [
    { 
      "paths": [ "/var/logs/app1/*.log", "/var/log/app1/notice.log.*", "/var/log/app1/back2015????.log"],
      "fields": { "type": "app1" }
    }
   ]
}

so one server is shipping everything as expected but the other ones are not. Any reason why? They are only shipping logs that go a week back.

[driskell]

They are only shipping logs that go a week back

Do you mean all four, and do you mean they are configured for that? Or do you mean you are expecting them to ship more than that?

Just trying to understand if the issue is 3/4 are shipping nothing or if 3/4 are only shipping recent files.

If you can provide logs for when it starts up that will assist greatly too.

[mr-oz-09]

I am expecting them to ship more than that. I have logs in there from last year.

16/03/31 13:50:18.766953 Started harvester at position 3725 (requested 3725): /var/log/app20160128-000.log 2016/03/31 13:50:18.803011 Connected to 10.110.90.38:9023 2016/03/31 13:50:18.803061 Send now open: Awaiting events for new payload

This is that status of the courier when started. The only thing is that the first line with the harvester is repeated for every log.

[driskell]

On the first run of Log Courier where it hasn't run before and has no previous state, it will scan all files and start shipping them from the end - so it will only ship new entries. On subsequent runs it remembers where it left off so it can resume where it left off.

Could this be the cause? If you need a brand new Log Courier to ship lots of old date you will need to configure dead time accordingly (as it will skip files older than certain age) and also run log-courier with the -from-beginning=true flag. This flag is only used on the first run of Log Courier where it has no previous state, to tell it to ship everything instead of starting from the end of all the files.

[mr-oz-09]

Why would this work as expected on one server but not the other 3. That's what i find weird. The log formats are all the same so I am grabbing them the same in each location. I would expect it to grab all the log files as it did in server 1.

[driskell]

I can only surmise that Log Courier had already shipped the old logs due to existing .log-courier file or that the modifications times were older so it was skipped due to dead time

If you could provide the entire log-courier logs to compare we might be able to find out. All harvesting decisions are logged and all decisions about resuming during startup are also logged. All the answers lie in those logs.

[driskell]

Did you manage to solve this? If you can provide fuller logs, from both a working server and a non-working server it'll help see what's happening and what's different. Without that it's hard to tell. But if both are using same Log Courier I suspect there is something different on the servers whether it be the .log-courier is there or the files are differing timestamps etc. and something about that is causing them to resume or start harvesting differently.

I'll close for now and reopen if you still having issues and would still like to investigate.