Cannot ship remote IIS log files

[mrdavidlaing]

When the file is copied locally it works:

    file {
        type => "iis_tradingapi"
        path => "C:\Dev\temp\u_ex130723.log"
        stat_interval => 1
        add_field => [ "service", "TradingApi_IIS" ]
        add_field => [ "@real_source_host", "pkh-ppe-web24" ]
        debug => true
        start_position => "beginning"
    }

When accessed remotely it doesn't ship anything

    file {
        type => "iis_tradingapi"
        path => "\\pkh-ppe-web24\c$\inetpub\logs\LogFiles\W3SVC1\u_ex130723.log"
        stat_interval => 1
        add_field => [ "service", "TradingApi_IIS" ]
        add_field => [ "@real_source_host", "pkh-ppe-web24" ]
        debug => true
        start_position => "beginning"
    }

Mapping the remote folder doesn't work either

    file {
        type => "iis_tradingapi"
        path => "X:\inetpub\logs\LogFiles\W3SVC1\u_ex130723.log"
        stat_interval => 1
        add_field => [ "service", "TradingApi_IIS" ]
        add_field => [ "@real_source_host", "pkh-ppe-web24" ]
        debug => true
        start_position => "beginning"
    }

Grr.

[dpb587]

Do you get any error output?

[mrdavidlaing]

For all the above settings, no, no errors.

However when acess the logs via a custom file share - like the other working logs are using - with this config:

    file {
        type => "iis_tradingapi"
        path => "\\pkh-ppe-web24\W3SVC1\u_ex130723.log"
        stat_interval => 1
        add_field => [ "service", "TradingApi_IIS" ]
        add_field => [ "@real_source_host", "pkh-ppe-web24" ]
        #debug => true
        #start_position => "beginning"
    }

I get the following errors:

{
:   timestamp=>"2013-07-23T18:36:52.305000+0100   ", :message=>"   Input thread exception",
:   plugin=><LogStash::   Inputs::File type=>"iis_tradingapi",
   path=>   [
      "\\\\pkh-ppe-web24\\W3SVC1\\u_ex130723.log"
   ],
   add_field=>   {
      "service"      =>"TradingApi_IIS",
      "@real_source_host"      =>"pkh-ppe-web24"
   },
   charset=>"UTF-8",
   start_position=>"end">,
:   exception=>#<SystemCallError:Unknown error - Unknown Error (20047) - //pkh-ppe-web24/W3SVC1/u_ex130723.log>,
:backtrace=>   [
      "org/jruby/RubyFileStat.java:338:in `initialize'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:139:in `_discover_file'",
      "org/jruby/RubyArray.java:1613:in `each'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:122:in `_discover_file'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:93:in `discover'",
      "org/jruby/RubyArray.java:1613:in `each'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:92:in `discover'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:106:in `subscribe'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/tail.rb:64:in `subscribe'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/inputs/file.rb:122:in `run'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:761:in `run_input'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:407:in `start_input'"
   ],
:   level=>:warn
}

Not sure if this gives any clues...

[dpb587]

Possibly relevant JRuby Bug: http://jira.codehaus.org/browse/JRUBY-4820

[mrdavidlaing]

Just got another error which seems related (I think)

{
:   timestamp=>"2013-07-23T19:10:47.808000+0100   ", :message=>"   Input thread exception",
:   plugin=><LogStash::   Inputs::File type=>"ci_log4net",
   path=>   [
      "\\\\pkh-ppe-web24\\Logs\\TradingApi.log20130723.log"
   ],
   charset=>"Windows-1252",
   add_field=>   {
      "service"      =>"TradingApi",
      "@real_source_host"      =>"pkh-ppe-web24"
   },
   start_position=>"end">,
:   exception=>#<Errno::   EIO:Input/output error - //pkh-ppe-web24/Logs/TradingApi.log20130723.log>,
:backtrace=>   [
      "org/jruby/RubyFileStat.java:338:in `initialize'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:139:in `_discover_file'",
      "org/jruby/RubyArray.java:1613:in `each'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:122:in `_discover_file'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/watch.rb:34:in `watch'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/filewatch/tail.rb:58:in `tail'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/inputs/file.rb:119:in `run'",
      "org/jruby/RubyArray.java:1613:in `each'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/inputs/file.rb:119:in `run'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:761:in `run_input'",
      "file:/C:/Dev/cityindex/logsearch-on-aws/ppe-cluster/bin/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:407:in `start_input'"
   ],
:   level=>:warn
}

[mrdavidlaing]

Java version is:

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

[mrdavidlaing]

Any idea how to figure our which version of JRuby is bundled inside logstash-1.1.13-flatjar.jar ?

[dpb587]

https://github.com/logstash/logstash/blob/v1.1.13/Makefile#L5

JRUBY_VERSION=1.7.3

[sopel]

@dpb587 - great find, sounds like that could be the culprit indeed.

I'm also irritated about ELASTICSEARCH_VERSION=0.20.6 (has been downgraded from 0.90.0, see https://github.com/logstash/logstash/commit/05be9dcdab303da7d25e78d7473311594f30cbbd), whereas this cluster is using the current version 0.90.1 throughout - do you think that could yield problems as well?

Both have apparently been addressed in master already, which now uses JRuby 1.7.4 and at least Elasticsearch 0.90.0 again, any hint on when a new version might be ready?

[dpb587]

I don't believe 0.90.1 is causing any issues that we're seeing (either in this specific issue or elsewhere). The elasticsearch version primarily refers to the embedded version. The elasticsearch HTTP API is what we're using right now, and that is functionally consistent.

In terms of JRuby, the referenced commit is indeed new to 1.7.4; hopefully it is a fix. I can't find a real roadmap or timeline, but according to jira, it looks like the next planned release for logstash is 1.2.0. Jira suggests all issues in that 1.2.0 milestone are resolved, but I'm not sure if that means anything in terms of guessing a timeline.

Last time I tried compiling the flatjar I ran into problems. I can give it another go tomorrow so we can see if the Windows bug is indeed fixed in addition to testing the other 1.2.0-pre changes.

[le-bott]

Lets not spend too much time trying to compile pre version of logstash, since we know we will get the same result by waiting a few weeks.

As a stopgap I can pipe a simple tail -f into a local file in a different process and the use logstash to ship the local copy to Redis.

In the long run I'd like to implement a solution where we ship logs directly from IIS into Redis. First writing to file, and having a separate machine doing a network file watch smells of excessive complexity to me. I'm not sure that running a Logstash shipper on the IIS machine is going to work, since it requires a JVM and seems quite memory hungry,

Are there any lightweight IIS/Windows native alternatives?

On Tuesday, July 23, 2013, Danny Berger wrote:

I don't believe 0.90.1 is causing any issues that we're seeing (either in this specific issue or elsewhere). The elasticsearch version primarily refers to the embedded version. The elasticsearch HTTP API is what we're using right now, and that is functionally consistent.

In terms of JRuby, the referenced commit is indeed new to 1.7.4; hopefully it is a fix. I can't find a real roadmap or timeline, but according to jira, it looks like the next planned release for logstash is 1.2.0. Jira suggests all issues in that 1.2.0 milestone are resolved, but I'm not sure if that means anything in terms of guessing a timeline.

Last time I tried compiling the flatjar I ran into problems. I can give it another go tomorrow so we can see if the Windows bug is indeed fixed in addition to testing the other 1.2.0-pre changes.

— Reply to this email directly or view it on GitHubhttps://github.com/cityindex/logsearch-development-flow/issues/71#issuecomment-21448560 .

David Laing Open source @ City Index - github.com/cityindex http://davidlaing.com Twitter: @davidlaing

[mrdavidlaing]

How embarassing. I was running the shipping service as a user without permissions to access those file shares :-#

https://github.com/cityindex/logsearch-on-aws/commit/f5f76f9458df38e6702b8ab12ce917af6da85e2f