search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
67 stars 3.5k forks source link

NotImplementedError: block device detection unsupported or native support failed to load #3127

Closed mattstibbs closed 7 years ago

mattstibbs commented 9 years ago

Issue

I am encountering the following error when trying to use the file input to watch /var/log/syslog and /var/log/auth.log

I have run the following on the log files:

setfacl -m u:logstash:r /var/log/{syslog,auth.log}

The following exception stack trace is from /var/log/logstash/logstash.err

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/tail.rb:230
   sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/tail.rb:203
        teardown at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.9/lib/logstash/inputs/file.rb:151
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:203
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:203
     start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:171

If I run interactively using sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --verbose

I get the same exception stack trace plus this:

The error reported is:
  Bad file descriptor - Bad file descriptor

I have a number of file inputs in the pipeline, but it seems only be this file input that causes a problem. Removing this file input allows it to start fine.

Environment Details

root@logstash1:/# lsb_release -d
Description:    Ubuntu 14.04.1 LTS
root@logstash1:/# /opt/logstash/bin/logstash -V
logstash 1.5.0-rc3
root@logstash1:/# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
mattstibbs commented 9 years ago

I have now significantly simplified the pipeline to try and narrow this problem down. Pipeline is now:

input {
  file {
    sincedb_path => [ "/var/tmp/log.sincedb" ]
    path => [ "/var/tmp/log.log" ]
    type => "syslog"
  }
}
output {
  stdout { codec => rubydebug }
}

I have also changed permissions on /var/tmp with:

sudo chmod 777 -R /var/tmp/

Running logstash with the following still generates the original error:

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --verbose

Stuck as to where to go next!

mattstibbs commented 9 years ago

Having started with a fresh install of Ubuntu, I am getting the same problem but am now noticing the following error intermittently as well as the reported error.

I can't find any pattern as to what causes which to appear, although it seems that if I only use RC2 I only get the below error (and not the block device error)

NotImplementedError: stat.st_dev unsupported or native support failed to load
       dev_major at org/jruby/RubyFileStat.java:188
  _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:150
            each at org/jruby/RubyArray.java:1613
  _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:132
           watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:38
            tail at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/tail.rb:68
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.6/lib/logstash/inputs/file.rb:133
            each at org/jruby/RubyArray.java:1613
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.6/lib/logstash/inputs/file.rb:133
     inputworker at /opt/logstash/lib/logstash/pipeline.rb:174
     start_input at /opt/logstash/lib/logstash/pipeline.rb:168

Also confirmed not an issue when reverting to logstash-1.4.2 using same config on same machine

suyograo commented 9 years ago

@mattstibbs I am unable to reproduce this :( It is weird because we have many folks hitting this issue -- https://github.com/elastic/logstash/issues/2962, https://github.com/elastic/logstash/issues/3033

ubuntu@ip-10-0-1-9:/opt/logstash$ lsb_release -d
Description:    Ubuntu 14.04.1 LTS
ubuntu@ip-10-0-1-9:/opt/logstash$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
ubuntu@ip-10-0-1-9:/opt/logstash$ bin/logstash version
logstash 1.5.0-rc3
ubuntu@ip-10-0-1-9:/opt/logstash$ bin/logstash -e 'input { file { path => "/var/log/syslog" start_position => "beginning" } }'
Logstash startup completed
{
       "message" => "Apr 30 07:00:49 ip-10-0-1-9 dhclient: DHCPREQUEST of 10.0.1.9 on eth0 to 10.0.1.1 port 67 (xid=0x54d2a924)",
      "@version" => "1",
    "@timestamp" => "2015-04-30T21:22:40.079Z",
          "host" => "ip-10-0-1-9",
          "path" => "/var/log/syslog"
}
mattstibbs commented 9 years ago

@suyograo :(

All my previous tests have been on digitalocean droplets. Have just followed below steps and am getting exactly same error:

  1. Download Ubuntu 14.04.2 Server iso from http://www.ubuntu.com/download/server
  2. Create standard Ubuntu install using VirtualBox (RAM 512Mb, HDD 16GB).
  3. During startup create a user as instructed and then log into machine as that user
  4. Run following commands in order:
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
reboot

wget http://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.0-rc3-1_all.deb
sudo dpkg -i logstash_1.5.0-rc3-1_all.deb
reboot

cd /opt/logstash
sudo bin/logstash -e 'input { file { path => "/var/log/syslog" start_position => "beginning" } }'
suyograo commented 9 years ago

@mattstibbs Thanks, I was able to reproduce -- we'll look into this. 

ubuntu@ip-10-0-1-10:/opt/logstash$ lsb_release -d
Description:    Ubuntu 14.04.2 LTS
ubuntu@ip-10-0-1-10:/opt/logstash$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
ubuntu@ip-10-0-1-10:/opt/logstash$ bin/logstash version
logstash 1.5.0-rc3
ubuntu@ip-10-0-1-10:/opt/logstash$
suyograo commented 9 years ago

@jsvd the device check was added in https://github.com/jordansissel/ruby-filewatch/pull/54. Can you take a look please?

jsvd commented 9 years ago

I tried using Oracle Java 7 or Java 8 from the repo @mattstibbs suggested in an ec2 ubuntu 14.04 instance:

$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
$ bin/logstash irb
irb(main):001:0> File.blockdev? "/"
NotImplementedError: block device detection unsupported or native support failed to load
    from org/jruby/RubyFileTest.java:67:in `blockdev?'
    from (irb):1:in `evaluate'
    from org/jruby/RubyKernel.java:1107:in `eval'
    from org/jruby/RubyKernel.java:1507:in `loop'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:77:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:131:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/task.rb:12:in `initialize'
irb(main):002:0>

Then tried stock openjdk 7:

$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
(reverse-i-search)`': ^C
$ bin/logstash irb
irb(main):001:0> File.blockdev? "/"
=> false

So it seems to be an issue with oracle jvm, looking deeper into this..

jsvd commented 9 years ago

Same happens for Stat#dev_minor

Oracle JDK 7

$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
$ bin/logstash irb
irb(main):001:0> File::Stat.new("/var/tmp/log").dev_minor
NotImplementedError: stat.st_dev unsupported or native support failed to load
    from org/jruby/RubyFileStat.java:193:in `dev_minor'
    from (irb):3:in `evaluate'
    from org/jruby/RubyKernel.java:1107:in `eval'
    from org/jruby/RubyKernel.java:1507:in `loop'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from org/jruby/RubyKernel.java:1270:in `catch'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:77:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/lib/logstash/runner.rb:131:in `run'
    from org/jruby/RubyProc.java:271:in `call'
    from /home/ubuntu/logstash-1.5.0-rc3/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/task.rb:12:in `initialize'

OpenJDK7

$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
$ bin/logstash irb
irb(main):001:0> File::Stat.new("/var/tmp/log").dev_minor
=> 51713
jsvd commented 9 years ago

@mattstibbs can you try installing the package "libc6-dev" (and its dependencies) in your ubuntu instance and check if the issue goes away?

suyograo commented 9 years ago

@jordansissel is it a viable option to have libc6-dev as a dependency for the deb/rpm packages?

jordansissel commented 9 years ago

@suyograo Yep! I"ll file a bug about it.

jordansissel commented 9 years ago

Do we know if this only affects Ubuntu? or Red Hat systems also?

jsvd commented 9 years ago

@jordansissel it affects only ubuntu if using Oracle jdk (not openjdk).

jordansissel commented 9 years ago

For further clarification, this is only known (for now) to reproduce with Ubuntu 14.04 and Oracle Java. OpenJDK's java packages seems unaffected by this bug.

The libc6-dev package requirement had me do some investigation:

Ultimately, this is the fix:

ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so

How did I find this? Here's how -

% strace -fo /tmp/jruby.failing -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'
NotImplementedError: block device detection unsupported or native support failed to load
% strace -fo /tmp/jruby.working -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'
false

Now we have two files jruby.failing and jruby.working which only contain file-related syscalls made by any thread (or subprocess) of java. Let's find the unique files:

% awk -F\" '{print $2}' /tmp/jruby.working | sort -u > /tmp/w
% awk -F\" '{print $2}' /tmp/jruby.failing | sort -u > /tmp/f
% diff /tmp/{f,w}
[noise trimmed for clarity]
< /lib/libcrypt.so
< /lib/x86_64-linux-gnu/libcrypt.so
< /usr/lib/libcrypt.so

So, libcrypt, eh?

# Observe the working strace
root@asdf:~/opt/logstash# grep libcrypt /tmp/jruby.working
13764 open("/root/jdk1.8.0_45/bin/../lib/amd64/jli/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
13764 open("/root/jdk1.8.0_45/bin/../lib/amd64/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
13764 open("/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
13764 open("/usr/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = 12

# observe the failing strace
root@asdf:~/opt/logstash# grep libcrypt /tmp/jruby.failing
11222 open("/root/jdk1.8.0_45/bin/../lib/amd64/jli/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/root/jdk1.8.0_45/bin/../lib/amd64/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/usr/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/lib/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
11222 open("/usr/lib/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

libcrypt.so wasn't found! Turns out Ubuntu 14.04's libc6 package doesn't have such a file:

root@asdf:~/opt/logstash# dpkg -L libc6 | grep libcrypt
/lib/x86_64-linux-gnu/libcrypt-2.19.so
/lib/x86_64-linux-gnu/libcrypt.so.1

But the libc6-dev package does:

root@asdf:~/opt/logstash# dpkg -L libc6-dev | grep libcrypt
/usr/lib/x86_64-linux-gnu/libcrypt.a
/usr/lib/x86_64-linux-gnu/libcrypt.so

Because libc6-dev transitively depends on gcc, and some users may not want gcc installed as a dependency of logstash, Ubuntu 14.04 users can patch this by doing this symlink, I think:

ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so
jordansissel commented 9 years ago

Using CentOS 7, Oracle JDK 1.8.0_45, JRuby 9.0.0.0.pre2, I cannot reproduce. File.blockdev?("/") is OK on this environment.

mattstibbs commented 9 years ago

@jsvd Sorry for the delay - am on hols in Singapore!

Have just tested after installing libc6-dev and it works fine - as you probably expected!

KyleCartmell commented 9 years ago

Encountered this exact error after stepping up from 1.4.2 to 1.5.0 RC4 under Solaris 10.5. Problem was an old JRE; upgraded from Oracle Java 6 to 8 and logs are once again being stashed.

pako-pl commented 9 years ago

I had the same issue on Debian 7.8 Wheezy, but adding the symlink solved the problem. Thanks.

atman9 commented 9 years ago

I am seeing this as well on RHEL6, java 1.7

java -version
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)
./logstash -V
logstash 1.5.0
SMMAN commented 9 years ago

Hi, I confirm we are having the same issue with: java version "1.7.0_79" OpenJDK Runtime Environment (rhel-2.5.5.3.el6_6-x86_64 u79-b14) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

CentOS release 6.6 (Final)

logstash-1.5.0

What about this news: http://jruby.org/2015/05/05/jruby-1-7-20.html

Github Issues resolved for 1.7.20:

Could you guys not just use this new JRuby version? greetings

mjadas commented 9 years ago

And I can confirm the error on Solaris 10 on SPARC, logstash 1.5.0, Oracle JDK 8u45. Oddly, it was shipping logs just fine for a week or so, but this error occurred when I restarted it.

mvonthron commented 9 years ago

Reproduced on Debian 8.1 64b + Java 1.8.0_45

The symlink trick or installing libc6-dev solves the problem. Thanks @jordansissel

naisanza commented 9 years ago

@jordansissel Awesome troubleshooting! I can confirm your soft link fixed this issue on Ubuntu 14.10 as well.

hkhail commented 9 years ago

I have the same error on Solaris 10 on SPARC, Logstash 1.5.1 & Oracle JDK 7u71

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:231
   sincedb_write at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:204
        teardown at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.10/lib/logstash/inputs/file.rb:151
     inputworker at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202
     start_input at /opt/logstash/1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:170
srijiths commented 9 years ago

have the same error in ubuntu NotImplementedError: stat.st_gid unsupported or native support failed to load gid at org/jruby/RubyFileStat.java:247 atomic_write at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/helper.rb:45 _sincedb_write at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:234 sincedb_write at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb:204 teardown at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.10/lib/logstash/inputs/file.rb:151 inputworker at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202 synchronize at org/jruby/ext/thread/Mutex.java:149 inputworker at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:202 start_input at /mnt/ebs/IoT/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:170

atman9 commented 9 years ago

Still seeing this error on LS 1.5.2 even on OpenJDK and Oracle 1.8:

java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
 cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

The error:

Sending logstash logs to /db/dba/logstash/logstash/logstash.log.
NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/helper.rb:67
  _sincedb_write at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:233
   sincedb_write at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:206
        teardown at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.0/lib/logstash/inputs/file.rb:152
     inputworker at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202
     start_input at /db/dba/logstash/logstash-1.5.2/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:170

Anyone hit this on Redhat and solve it with a symlink? I am seeing a libcrypt.so in /usr/lib64:

 ls -l /usr/lib64/libcrypt.*
lrwxrwxrwx. 1 root root 25 Jul  8 11:01 /usr/lib64/libcrypt.so -> ../../lib64/libcrypt.so.1
ebuildy commented 9 years ago

I can confirm the symlink fix issue on Ubuntu 14.04 with Oracle Java 1.8!

Thanks,

atman9 commented 9 years ago

LS 1.5.2 works on RHEL6 on openJDK 1.8 with no linking necessary. I had a pathing issue that was sourcing the wrong version of Java.

Ciciloveyou commented 9 years ago

@jordansissel hello jordan, i have the same issue with Red Hat 5.9, logstash version 1.5.2 and java 1.7.1_71, if can you help to fix this issue? thank you

rlwmmw commented 9 years ago

Thanks, Jordan. The symlink patch appears to have solved my issue as well. For posterity. I'm including my errors as well.

Fix: ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so

Ubuntu 14.04.3 LTS logstash 1.5.2 java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

stdout Error:

NotImplementedError: block device detection unsupported or native support failed to load blockdev? at org/jruby/RubyFileTest.java:67 device? at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/helper.rb:67 _sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:233 sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.4/lib/filewatch/tail.rb:206 teardown at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.0/lib/logstash/inputs/file.rb:152 inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202 synchronize at org/jruby/ext/thread/Mutex.java:149 inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:202 start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:170

mjadas commented 9 years ago

Appears it was a library issue on Solaris10/SPARC as well. Putting the sparcv9 lib dir at the head of the library path fixed it for me:

export LD_LIBRARY_PATH=/usr/lib/sparcv9/:$LD_LIBRARY_PATH

scottnonnenberg commented 9 years ago

This was a nasty surprise when attempting to upgrade from 1.4 to 1.5 on Ubuntu 14.04 with Oracle Java 1.7.0_76. Reverting back to 1.4 for now. Maybe the intro docs should be updated to say that the Oracle JVM is no longer recommended?

jordansissel commented 9 years ago

Oracle and OpenJDK JVM are both fine. Maybe this is a bug, and not a feature?

On Sunday, September 13, 2015, Scott Nonnenberg notifications@github.com wrote:

This was a nasty surprise when attempting to upgrade from 1.4 to 1.5 on Ubuntu 14.04 with Oracle Java 1.7.0_76. Reverting back to 1.4 for now. Maybe the intro docs https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html should be updated to say that the Oracle JVM is no longer recommended?

— Reply to this email directly or view it on GitHub https://github.com/elastic/logstash/issues/3127#issuecomment-139932387.

scottnonnenberg commented 9 years ago

jordansissel: You must be referring to the fact that it's some sort of symlink issue, and not a problem with the JVM itself. Fair enough. I'll just be more explicit with my request. I'd like to see a page, linked to from the getting started docs, which talks about the known issues on the various platforms. Or, better, an update to the apt package to provide the symlink on ubuntu/debian.

jonatanblue commented 9 years ago

Fix in https://github.com/elastic/logstash/issues/3127#issuecomment-101068714 helped me. My system: OS: Ubuntu 14.04 Kernel: 3.13.0-63-generic Logstash: 1.5.4-1 Java: 1.8.0_60

montaropdf commented 9 years ago

Hello,

I encounter a similar problem on Fedora 21. In my case, it seems to be the filter that causes problems and not the input.

My system: OS: Fedora 21 x86_64 Kernel: 4.1.7-100.fc21 Logstash: 1.5.4 Java: openjdk 1.8.0_60

Executing and looking at strace, I see that it find libcrypt.so. Although, there many other files that are not found.

As there are a lot of entries, If you can tell me what I should look for I can post it.

tobias-urdin commented 9 years ago

If anybody is still having issues with this, this only happened to me when using the file input plugin. I'm running Ubuntu 14.04 x86_64 with Oracle Java JDK and Logstash 1.5.4-1

Installing libc6-dev package resolved the issue as it provides the /usr/lib/x86_64-linux-gnu/libcrypt.so library. apt-get install libc6-dev

montaropdf commented 9 years ago

Hi,

Any more advice to fix that issue when using OpenJDK?

I perform further investigation and it seems that the problem is using the file plugins and the grok filter that produce the error.

If I use a filter like kv, I have no error, but nothing is processed, see below:

config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@path = ["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@type = "mep_access" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@start_position = "beginning" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@debug = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"kv", :path=>"logstash/filters/kv", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
config LogStash::Filters::KV/@field_split = " " {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@value_split = ":" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@prefix = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@source = "message" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@include_keys = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@exclude_keys = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@default_keys = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Filters::KV/@allow_duplicate_values = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"rubydebug", :path=>"logstash/codecs/rubydebug", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
config LogStash::Codecs::RubyDebug/@metadata = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug metadata=>false> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
config LogStash::Outputs::Stdout/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
Registering file input {:path=>["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"77", :method=>"register"}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_4a293cebe3d2635d1c8dca2cc68d13a7", :path=>["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"120", :method=>"register"}
_sincedb_open: /root/.sincedb_4a293cebe3d2635d1c8dca2cc68d13a7: No such file or directory - /root/.sincedb_4a293cebe3d2635d1c8dca2cc68d13a7 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"215", :method=>"_sincedb_open"}
_discover_file_glob: /home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log: glob is: ["/home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
Pipeline started {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"87", :method=>"run"}
_discover_file: /home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log: new: /home/reveatwork/work/promotic/client/ewbs/tmp/eventmep-REQ060793-20151027.log (exclude is []) {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"141", :method=>"_discover_file"}
Logstash startup completed
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"204", :method=>"sincedb_write"}
Plugin is finished {:plugin=><LogStash::Outputs::Stdout codec=><LogStash::Codecs::RubyDebug metadata=>false>, workers=>1>, :level=>:info, :file=>"logstash/plugin.rb", :line=>"61", :method=>"finished"}
Pipeline shutdown complete. {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"101", :method=>"run"}
Logstash shutdown completed
sebadiaz commented 8 years ago

libc6-dev solve the problem too on logstash 2.1.0

ghost commented 8 years ago

Hi, My system Centos 7, java-openjdk 1.7 or 1.8, logstash 2.1.0 => same error

Thanks to jordansissel post i did following things:

trace -fo /tmp/jruby.failing -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'

reading jruby.failing ... libcrypt.so was found! but file crypt wasn't found! :(

I did symlink and everything is ok now

ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/crypt

Thank you boys

montaropdf commented 8 years ago

I apply the same fix as @neradpeter , and now the file plugin is working great.

My system: Fedora 21 java-openjdk 1.8 logstash 1.5.5

A question:

As this fix will break next time libcrypto.so is updated, who should fix its package/application? A. The maintainer of logstash or its RPM package, to better discover the presence of libcrypto? B. The maintainer of fedora/centos/rh/... to add the link to /usr/lib64/crypt?

Thanks.

evidal commented 8 years ago

Hi, I have the same problem on Red Hat Enterprise Linux Server release 6.7 (Santiago) I've tried with logstash 1.5.5 and 2.0.0 and with

Everytime, I've the same error

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:232
   sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:205
        teardown at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.1/lib/logstash/inputs/file.rb:157
     inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203
     start_input at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:171

Finally I've made the following link and it works ?!

ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/crypt
montaropdf commented 8 years ago

@Eric Vidal, have you tried the solution from neradpeter?

On Wed, Dec 2, 2015 at 12:10 PM, Eric Vidal notifications@github.com wrote:

Hi, I have the same problem on Red Hat Enterprise Linux Server release 6.7 (Santiago) I've tried with logstash 1.5.5 and 2.0.0 and with

  • jdk1.8.0_65, jdk1.8.0_60 and jdk1.8.0_51
  • openjdk jre-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64

Everytime, I've the same error

NotImplementedError: block device detection unsupported or native support failed to load blockdev? at org/jruby/RubyFileTest.java:67 device? at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/helper.rb:67 _sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:232 sincedb_write at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/filewatch-0.6.6/lib/filewatch/tail.rb:205 teardown at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.1/lib/logstash/inputs/file.rb:157 inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203 synchronize at org/jruby/ext/thread/Mutex.java:149 inputworker at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:203 start_input at /opt/newstatpoc/logstash-1.5.5/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.5-java/lib/logstash/pipeline.rb:171

— Reply to this email directly or view it on GitHub https://github.com/elastic/logstash/issues/3127#issuecomment-161263344.

blubbi321 commented 8 years ago

Also have this issue with

The fix with the symlink does not seem to help. I am able to reproduce the steps from @jordansissel but when it comes to

grep libcrypt /tmp/jruby.working

i get 46437 open("/usr/lib/jvm/java-7-oracle/bin/../lib/amd64/jli/libcrypt.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 46437 open("/usr/lib/x86_64-linux-gnu/libcrypt.so", O_RDONLY|O_CLOEXEC) = 9

any idea what could be wrong with my system?

adiworkoholic commented 8 years ago

Environment

$ lsb_release -d
Description:    Ubuntu 14.04.2 LTS
$ /opt/logstash/bin/logstash -V
logstash 2.2.0
$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

I also have the symlink as suggested lrwxrwxrwx 1 root root 35 Feb 25 2015 libcrypt.so -> /lib/x86_64-linux-gnu/libcrypt.so.1

Still see this problem. otImplementedError: stat.st_dev unsupported or native support failed to load dev_major at org/jruby/RubyFileStat.java:205 nix_inode at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:28 inode at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:32 inode at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:106 watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:96 _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:293 each at org/jruby/RubyArray.java:1613 each at org/jruby/RubyEnumerator.java:274 _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:284 watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:95 call at org/jruby/RubyProc.java:281 synchronized at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:337 synchronize at org/jruby/ext/thread/Mutex.java:149 synchronized at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:337 watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/watch.rb:92 tail at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.8.0/lib/filewatch/tail_base.rb:73 tail at /opt/logstash/vendor/jruby/lib/ruby/1.9/forwardable.rb:201 begin_tailing at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.0/lib/logstash/inputs/file.rb:284 each at org/jruby/RubyArray.java:1613 begin_tailing at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.0/lib/logstash/inputs/file.rb:284 run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.0/lib/logstash/inputs/file.rb:288 inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.0-java/lib/logstash/pipeline.rb:316 start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.0-java/lib/logstash/pipeline.rb:310 The error reported is: Bad file descriptor - Bad file descriptor

Any idea how this can be resolved ? This is a blocking issue since we need the file input and this happens when we use the file input.

adiworkoholic commented 8 years ago

Any update on the ETA for the fix ?

jconlon commented 8 years ago

Also still see the problem with this configuration:

Description: Ubuntu 14.04.3 LTS

logstash 2.2.2

java version "1.8.0_51" Java(TM) SE Runtime Environment (build 1.8.0_51-b16) Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

lrwxrwxrwx 1 root root 35 Feb 25 2015 /usr/lib/x86_64-linux-gnu/libcrypt.so -> /lib/x86_64-linux-gnu/libcrypt.so.1

jordansissel commented 8 years ago

For those of you seeing this problem on Ubuntu, my recommendation for a fix is still to install the libc6-dev package.

adiworkoholic commented 8 years ago

See the problem even after installing the libc6-dev package. :(

jhaeyaert commented 8 years ago

Same issue for me on :

I've made the following link and it works : sudo ln -s /lib64/libcrypt.so.1 /usr/lib64/libcrypt.so