Logstash error ELK 5.4 - java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed

[yami12376]

Hey, i have two servers of applications (using the same version of application), and only on one of them i see this issue. Application is stored in Weblogic. They may differ slightly by configuration ear - but i did not find there anything that would have meaning in this matter.

[2017-06-26T11:16:29,552][DEBUG][logstash.inputs.log4j    ] Accepted connection {:client=>"X", :server=>"0.0.0.0:4561"}
[2017-06-26T11:16:29,572][DEBUG][logstash.inputs.log4j    ] Closing connection {:client=>"X", :exception=>java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed.}

not sure what is the cause ? ELK 5.4 - elsewhere working fine Configuration on both of them is the same.

What can be wrong in one of them ? I tailed the log - and both seems to send the same log - but one is ok - the second one is not ok.

I run logstash by these commands:

bin/logstash -f centralLogstash.conf   --debug --path.data usr/share/logstash/tmp

bin/logstash -f  tcp_agent_4561.conf   --debug --path.data usr/share/logstash/tmp2

centralLogstash.conf

input {
              stomp{
                password => "admin"
                user => "admin"
                host => "X"
                destination => "stomp"
                codec => "json"
        }
}

output {
        elasticsearch {
                hosts => "X"
                codec => "plain"
        }
  }

tcp_agent_4561.conf

input {
  log4j {
        port => 4561
        codec => multiline{
                pattern => "^%{TIMESTAMP_ISO8601}"
                negate => "true"
                what => previous
                }
        }
}

filter {
        if[priority] == "ERROR" {
        mutate {
              add_tag => [ "error" ]
                }
        grok {
                        match => ["stack_trace", "(?<exception>%{JAVAEXCEPTION})"]
                }
        }

        mutate {
                add_field => { "WeblogicName" => "X" "Appender" => "Y" }
                remove_field => [ "path", "method", "file", "class", "thread" ]
        }
}

output {
                stomp {
                        password => "X"
                        user => "X"
                        host => "X"
                        destination => "stomp"
                        codec => "json"
                }
}

Java configuration for both of application is the same:

package X.utils;

import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.log4j.net.SocketAppender;

import X.ApplicationProperties;

public class Log4jInitialization {

    public static void init(ApplicationProperties properties){

        String LOGSTASH_HOST = properties.getLOGSTASH_HOST();
        String LOGSTASH_PORT_1 = properties.getLOGSTASH_PORT_1();
        String LOGSTASH_PORT_2 = properties.getLOGSTASH_PORT_2();
        String LOGSTASH_PORT_3 = properties.getLOGSTASH_PORT_3();

            if (LOGSTASH_HOST == null || LOGSTASH_PORT_1 == null ||
                LOGSTASH_PORT_2 == null ||LOGSTASH_PORT_3 == null){
                return;
            }

        if (LOGSTASH_HOST.equals("") || LOGSTASH_PORT_1.equals("") ||
            LOGSTASH_PORT_2.equals("") ||LOGSTASH_PORT_3.equals("")){       
            return;
        }

        int PORT_1 = Integer.parseInt(LOGSTASH_PORT_1);
        int PORT_2 = Integer.parseInt(LOGSTASH_PORT_2);
        int PORT_3 = Integer.parseInt(LOGSTASH_PORT_3);

        SocketAppender Y = new SocketAppender();
        Y.setRemoteHost(LOGSTASH_HOST);
        Y.setPort(PORT_1);
        Y.setReconnectionDelay(10000);
        Y.setThreshold(Level.DEBUG);
        Y.setName("Y");
        Y.activateOptions();

        Logger z = Logger.getLogger("Z");
        z.addAppender(Y);

        SocketAppender g = new SocketAppender();
        g.setRemoteHost(LOGSTASH_HOST);
        g.setPort(PORT_2);
        g.setReconnectionDelay(10000);
        g.setThreshold(Level.DEBUG);
        g.setName("g");
        g.activateOptions();

        Logger f = Logger.getLogger("G");
        f.addAppender(g);

        SocketAppender l = new SocketAppender();
        l.setRemoteHost(LOGSTASH_HOST);
        l.setPort(PORT_3);
        l.setReconnectionDelay(10000);
        l.setThreshold(Level.DEBUG);
        l.setName("l");
        l.activateOptions();

        Logger la = Logger.getLogger("L");
        la.addAppender(l);

    }
}

Both of them share the same application.properties file regarding these properties.

Both of them send pretty much the same message (except for time)

changed some stuff to X Y etc. just in case.

I'm not sure how to find reason why this is happening - any help narrowing down the issue will be great.

[lours1]

I got the same issue. If i connect servicemix with TCPAppender to logstash, got the same error on version 5.4.3. But if I do the same test with an older version of logstash (5.2.2) the log are capture by logstash

[yami12376]

Issue still occurs on Logstash 5.5

[yami12376]

I confirm what lours1 said - on logstash 5.2.2 the issue does not appear

[wkruse]

We've got the same issue in JBoss. It is related to Limit objects accepted to be only log4j objects.

When MDC/NDC is touched (even if it is removed before the actual logging call), it gets serialized and triggers the error:

[2017-07-26T15:32:05,690][DEBUG][logstash.inputs.log4j    ] Closing connection {:client=>"x.x.x.x:62734", :exception=>java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed.}

The last working version of logstash-input-log4j with MDC/NDC is 3.0.3.

See also The future of Log4j input in Logstash

[laughingcai]

Issue still occurs on Logstash 7.8,the following is issue detail: Closing connection {:client=>"xxx", :exception=>java.io.InvalidObjectException: Object type org.apache.log4j.spi.LocationInfo is not allowed.}