search

logstash-plugins / logstash-input-jms

12 stars 27 forks source link

Missing valuefier for java.lang.Byte when reading messages from IBM MQ #21

Open alexander-milovidov opened 7 years ago

alexander-milovidov commented 7 years ago

I have problem reading messages from IBM WebSphere MQ. It seems that some field(s) in MQMD header has type which is not handled properly. Some messages are read without exception. When I turn off include_header setting in jms input section, all messages are handled without any errors.

When working with MQ manager on AIX: java.lang.IllegalArgumentException: Missing Valuefier handling for full class name=java.lang.Byte, simple name=Byte MQ manager and logstash on Linux: java.lang.IllegalArgumentException: Missing Valuefier handling for full class name=java.lang.Short, simple name=Short

Logstash version: 5.3.0 Plugin version: logstash-input-jms-3.0.1-java java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) Operating systems: Client: Windows 7 Ent SP1, Linux RHEL (uname -r output: 2.6.32-642.4.2.el6.x86_64) Server: AIX 7.1 (Power PC), Linux Mint Serena (x86 32-bit) WebSphere MQ versions 7.5.0.6, 8.0.0.6 WebSphere MQ client version: 9.0.0.0 (Level: p900-L160512.4)

Update: I have just tried to run this configuration with MQ client 7.5.0.6 for Linux, it makes no difference.

How to reproduce. You need a Windows or Linux machine with MQ java libraries. I have installed MQ client 9 on Windows. I have attached two sample messages which can be loaded to MQ queue using RFHUTIL (IBM SupportPac IH03) (use "Load Q" button with default settings). RFHUTIL requires MQ client. One of these messages is processed without error, another message causes exception. test_messages.zip

Test messages for qload utility (SupportPac MO03): qload_test_messages.zip

Logs: logstash_jms.log.zip

Config input section:

input {
    jms {
        yaml_file => "C:/logstash-5.3.0/config/jms.yml"
        yaml_section => "wmq"
        include_header => true
        include_properties => true
        include_body => true
        use_jms_timestamp => false
        destination => "TEST.LOG"
        interval => 3
        runner => async
        add_field => [ "host", "computer_name" ]
    }
}

jms.yml:

---
# IBM WebSphere MQ
wmq:
  :factory: com.ibm.mq.jms.MQQueueConnectionFactory
  :queue_manager: QMNAME
  :host_name: server_name.domain_name
  :channel: LOGSTASH.CHL
  :port: 1414
  # Transport Type: com.ibm.mq.jms.JMSC::MQJMS_TP_CLIENT_MQ_TCPIP
  :transport_type: 1
  :username: test
  :require_jars:
    - C:/IBM/MQ/java/lib/com.ibm.mq.allclient.jar

You may have to include other jars from MQ java/lib, but this configuration works on my computer.

I have tried to ask at the forum, but no one replied. https://discuss.elastic.co/t/logstash-input-jms-problem-reading-messages-from-ibm-websphere-mq-missing-valuefier/81993

alexander-milovidov commented 7 years ago

Steps to create MQ server test environment (Linux example).

  1. Download IBM WebSphere MQ trial edition from IBM: https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=ESD-WSMQ-EVAL&S_CMP=web_dw_rt_swd After registering with IBM ID (free of charge) and accepting tons of license agreements, you will be allowed to download the installation package. I have downloaded MQ 7.5.0.2 for Linux x86 32-bit because I have used my old laptop for the test environment.
  2. Install MQ. Unpack downloaded tar.gz file, install packages: sudo rpm -i MQSeriesRuntime-7.5.0-2.i386.rpm MQSeriesClient-7.5.0-2.i386.rpm MQSeriesJava-7.5.0-2.i386.rpm MQSeriesServer-7.5.0-2.i386.rpm I have also installed MQSeriesExplorer-7.5.0-2.i386.rpm, MQSeriesJRE-7.5.0-2.i386.rpm, MQSeriesMan-7.5.0-2.i386.rpm but did not used.

If you have Ubuntu Linux like me, you need to convert rpm packages to deb with Alien: sudo alien --scripts *.rpm and install with dpkg:

dpkg -i mqseriesruntime_7.5.0-3_i386.deb
dpkg -i mqseriesserver_7.5.0-3_i386.deb
dpkg -i mqseriesclient_7.5.0-3_i386.deb
dpkg -i mqseriesjava_7.5.0-3_i386.deb

MQSeriesRuntime is required for other packages, so it needs to be installed first.

Accept license: sudo ./mqlicense.sh

  1. Create and run MQ server. Open another terminal and switch to mqm user: sudo -u mqm -i bash Create and configure MQ manager:

    crtmqm QMTEST
strmqm QMTEST
runmqsc QMTEST

    Enter MQS commands in mqsc shell:

    * Comments begin with asterisk
* Disable channel authentication records (not recommended for production):
alter qmgr chlauth(disabled)
* Create local queue:
define qlocal(TEST.LOG)
* Create backout local queue (not necessary for testing)
define qlocal(TEST.LOG.ERR)
* Assign backout queue setting (not necessary for testing)
alter qlocal(TEST.LOG) BOQNAME(TEST.LOG.ERR) BOTHRESH(1)
* Create server-connection channel for incoming client connections
* (do not use mca user mqm in production)
define channel(LOGSTASH.CHL) chltype(SVRCONN) mcauser('mqm')
* Create and start TCP listener
define listener(LISTENER.TCP) TRPTYPE(TCP) PORT(1414)
start listener(LISTENER.TCP)
* Quit MQSC command shell
end

    After these steps, MQ server is ready. You can start Logstash and connect to MQ manager QMTEST, queue TEST.LOG.

  2. Load test messages to TEST.LOG queue. Download qload utility: wget ftp://public.dhe.ibm.com/software/integration/support/supportpacs/individual/mo03.zip Unzip to any location. Choose qload executable according to your OS. Download test messages which are attached to this issue: wget https://github.com/logstash-plugins/logstash-input-jms/files/951560/qload_test_messages.zip Unzip and load messages with qload utility:

    ./qload -m QMTEST -f qload_msg_test_success -o TEST.LOG
./qload -m QMTEST -f qload_msg_test_error -o TEST.LOG

After reproducing this problem on my laptop, I mentioned that exception is slightly differs from the exception in work environment - problem with Short class instead of Byte: java.lang.IllegalArgumentException: Missing Valuefier handling for full class name=java.lang.Short, simple name=Short