jjjw420 / splunk-ibm-websphere-mq-add-on

IBM Websphere MQ add-on for Splunk
3 stars 6 forks source link

Error 'Exception in ChannelStatusPoller: User and password must be str or bytes' #2

Closed thilles closed 5 years ago

thilles commented 5 years ago

Hi,

Trying to get the add-on to work, but am getting the following error:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mq_ta/bin/mqchs.py" Stopping. Exception occurred in ChannelStatusPoller: Both user and password must be instances of str or bytes mqchs_stanza:mqchs://Test2_channel_status

Username is in form _myuser and pwd is in form mYs3Cr3tPwD42

Any pointers?

jjjw420 commented 5 years ago

Suspect It's the pymqi version and specifically the python2/3 compatibility. I'm busy converting to cater for python 3, pymqi 1.9+ and splunk 8.

Can u please advise on the following Splunk version Pymqi version. MQ client version

As a quick fix u could go back to pymqi 1.5. But I may have a patch out for this in the next few days.

thilles commented 5 years ago

Sure, here you go: Splunk 7.2.1 Pymqi 1.9.2 MQ client 9.1.3.0

I'll try pymqi 1.5 in the meanwhile

thilles commented 5 years ago

Still an error using pymqi 1.5.4, but now it's morphed somewhat: ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mq_ta/bin/mqinput.py" Stopping. Exception in QueuePoller: MQCD wrong size. Given: 1984, expected 1992 mqinput_stanza:mqinput://Test1_mq_queue

Both input types throw the same error. This is also the same error I got when I, with version 1.9.2, casted the mq_user_name and mq_password to strings in mqinput.py and mqchs.py.

jjjw420 commented 5 years ago

Indeed. My sincerest apologies. I'm busy resolving these compatibility issues as we speak. That particular one is due to the MQCD modifications in the later MQ versions.

This sucks. I should not have left this conversion for so late.
If you can wait a few days great. Otherwise - you need to downgrade all your MQ client software which is not something I would suggest.

Dont stress though. This plugin runs in a couple of production environments reading events from MQ - so this conversion is top priority.

jjjw420 commented 5 years ago

I have made progress. I should have a new version available on Monday.

jjjw420 commented 5 years ago

OK. Much better!

You can test with the latest code on the python2-to-python3-conversion branch if you like. See here: https://github.com/jjjw420/splunk-ibm-websphere-mq-add-on/tree/python2-to-python3-conversion

PLEASE NOTE: The current version of pymqi does not support MQ 9.1.3 yet (9.1.3 added an additional field to the MQCD structure which is the reason for your last error). However - you can find a branch that adds support for MQ 9.1.3 here: https://github.com/SeyfSV/pymqi/tree/mqcd-913/code/pymqi

Also note that Splunk V8 now has the "ctypes" library installed by default. So you no longer have to find a Splunk Python compatible version of "ctypes" and "_ctypes.so". Just the PyMQI library is required with Splunk V8 which makes the installation much simpler.

I am currently testing this build on:

If all is well I will be merging this back into master soonest.

jjjw420 commented 5 years ago

On request I merged the latest changes for python3 and pymqi compatibility into master.
Please try it and let me know. Remember that if you use MQC 9.1.3 that you will need the branch from https://github.com/SeyfSV/pymqi/tree/mqcd-913/code/pymqi

thilles commented 5 years ago

Thanks for the fast work and quick responses!

Installed the mqcd-913 brach of pymqi and the latest master version of mq_ta, but still experiencing the same error ChannelStatusPoller: Both user and password must be instances of str or bytes mqchs_stanza:mqchs://Test2_channel_status

However, by casting the mq_user_name and mq_password to string in mqinput.py and mqchs.py the error disappeared and the input connected. Still get an error MQ Exception occurred: MQI Error. Comp: 2, Reason 2538: FAILED: MQRC_HOST_NOT_AVAILABLE but that's more our problem I believe.

jjjw420 commented 5 years ago

That was my fault. I only migrated the last of the queue managers which have authentication now. I fixed the user name and password unicode strings. You can pull the latest code. I'm going to check all the strings again.

MQRC 2538 only really happens for a few reasons - the most common being: 1.) You have the wrong host name, ip or the dns lookup fails. 2.) you have the wrong port. Please do me favour and actualy enter the port - even if its the 1414 default. I will verify this again too.

So check your connection parameters. However - We can easily debug the mqrc 2538 and your connection to MQ.
I normally use the "amqscnxc" test program which is installed with the MQ samples but you could also use "MQ Explorer".

eg.

/opt/mqm/samp/bin/amqscnxc -x 'myhostnameorip(listener port)' -c <server conn channel> [-u <username>] <queue manager name>

So given Queue Manager "LDB0" at "192.168.0.127" on port 1414 and using the "LDB0.CLIENT" server connection channel with no authentication - a good connection will look like this:

$ /opt/mqm/samp/bin/amqscnxc -x '192.168.0.127(1414)' -c LDB0.CLIENT LDB0 Sample AMQSCNXC start Connecting to queue manager LDB0 using the server connection channel LDB0.CLIENT on connection name 192.168.0.127(1414). Connection established to queue manager LDB0 Sample AMQSCNXC end

If I use the wrong port or host(see the MQRC 2538): $ /opt/mqm/samp/bin/amqscnxc -x '192.168.0.127(1417)' -c LDB0.CLIENT LDB0 Sample AMQSCNXC start Connecting to queue manager LDB0 using the server connection channel LDB0.CLIENT on connection name 192.168.0.127(1417). MQCONNX ended with reason code 2538

With authentication it looks like this - given queue manager "LDB1" at "10.182.22.23(1415)" using the "LDB1.CLIENT" server connection channel and user "hannes"(you are prompted for the password):

/opt/mqm/samp/bin/amqscnxc -x '10.182.22.23(1415)' -c LDB1.CLIENT -u hannes LDB1 Sample AMQSCNXC start Connecting to queue manager LDB1 using the server connection channel LDB1.CLIENT on connection name 10.182.22.23(1415). Enter password: ******** Connection established to queue manager LDB1 Sample AMQSCNXC end

If you can connect using this mechanism or "MQ Explorer" - then Splunk will connect too - as long as you use the same settings.

Overall all though it looks like your environment is working correctly. That means we can overcome any issue you may have.

Let me know how it goes! Thanks for your support.

jjjw420 commented 5 years ago

I've done quite a bit the last few days. It running perfectly since Monday evening (after fixing the user and password issue). It's indexing 10+ million events from messages on MQ per day using a variery of response handlers. IBM Message Broker montiroring events(using the "BrokerEvents" response handler is at the top of that volume list.

thilles commented 5 years ago

Thanks for all the help! I'm quite certain the issue is now at our side - the amqscnxc test gave some very useful hints especially . I'm closing this issue, but as a sidenote: Do you know what the minimum of components of MQ client that are neccessary to install? Ref https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ins.doc/q008350_.htm

jjjw420 commented 5 years ago

This is what I just used for my 9.0.0.4 installation (9.0.0.4-IBM-MQC-LinuxX64.tar.gz) on CentOS(ubuntu library names in V9.1 are the same)..

After accepting the license I installed the following:

MQSeriesRuntime-9.0.0-4.x86_64.rpm MQSeriesSDK-9.0.0-4.x86_64.rpm MQSeriesSamples-9.0.0-4.x86_64.rpm MQSeriesClient-9.0.0-4.x86_64.rpm MQSeriesJava-9.0.0-4.x86_64.rpm MQSeriesJRE-9.0.0-4.x86_64.rpm MQSeriesGSKit-9.0.0-4.x86_64.rpm MQSeriesMan-9.0.0-4.x86_64.rpm

eg.
rpm -ivh MQSeriesRuntime-9.0.0-4.x86_64.rpm MQSeriesSDK-9.0.0-4.x86_64.rpm MQSeriesSamples-9.0.0-4.x86_64.rpm MQSeriesClient-9.0.0-4.x86_64.rpm MQSeriesJava-9.0.0-4.x86_64.rpm MQSeriesJRE-9.0.0-4.x86_64.rpm MQSeriesGSKit-9.0.0-4.x86_64.rpm MQSeriesMan-9.0.0-4.x86_64.rpm

Not all of them are required. If you do not want manual pages you can leave MQSeriesMan.
If you will not do any key management you can leave MQSeriesGSKit. If you do not want the samples you can leave MQSeriesSamples.

You can also leave off the MQSeriesJava package if you have no intentions of using the IBM MQ series java classes.

But the rest you really should install for connectivity and building (MQSeriesRuntime-9.0.0-4.x86_64.rpm ,MQSeriesSDK-9.0.0-4.x86_64.rpm, MQSeriesClient-9.0.0-4.x86_64.rpm, MQSeriesJRE-9.0.0-4.x86_64.rpm)

If you are having issues connecting to your Queue Managers - post the problem here and I will assist where I can.