ibm-messaging / mq-jms-spring

Components to assist MQ JMS integration with Spring frameworks
Apache License 2.0
189 stars 102 forks source link

FFST is printed in logs and spams kibana #111

Open ckruczek opened 3 months ago

ckruczek commented 3 months ago

Please include the following information in your ticket.

In our current spring boot(3.2.4) project we encounter issues with logging from the library. Namely there are logs written to the stderr that we can't control and spam our kibana. Whenever we cannot connect to the ibm mq the library responds with an FFST log that is written to our kibana. We see ton's of log entries with the following format: XMSC_WMQ_BROKER_CONTROLQ or FFST Location We tried to use: -Dcom.ibm.msg.client.commonservices.log.status=OFF as well as -Dcom.ibm.msg.client.commonservices.log.maxBytes=0 but non of them prevented the library from writing these logs.

ibmmqmet commented 3 months ago

Not really a Spring Boot thing, as FFDCs are controlled by the underlying MQ JMS client.

But anyway ... there's no way I know of to completely block all FFST. They are not the same as trace or log data. But you might be able to set up directories so that the FFDCs are written somewhere else instead of stderr. (I think by default it tries to create and write into an FFDC directory under your current working directory and only uses stderr if that fails.)

What you CAN do however is significantly reduce the number that are generated.

The product documentation explains how you can suppress multiple instances of the same FFST probes from a given process. Setting the com.ibm.msg.client.commonservices.ffst.suppress property to -1 or a large positive integer would only give the first occurrence.

ckruczek commented 3 months ago

Hey @ibmmqmet. Thanks for your fast response!

Setting this com.ibm.msg.client.commonservices.ffst.suppress might help already a lot. For your other suggestions. We can't redirect it to another directory as this is not in our hands and it would also bloat the pod where our mq client is running. But anyway, thanks for pointing out as an alternative idea!

I would still suggest to provide a patch or change in the library that fixes this behaviour or at least gives more control over it.

Thanks a lot @ibmmqmet

ckruczek commented 3 months ago

We tried to set -Dcom.ibm.msg.client.commonservices.log.outputName=/dev/null as a java tool option, but not even this prevented the logs from being written to the logs. Is there any other way to write them?

ckruczek commented 3 months ago

Hey again!

We have serious issues with this behaviour. Million of logs are written into our kibana and there is no way to turn off this behaviour. We are currently trying to redirect it to a file and then remove the file every minute with a cron job. But this is frankly not an acceptable solution. Considering that this library is meant to be run in a spring environment, then there also must be possibilities to configure it with standard spring options, like logging level. Just writing to stderr without error level and without configuration possibilities to turn of the FFST logs is nothing I would consider production ready.

ibmmqmet commented 3 months ago

The product documentation describes how to do additional configuration of the diagnostics components.

So if I run with -Dcom.ibm.mq.commonservices=diag.props and that file contains Diagnostics.Java.FFDC.Destination.Pathname=/tmp/jms_ffdc. then that is the directory the FDCs go to - provided it's an existing writable directory.

There are ways to hook into the Logging framework for Tracing as shown here but that doesn't really cover FFDCs which are considered more important. And as I said before, all of thise is really outside the scope of this package - it's handled by the core MQ Java client libraries, and issues or changes to it would have to be handled initially by IBM support or by feature enhancement requests.