IBM i2EIA - Unable to perform quicksearch after Configuring the 'audit logging to database'

[harz789]

I'm implementing and Configuring the 'audit logging to database' by following the instructions at https://github.com/IBM-i2/Analyze/blob/master/documentation/Configuring-the-audit-logging-to-database-example-project.md

I’ve not found the windows installer for 4.3.0. Hence i copied over the toolkit folders from the QA linux install over to the windows machine that has eclipse installed. after updating the TOOLKIT_ROOT variable, I was able to compile the jar successfully and copied it over to the QA linux server’s target directories. I’ve made the required changes in the topology.xml file and ApolloServerSettingsMandatory.properties file.

I’ve since redeployed i2. Now, I’m unable to perform a quicksearch. The logs state that they are unable to write to the audit db due to an LDAP issue. However, I’m getting authenticated and logged in to i2 using the same LDAP settings.

I’ve also checked the permissions of the user id that was mentioned in the serverdatasources.xml file.

[harz789]

Operating Systems: Operating System version running on the Application Server host: [dga@03l ~] $ cat /etc/*release NAME="Red Hat Enterprise Linux Server" VERSION="7.8 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.8" PRETTY_NAME="Red Hat Enterprise Linux Server 7.8 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.8" Red Hat Enterprise Linux Server release 7.8 (Maipo) Red Hat Enterprise Linux Server release 7.8 (Maipo) [dga@03l ~] $

Operating System version running on the database host: 68l:vdga[131]$ uname -a AIX 68l 2 7 00C09FB84C00 68l:dga[132]$

[harz789]

Analysis Repository database type & version: 68l:dga[132]$ db2level DB21085I This instance or install (instance name, where applicable: "dga") uses "64" bits and DB2 code release "SQL11014" with level identifier "0205010F". Informational tokens are "DB2 v11.1.4.5", "s1911120100", "DYN1911120100AIX", and Fix Pack "5". Product is installed at "/opt/IBM/db2/dga/sqllib". 68l:dga[133]$

[harz789]

When the quick search failed after implementing ‘audit logging to database’, I reconfigured the previous implementation of ‘audit logging to csv’ and redeployed i2. I had no issues with quick search.

The issue re-appeared after I implemented ‘audit logging to database’.

[TonyJon]

Hi Harz789

I have not had to do anything special with the DB version as opposed to the others to get it to work on my VM here at IBM, but I do tend to only use a basic user registry for that so do not have LDAP involvement for the directory lookup. Out of curiosity, what software do you use for your user directory management?

Is this a test system that you can share the logs /topology file from on here, or would you rather take this offline whilst we investigate with you and then we can just post back when we get a solution?

At the latest release 4.3.3 we no longer support any older Onyx/Analysis Repository related Developer Essentials so the new documentation you linked only includes the Opal parts of the topology and the Opal auditing example.

Are you using both Opal Information Store and and Onyx Analysis repository in your deployment?

As for copying the Linux version you had to windows and working on it there then copying that back to Linux . That should not affect the workings of your example at least.

However, to be sure of being correct you can get the proper example documentation and software from the Releases option on the right hand side of the page specifically written for your release.

Which takes you here

https://github.com/IBM-i2/Analyze/releases

and from there you can get to the 4.3.0 you want.

https://github.com/IBM-i2/Analyze/releases/tag/v8.0.1

The documentation for that release is downloaded in the tar or zip so you can access it locally if you wish or you can also get to the correct documentation online via the history button

on the documentation page.

Here is a direct link to the online 4.3.0 documentation:

https://github.com/IBM-i2/Analyze/tree/a922a5c4ef52ede81b97c9e75254c75994f6245c

It would be worth downloading and using the 4.3.0 example software with the 4.3.0 documentation and just trying the DB example again in case something has changed under the hood.

If that gives you the same result, let us know how you would like to continue communicating and we can take things from there.

Cheers

FYI : I am out of the office from 4:15pm today (UK time) until 7am (UK time) Monday 8 Feb so will not see your replies during that period. (I will however check in again before I leave today and first thing on Monday) .

Cheers

[TonyJon]

Just checking in at the end of my day in case there was an update. I guess we have a timezone difference maybe? Cheers

[TonyJon]

Hi Harz789 Just wondering how you were getting on with this... Have you made any progress? Cheers

[harz789]

Hi Tony,

We use Microsoft Active Directory. This is a test environment and I can share the logs and the required configs. However, I wasn’t sure if sharing these publicly on github is a good idea.

I’ve already used the git hub repo code from 4.3.0 and still I’ve the same issue.

[TonyJon]

Hi harz789

Thanks for the update.

As long as this is just your test system and has no sensitive data in the logs then please feel free to zip things up and send them in to my IBM email address.

tony.johnson@uk.ibm.com

What will be useful is if you stop your Liberty test server and clear down the logs folders here:

.../deploy/wlp/usr/servers/opal-server/logs

and here:

.../toolkit/configuration/logs

and then edit the log4j.properties file in the Liberty test servers deployed war here:

.../deploy/wlp/usr/servers/opal-server/apps/opal-services-is.war/WEB-INF/classes

to enable debugging by changing this:

to this:

Then if you could please start your test system back up and run your search so that it errors and then stop it again.

You can then zip those two log folders up (including and sub folders) and email them to me including your liberty server.xml and serverdatasources.xml and your toolkit topology.xml files and I'll take a look.

Please password protect your zip/zips.

Cheers

[TonyJon]

After receiving the logs, and with the help of the additional debug information, we were able to see that the underlying problem was actually due to there being too much group information to fit into the User_Security_Groups fields in the tables used for the auditing example.

This is only an example, and can be freely altered, so the script provided

create-audit-database.sql

in the unpacked Developer Essentials download here:

developer-essentials\SDK\sdk-projects\opal-audit-database-example\scripts\database\db2

to create the tables used in auditing was modified to create these fields as CLOBs so they had a lot more storage space.

e.g.

User_Security_Groups VARCHAR(1000) NOT NULL,

was replaced by

User_Security_Groups CLOB(2147483647) NOT NULL

This resolved the issue for the example in this context, however, please remember that this is an example implementation and not intended for use in production. We would expect any auditing tables you create for production use to be carefully designed to fit your specific needs.