search

i2group / analyze

Develop and deploy custom Java extensions and REST API client code for i2 Analyze. View the Java API documentation.
https://i2group.github.io/analyze/
MIT License
37 stars 30 forks source link

EIA 2.4.1 - Audit Enablement causing Expand issue in Analyst Notebook #67

Closed aq94 closed 2 years ago

aq94 commented 2 years ago

Hi Team,

We've been facing some challenges after implementing the database audit logging feature following the steps outlined in the links below:

To provide some context, in our previous version of EIA (2.3.0) we had followed the same steps and faced no issues. Earlier this year in May we upgraded to EIA 2.4.1 which is where we began to face the expand issue - after following the steps in the links above and redeploying to enable auditing, the expand operation in Analyst Notebook does not work and gives a couple of pop-up errors messages (see attached).

We raised a case on this matter with i2 Support (Case: 00007639) where we were able to establish that the audit logging implementation was causing the expand error in ANB, so we disabled the audit logging to allow the expand to function. As part of these discussions we thought it was due to us using a jar file compiled using the older version of EIA (2.3.0) which was not compatible with our current version of EIA (2.4.1)

We are currently on i2 Analyze 4.3.5, and so I've downloaded the Developer Essentials repository v4.3.5 supplied in the i2 Github. I've followed the steps in the links and redeployed however we are still facing the expand error, which may suggest there may be another underlying issue not related to jar file compatibility.

Note: There is no loss of expand functionality in WebClient, and logs for the other actions (quick_search, visual query, etc.) are being captured in our audit tables. Expands performed in WebClient are not being captured in our audit table.

I deployed the audit logging just before 10pm last night (29/09/22) and between 10pm - 10.15pm performed some front-end tests. I've attached the console.log for reference.

Please let me know if there are any other details you need that may assist in troubleshooting.

Thanks, Adam

console.zip

ANB Expand Error Message 1 ANB Expand Error Message 2
Anthony-Johnson-i2 commented 2 years ago

Hi Adam

It would be worth knowing if an expand from a single seed item that only has a couple of links and entities that you expect to see as results also fails like this. (You could find one with a small number of connections when you don't have the audit logging enabled and then try that with it enabled).

I did get a very similar error , possibly the same error as you are seeing

Caused by: com.ibm.db2.jcc.am.SqlDataException: The value of a host variable in the EXECUTE or OPEN statement is out of range for its corresponding use.. SQLCODE=-302, SQLSTATE=22001, DRIVER=4.21.29

a couple of years ago with expands, when the number of seed or result IDs it was trying to add to the audit tables it uses in the database for logging was causing it to blow the max length of the field it was using.

The problem back then was that:

The i2Audit.Expand table had a Seeds field and the i2Audit.Record_Retrieval table had a Records field that could not cope with the amount of information we were trying to log to them.

We swapped the definition for those to be Seeds CLOB(2147483647) NOT NULL and Records CLOB(2147483647) NOT NULL and that resolved the issue.

I would not expect that you have such an old version of the tables but it would be worth just checking that in your Expand and Record_Retrieval tables that you are writing to you have the CLOB definitions for those fields instead of the older ones so that we can rule that out.

Cheers

aq94 commented 2 years ago

Hi Anthony,

Thanks for the information above, the good news is that we've been able to successfully troubleshoot the issue.

We decided to drop our previous tables and re-create them with the new script provided in the Developer Essentials repository. Noting the two tables and columns (expand.seeds and record_retrieval.records) they were assigned the CLOB(2147483647) data type.

When we re-started ANB we were unable to perform a quick search however. This was an issue one of our other engineers faced as per the link below, so we ended up recreating the tables again and assigning CLOB(2147483647) to the USER_SECURITY_GROUPS and USER_SECURITY_PERMISSIONS columns for all of our tables.

https://github.com/i2group/analyze/issues/59

After redeploying with these changes, we haven't seen any issues in the front end and all the tables seem to be capturing the logs as expected.

Happy to close this issue as resolved and thanks again for the help.

Cheers, Adam

Anthony-Johnson-i2 commented 2 years ago

Excellent news Adam and thanks for the update. As it happens, the other case you mention #59 from Feb 2019 was also one I was looking at but under my then IBM user name TonyJon :-)