[BUG] Unable to configure KafkaAppender for audit logging with Log4J due to possible library loading race condition

[fazir83]

What is the bug? Unable to configure Auditlogging to use Log4J and the KafkaAppender therein.

How can one reproduce the bug? Steps to reproduce the behavior:

1. Follow implementation example in the documentation
2. Configure a KafkaAppender in log4j.properties
3. Restart OpenSearch
4. See error (Audit logs end up on ConsoleAppender with no events being shipped to the KafkaAppender

What is the expected behavior? OpenSearch audit logging being shipped to Kafka Topic

What is your host/environment?

• OS: Ubuntu
• Version 20.04
• Plugins Aiven-Open/prometheus-exporter-plugin-for-opensearch
• Using the tar-release from

Do you have any screenshots? No

Do you have any additional context?

I'll start off by saying I'm not well versed in Java, so I might have taken some unorthodox steps in trying to debug this, and thus any conclusions or theory I might present might be way off basis and have no grounds in reality.

I had need of pushing the audit events from my OpenSearch nodes to a Kafka topic (namely RedPanda, but that shouldn't matter).

So I followed the instructions in documentation and set up audit to log to log4j

plugins.security.audit.config.log4j.logger_name: audit
plugins.security.audit.config.log4j.level: INFO

And then I configured a KafkaAppender in /etc/opensearch/log4j.properties

Restarting OpenSearch at this point results in NoClassDefFoundError due to kafka-clients being missing.

I thought this strange, as it seems by automated commit history that kafka-clients was being built as a dependency.

In opensearch-security, kafka-clients is present in the plugins lib-path. It is however missing from opensearch.

In an attempt to debug this, I decided to move the kafka-clients-3.5.1.jar from the $OPENSEARCH_HOME/plugins/opensearch-security/-directory to the libpath of opensearch: $OPENSEARCH_HOME/lib/ And did the same for the dependencies slf4j-api and log4j-slf4-impl This resusltet in opensearch snagging on the bootstrapcheck for "JarHell". In yet another attempt to work around this I decided to remove all plugin libraries of the aforementioned JAR-files.

Doing this resultet in some JavaSecurityPolicy errors that I worked around using:

    permission javax.management.MBeanTrustPermission "register";
    permission javax.management.MBeanServerPermission "createMBeanServer";
    permission javax.management.MBeanPermission "*", "registerMBean";
    permission javax.management.MBeanPermission "*", "unregisterMBean";
    permission java.net.SocketPermission "<ip:port of kafka bootstrapserver>", "connect,resolve";

Applied under the grant {-statement in $OPENSEARCH_HOME/jdk/conf/security/java.policy

Doing all these steps, OpenSearch was successfully loading the right libraries in the right order to be able to start. And was thus posting my audit messages to my Kafka topic.

My theory is that the kafka-clients library being loaded by opensearch-security is creating a sort of racecondition, seems opensearch is loading up Log4J, parses the log4j.properties and tries loading classes for the KafkaAppender, though this class is being loaded at a later place during bootstrap, it returns at NoClassDefFoundError, thus preventing OpenSearch from starting up.

[stephen-crawford]

[Triage] Hi @fazir83 thank you for filing this issue. Looks like the security policy may be causing some issues here. Someone will need to take a look.

[fazir83]

Yes, that seems to be part of it. Though there mainly seems to be a dependency issue from what I gather, as I had to apply some what seemed to me pretty unorthodox changes (from baseline tar-release, that is) in regards to dependencies to even get the security policy issues to exert themselves

[stephen-crawford]

This may be helpful in the meantime: https://github.com/opensearch-project/security/issues/820

Looks like at some this was possible.

Alternatively you could look at Aiven's website: https://aiven.io/docs/products/kafka/kafka-connect/howto/opensearch-sink

@willyborankin may have some idea about the latter.

[fazir83]

820 Doesn't indicate that this was possible at all. That issue is closed with an unhelpful reference to Log4J documentation. Following these steps (Like I have, above) You'd end up with the issues discussed above.

The Aiven connector you're refering to performs opposite function of what I'm after here. I am trying to get auditlogs generated by OpenSearch to be appended to a Kafka topic. The Aiven connector is a sink functionality in kafka to put events from a kafka topic into an opensearch cluster.

[stephen-crawford]

@willyborankin @reta do either of you know anything about this? @fazir83 is trying to use Prometheus so I thought you may have come across this scenario

[fazir83]

To clarify, I specified the prometheus plugin just to follow the issue template, it has nothing to do with the issue at hand, as far as I'm aware. I have been able to recreate this issue in a freshly built opensearch with opensearch-security from mainline as of right now.

[reta]

@fazir83 I believe that there is quite a gap in security plugin documentation related to detailed instruction of how additional appenders should be configured: modifying /etc/opensearch/log4j.properties is clearly not the way to do that since it alters the whole OpenSearch instance but shouldn't.

[willyborankin]

@fazir83 Finally found the solution. You need to use: plugins.security.audit.type: kafka so as result audit will start to sync in Kafka topic instead of index or log file. here is the code: https://github.com/opensearch-project/security/blob/20c524ad994a9cc7d8757999f92f6d2fec6cb8ca/src/main/java/org/opensearch/security/auditlog/sink/SinkProvider.java#L161

Additional properties same as for Kafka client settings:

• bootstrap_servers
• topic_name

Important note: Teh sec plugin Kafka Audit Sync does not work with SSL connections

[fazir83]

@willyborankin Please take a look at the comment I posted in this issue I had to run through some extreme hoops to actually get that to work since opensearch-security doesn't announce those configuration directives to opensearch, preventing configuration.

[willyborankin]

@willyborankin Please take a look at the comment I posted in this issue I had to run through some extreme hoops to actually get that to work since opensearch-security doesn't announce those configuration directives to opensearch, preventing configuration.

@fazir83 Got it. I will re-open the issue you mentioned.

[willyborankin]

@fazir83, thank you for the investigation. It definitely does not work properly, given the fact that we have a test for it.