Closed sourikghosh closed 1 year ago
Can #45 cause the above issue ?
Hi @sourikghosh. Can you please share the dependency tree? looks like there are two slf4j bindings.
mvn dependency:tree
Hey @yhl25 Thanks for the quick response. After running gradle dependencies
Also adding the runtimeClasspath.
Is it a warning or an error?
The error log shows rg.apache.logging.log4j/log4j-slf4j-impl/2.17.2
and org.slf4j/slf4j-log4j12/1.7.30
are causing the issue, based on your RuntimeClassPath both these dependencies are not part of numaflow sdk. Not sure how it is working when you remove the numaflow sdk dependency. Can you try excluding one of these dependencies and see if that works? https://www.slf4j.org/codes.html#multiple_bindings
i see there are two slf4j bindings here one brought in by spark (log4j-slf4j-impl) and one by hadoop (slf4j-log4j12) since numaflow uses slf4j-api and numaflow is the main function, it cannot decide which one to bind to. quickest way is to see which one we can exclude and get it to working.
as to why this issue is seen only with numaflow sdk, we need to see how the spark / hadoop are being used in the udf
Thanks @ashwinidulams @yhl25. for the help.
Tried excluding both spark (log4j-slf4j-impl) and hadoop (slf4j-log4j12)
one at a time to check which one works. But it seems none of them are working after excluding.
configurations.implementation {
// exclude(group = "org.apache.logging.log4j",module = "log4j-slf4j-impl")
exclude(group = "org.slf4j",module = "slf4j-log4j12")
}
Sure the warning of multiple binding has gone away but no logs in STDOUT
.
whats the log4j xml or the configuration look like?
This is now resolved after adding the log4j2 xml configuration.
Apparently by default the slf4j-simple
uses loglevel="INFO" and after adding numaflow dependency it changes the underlying logging backend to org.slf4j.impl.Log4jLoggerAdapter
which defaultLogLevel="ERROR".
log4j2.xml
configuration used
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<!-- Configure appenders -->
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<!-- Root logger -->
<Root level="info">
<AppenderRef ref="Console"/>
</Root>
<!-- Logger for org.slf4j.impl.Log4jLoggerAdapter -->
<Logger name="org.slf4j.impl.Log4jLoggerAdapter" level="debug"/>
</Loggers>
</Configuration>
Context on Environment
Writing numaflow udf in Kotlin using numaflow java sdk ( numaflow-java:0.4.6 ). Logger used in udf - KotlinLogging.logger ( mu.KotlinLogging ) Logging generally works after removing numaflow dependency.
Console Error log
build.gradle.kts