Open linghengqian opened 1 month ago
Hi @linghengqian,
Could you point us to how you generate the GraalVM image?
As almost all filters (except DenyAllFilter
and Filters
), MarkerFilter
has a onMismatch
property. I suspect that the GraalVM reflection metadata you use might lack the property, hence the problem in the native image.
Hi @linghengqian,
Could you point us to how you generate the GraalVM image?
As almost all filters (except
DenyAllFilter
andFilters
),MarkerFilter
has aonMismatch
property. I suspect that the GraalVM reflection metadata you use might lack the property, hence the problem in the native image.
Hi, the steps I mentioned in the issue description never involve using log4j2 under GraalVM Native Image. I assume you are referring to https://github.com/apache/shardingsphere/pull/31526 . On the shardingspehre side, all nativeTest under GraalVM Native Image have excluded log4j2 in maven pom, and I am actually waiting for the release of log4j2 2.24.0.
On the hive side, the Error Log of ERROR MarkerFilter contains an invalid element or attribute "onMismatch"
is obtained by executing unit tests under Hotspot JDK 8. Hive can still only be compiled with openjdk 8, and it is too early to discuss graalvm native image in the hive community.
@linghengqian,
The problem is due to a typo in your configuration file:
logger.CBORuleLogger.filter.marker.type = MarkerFilter
logger.CBORuleLogger.filter.marker.marker = FULL_PLAN
logger.CBORuleLogger.filter.marker.onMatch = DENY
logger.CBORuleLogger.filter.marker.onMisMatch = NEUTRAL
You have onMisMatch
instead of onMismatch
. The misleading error message is due to the following code in PropertiesConfigurationBuilder
:
Since you don't have an onMismatch
attribute, the code creates a synthetic onMismatch
attribute with an invalid null
value.
Remark: Fixing this also fixes LOG4J2-2931, so I am redirecting that issue here.
PS: I noticed that Apache Hive uses a pre-2.6 properties configuration format (the loggers
is not required since version 2.6).
Would you consider switching to a more easily readable configuration format? Version 3.x will introduce breaking changes to the properties format, but not the hierarchical ones.
@ppkarwasz
Late reply, because I need to wait for verification on apache/hive
's CI side. apache/hive
's CI resources are too tight, and apache/hive
's jenkins node limits each contributor to 5 hours between 2 consecutive CIs. To be honest, https://github.com/apache/hive/pull/5375 is the first PR I opened on apache/hive's side, and I know almost nothing about apache/hive
's logging design.
I updated https://github.com/apache/hive/pull/5375 not long ago, I removed the packages
, appenders
, loggers
nodes of /hive/data/conf/hive-log4j2.properties
, and changed logger.CBORuleLogger.filter.marker.onMisMatch
to logger.CBORuleLogger.filter.marker.onMismatch
. apache/hive
's CI is now back on apache/logging-log4j2:2.23.0. I'm assuming this is a reasonable way forward to not break compatibility with future versions of log4j2. I'm assuming the changes are in line with what's documented in https://github.com/apache/pulsar/pull/22327 , https://logging.apache.org/log4j/2.x/manual/configuration.html , and https://logging.apache.org/log4j/2.x/manual/plugins.html . I note from https://issues.apache.org/jira/browse/LOG4J2-3644 that the packages
property seems to be straightforward to remove, though https://github.com/apache/jmeter/issues/5937 on the apache/jmeter
side is still unclear as to the way forward.
2024-08-02T10:55:16.634Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-08-02T10:55:16.650Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-08-02T10:55:16.670Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-08-02T10:55:16.686Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
On the apache/hive
side, I noticed a new issue when bump from apache/logging-log4j2:2.23.0
to apache/logging-log4j2:2.23.1
, the status = INFO
defined in hive-log4j2.properties
is no longer valid, and Log4j2 internal logs at TRACE
and DEBUG
levels will be printed starting from apache/logging-log4j2:2.23.1
. Since the current issue is bound to https://issues.apache.org/jira/browse/LOG4J2-2931 , I opened #2794 to avoid ambiguity.
I will say that I personally don't know how to fix #2791 with the good first issue
tag.
@ppkarwasz
I note from https://issues.apache.org/jira/browse/LOG4J2-3644 that the
packages
property seems to be straightforward to remove, though Migrate log4j2 to logback jmeter#5937 on theapache/jmeter
side is still unclear as to the way forward.
There is no problem in Apache Hive: I checked that the appropriate annotation processor is configured in the hive-exec
project and there is a Log4j2Plugins.dat
file in the JAR. You can safely remove the packages
configuration attribute.
* I will say that I personally don't know how to fix [Fix handling of `onMatch` and `onMismatch` attributes in the properties configuration format #2791](https://github.com/apache/logging-log4j2/issues/2791) with the `good first issue` tag.
I believe that the problem breaks down to a missing null-check: ComponentBuilder#addAttribute
should add an attribute if the value is not null
and remove it if the value is null
.
null
is never a valid value for an attribute.
BTW: All the problems you are having are due to the fact that TestHplsqlLocal
and TestHplsqlOffline
are fragile tests. They write something to System.out
and they make assertions about that. The problem is: other components (like the status logger) might also write to System.out
. I proposed a solution to this in apache/hive#5381
Hi @linghengqian,
Could you point us to how you generate the GraalVM image?
As almost all filters (except
DenyAllFilter
andFilters
),MarkerFilter
has aonMismatch
property. I suspect that the GraalVM reflection metadata you use might lack the property, hence the problem in the native image.
Description
onMismatch
ofMarkerFilter
is no longer available. This was actually found from https://github.com/apache/hive/pull/5375 . It can be reproduced in both2.23.0
and2.23.1
. This will be reflected in a log similar toERROR MarkerFilter contains an invalid element or attribute "onMismatch"
.Configuration
Version: [Log4j version]
2.23.0
Operating system: [OS and version]
Ubuntu WSL 22.04.4
JDK: [JDK distribution and version]
8.0.422-tem
onSDKMAN!
Logs
Reproduction
/hive/data/conf/hive-log4j2.properties
.property.hive.log.level = DEBUG property.hive.root.logger = DRFA property.hive.log.dir = ${sys:test.tmp.dir}/log property.hive.log.file = hive.log property.hive.test.console.log.level = INFO
appenders = console, DRFA
appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.DRFA.type = RollingRandomAccessFile appender.DRFA.name = DRFA appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file} appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd} appender.DRFA.layout.type = PatternLayout appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n appender.DRFA.policies.type = Policies appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy appender.DRFA.policies.time.interval = 1 appender.DRFA.policies.time.modulate = true appender.DRFA.strategy.type = DefaultRolloverStrategy appender.DRFA.strategy.max = 30
loggers = HadoopIPC, HadoopSecurity, Hdfs, HdfsServer, HadoopMetrics2, Mortbay, Yarn, YarnServer, Tez, HadoopConf, Zookeeper, ServerCnxn, NIOServerCnxn, ClientCnxn, ClientCnxnSocket, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX, Operator, Serde2Lazy, ObjectStore, CalcitePlanner, AmazonAws, ApacheHttp, Thrift, Jetty, BlockStateChange, swo, CBORuleLogger
logger.HadoopIPC.name = org.apache.hadoop.ipc logger.HadoopIPC.level = WARN
logger.HadoopSecurity.name = org.apache.hadoop.security logger.HadoopSecurity.level = INFO
logger.Hdfs.name = org.apache.hadoop.hdfs logger.Hdfs.level = INFO
logger.HdfsServer.name = org.apache.hadoop.hdfs.server logger.HdfsServer.level = WARN
logger.HadoopMetrics2.name = org.apache.hadoop.metrics2 logger.HadoopMetrics2.level = INFO
logger.Mortbay.name = org.mortbay logger.Mortbay.level = INFO
logger.Yarn.name = org.apache.hadoop.yarn logger.Yarn.level = INFO
logger.YarnServer.name = org.apache.hadoop.yarn.server logger.YarnServer.level = WARN
logger.Tez.name = org.apache.tez logger.Tez.level = INFO
logger.HadoopConf.name = org.apache.hadoop.conf.Configuration logger.HadoopConf.level = ERROR
logger.Zookeeper.name = org.apache.zookeeper logger.Zookeeper.level = INFO
logger.ServerCnxn.name = org.apache.zookeeper.server.ServerCnxn logger.ServerCnxn.level = WARN
logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn logger.NIOServerCnxn.level = WARN
logger.ClientCnxn.name = org.apache.zookeeper.ClientCnxn logger.ClientCnxn.level = WARN
logger.ClientCnxnSocket.name = org.apache.zookeeper.ClientCnxnSocket logger.ClientCnxnSocket.level = WARN
logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO logger.ClientCnxnSocketNIO.level = WARN
logger.DataNucleus.name = DataNucleus logger.DataNucleus.level = ERROR
logger.Datastore.name = Datastore logger.Datastore.level = ERROR
logger.JPOX.name = JPOX logger.JPOX.level = ERROR
logger.Operator.name = org.apache.hadoop.hive.ql.exec.Operator logger.Operator.level = INFO
logger.Serde2Lazy.name = org.apache.hadoop.hive.serde2.lazy logger.Serde2Lazy.level = INFO
logger.ObjectStore.name = org.apache.hadoop.hive.metastore.ObjectStore logger.ObjectStore.level = INFO
logger.CalcitePlanner.name = org.apache.calcite.plan.RelOptPlanner logger.CalcitePlanner.level = INFO
logger.CBORuleLogger.name = org.apache.hadoop.hive.ql.optimizer.calcite.RuleEventLogger logger.CBORuleLogger.level = OFF logger.CBORuleLogger.filter.marker.type = MarkerFilter logger.CBORuleLogger.filter.marker.marker = FULL_PLAN
logger.CBORuleLogger.filter.marker.onMatch = DENY logger.CBORuleLogger.filter.marker.onMisMatch = NEUTRAL
logger.AmazonAws.name=com.amazonaws logger.AmazonAws.level = INFO
logger.ApacheHttp.name=org.apache.http logger.ApacheHttp.level = INFO
logger.Thrift.name = org.apache.thrift logger.Thrift.level = WARN
logger.Jetty.name = org.eclipse.jetty logger.Jetty.level = WARN
logger.BlockStateChange.name = BlockStateChange logger.BlockStateChange.level = WARN
rootLogger.level = ${sys:hive.log.level} rootLogger.appenderRefs = root, console rootLogger.appenderRef.root.ref = ${sys:hive.root.logger} rootLogger.appenderRef.console.ref = console rootLogger.appenderRef.console.level = ${sys:hive.test.console.log.level}
logger.swo.name = org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer logger.swo.level = DEBUG