Log4j2PluginsCacheFileTransformer not merging .dat files?

MacTrophy commented 5 years ago

Shadow Version

4.0.2

Gradle Version

4.10.2

Expected Behavior

The Log4j2Plugins.dat file should contain a list of all matching Log4j2 plugins, as the plugin from TheBoegl was doing

Actual Behavior

I am under the impression that only the last (or another criteria IDK) plugin gets in the generated file. May be the problem comes from the fact that I have a custom AbstractAppender in my project? Only my custom appender gets inserted in the file.

BTW my custom appender is a slightly tuned version of https://github.com/dblock/log4jna/blob/master/log4jna-api/src/main/java/org/dblock/log4jna/nt/Win32EventLogAppender.java

Gradle Build Script(s)

I think only those files are relevant Old lines for shadow 2.0.2 with TheBoegl 2.2.0 that were working:

buildscript {
    dependencies {
        classpath "com.github.jengelman.gradle.plugins:shadow:2.0.2"
        classpath 'de.sebastianboegl.gradle.plugins:shadow-log4j-transformer:2.2.0'
    }
}
shadowJar {
    {...}
    transform(de.sebastianboegl.gradle.plugins.shadow.transformers.Log4j2PluginsFileTransformer)
}

New lines for shadow 4.0.2 that are not working

shadowJar {
    {...}
    transform(com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer)
}

Content of Shadow JAR (`jar tf <jar file>` - post link to GIST if too long)

Only the content of Log4j2Plugins.dat is problematic, I attached the 2 versions Log4j2Plugins.dat.4.0.2.txt Log4j2Plugins.dat.2.0.2.txt

johnrengelman commented 5 years ago

Not sure. If you remove your appender does it work? Would it be possible to provide a github repo with a project that exhibits the issue? (I don't have a windows machine, so I won't be able to run anything with that code in it)

MacTrophy commented 5 years ago

I don't have any github repo nor do I know how to make one but I built a simple project and attached it, hopefully you can open it and call Gradle if you require.

You do not need Windows (I work on MacOS X) nor do you need to execute the Java code, only building the project with gradle (task shadowJar) is enough to reproduce the problem. The goal is to get all the appenders in the Log4j2Plugins.dat file already present on runtime.

In the archive I provided under build/distributions are 2 directories with with only the pertinent Log4j2Plugins.dat in them that I extracted from the 2 generated Jar files. 2.0.2_Log4j2PluginsCacheFileTransformer : the file is 20KB and has a lot of appenders, including the Win32EventLogAppender 4.0.2_Log4j2PluginsFileTransformer : the file is 64bytes and only has the Win32EventLogAppender

The Gradle file is currently setup for your plugin, to use TheBoegl's plugin you have to:

uncomment the 5 first lines
comment your plugin
uncomment TheBoegl's transform
comment your transform.

Let me know if something I specified/provided doesn't work

Log4j2Plugins.zip

MacTrophy commented 5 years ago

Hi, I would like to know, has my example been tried? Thank you

MacTrophy commented 5 years ago

I've tried again with Gradle 5.2.1 and the newer Shadow 4.0.4.

With your transformation called the Log4j2Plugins.dat file is not present at all. Without your transformation called, the file is present at around 20KB but doesn't contain my plugin. Seems to include the plugins in all JARs but not from the project I find this a weird coincidence... Does the transformation gets called automatically and if I call it it fails for some reason? IDK.

Another thing I found is that it's possible to call the annotations processor of Log4j2 but this will produce me a Log4j2Plugins.dat with only my custom plugin.

dependencies {
    implementation 'org.apache.logging.log4j:log4j-api:2.11.1'
    annotationProcessor 'org.apache.logging.log4j:log4j-core:2.11.1'
    {....}
}

I have the impression if this plugin would be called somehow by Shadow, everything would be working.

OrangeFlag commented 5 years ago

@johnrengelman its example of problem https://github.com/OrangeFlag/issue_427_shadow_jar_example Simple handler for aws lambda with log4j2 To use log4j2 we should add custom appender from aws(com.amazonaws:aws-lambda-java-log4j2). But as say @MacTrophy in /org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat there is only this custom plugin and no default plugins that should be

Log4j2Plugins.dat(by vim)

^@^@^@^A^@^Dcore^@^@^@^A^@^Flambda^@;com.amazonaws.services.lambda.runtime.log4j2.LambdaAppender^@^Happender^A^@

Gradle - 5.5 Shadow - 5.1.0

Log from Lambda in trace mode:

Picked up JAVA_TOOL_OPTIONS: -Dlog4j.configurationFile=./log4j2.properties -Dlog4j2.debug=True -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE -Dlog4j2.loggerContextFactory=org.apache.logging.log4j.core.impl.Log4jContextFactory
DEBUG StatusLogger Using ShutdownCallbackRegistry class org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry
DEBUG StatusLogger Loaded Provider Provider[priority=10, className=org.apache.logging.log4j.core.impl.Log4jContextFactory, url=file:/var/task/META-INF/log4j-provider.properties, classLoader=java.net.URLClassLoader@4d405ef7]
DEBUG StatusLogger Not in a ServletContext environment, thus not loading WebLookup plugin.
DEBUG StatusLogger AsyncLogger.ThreadNameStrategy=CACHED
TRACE StatusLogger Using default SystemClock for timestamps.
DEBUG StatusLogger Not in a ServletContext environment, thus not loading WebLookup plugin.
DEBUG StatusLogger Took 0.006083 seconds to load 1 plugins from java.net.URLClassLoader@4d405ef7
DEBUG StatusLogger PluginManager 'Converter' found 0 plugins --- *Important row*
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
DEBUG StatusLogger Starting OutputStreamManager SYSTEM_OUT.false.false-1
DEBUG StatusLogger Starting LoggerContext[name=4d405ef7, org.apache.logging.log4j.core.LoggerContext@5a10411]...
DEBUG StatusLogger Reconfiguration started for context[name=4d405ef7] at URI null (org.apache.logging.log4j.core.LoggerContext@5a10411) with optional ClassLoader: null
DEBUG StatusLogger Not in a ServletContext environment, thus not loading WebLookup plugin.
DEBUG StatusLogger PluginManager 'ConfigurationFactory' found 0 plugins
DEBUG StatusLogger Using configurationFactory org.apache.logging.log4j.core.config.ConfigurationFactory$Factory@2ef1e4fa
ERROR StatusLogger Reconfiguration failed: No configuration found for '4d405ef7' at 'null' in 'null'
DEBUG StatusLogger Shutdown hook enabled. Registering a new one.
DEBUG StatusLogger LoggerContext[name=4d405ef7, org.apache.logging.log4j.core.LoggerContext@5a10411] started OK.

And then only standard access logs, no logs from log4j2

OrangeFlag commented 5 years ago

Test different version On shadow 4.0.0 with TheBoegl plugin ver 2.2.0 - all is normal But when disable plugin - bug manifests itself

OrangeFlag commented 5 years ago

Something strange, now recheck with transform(com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer()) And it seems everything works. Maybe I just forgot to add a call to this transformer🤦

kolod commented 2 years ago

I have a similar problem. My Log4j2PluginsCache.dat file replaces the original file instead of being merged.

sample project

Shadow: 7.1.0 & 7.0.0
Product Version: Apache NetBeans IDE 12.5
Gradle: 7.0
Kotlin: 1.4.31
Groovy: 3.0.7
Ant: Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM: 1.8.0_202 (Oracle Corporation 25.202-b08)
OS: Windows 10 10.0 amd64

kolod commented 2 years ago

Build log

> Task :compileJava
Note: Processing Log4j annotations
Note: Annotations processed
Note: Processing Log4j annotations
Note: No elements to process
...
> Task :shadowJar
canTransformResource: META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat
canTransformResource: META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat

transform:
com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext(META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat, shadow.org.apache.tools.zip.ZipFile$1@2bddc88e, [], *******************
GRADLE SHADOW STATS

Total Jars: 1 (includes project)
Total Time: 0.0s [0ms]
Average Time/Jar: 0.0s [0.0ms]
*******************)
Add temp file: '/tmp/Log4j2Plugins7725904110337239769.dat'
relocators:
hasTransformedResource()
hasTransformedMultipleFiles  : false
hasAtLeastOneFileAndRelocator: false
hasTransformedResources      : false

The canTransformResource() method is called twice for only one file. But the transform() method is called once. As a result, hasTransformedResource() returns false.

kolod commented 2 years ago

I have moved the TextAreaAppender class to a separate package. And connected it as an external dependency. And it all worked.

The problem was that shadow looks for Log4j2Plugins.dat files only in dependencies and does not search in the project itself.

ppkarwasz commented 5 months ago

The Log4j2PluginsCacheFileTransformer seems to be affected by two bugs:

it does not transform Log4j2Plugins.dat files in your own project, but only those found in JAR files. The problems seems to be in the canTransformResource method:

https://github.com/johnrengelman/shadow/blob/9c5182d2d9c5f7141e4d2a525ec94d6111283cd9/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/transformers/Log4j2PluginsCacheFileTransformer.groovy#L57-L60

Since PLUGIN_CACHE_FILE is a path, the check should probably be against element.path, although I don't know the details of the Gradle API.
it deletes all Log4j2Plugins.dat files if there is only one such file (this is similar to apache/logging-log4j-transform#87). This problem was inherited from the original plugin:

https://github.com/johnrengelman/shadow/blob/9c5182d2d9c5f7141e4d2a525ec94d6111283cd9/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/transformers/Log4j2PluginsCacheFileTransformer.groovy#L84-L92

The correct logic is the simplest one !temporaryFiles.isEmpty().

When I have time to write tests, I could supply a PR, but this could take some time.

erathorus commented 5 months ago

it does not transform Log4j2Plugins.dat files in your own project, but only those found in JAR files. The problems seems to be in the canTransformResource method

I got this issue as well. I think the problem is when the Log4j2Plugins.dat file is in my own project, FileTreeElement is a DefaultFileCopyDetails object, which has the getName() method implemented as:

@Override
public String getName() {
    return getRelativePath().getLastName();
}

The implementation of getLastName() is:

public String getLastName() {
    if (segments.length > 0) {
        return segments[segments.length - 1];
    } else {
        return null;
    }
}

As a result, getLastName() will return Log4j2Plugins.dat, while PLUGIN_CACHE_FILE is META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat, hence the canTransformResource() check will fail.

dimabran commented 1 week ago

Is there any workaround or fix that does not require moving the plugin to a separate module?

GradleUp / shadow