JDK17 compatibility

[nmandrescu]

Problem

JEP 403 targeted for JDK 17 (release planned for Sept 2021) will remove --illegal-access flag. That will be equivalent to --illegal-access=deny.

Running with --illegal-access=warn apps using jackson-databind and other jackson modules reported here https://github.com/FasterXML/jackson-databind/issues/2856#issuecomment-700581797 can discover issues like WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil

Workaround

For illegal reflective access pointing to jackson use --add-open for the package containing the restricted field.

Update Not related to jackson library, since JDK 9 a number of standard replacements are provided for the most commonly used internal elements of the JDK.

[cowtowncoder]

Unfortunately most if not all remaining issues that surface with that rather non-descriptive warning are due to dynamic introspection for various JDK types and depend on actual usage. Jackson core itself does not force access to JDK, but to access fields, getters, setters, constructors, introspection is used. ClassUtil specifically gets called to try to force access when it is deemed necessary based on types user tries to serialize/deserialize; it has no "agenda of its known".

So unfortunately this is not as easy as (for example) replacing use of sun.misc.Unsafe with alternatives.

So what tends to be necessary are user reports for specific usage that triggers these issues: this helps figure out how to prevent specific problems. I am not confident there is a general simple solution; although more and more general improvements can be developed when finding patterns common to reported problems.

[nmandrescu]

@cowtowncoder, thank you for clarification, fully agree! I was about to update the description that this is not solvable by jackson itself.

A way to deal with it is to expose such illegally accessed packages with --add-opens.

[cowtowncoder]

@nmandrescu Agreed! And this is one area that I really hope we can get improvements on, from better diagnostics to tooling perhaps. One thing that has been unfortunate is that from what I have seen, stack traces do not always seem to identify context in which problem occurs: while it is ClassUtil that tries to apply operation, it is the target class and (parts of) stack trace that really matter. I realize that it may not be something JDK can easily provide, but maybe there are settings that would let developers get more information.

[GedMarc]

This is something that I am specifically looking at -

The problem is closer to usage, than implementation though, through devious attempts of resolution -

1) Field Accessors to JDK Internals is banned across the board, this is correct according to the principle of encapsulation. This issue is only specific to referencing JDK Internal Value Objects, you must use Property fetch and Set. As value objects though, these don't bring back the final result, only whats necessary to calculate the final result -> LocalDateTime/OffsetDateTime best example here 1,a) Our final result ended up in us simply registering a serializer and deserializer for each of the value objects in the JDK, and handling it properly by property, getters and setters 1.b) The list of value objects in JDK is not exhaustive, setting all objects found in package java. to find by property only removes all invalid encapsulation requests

2) Kotlin so-far does not seem to be adopting JDK 9 and up, and are following the JDK 8 pathway, going non-modular (even on higher jdk's making a move to JDK 11 very hard, JDK 17 is completely impossible for them with the block). This library would have the largest impact from forcing modularizaiton

3) Version 3.0 is under way with Jakarta specific releases for anything I've found on that front, and once finalized will port across. Jakarta EE 9.1 is still not even JDK 9 compatible which is a huge hammer in a lot of people plans, and are so far away from being 11 compatible , that 16 or 17 right now for them seems out of reach for at least 2 years. (VERY ARRRRGGGGGHHHHH)

For now I think Jackson caters perfectly for modularization, and adheres to all encapsulation limitations placed - The user however must be aware of the new encapsulation rules (which are correct) and bend accordingly to them.

We can perhaps improve on the time-jsr a bit more to assist users move across properly, but right now, I am looking at the jakarta module split, and seeing what's happening in Version 3.

Outcome Analysis - User Issue

[cowtowncoder]

Good overview @GedMarc. One specific area where we can probably make progress is to start preventing what must be illegal accesses into java.* types (and probably javax., if any remain). And then as you pointed out trying to cover more of JDK types that formerly "happened to work", f.ex by using non-public single-String constructors, based on user reports (or active searching if feasible).

[nmandrescu]

If it helps and possible to do anything about, here are some java.* samples:

WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.reflect.Proxy.h
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.ref.Reference.referent
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.ref.Reference.queue
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.ref.Reference.next

For javax.*

WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field javax.naming.InitialContext.myProps
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field javax.naming.InitialContext.defaultInitCtx
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field javax.naming.InitialContext.gotDefault

[cowtowncoder]

Thank you @nmandrescu! It'd be great to know the path through which these came about (for reproduction), but this is helpful as-is.

[amCap1712]

Not sure if this is helpful but you can pass --illegal-access=debug on the command line to print a stack trace along with the message for each illegal access.

[JJBRT]

Please take a look at my new article that explain how to export all modules to all modules at runtime in Java 16 and later without using any JVM parameter

[GedMarc]

Look at the new frameworks for how write modular and encapsulated applications, something like guicedee.com, that works completely on proper encapsulation. What is the point - best that you'll do is force oracle to lock it down even more, load up proxy classes in anonymous modules and force 'exports' for all packages because of this? you can't modify a module chain once in JRT!! it is a read-only vfs!

You don't "Solve" this problem with --add-exports and add-modules, or any other temporary work around in order to give people time to adapt, you Adapt to the shift in language by just doing it properly, so you can get all the benefits from it, which far out way and out perform any classpath-based application, by miles.

I apologize for the below tone but..... in order to get my facial expression across xD (in jest please) No. Hard No, Definitely. No. I'm sorry - No. insert meme here NO!

[AISCo]

Could these details help me in my specific case: [ERROR] WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:/home/user/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.2/jackson-databind-2.10.2.jar) to constructor java.util.Optional() [ERROR] at com.fasterxml.jackson.databind.util.ClassUtil.checkAndFixAccess(ClassUtil.java:939) [ERROR] at com.fasterxml.jackson.databind.deser.impl.CreatorCollector._fixAccess(CreatorCollector.java:263) [ERROR] at com.fasterxml.jackson.databind.deser.impl.CreatorCollector.setDefaultCreator(CreatorCollector.java:123) [ERROR] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:383) [ERROR] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:283) [ERROR] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:224) [ERROR] at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:220) [ERROR] at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) [ERROR] at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:443) [ERROR] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.createContextual(MapDeserializer.java:262) [ERROR] at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:650) [ERROR] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:484) [ERROR] at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.resolve(SuperSonicBeanDeserializer.java:79) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) [ERROR] at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:476) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4389) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4198) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3205) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3173) [ERROR] at com.application.commun.echange.utils.JsonUtils.deserializerJSON(JsonUtils.java:53)

with this sample code: public static <T> T deserializerJSON(final String jsonMessage, final Class<T> classObject) { try { final var mapper = new ObjectMapper(); mapper.registerModule(new AfterburnerModule()); mapper.configure(FAIL_ON_UNKNOWN_PROPERTIES, false); return mapper.readValue(jsonMessage, classObject); } catch (final IOException e) { throw new TechnicalException(e); } }

the error is on this line : return mapper.readValue(jsonMessage, classObject);

[sruffatti]

So do we support JDK 17 now?

[GedMarc]

Yep - including full support for jpms

[nmandrescu]

Thank you! We've used successfully several jackson-databind newer releases with JDK17. No more issues reported. Closing.

[Aksh125]

Can Anyone Help Me to resolved thw following error I have migrated Spring Boot application from jdk 11 to jdk 17. I got the error java.lang.illegalAccessError for jackson Module afterburner

[JJBRT]

Can Anyone Help Me to resolved thw following error I have migrated Spring Boot application from jdk 11 to jdk 17. I got the error java.lang.illegalAccessError for jackson Module afterburner

The fastest solution is to export all modules to all modules at runtime

[GedMarc]

@Aksh125 please provide a little more detail

[Roberto-Gentili]

@Aksh125 please provide a little more detail

Please stop down-voting working solutions even if they use some hack because they might be interpreted as broken solutions. And very often people don't have time to fix problems caused by new implementations of the jdk

[GedMarc]

@Roberto-Gentilli As the developer for the modularization of the Jackson library and assisting with the usage of modules, unfortunately no - Giving the impression to say, new developers, that that process is acceptable is giving the wrong guiding and direction entirely - To override all module links and dependencies in a JDK above JDK9, the solution is not valid - but I appreciate your constructive feedback :)

Reference : https://stackoverflow.com/questions/68867895/in-java-17-how-do-i-avoid-resorting-to-add-opens

[Roberto-Gentili]

@Roberto-Gentilli As the developer for the modularization of the Jackson library and assisting with the usage of modules, unfortunately no - To override all module links and dependencies in a JDK above JDK9, the solution is not valid - but I appreciate your constructive feedback :)

You can tell people that there are better solutions, but if you vote against a solution it might appear that the solution doesn't work. I can assure that the solution proposed by @JJBRT works perfectly (except on Android) and I can also assure you that people's frustration when they see that their software no longer works with the new jdk implementation is often very high

[Aksh125]

@GedMarc @Roberto-Gentili when my spring boot application running on JDK 11 its show the warning Warning: illegal reflective access by com.fasterxml.jackson.module.afterburner.util.myclassloader And When I Updated to JDK 17 it get the error
java.lang.illegalAccessError My Springboot version is 2.6.5

[GedMarc]

@Aksh125 A stack trace please, and which version of Jackson are you using? :)

[GedMarc]

@Aksh125 can you please log it against the afterburner project?

[Roberto-Gentili]

@GedMarc @Roberto-Gentili when my spring boot application running on JDK 11 its show the warning Warning: illegal reflective access by com.fasterxml.jackson.module.afterburner.util.myclassloader And When I Updated to JDK 17 it get the error java.lang.illegalAccessError My Springboot version is 2.6.3

If you need a fast solution and you are not developing on Android you need only to add this to your pom:

<dependency>
    <groupId>org.burningwave</groupId>
    <artifactId>core</artifactId>
    <version>12.62.0</version>
</dependency>

... And then you need only to call the method org.burningwave.core.assembler.StaticComponentContainer.Modules.exportAllToAll() in a static block of your main class e.g.:

public class MainClass {

    static {
        org.burningwave.core.assembler.StaticComponentContainer.Modules.exportAllToAll();
    }

    public static void main(String[] args) {
        ... Do the stuff
    }
}

if you see a lot of logs, they can be disabled

[GedMarc]

and only if it's not enterprise, and if it is enterprise, only if the burningwave is on the approved dependencies list for that organisation... and you are still circumventing standard Java JPMS so, from my perspective, not a solution

[Roberto-Gentili]

and only if it's not enterprise, and if it is enterprise, only if the burningwave is on the approved dependencies list for that organisation... and you are still circumventing standard Java JPMS so, from my perspective, not a solution

if this is a solution to @Aksh125's problem, I'd let him decide And if the jackson library is throwing these errors, it's doing non-jdk compliant operations, so some moralizing should be avoided

[GedMarc]

Rather just learn how JPMS works and just fix it, Classpath mode is fading away - all of these -- hacks --- at one point will stop working, each Java major release adds another layer of modular security, and the benefit's of modularization far outweighs the push-back against it. When looking for support, you want solutions, not hack-patches that will only work for x amount of time.

[Roberto-Gentili]

When looking for support, you want solutions, not hack-patches that will only work for x amount of time.

Ok, you made it clear, but let him decide which solution to adopt especially when your library is the first to use illegal reflection operations

[GedMarc]

Only on private field-access properties - which is against strict-encapsulation, but we want to keep that option available for backwards compatibility...... But everything can be resolved, in a proper, viable and operational way, without breaking or abusing the internals of Java to get "JDK8"-like functionality back in JDK's that are modular by definition,

It could just be as simple as an opens clause in their module-info file, or proxying the classes referenced from afterburner....

[Vishal-0319]

Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @1d8b0500

Getting the above error while upgrading from jdk 11 to 17. Could you please help here?

[pjfanning]

When calling Java command, you can add an add-opens option. See https://docs.oracle.com/javase/9/migrate/toc.htm#JSMIG-GUID-7BB28E4D-99B3-4078-BDC4-FC24180CE82B

[cowtowncoder]

@Vishal-0319 that error message would not be enough to help, fwtw, as it does not show which class is being problematic (or reproduction to figure it out). You also did not include Jackson version. Going forward it'd make sense to add more information as well as file a separate issue.

[Vishal-0319]

I will try out and get back with the info if it doesnt workout, thanks.