search

FasterXML / jackson-databind

General data-binding package for Jackson (2.x): works on streaming API (core) implementation(s)
Apache License 2.0
3.52k stars 1.38k forks source link

ObjectMapper throws InvalidDefinitionException with JDK 9 #1939

Closed yrachid closed 3 years ago

yrachid commented 6 years ago

Jackson-databind JDK9 issue

When used with JDK 9, Jackson's ObjectMapper launches a com.fasterxml.jackson.databind.exc.InvalidDefinitionException while trying to read JSON content into a class that declares constructors with arguments. It works when the same is attempted with a class that declares a constructor with no arguments.

The issue 393 shows a very similar problem. The only difference is that in such issue the problem was related to Lombok annotations whereas in the current scenario the problem happens in both Pojos and Lombok annotated classes.

At the aforementioned issue's discussions someone suggested disabling the INFER_CREATOR_FROM_CONSTRUCTOR_PROPERTIES feature from the Mapper but that did not solve the current problem and the exception is still being thrown.

Still in that discussion, the @JsonCreator annotation was mentioned as a possible way to solve the problem. Such annotation was added on the cases with constructor arguments but the problem still persists (with or without @JsonCreator).

Test Scenarios

This repository was created as an attempt to reproduce the error.

There are four scenarios reproduced in the given repository:

Pojos:

Lombok generated constructors:

Given scenarios were run with and without applying the @JsonCreator annotation to the classes.

When running the tests, the following output is presented:

When @JsonCreator is applied:


ObjectMapperTest > lombok_all_args_fails FAILED
    com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Invalid type definition for type `lombok.PersonAllArgs`: Argument #0 has
no property name, is not Injectable: can not use as Creator [constructor for lombok.PersonAllArgs, annotations: {interface
com.fasterxml.jackson.annotation.JsonCreator=@com.fasterxml.jackson.annotation.JsonCreator(mode=DEFAULT)}]
     at [Source: (String)"{"name": "Joe", "age": 10}"; line: 1, column: 1]
        at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:62)
        at com.fasterxml.jackson.databind.DeserializationContext.reportBadTypeDefinition(DeserializationContext.java:1429)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitPropertyCreator(BasicDeserializerFactory.java:684)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitAnyCreator(BasicDeserializerFactory.java:715)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:465)
        at
com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:337)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:255)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:214)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:137)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
        at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
        at ObjectMapperTest.lombok_all_args_fails(ObjectMapperTest.java:46)

ObjectMapperTest > pojo_all_args_fails FAILED
    com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Invalid type definition for type `pojo.PersonAllArgs`: Argument #0 has no
property name, is not Injectable: can not use as Creator [constructor for pojo.PersonAllArgs, annotations: {interface
com.fasterxml.jackson.annotation.JsonCreator=@com.fasterxml.jackson.annotation.JsonCreator(mode=DEFAULT)}]
     at [Source: (String)"{"name": "Joe", "age": 10}"; line: 1, column: 1]
        at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:62)
        at com.fasterxml.jackson.databind.DeserializationContext.reportBadTypeDefinition(DeserializationContext.java:1429)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitPropertyCreator(BasicDeserializerFactory.java:684)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitAnyCreator(BasicDeserializerFactory.java:715)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:465)
        at
com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:337)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:255)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:214)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:137)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
        at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
        at ObjectMapperTest.pojo_all_args_fails(ObjectMapperTest.java:55)

When @JsonCreator is absent:


ObjectMapperTest > lombok_all_args_fails FAILED
    com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `lombok.PersonAllArgs` (no Creators, like
default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
     at [Source: (String)"{"name": "Joe", "age": 10}"; line: 1, column: 2]
        at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
        at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1451)
        at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1027)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1290)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
        at ObjectMapperTest.lombok_all_args_fails(ObjectMapperTest.java:46)

ObjectMapperTest > pojo_all_args_fails FAILED
    com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `pojo.PersonAllArgs` (no Creators, like
default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
     at [Source: (String)"{"name": "Joe", "age": 10}"; line: 1, column: 2]
        at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
        at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1451)
        at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1027)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1290)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
        at ObjectMapperTest.pojo_all_args_fails(ObjectMapperTest.java:55)

Reproducing

There are two ways to reproduce the error in the aforementioned repository:

Without Docker:

./gradlew test --info

With Docker:

Using OpenJDK 9.0.1 image

./reproduce-with-docker.sh

Information

The problem was produced under the following circumstances:

Jackson versions tested:

Operating System:

Linux Ubuntu 17.10
Kernel 4.13.0-32-generic

JDK Version:


$ java -version

java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

Gradle version:


------------------------------------------------------------
Gradle 4.2.1
------------------------------------------------------------

Build time:   2017-10-02 15:36:21 UTC
Revision:     a88ebd6be7840c2e59ae4782eb0f27fbe3405ddf

Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          9.0.4 (Oracle Corporation 9.0.4+11)
OS:           Linux 4.13.0-32-generic amd64

Sorry if this is the wrong place or if this issue was already reported. Would you have any advice on that?

Thanks in advance!

yrachid commented 6 years ago

Updates

After some further testing I found out that using the @JsonProperty annotation might solve the problem for the Pojo scenarios given that it will only work when applied to constructor parameters instead of class fields:

cowtowncoder commented 6 years ago

Just one note: although use of Jackson with Lombok should be fine as things are, all reproductions of problems need to be without use of Lombok itself -- just using annotations like Lombok pre-processor adds into byte code.

There are no known issues with Java 9 at this point (wrt 2.9.4, unit test suite) that I am aware of.

But one thing to note, wrt constructor/field/method annotations -- unless you also register Java 8 parameter names module, constructor parameter names are not available, and as such linkage can not be established. This is probably relevant here. So whereas fields have "implicit" name (name of field in bytecode) and possibly "explicit" name, override (from annotations), constructor parameters do not. This matters when combining individual accessors (ctor parameter, field, getter, setter) into single logical property, and combining information from all annotations, to avoid having to add multiple instances of same annotation.

yrachid commented 6 years ago

Thank you for the details. I have a few questions:

all reproductions of problems need to be without use of Lombok itself

In the repository I sent previously there are test cases where the classes do not use Lombok at all (please see pojo.PersonAllArgs class) but Lombok is still in the classpath of that project, do you believe that can be an issue?

just using annotations like Lombok pre-processor adds into byte code.

Do you mean that just by having Lombok in the classpath the results can be affected? Sorry for asking this kind of unrelated question, I was just wondering if you would know anything about it.

unless you also register Java 8 parameter names module, constructor parameter names are not available, and as such linkage can not be established.

In that same repository the Jdk8Module is registered in the ObjectMapper (see here), are you referring to that module specifically? Or is there another module to configure Java 8 parameter names?

I have another project (using JDK 8) that uses Lombok and Jacksons' JDK8Module and in that project the use of JsonCreator and JsonProperty annotations are not required, Jackson works fine without them even in conjunction with Lombok annotations like @RequiredArgsConstructor or @AllArgsConstructor and after trying to migrate that project to JDK 9, things stopped working. That's the reason why I believe this could be JDK 9 related.

Once again thank you very much for your attention! I will create another repository without having Lombok in the classpath to try reproducing the issue.

yrachid commented 6 years ago

Update

I was able to reproduce the problem in a project without Lombok as a dependency:

screenshot from 2018-02-23 12-13-13

Please see the project here.

cowtowncoder commented 6 years ago

What I meant was not that Lombok couldn't be somewhere, but rather that it can not be a required dependency by test(s). This because as far as I know, it is not enough to add a dependency, but additional jar has to be deployed with jdk, and I want to keep project free of any external dependencies beyond Maven. That is, when cloning git repo, all you need is Maven (and JDK for it to use), after which build should should succeed from clean slate. (I also in general try to minimize lib dependencies by core components, but in some cases new test dependencies may be added)

Thank you for reproduction!

MiWeiss commented 6 years ago

For reference: This seems to be the related issue in the Lombok project: https://github.com/rzwitserloot/lombok/issues/1563

maffe commented 4 years ago

unless you also register Java 8 parameter names module, constructor parameter names are not available, and as such linkage can not be established.

In that same repository the Jdk8Module is registered in the ObjectMapper (see here), are you referring to that module specifically? Or is there another module to configure Java 8 parameter names?

Yes, namely ParameterNamesModule.

tup916 commented 4 years ago

It seems to be a problem in java 11 as well.

sergey-morenets commented 4 years ago

It seems to be a problem in java 11 as well.

I was able to fix this issue with JDK 14 by adding @ConstructorProperties annotation.

cowtowncoder commented 4 years ago

Since core jackson-databind does not automatically find parameter names -- jackson-module-parameter-names (from jackson-modules-java8) is needed -- then one of following is needed:

  1. Use of @JsonCreator to indicate constructor to use, along with @JsonProperty -- or, @ConstructorProperties
  2. Move/re-create this issue for jackson-modules-java8, along with reproduction (possibly one that was linked to earlier -- but if possible, just inline minimal repro here)

As to jackson-databind, I think a better exception would make sense, if possible, to try to indicate problem better. But that may be challenging too, depending on where all information is available (wrt. potential candidates)

cowtowncoder commented 3 years ago

At this point not sure what could be done here: closing for now; may be re-filed with up-to-date reproduction with 2.11.x (or later).

xenoterracide commented 3 years ago

unless you also register Java 8 parameter names module

I just spent so much time on this... so much time... because I are dumb (or rather spoiled by spring boot) I was missing both the databind and the parameters module, is there any way the thing that throws this error could detect that they, or at least the latter is missing and throw a better error?

cowtowncoder commented 3 years ago

@xenoterracide I am open to improvements, but it is typically quite difficult to figure out something that might be needed is missing. This is problematic for another common problem case: javac not being configured to include parameter names in bytecode -- but since that alone may be intentional (older code compiled with older jdk; JDK itself I think leaves names out on purpose), missing names is not an error condition.

For 2.13 I will probably suggest upgrade of baseline to be Java 8 (same as what 3.0/master already requires) and if so, parameter names functionality could be included directly and not as module. That would probably reduce these issues.

xenoterracide commented 3 years ago

maybe the error message could just be improved? like "make sure you have -parameters enabled, and have the parameters-name-module` registered. I want paragraph long error messages! (or like, links to them?)

hfsugar commented 2 years ago

It seems to be a problem in java 11 as well.

Adding @ConstructorProperties annotation works in java 11 as well.