NullPointerException during JSON deserialization

[leonardehrenfried]

I'm seeing this very hard to reproduce NullPointerException when I'm trying to deserialize JSON with play-json into an enumeratum enum.

java.lang.NullPointerException: null
at enumeratum.Enum$$anonfun$lowerCaseNamesToValuesMap$1.apply(Enum.scala:47)
at enumeratum.Enum$$anonfun$lowerCaseNamesToValuesMap$1.apply(Enum.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at enumeratum.Enum$class.lowerCaseNamesToValuesMap(Enum.scala:47)

It appears the the value of lowerCaseNamesToValuesMap is null. My guess is that it is connected to the variable being declared lazy.

If I access the variable lowerCaseNamesToValuesMap before the deserilisation the problem goes away. I used to only see this in test code but now also in production code.

Have you seen anything like this before?

[lloydmeta]

Strange, I've not seen this in my usage of Enumeratum per se, but I have seen something similar when I've had cyclic dependencies between different objects. I don't remember the specifics, but I do remember it was hard to track down because it was non-obvious, but I think I resolved the problem by using lazy val though. I think it's unlikely that there's a NPE bug inherent to the usage of lazy val because it's such a core part of the language..

Can you put up a minimal example that reproduces the problem so we can take a look together?

[leonardehrenfried]

I'm going to try to extract a minimal example from my large-ish application. Not sure when I will get around to doing this.

[sergey-tikhonenko]

Hello, I've faced with the same error, so far hard to determine what part introduces the error. BTW I've managed to extract a minimal example to reproduce, might be some are excessive...

Could you look at the test in the attachment? error-example.tar.gz

[lloydmeta]

@sergey-tikhonenko can you throw that in a repo? Much easier to take a look then.

[sergey-tikhonenko]

Please, look at https://github.com/sergey-tikhonenko/org.sandbox.enumeratum

[lloydmeta]

Cool, thanks for that.

I think the culprit is the same as in #138 and #144; classic circular dependency going on between companion object and class init. I've submitted sergey-tikhonenko/org.sandbox.enumeratum#1 which demonstrates how this can be addressed.

[sergey-tikhonenko]

I see what is the cause. Thanks for pointing to right direction.

[mlvn23]

I see the null exception with Jackson and deadlocks on the others when I run many tests in parallel that uses the case objects directly (with mixed Scala and Java code - not sure if that's related). If I assign the case objects to val declarations and use that in my tests, then the problem goes away. I'm using a 14-core CPU so it's pretty easy to reproduce. This is true especially when the enum has a large number of entries and it's probably related to the way Scala instantiates objects.

[lloydmeta]

@mlvn23 interesting; would you happen to have any dependencies (references to each other) amongst the different enums? In any case, if you could share a minimised example, it would be very helpful :)

I have a project where there are ~3000 enum members for one of the enums (don't ask) and haven't gotten any NPEs in parallel testing or in prod. That said, this is a Play project using Play-JSON.

[yellowstonesoftware]

I'm encountering the same problem.

I also encountered a variation whereby a val I was declaring in my PlayEnum object also consistently had a null value in findValue (seemingly always the same one), I managed to work around this by declaring it lazy:

    lazy val displayValues = findValues.foldLeft(SortedMap.empty[Category, List[Category]](DisplayOrdering)) { (z, b) =>
      b.parentCategory match {
        case Some(p) =>
          z + (p -> (z.get(p).getOrElse(List.empty[Category]) :+ b))
        case None =>
          z
      }
    }.mapValues(v => v.sortBy(_.entryName))

As the OP mentioned, if I access Enum.namesToValuesMap and Enum.lowerCaseNamesToValuesMap beforehand (I do it on startup in an eager singleton). If I don't, the first time I submit a form and trying to parse the form into a domain object using Enum.withName(name) it throws the NullPointerException. Curiously this worked fine until I've reworked this particular enumeration a bit, adding some properties and additional enumeration constants (case object)

[bigmoby]

The problem I think is strictly coupled with supplemental params added in a enum class and the macro applyed with findValues. I solved the issue declaring lazy findValue method.

[cchepelov]

Encountered this. The problem was triggered under a large ScalaTest suite.

It seems the scenario includes two threads racing on the initialization of SomeEnum$, SomeEnum$.values, and the various instances of SomeEnum$.Foo$, SomeEnum$.Bar$, SomeEnum$.Etc$. The problem is that when the various Foo/Bar/Etc values of SomeEnum are initialized by a different thread, the findValues macro might not see the value of Foo$.MODULE$ and accidentally use a null instead.

Happened under scala 2.12.8 on two distinct multicore Intel processors (i7-7700HQ which is 4C8T, and another of a similar vintage which is 2C4T).

Not sure whether the "lazy val" causes a proper read barrier to be in place to guarantee the visibility of the other thread's initialization of SomeEnum$.Foo$, or whether that invariant would hold in scala 2.13.0 (not in position to test/reproduce under 2.13 at this moment) or whether "lazy val" just causes enough delay that it now works by luck, but so far going with this.

Suggestion: the implementation of the findValues might want to strive to ensure the appropriate read barrier is in place before attempting to read each of the values' respective MODULE$ static variables. Possibly, this happens in https://github.com/lloydmeta/enumeratum/blob/master/macros/src/main/scala/enumeratum/EnumMacros.scala#L160 but might require a JVM-specific hack to locate the appropriate thing to synchronize on?