spotify / scio

A Scala API for Apache Beam and Google Cloud Dataflow.
https://spotify.github.io/scio
Apache License 2.0
2.56k stars 513 forks source link

Coder: Mutation detector warning about structural values #4541

Closed f-loris closed 2 years ago

f-loris commented 2 years ago

I'm currently testing with 0.12-RC2 and in my tests I see a lot of messages from org.apache.beam.sdk.util.MutationDetectors are appearing. I could trace it down due to the usage of floating point number in my case classes.

Please see the following reproducer:

import com.spotify.scio.coders.{Coder, CoderMaterializer}
import org.apache.beam.sdk.util.MutationDetectors

case class Test(value: Double)

object CoderTest {

  implicit val testCoder: Coder[Test] = Coder.gen[Test]

  def main(args: Array[String]): Unit = {
    val value = Test(value = 3.9)
    val detector = MutationDetectors.forValueWithCoder(value, CoderMaterializer.beamWithDefault(testCoder))
    detector.verifyUnmodified()
  }

}

This produces:

WARN org.apache.beam.sdk.util.MutationDetectors - Coder of type class com.spotify.scio.coders.MaterializedCoder has a #structuralValue method which does not return true when the encoding of the elements is equal. Element Test(3.9)

I don't know if the root cause issue originates from scio or Apache Beam.

Would be great if you can take a look at that before releasing 0.12. Thanks a lot!

turb commented 2 years ago

Same here, on Apple Silicon / JDK 11 / Scala 2.13. Everything works, but the warning is worrying (all tests and jobs log it).

I have no warning on 0.11.10 with Beam forced to 0.41.

RustedBones commented 2 years ago

Thanks for reporting. I've modified the coder hierarchy and may have missed smth there