search

AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

TestMongoDbStreamWriterIntegration test fails on Mac M1 #277

Closed kevinwallimann closed 1 year ago

kevinwallimann commented 1 year ago

Description

TestMongoDbStreamWriterIntegration fails on a Mac M1 (ARM64) chip.

How to reproduce

Run mvn clean package -Pspark-3,scala-2.12. Fails with the following exception:

za.co.absa.hyperdrive.ingestor.implementation.writer.mongodb.TestMongoDbStreamWriterIntegration *** ABORTED ***
  java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: za.co.absa.hyperdrive.ingestor.implementation.writer.mongodb.TestMongoDbStreamWriterIntegration
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)
  at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
  at scala.collection.Iterator.foreach(Iterator.scala:943)
  at scala.collection.Iterator.foreach$(Iterator.scala:943)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  at scala.collection.IterableLike.foreach(IterableLike.scala:74)
  at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
  at scala.collection.TraversableLike.map(TraversableLike.scala:273)
  ...
  Cause: java.lang.IllegalArgumentException: this version does not support 32Bit: V4_0:OS_X:B32
  at de.flapdoodle.embed.mongo.Paths.getBitSize(Paths.java:157)
  at de.flapdoodle.embed.mongo.Paths.getPath(Paths.java:97)
  at de.flapdoodle.embed.process.store.LocalArtifactStore.getArtifact(LocalArtifactStore.java:71)
  at de.flapdoodle.embed.process.store.LocalArtifactStore.checkArtifact(LocalArtifactStore.java:38)
  at de.flapdoodle.embed.process.store.ArtifactStore.checkDistribution(ArtifactStore.java:65)
  at de.flapdoodle.embed.process.store.ExtractedArtifactStore.checkDistribution(ExtractedArtifactStore.java:60)
  at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:52)
  at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:45)
  at za.co.absa.hyperdrive.ingestor.implementation.testutils.mongodb.EmbeddedMongoDbSingleton$.startEmbeddedMongoDb(EmbeddedMongoDbSingleton.scala:53)
  at za.co.absa.hyperdrive.ingestor.implementation.testutils.mongodb.EmbeddedMongoDbSingleton$.embeddedMongoDb$lzycompute(EmbeddedMongoDbSingleton.scala:28)

Expected behaviour

Test should run without failure.

Suggested fixes

Upgrading de.flapdoodle.embed.mongo to >= 3.4.3 should resolve the issue, according to the comments here: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues/337

Workaround

Explicitly set System.setProperty("os.arch", "i686_64"); on mac m1 local machines (not for production!!)