search

opensearch-project / opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.
Apache License 2.0
14 stars 22 forks source link

[BUG] FlintJob failed "Exception in thread "check-create-index-0" java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.JsonFactory" #442

Closed penghuo closed 2 weeks ago

penghuo commented 1 month ago

What is the bug?

Exception in thread "check-create-index-0" java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.JsonFactory com.fasterxml.jackson.core.JsonFactory.setStreamReadConstraints(com.fasterxml.jackson.core.StreamReadConstraints)'
    at org.opensearch.common.xcontent.json.JsonXContent.<clinit>(JsonXContent.java:84)
    at org.opensearch.common.xcontent.XContentType$1.xContent(XContentType.java:76)
    at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2234)
    at org.opensearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$13(RestHighLevelClient.java:1865)
    at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1935)
    at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1901)
    at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1865)
    at org.opensearch.client.IndicesClient.get(IndicesClient.java:768)
    at org.opensearch.flint.core.RestHighLevelClientWrapper.lambda$getIndex$7(RestHighLevelClientWrapper.java:99)
    at org.opensearch.flint.core.RestHighLevelClientWrapper.execute(RestHighLevelClientWrapper.java:154)
    at org.opensearch.flint.core.RestHighLevelClientWrapper.getIndex(RestHighLevelClientWrapper.java:99)
    at org.apache.spark.sql.OSClient.$anonfun$getIndexMetadata$1(OSClient.scala:47)
    at org.apache.spark.sql.OSClient.using(OSClient.scala:107)
    at org.apache.spark.sql.OSClient.getIndexMetadata(OSClient.scala:44)
    at org.apache.spark.sql.FlintJobExecutor.checkAndCreateIndex(FlintJobExecutor.scala:364)
    at org.apache.spark.sql.FlintJobExecutor.checkAndCreateIndex$(FlintJobExecutor.scala:362)
    at org.apache.spark.sql.JobOperator.checkAndCreateIndex(JobOperator.scala:24)
    at org.apache.spark.sql.JobOperator.$anonfun$start$2(JobOperator.scala:53)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
    at scala.util.Success.$anonfun$map$1(Try.scala:255)
    at scala.util.Success.map(Try.scala:213)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
    at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
    at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)

How can one reproduce the bug? submit EMR-S job --class org.apache.spark.sql.FlintJob

What is the expected behavior? no exception

What is your host/environment?

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

penghuo commented 1 month ago

Upgrade play-json failed

scala module 2.13.4 requires jackson databind version >= 2.13.0 and < 2.14.0 - found jackson-databind version 2.14

as spark 3.3.2 depend on jackson 2.13.4. https://github.com/apache/spark/blob/v3.3.2/pom.xml

    <fasterxml.jackson.version>2.13.4</fasterxml.jackson.version>
    <fasterxml.jackson.databind.version>2.13.4.2</fasterxml.jackson.databind.version>