search

awslabs / spark-sql-kinesis-connector

Spark Structured Streaming Kinesis Data Streams connector supports both GetRecords and SubscribeToShard (Enhanced Fan-Out, EFO)
Apache License 2.0
22 stars 9 forks source link

Getting error when I reduce executor count. #19

Open MJFND opened 1 month ago

MJFND commented 1 month ago

I am using 4 executors with the following config:

        "cores"  = 3
        "memory" = "22g"

It works fine for 4 executors, but when I reduce the count to 3 it starts to throw the following:

 at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:332)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$1(StreamExecution.scala:211)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:211)
Caused by: org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
        at org.apache.spark.SparkException$.internalError(SparkException.scala:107)
        at org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:536)
        at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:548)
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:219)

Seems like a bug in the plugin.

hwanghw commented 1 month ago

can you upload the full stack trace?

MJFND commented 1 month ago

Will send, it seems like the issue is resolved by deleting the checkpoint directory, I will keep posted if I see same behavior. Thanks

On Thu, May 23, 2024, 7:22 PM hwanghw @.***> wrote:

can you upload the full stack trace?

— Reply to this email directly, view it on GitHub https://github.com/awslabs/spark-sql-kinesis-connector/issues/19#issuecomment-2128184734, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC65XLX2VA3GU2H3BLQ6ZJTZDZ22RAVCNFSM6AAAAABIGNFVNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRYGE4DINZTGQ . You are receiving this because you authored the thread.Message ID: @.***>