search

apache / incubator-streampark

Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
https://streampark.apache.org/
Apache License 2.0
3.91k stars 1.01k forks source link

[Bug] When launching a job on a Kubernetes cluster on Windows using an IDE, the job status cannot be properly updated. #4112

Open FrommyMind opened 1 month ago

FrommyMind commented 1 month ago

Search before asking

Java Version

JDK1.8

Scala Version

2.12.x

StreamPark Version

  1. release-2.1.4
  2. release-2.1.5-rc1

Flink Version

1.17.2

deploy mode

kubernetes-session

What happened

After building Streampark, I started the console-server using the IDE, and then tried to start an Flink SQL Demo sample job in a Kubernetes session using Flink, but the status of the job was not updated correctly and remained "STARING". But on the Flink cluster Web UI, the job is already started and finally completed.

image

image

And the console log show that "tracking flink job status on kubernetes native session mode timeout"

10:26:49.972 [pool-10-thread-1] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] [FlinkJobStatusWatcher] tracking flink job status on kubernetes native session mode timeout, limitSeconds=120, trackIds=TrackId(kubernetes-session,pre-flink,dev-k8s,100000,19308092faf9eb315b31fcf01fe0fdd3,100000,{jobmanager.execution.failover-strategy=region, fs.s3a.secret.key=test-user, state.checkpoints.dir=s3://flink-checkpoint/checkpoints/, rest.bind-address=localhost, taskmanager.numberOfTaskSlots=1, jobmanager.rpc.address=localhost, state.savepoints.dir=s3://flink-checkpoint/savepoints/, taskmanager.bind-host=localhost, jobmanager.bind-host=localhost, jobmanager.memory.process.size=1600m, state.backend.incremental=true, parallelism.default=1, fs.s3a.endpoint=10.191.80.169:9001, containerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.17.2.jar, containerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.17.2.jar, rest.address=localhost, fs.s3a.list.version=1, jobmanager.rpc.port=6123, fs.s3a.connection.ssl.enabled=false, fs.s3a.access.key=test-user, taskmanager.host=localhost, taskmanager.memory.process.size=1728m, s3.path.style.access=true})

Then I tried to figure out what was going on.

After some debugging, I modified the code as follows.

  private def callJobsOverviewsApi(restUrl: String): Option[JobDetails] = {

  try {
    JobDetails.as(
      Request
        .get(s"$restUrl/jobs/overview")
        .connectTimeout(KubernetesRetriever.FLINK_REST_AWAIT_TIMEOUT_SEC)
        .responseTimeout(KubernetesRetriever.FLINK_CLIENT_TIMEOUT_SEC)
        .execute
        .returnContent()
        .asString(StandardCharsets.UTF_8)
    )
  }catch {
    case e: Throwable => logWarn(e.toString)
      None
  }
  }

The I found some exeception output.

10:37:42.660 [pool-10-thread-3] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] java.lang.NoClassDefFoundError: org/json4s/Formats
10:37:42.661 [pool-10-thread-3] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] java.lang.NoClassDefFoundError: org/json4s/Formats
10:37:47.660 [pool-10-thread-7] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] java.lang.NoClassDefFoundError: org/json4s/Formats

Error Exception

10:37:42.660 [pool-10-thread-3] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] java.lang.NoClassDefFoundError: org/json4s/Formats
10:37:42.661 [pool-10-thread-3] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] java.lang.NoClassDefFoundError: org/json4s/Formats
10:37:47.660 [pool-10-thread-7] WARN org.apache.streampark.flink.kubernetes.watcher.FlinkJobStatusWatcher - [StreamPark] java.lang.NoClassDefFoundError: org/json4s/Formats

Screenshots

No response

Are you willing to submit PR?

Code of Conduct