search

spark-jobserver / spark-jobserver

REST job server for Apache Spark
Other
2.84k stars 1k forks source link

streaming job is still in progress after killing it #690

Open uladzimir-shelhunou opened 7 years ago

uladzimir-shelhunou commented 7 years ago

Hi,

I am using spark-2.0-preview build.

I started streaming job, after some time I killed this job and job server ui showed my job as killed, but I see that job is still in progress as it still retrieves the information and performs my business logic.

Is it a known issue?

Thanks, Uladzimir

maxosprojects commented 7 years ago

Try to kill the context after killing the job.

DELETE /jobs/<jobId>
DELETE /contexts/<name>
noorul commented 7 years ago

@vvsh Are you using context-per-jvm = true in your config?

uladzimir-shelhunou commented 7 years ago

@noorul, yes, I am.

uladzimir-shelhunou commented 7 years ago

@maxosprojects, I am using dynamic context, not a single one (I am running a lot of streaming jobs).

bsikander commented 6 years ago

Any update on this ? @noorul DELETE /jobs/ does not work. It says that the job was killed, this is also written to database but the job is actually running.

You can reproduce this issue on current master.

velvia commented 6 years ago

When you say the job is still running, do you mean in Spark UI it still shows as alive? Or is it dead but the process is still around?

On Aug 23, 2017, at 6:57 AM, Behroz Sikander notifications@github.com wrote:

Any update on this ? DELETE /jobs/ does not work. It says that the job was killed, this is also written to database but the job is actually running.

You can reproduce this issue on current master.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/spark-jobserver/spark-jobserver/issues/690#issuecomment-324340623, or mute the thread https://github.com/notifications/unsubscribe-auth/ABA32wVm-IPCpIZuexLiV3NyjnQNaZkpks5sbC_PgaJpZM4LWZf_.

bsikander commented 6 years ago

I was doing the experimentation locally (so no Spark involved and no context-per-jvm). The Spark Jobserver UI showed that the job has been killed (in the failed table of UI) but the Driver was still returning output.

I was running a Streaming context.

velvia commented 6 years ago

Oh. There is a known issue with streaming contexts. They basically can’t be killed unless you use a separate process (context-per-ivm).

On Aug 24, 2017, at 12:25 AM, Behroz Sikander notifications@github.com wrote:

I was doing the experimentation locally (so no Spark involved and no context-per-jvm). The Spark Jobserver UI showed that the job has been killed (in the failed table of UI) but the Driver was still returning output.

I was running a Streaming context.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spark-jobserver/spark-jobserver/issues/690#issuecomment-324555614, or mute the thread https://github.com/notifications/unsubscribe-auth/ABA32431EGcy-ygATc8RZvjSUeDZ3TTTks5sbSVWgaJpZM4LWZf_.

bsikander commented 6 years ago

Job cannot be killed even with context-per-jvm=true. Also see the comment by @vvsh in this PR. He was also using context-per-jvm=true.

bsikander commented 5 years ago

Did anybody found a solution for this?

vglagoleva commented 3 years ago

The scenario should be tested (streaming job, context-per-jvm=true, try to stop the job) and if this is a limitation of Spark, then we should add it into the documentation.

AreRex14 commented 2 years ago

Is the streaming job java or python?