justanhduc / task-spooler

A scheduler for GPU/CPU tasks
https://justanhduc.github.io/2021/02/03/Task-Spooler.html
GNU General Public License v2.0
273 stars 24 forks source link

Getting strange errors when trying to submit tasks #1

Closed kiklop74 closed 3 years ago

kiklop74 commented 3 years ago

Hello,

I am trying to utilize ts as a cpu task queue. Currently using ts 1.0 (since that is being distributed with Ubuntu). I have some code in PHP that periodically submits shell script to be executed as task in ts. The task does get submitted but process is immediately finished and stuff is written to socket error file.

PHP code example does this:

shell_exec('TS_SOCKET=/run/task-spooler/tsp.sock /usr/bin/tsp '/path/to/my/script.sh' 2>&1');

Task does get added, it is listed but always shows result -1 and socket error lists stuff like this:

-------------------Warning
 Msg: JobID 4 quit while running.
 errno 32, "Broken pipe"
date Thu Jan  7 13:52:04 2021
pid 721
type SERVER
New_jobs
  new_job
    jobid 4
    command "/tmp/stacks/specialclient/stack124/apply-all.sh"
    state running
    result.errorlevel 0
    output_filename "/tmp/ts-out.h0dQen"
    store_output 1
    pid 5186
    should_keep_finished 1
  new_job
    jobid 5
    command "/tmp/stacks/specialclient/stack124/output-all.sh"
    state queued
    result.errorlevel 0
    output_filename "NULL"
    store_output 1
    pid 0
    should_keep_finished 1

Do you have any hint or suggestion here of what am I doing wrong here?

kiklop74 commented 3 years ago

Ok, just figured it out. It seems we need to wait a bit before terminating the shell that connected to the TS server.

This makes it work:

shell_exec('TS_SOCKET=/run/task-spooler/tsp.sock /usr/bin/tsp '/path/to/my/script.sh' 2>&1 && /usr/bin/sleep 1s');
justanhduc commented 3 years ago

Nice. My two cent guess is that the shell is closed before some communication channel is closed properly that leads to the Broken pipe error. Thanks for sharing your solution!