gpu 任务管理 -- task spooler

用法: 加入任务：ts xxx

python test_global_ss.py --test=raw_loss_weight

查看任务：ts -l

ID   State      Output               E-Level  Times(r/u/s)   Command [run=1/1]
23   running    /tmp/ts-out.C0KCTc                           python edge_segmentation.py --test=last_field_of_view
24   queued     (file)                                       python test_global_ss.py --test=global_type
25   queued     (file)                                       python test_global_ss.py --test=raw_loss_weight
17   finished   /tmp/ts-out.Sps3GT   0        17625.35/22538.54/684.22 python imagenet_ss.py
19   finished   /tmp/ts-out.kk7SVM   0        17120.36/22011.75/599.55 python imagenet_ss.py --test=activation
20   finished   /tmp/ts-out.eLLlu9   0        8701.79/11326.12/370.11 python edge_ss.py
21   finished   /tmp/ts-out.GdZ1sL   0        35721.01/46083.40/1648.31 python edge_segmentation.py --test=edge_width
22   finished   /tmp/ts-out.9tROXc   0        35714.44/46234.66/1464.04 python edge_segmentation.py --test=edge_loss_weight

查看任务输出：tail -f /tmp/ts-out.C0KCTc

(new) ➜  ~ tail -f /tmp/ts-out.C0KCTc
55/65 [========================>.....] - ETA: 42s - loss: 0.4320 - seg_loss: 0.5955 - edge_loss: 0.2685 - seg_switch_mean_iou: 3.6293e-05 - seg_precision: 0.8140 - seg_recall: 0.8412 - seg_fmeasure: 0.7802 - edge_switch_mean_iou: 4.6049e-04 - edge_precision: 0.8774 - edge_recall: 0.8774 - edge_fmeasure: 0.8299
56/65 [========================>.....] - ETA: 38s - loss: 0.4322 - seg_loss: 0.5958 - edge_loss: 0.2687 - seg_switch_mean_iou: 3.6145e-05 - seg_precision: 0.8139 - seg_recall: 0.8410 - seg_fmeasure: 0.7801 - edge_switch_mean_iou: 4.6246e-04 - edge_precision: 0.8774 - edge_recall: 0.8774 - edge_fmeasure: 0.8299
57/65 [=========================>....] - ETA: 34s - loss: 0.4323 - seg_loss: 0.5958 - edge_loss: 0.2688 - seg_switch_mean_iou: 3.5992e-05 - seg_precision: 0.8139 - seg_recall: 0.8411 - seg_fmeasure: 0.7801 - edge_switch_mean_iou: 4.6477e-04 - edge_precision: 0.8774 - edge_recall: 0.8772 - edge_fmeasure: 0.8298

查看帮助：ts -h

➜  ~ ts -h
usage: ts [action] [-ngfmdE] [-L <lab>] [-D <id>] [cmd...]
Env vars:
  TS_SOCKET  the path to the unix socket used by the ts command.
  TS_MAILTO  where to mail the result (on -m). Local user by default.
  TS_MAXFINISHED  maximum finished jobs in the queue.
  TS_MAXCONN  maximum number of ts connections at once.
  TS_ONFINISH  binary called on job end (passes jobid, error, outfile, command).
  TS_ENV  command called on enqueue. Its output determines the job information.
  TS_SAVELIST  filename which will store the list, if the server dies.
  TS_SLOTS   amount of jobs which can run at once, read on server start.
  TMPDIR     directory where to place the output files and the default socket.
Actions:
  -K       kill the task spooler server
  -C       clear the list of finished jobs
  -l       show the job list (default action)
  -S [num] get/set the number of max simultaneous jobs of the server.
  -t [id]  "tail -n 10 -f" the output of the job. Last run if not specified.
  -c [id]  like -t, but shows all the lines. Last run if not specified.
  -p [id]  show the pid of the job. Last run if not specified.
  -o [id]  show the output file. Of last job run, if not specified.
  -i [id]  show job information. Of last job run, if not specified.
  -s [id]  show the job state. Of the last added, if not specified.
  -r [id]  remove a job. The last added, if not specified.
  -w [id]  wait for a job. The last added, if not specified.
  -k [id]  send SIGTERM to the job process group. The last run, if not specified.
  -u [id]  put that job first. The last added, if not specified.
  -U <id-id>  swap two jobs in the queue.
  -B       in case of full queue on the server, quit (2) instead of waiting.
  -h       show this help
  -V       show the program version
Options adding jobs:
  -n       don't store the output of the command.
  -E       Keep stderr apart, in a name like the output file, but adding '.e'.
  -g       gzip the stored output (if not -n).
  -f       don't fork into background.
  -m       send the output by e-mail (uses sendmail).
  -d       the job will be run only if the job before ends well
  -D <id>  the job will be run only if the job of given id ends well.
  -L <lab> name this task with a label, to be distinguished on listing.
  -N <num> number of slots required by the job (1 default).

ISCAS007 / PaperReading

gpu 任务管理 -- task spooler #4

pueue