golemfactory / clay

Golem is creating a global market for computing power.
https://golem.network
GNU General Public License v3.0
2.91k stars 286 forks source link

golemcli tasks subtasks list is not working for tasks in timeout #5134

Closed ederenn closed 4 years ago

ederenn commented 4 years ago

Description

Golem Version: any 22.1 based

Golem-Messages version (leave empty if unsure):

Electron version (if used):

OS [e.g. Windows 10 Pro]: any

Branch (if launched from source):

Mainnet/Testnet: mainnet

Priority label is set to the lowest by default. To setup higher priority please change the label P0 label is set for Severity-Critical/Effort-easy P1 label is set for Severity-Critical/Effort-hard P2 label is set for Severity-Low/ Effort-easy P3 label is set for Severity-Low/Effort-hard

Description of the issue:

After task has finished in timeout it is not possible to list subtasks of a task from cli

ederenn@ubuntu:~$ golemcli tasks subtasks list 37ffacd4-57cd-11ea-ada7-170b16b9d875
ERROR [golem_rpc_api::rpc] on comp.task.subtasks unable to parse: [Array([Object({"subtask_id": String("4677c670-57cd-11ea-96ae-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582634975), "node_id": String("6ed84622e32b638f6def153fdf919679ae025908ce67a3a534e06af39227bd226f60ca8e072d64fb763dc994a460df367244f9fd41b666808b966dfa4e359457"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(1)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(1), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_1"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.8325), Number(1.0)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/4677c670-57cd-11ea-96ae-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/4677c670-57cd-11ea-96ae-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_10001.png")])}), Object({"subtask_id": String("467f60b8-57cd-11ea-9d69-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582634975), "node_id": String("6d33259d75da2ed9ddf1b8c131a520d998362f338a2e337920053718209674e9a4916d568a81a6f60d39f3b3997c4fd3d2e3ba54e38ce1af249f7811e30b708c"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(1)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(2), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_2"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.665), Number(0.8325)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/467f60b8-57cd-11ea-9d69-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/467f60b8-57cd-11ea-9d69-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_20001.png")])}), Object({"subtask_id": String("6706a442-57cd-11ea-9116-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635029), "node_id": String("6ed84622e32b638f6def153fdf919679ae025908ce67a3a534e06af39227bd226f60ca8e072d64fb763dc994a460df367244f9fd41b666808b966dfa4e359457"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(1)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(3), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_3"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.4975), Number(0.665)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/6706a442-57cd-11ea-9116-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/6706a442-57cd-11ea-9116-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_30001.png")])}), Object({"subtask_id": String("762385de-57cd-11ea-a1f4-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635055), "node_id": String("6d33259d75da2ed9ddf1b8c131a520d998362f338a2e337920053718209674e9a4916d568a81a6f60d39f3b3997c4fd3d2e3ba54e38ce1af249f7811e30b708c"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(1)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(4), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_4"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.33), Number(0.4975)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/762385de-57cd-11ea-a1f4-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/762385de-57cd-11ea-a1f4-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_40001.png")])}), Object({"subtask_id": String("88a1a2c2-57cd-11ea-98ee-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635086), "node_id": String("6ed84622e32b638f6def153fdf919679ae025908ce67a3a534e06af39227bd226f60ca8e072d64fb763dc994a460df367244f9fd41b666808b966dfa4e359457"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(1)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(5), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_5"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.165), Number(0.33)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/88a1a2c2-57cd-11ea-98ee-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/88a1a2c2-57cd-11ea-98ee-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_50001.png")])}), Object({"subtask_id": String("96ed21e8-57cd-11ea-8436-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635110), "node_id": String("6d33259d75da2ed9ddf1b8c131a520d998362f338a2e337920053718209674e9a4916d568a81a6f60d39f3b3997c4fd3d2e3ba54e38ce1af249f7811e30b708c"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(1)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(6), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_6"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.0), Number(0.165)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/96ed21e8-57cd-11ea-8436-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/96ed21e8-57cd-11ea-8436-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_60001.png")])}), Object({"subtask_id": String("a111b962-57cd-11ea-9651-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635127), "node_id": String("6ed84622e32b638f6def153fdf919679ae025908ce67a3a534e06af39227bd226f60ca8e072d64fb763dc994a460df367244f9fd41b666808b966dfa4e359457"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(2)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(7), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_7"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.8325), Number(1.0)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/a111b962-57cd-11ea-9651-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/a111b962-57cd-11ea-9651-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_70002.png")])}), Object({"subtask_id": String("b551e80c-57cd-11ea-a6fe-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635161), "node_id": String("6d33259d75da2ed9ddf1b8c131a520d998362f338a2e337920053718209674e9a4916d568a81a6f60d39f3b3997c4fd3d2e3ba54e38ce1af249f7811e30b708c"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(2)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(8), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_8"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.665), Number(0.8325)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/b551e80c-57cd-11ea-a6fe-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/b551e80c-57cd-11ea-a6fe-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_80002.png")])}), Object({"subtask_id": String("c0117328-57cd-11ea-ba52-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635179), "node_id": String("6ed84622e32b638f6def153fdf919679ae025908ce67a3a534e06af39227bd226f60ca8e072d64fb763dc994a460df367244f9fd41b666808b966dfa4e359457"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(2)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(9), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_9"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.4975), Number(0.665)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/c0117328-57cd-11ea-ba52-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/c0117328-57cd-11ea-ba52-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_90002.png")])}), Object({"subtask_id": String("de75d4f6-57cd-11ea-8fe2-170b16b9d875"), "progress": Number(1.0), "time_started": Number(1582635230), "node_id": String("6ed84622e32b638f6def153fdf919679ae025908ce67a3a534e06af39227bd226f60ca8e072d64fb763dc994a460df367244f9fd41b666808b966dfa4e359457"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(2)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(10), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_10"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.33), Number(0.4975)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Finished"), "stdout": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/de75d4f6-57cd-11ea-8fe2-170b16b9d875/stdout.log"), "stderr": String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/de75d4f6-57cd-11ea-8fe2-170b16b9d875/stderr.log"), "results": Array([String("/home/ederenn/.local/share/golem/default/rinkeby/ComputerRes/37ffacd4-57cd-11ea-ada7-170b16b9d875/tmp/small_shark100_100002.png")])}), Object({"subtask_id": String("de7fe34c-57cd-11ea-b85e-170b16b9d875"), "progress": Number(0.0), "time_started": Number(1582635230), "node_id": String("6d33259d75da2ed9ddf1b8c131a520d998362f338a2e337920053718209674e9a4916d568a81a6f60d39f3b3997c4fd3d2e3ba54e38ce1af249f7811e30b708c"), "node_name": String(""), "deadline": Number(1582635250), "price": Number(100000000000000000), "extra_data": Object({"scene_file": String("/golem/resources/small_shark100.blend"), "resolution": Array([Number(400), Number(400)]), "use_compositing": Bool(false), "samples": Number(1000), "frames": Array([Number(2)]), "output_format": String("PNG"), "path_root": String("/home/ederenn/Desktop"), "start_task": Number(11), "total_tasks": Number(12), "crops": Array([Object({"outfilebasename": String("small_shark100_11"), "borders_x": Array([Number(0.0), Number(1.0)]), "borders_y": Array([Number(0.165), Number(0.33)])})]), "entrypoint": String("python3 /golem/entrypoints/render_entrypoint.py")}), "status": String("Timeout"), "stdout": String(""), "stderr": String("[GOLEM] Timeout"), "results": Array([])})])]: unknown variant `Timeout`, expected one of `Starting`, `Downloading`, `Verifying`, `Failed - Resent`, `Finished`, `Failure`, `Restart`, `Cancelled`
ERROR [golemcli] : unknown variant `Timeout`, expected one of `Starting`, `Downloading`, `Verifying`, `Failed - Resent`, `Finished`, `Failure`, `Restart`, `Cancelled`
ederenn@ubuntu:~$ 

Steps To Reproduce

Short description of steps to reproduce the behavior: e.g.

  1. Launch the app
  2. start a small task with several subtasks. For example helicopter.blend samples from file, res 400x400, 1 frame, 4 subtasks, bid 0.1, task timeout 10 min, subtask timeout 5 min
  3. Wait for task to start, and 2 subtasks to be finished
  4. Close golemapp and wait for task to be timed out
  5. Restart an app
  6. Check subtasks of a timed out task from cli golemcli tasks subtasks list <task_id>

Proposed Solution?

(Optional: What could be a solution for that issue)

kmazurek commented 4 years ago

FYI: this issue is about the subtask status rather than the task status. To reproduce it, one of the task's subtasks must be in Timeout state.