golemfactory / clay

Golem is creating a global market for computing power.
https://golem.network
GNU General Public License v3.0
2.91k stars 286 forks source link

restart of timed out subtasks is not working #3248

Closed ederenn closed 6 years ago

ederenn commented 6 years ago

Encountered on:

Golem ver. 0.17.0+dev94.g09f3405

Steps to reproduce:

in golem log

18-08-29 06:06:50 INFO     golem.ethereum.fundslocker          Locking funds for task: '64837992-ab8c-11e8-a929-b4b0ce494777' price: 16666666666666668.000000 num: 10
2018-08-29 06:06:50 INFO     golem.ethereum.transactionsystem    Locking 0.166667 GNT and ETH for 10 payments
2018-08-29 06:06:50 INFO     golem.client                        Enqueue new task "'64837992-ab8c-11e8-a929-b4b0ce494777'"
2018-08-29 06:06:50 INFO     golem.network.p2p.p2pservice        Estimated network size: 64
2018-08-29 06:06:50 WARNING  golem.network.p2p.p2pservice        Cannot compute percentile rank. No host performance info is available
2018-08-29 06:06:50 INFO     golem.client                        Task 64837992-ab8c-11e8-a929-b4b0ce494777 initial mask size: 0 expected number of providers: 20 potential number of providers: 0
2018-08-29 06:06:50 INFO     golem.task.taskmanager              Task 64837992-ab8c-11e8-a929-b4b0ce494777 added
2018-08-29 06:06:51 INFO     golem.task.taskmanager              Task 64837992-ab8c-11e8-a929-b4b0ce494777 started
2018-08-29 06:06:55 WARNING  apps.core                           This is not my subtask 66035c30-ab8c-11e8-8b7f-b4b0ce494777
Traceback (most recent call last):
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/core/common.py", line 147, in func_wrapper
  File "/home/buildbot-worker/worker/buildpackage_linux/build/apps/core/task/coretask.py", line 470, in _mark_subtask_failed
KeyError: '0e291e07-10a8-40ab-a5fe-986ba37a5d56'
2018-08-29 06:06:55 WARNING  apps.core                           This is not my subtask 6615ff70-ab8c-11e8-9039-b4b0ce494777
Traceback (most recent call last):
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/core/common.py", line 147, in func_wrapper
  File "/home/buildbot-worker/worker/buildpackage_linux/build/apps/core/task/coretask.py", line 470, in _mark_subtask_failed
KeyError: 'cb49e43f-dcc0-48fe-8f80-bc13f091c02d'
2018-08-29 06:06:56 WARNING  apps.core                           This is not my subtask 662fea54-ab8c-11e8-86d4-b4b0ce494777
Traceback (most recent call last):
  File "/home/buildbot-worker/worker/buildpackage_linux/build/golem/core/common.py", line 147, in func_wrapper
  File "/home/buildbot-worker/worker/buildpackage_linux/build/apps/core/task/coretask.py", line 470, in _mark_subtask_failed
KeyError: 'b41e1b1a-17e0-41eb-8a51-9d0b3253fefe'
2018-08-29 06:06:56 INFO     golem.task.taskserver               requestor node is blacklisted; 380f9240faff8d2076da44447ed62e3a9319ac8da13c73c4438027347feeda2f7130d8da23804777444f267920efded556484fe48d5adc34dac9a81da26ba7c5
2018-08-29 06:06:57 INFO     golem.ethereum.paymentprocessor     Next sendout in 78748 s
2018-08-29 06:07:01 INFO     golem.task.taskserver               requestor node is blacklisted; 380f9240faff8d2076da44447ed62e3a9319ac8da13c73c4438027347feeda2f7130d8da23804777444f267920efded556484fe48d5adc34dac9a81da26ba7c5
2018-08-29 06:07:07 INFO     golem.task.taskserver               requestor node is blacklisted;

from Golem cli:

ederenn@ubuntu:~$ golemcli tasks subtasks 64837992-ab8c-11e8-a929-b4b0ce494777
node             id                                    ETA      status    completion
---------------  ------------------------------------  -------  --------  ------------
SerbianGolem     655d4886-ab8c-11e8-a7fa-b4b0ce494777  0:00:00  Finished  100.00 %
defitek          6566c9e8-ab8c-11e8-988d-b4b0ce494777  0:00:00  Finished  100.00 %
                 66035c30-ab8c-11e8-8b7f-b4b0ce494777  0:00:00  Restart   0.00 %
                 6615ff70-ab8c-11e8-9039-b4b0ce494777  0:00:00  Restart   0.00 %
                 662fea54-ab8c-11e8-86d4-b4b0ce494777  0:00:00  Restart   0.00 %
LegendDedoGolem  6650d8d2-ab8c-11e8-bff2-b4b0ce494777  0:00:00  Finished  100.00 %
Cloud9           66781866-ab8c-11e8-bdfa-b4b0ce494777  0:00:00  Finished  100.00 %
defitek          66a5ad6e-ab8c-11e8-b869-b4b0ce494777  0:00:00  Finished  100.00 %
defitek          66ddd664-ab8c-11e8-a397-b4b0ce494777  0:00:00  Finished  100.00 %
defitek          6717a9b6-ab8c-11e8-97c7-b4b0ce494777  0:00:00  Finished  100.00 %

restarted subtasks have status in logs: This is not my subtask 662fea54-ab8c-11e8-86d4-b4b0ce494777

same situation is, when subtask is restarted manually from second preview window after the task ends in timeout

etam commented 6 years ago

The same happens, when restarting successfully finished subtask.

etam commented 6 years ago

3280

shadeofblue commented 6 years ago

I believe it's fixed