search

golemfactory / clay

Golem is creating a global market for computing power.
https://golem.network
GNU General Public License v3.0
2.91k stars 284 forks source link

Crash when updating preview #2059

Closed Elfoniok closed 6 years ago

Elfoniok commented 6 years ago

This traceback occur and not more results can be received: Unhandled Error Traceback (most recent call last): File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/log.py", line 103, in callWithLogger return callWithContext({"system": lp}, func, *args, kw) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/log.py", line 86, in callWithContext return context.call({ILogContext: newCtx}, func, *args, *kw) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, args, kw) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw) --- --- File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite why = selectable.doRead() File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/internet/tcp.py", line 205, in doRead return self._dataReceived(data) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/internet/tcp.py", line 211, in _dataReceived rval = self.protocol.dataReceived(data) File "/home/elfoniok/golem/golem/network/transport/tcpnetwork.py", line 370, in dataReceived self._interpret(data) File "/home/elfoniok/golem/golem/network/transport/tcpnetwork.py", line 554, in _interpret SafeProtocol._interpret(self, data) File "/home/elfoniok/golem/golem/network/transport/tcpnetwork.py", line 397, in _interpret self.session.interpret(m) File "/home/elfoniok/golem/golem/task/tasksession.py", line 108, in interpret BasicSafeSession.interpret(self, msg) File "/home/elfoniok/golem/golem/network/transport/session.py", line 91, in interpret action(msg) File "/home/elfoniok/golem/golem/task/tasksession.py", line 426, in _react_to_want_to_compute_task msg.num_cores, self.address) File "/home/elfoniok/golem/golem/task/taskmanager.py", line 315, in get_next_subtask node_name File "/home/elfoniok/golem/apps/core/task/coretask.py", line 528, in accepting_qed return query_extra_data_func(self, perf_index, num_cores, node_id, node_name) File "/home/elfoniok/golem/apps/blender/task/blenderrendertask.py", line 446, in query_extra_data self._update_task_preview() File "/home/elfoniok/golem/apps/rendering/task/renderingtask.py", line 152, in _update_task_preview img_task = self._open_preview() File "/home/elfoniok/golem/apps/rendering/task/renderingtask.py", line 246, in _open_preview return Image.open(self.preview_file_path) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/PIL/Image.py", line 2519, in open % (filename if filename else fp)) builtins.OSError: cannot identify image file '/home/elfoniok/.local/share/golem/default/res/1a08ad59-4040-43ea-b49b-40901cef51e2/tmp/current_preview.PNG'

Elfoniok commented 6 years ago

And second problem with same situation, note this happens very rarely, probably some race condition

CRITICAL [twisted ] Unhandled Error Traceback (most recent call last): File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/log.py", line 103, in callWithLogger return callWithContext({"system": lp}, func, *args, kw) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/log.py", line 86, in callWithContext return context.call({ILogContext: newCtx}, func, *args, *kw) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, args, kw) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw) --- --- File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite why = selectable.doRead() File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/internet/tcp.py", line 205, in doRead return self._dataReceived(data) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/twisted/internet/tcp.py", line 211, in _dataReceived rval = self.protocol.dataReceived(data) File "/home/elfoniok/golem/golem/network/transport/tcpnetwork.py", line 370, in dataReceived self._interpret(data) File "/home/elfoniok/golem/golem/network/transport/tcpnetwork.py", line 554, in _interpret SafeProtocol._interpret(self, data) File "/home/elfoniok/golem/golem/network/transport/tcpnetwork.py", line 397, in _interpret self.session.interpret(m) File "/home/elfoniok/golem/golem/task/tasksession.py", line 108, in interpret BasicSafeSession.interpret(self, msg) File "/home/elfoniok/golem/golem/network/transport/session.py", line 91, in interpret action(msg) File "/home/elfoniok/golem/golem/task/tasksession.py", line 426, in _react_to_want_to_compute_task msg.num_cores, self.address) File "/home/elfoniok/golem/golem/task/taskmanager.py", line 315, in get_next_subtask node_name File "/home/elfoniok/golem/apps/core/task/coretask.py", line 528, in accepting_qed return query_extra_data_func(self, perf_index, num_cores, node_id, node_name) File "/home/elfoniok/golem/apps/blender/task/blenderrendertask.py", line 446, in query_extra_data self._update_task_preview() File "/home/elfoniok/golem/apps/rendering/task/renderingtask.py", line 156, in _update_task_preview self._mark_task_area(sub, img_task, sent_color) File "/home/elfoniok/golem/apps/blender/task/blenderrendertask.py", line 574, in _mark_task_area self.mark_part_on_preview(subtask['start_task'], img_task, color, self.preview_updater) File "/home/elfoniok/golem/apps/blender/task/blenderrendertask.py", line 570, in mark_part_on_preview img_task.putpixel((i, j), color) File "/home/elfoniok/golem-env/lib/python3.5/site-packages/PIL/Image.py", line 1592, in putpixel self.load() File "/home/elfoniok/golem-env/lib/python3.5/site-packages/PIL/ImageFile.py", line 217, in load raise IOError("image file is truncated") builtins.OSError: image file is truncated

Elfoniok commented 6 years ago

Can you fix this? It will randomly crash gathering results, and ban node. With no obvious reason. At least handling this error would suffice.

etam commented 6 years ago

Not sure if related

ERROR    [golem.task.taskthread              ] Task computing error: [Errno 17] File exists
ERROR    [golem.task.taskthread              ] The following lines were printed as an array of strings:
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/docker/task_thread.py", line 101, in run
    self.task_computer.task_computed(self)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/task/localcomputer.py", line 94, in task_computed
    self.computation_success(task_thread)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/task/localcomputer.py", line 105, in computation_success
    self.success_callback(task_thread.result, self._get_time_spent())
  File "/home/etam/code/github/golemfactory/golem/preview_crash/apps/blender/task/verifier.py", line 188, in _crop_rendered
    self.make_verdict()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/apps/blender/task/verifier.py", line 213, in make_verdict
    self.success()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/apps/rendering/task/verifier.py", line 71, in success
    self.verification_completed()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/apps/core/task/verifier.py", line 35, in verification_completed
    result=self._get_answer())
  File "/home/etam/code/github/golemfactory/golem/preview_crash/apps/core/task/coretask.py", line 219, in verification_finished
    verification_finished_()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/task/taskmanager.py", line 498, in verification_finished
    verification_finished_()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/task/tasksession.py", line 300, in verification_finished
    self.dropped()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/task/tasksession.py", line 132, in dropped
    BasicSafeSession.dropped(self)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/network/transport/session.py", line 81, in dropped
    self.conn.close()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/network/transport/tcpnetwork.py", line 557, in close
   SafeProtocol.close(self)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/golem/network/transport/tcpnetwork.py", line 350, in close
    self.transport.loseConnection()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/venv/lib/python3.6/site-packages/twisted/internet/_newtls.py", line 217, in loseConnection
    FileDescriptor.loseConnection(self)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/venv/lib/python3.6/site-packages/twisted/internet/abstract.py", line 412, in loseConnection
    self.startWriting()
  File "/home/etam/code/github/golemfactory/golem/preview_crash/venv/lib/python3.6/site-packages/twisted/internet/abstract.py", line 446, in startWriting
    self.reactor.addWriter(self)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/venv/lib/python3.6/site-packages/twisted/internet/epollreactor.py", line 127, in addWriter
    EPOLLOUT, EPOLLIN)
  File "/home/etam/code/github/golemfactory/golem/preview_crash/venv/lib/python3.6/site-packages/twisted/internet/epollreactor.py", line 96, in _add
    self._poller.register(fd, flags)
WARNING  [golem.task                         ] Computation failed [Errno 17] File exists
WARNING  [apps.blender                       ] Crop for verification render failure '[Errno 17] File exists'
etam commented 6 years ago

At least it repeated subtask and finished rendering.