Error (multiprocessor related I think?)

[danishduckling]

Hi, after specifying: processes = 2 (to avoid it using all cores (I have 4) in my config file I get this error:

2012-05-16 09:41:33  Rendered 0 of 424981.  0% complete
Process MultiprocessingDispatcherProcess-2:
Traceback (most recent call last):
  File "multiprocessing\process.pyc", line 232, in _bootstrap
  File "overviewer_core\dispatcher.pyc", line 260, in run
  File "overviewer_core\tileset.pyc", line 477, in do_work
  File "overviewer_core\tileset.pyc", line 984, in _render_rendertile
  File "overviewer_core\optimizeimages.pyc", line 43, in optimize_image
  File "subprocess.pyc", line 623, in __init__
  File "subprocess.pyc", line 833, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified
Process MultiprocessingDispatcherProcess-3:
Traceback (most recent call last):
  File "multiprocessing\process.pyc", line 232, in _bootstrap
  File "overviewer_core\dispatcher.pyc", line 260, in run
  File "overviewer_core\tileset.pyc", line 477, in do_work
  File "overviewer_core\tileset.pyc", line 984, in _render_rendertile
  File "overviewer_core\optimizeimages.pyc", line 43, in optimize_image
  File "subprocess.pyc", line 623, in __init__
  File "subprocess.pyc", line 833, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified
Traceback (most recent call last):
  File "overviewer.py", line 493, in <module>
  File "overviewer.py", line 442, in main
  File "overviewer_core\dispatcher.pyc", line 80, in render_all
  File "overviewer_core\dispatcher.pyc", line 106, in _dispatch_jobs
  File "overviewer_core\dispatcher.pyc", line 331, in dispatch
  File "overviewer_core\dispatcher.pyc", line 359, in _handle_messages
  File "<string>", line 2, in get
  File "multiprocessing\managers.pyc", line 726, in _callmethod

It is running on windows server 2008 R2 (x64)

and also are there other ways of limiting how much cpu overviewer is using to render the map? because I am running it on the same server as the game server, and overviewer uses so much cpu that it causes lag in the game (it is running on a copy of the map, not the live map), RAM usage is no issue, but cpu is.

[Fenixin]

I've found myself in this same error and it's not multiprocessing related. Are you using optimizeimg=1, 2 or 3 by any chance?

This happens to me whenever I try to render my world in Ubuntu 12.04 with the the optimizimg option. After some testing the problems seems to be in files.py:FileReplacer. Adding a simple print statement to optimizeimages.py shows that the imgpath returned by FileReplacer has already got ".tmp" at the end, pointing to a non existing path. I don't really understand how the FileReplacer works, but if you need any more info to fix it just ask for it.

The interesting thing is that this doesn't happen in ubuntu 10.04 with the same overviewer version.

My traceback:

This is the error that occurred:
Traceback (most recent call last):
  File "overviewer.py", line 493, in 
    ret = main()
  File "overviewer.py", line 442, in main
    dispatch.render_all(tilesets, config['observer'])
  File "/home/alejandro/source/overviewer/minecraft-overviewer/overviewer_core/dispatcher.py", line 80, in render_all
    observer.add(self._dispatch_jobs())
  File "/home/alejandro/source/overviewer/minecraft-overviewer/overviewer_core/dispatcher.py", line 106, in _dispatch_jobs
    finished_jobs += self.dispatch(tileset, workitem)
  File "/home/alejandro/source/overviewer/minecraft-overviewer/overviewer_core/dispatcher.py", line 144, in dispatch
    tileset.do_work(workitem)
  File "/home/alejandro/source/overviewer/minecraft-overviewer/overviewer_core/tileset.py", line 477, in do_work
    self._render_rendertile(RenderTile.from_path(tilepath))
  File "/home/alejandro/source/overviewer/minecraft-overviewer/overviewer_core/tileset.py", line 985, in _render_rendertile
    optimize_image(tmppath, self.imgextension, self.options['optimizeimg'])
  File "/home/alejandro/source/overviewer/minecraft-overviewer/overviewer_core/optimizeimages.py", line 43, in optimize_image
    stderr=subprocess.STDOUT, stdout=subprocess.PIPE).communicate()[0]
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

[danishduckling]

Yea, I was using optimizeing. but I am not on ubuntu but windows.

[eminence]

Can you guys easily reproduce this? I can't see how this error could happen.

[FabianN]

I get a similar error, when the optimizeimg option is enabled. I'm running Ubuntu Server 12.04 64-bit.

Process MultiprocessingDispatcherProcess-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 260, in run
    ret = self.tilesets[ti].do_work(workitem)
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/tileset.py", line 482, in do_work
    self._render_rendertile(RenderTile.from_path(tilepath))
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/tileset.py", line 996, in _render_rendertile
    optimize_image(tmppath, self.imgextension, self.options['optimizeimg'])
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/optimizeimages.py", line 43, in optimize_image
    stderr=subprocess.STDOUT, stdout=subprocess.PIPE).communicate()[0]
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Process MultiprocessingDispatcherProcess-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 260, in run
    ret = self.tilesets[ti].do_work(workitem)
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/tileset.py", line 482, in do_work
    self._render_rendertile(RenderTile.from_path(tilepath))
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/tileset.py", line 996, in _render_rendertile
    optimize_image(tmppath, self.imgextension, self.options['optimizeimg'])
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/optimizeimages.py", line 43, in optimize_image
    stderr=subprocess.STDOUT, stdout=subprocess.PIPE).communicate()[0]
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Process MultiprocessingDispatcherProcess-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 260, in run
    ret = self.tilesets[ti].do_work(workitem)
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/tileset.py", line 482, in do_work
    self._render_rendertile(RenderTile.from_path(tilepath))
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/tileset.py", line 996, in _render_rendertile
    optimize_image(tmppath, self.imgextension, self.options['optimizeimg'])
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/optimizeimages.py", line 43, in optimize_image
    stderr=subprocess.STDOUT, stdout=subprocess.PIPE).communicate()[0]
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
^C  0% [                                                                                                                ] 0 0.00t/s eta ----Traceback (most recent call last):
  File "overviewer.py", line 493, in <module>
    ret = main()
  File "overviewer.py", line 442, in main
    dispatch.render_all(tilesets, config['observer'])
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 80, in render_all
    observer.add(self._dispatch_jobs())
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 106, in _dispatch_jobs
    finished_jobs += self.dispatch(tileset, workitem)
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 331, in dispatch
    finished_jobs += self._handle_messages()
  File "/home/fabiann/overviewer_test/Minecraft-Overviewer/overviewer_core/dispatcher.py", line 359, in _handle_messages
    name, args, kwargs = self.signal_queue.get(True, timeout)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt

[pironic]

I am marking this as a Bug that can be solved by Phase 1 of OIL. If it still comes up once we complete phase 1 we may need to investigate further...