search

Rockhopper-Technologies / enlighten

Enlighten Progress Bar for Python Console Apps
https://python-enlighten.readthedocs.io
Mozilla Public License 2.0
416 stars 25 forks source link

Does not work with multiple processes #10

Closed Beatris closed 4 years ago

Beatris commented 4 years ago

We have code executed by multiple parallel processes and when I call get_manager from it the below exception is raised. It seems as race condition in the blessed library but I haven't investigated it further.


  File "/venvdir/lib64/python3.5/site-packages/enlighten/_manager.py", line 401, in get_manager
    return Manager(stream=stream, counterclass=counterclass, **kwargs)
  File "/venvdir/lib64/python3.5/site-packages/enlighten/_manager.py", line 78, in __init__
    self.term = Terminal(stream=self.stream)
  File "/venvdir/lib64/python3.5/site-packages/enlighten/_terminal.py", line 26, in __init__
    super(Terminal, self).__init__(*args, **kwargs)
  File "/venvdir/lib64/python3.5/site-packages/blessed/terminal.py", line 176, in __init__
    self._keyboard_fd = sys.__stdin__.fileno()
ValueError: I/O operation on closed file```
avylove commented 4 years ago

The intent in both Enlighten and Blessed is that most of the code for interacting with the console lives in the parent process. You'd then pass updates to the parent through some sort of IPC. One way I've seen this done is to have the children write back to a queue and have the parent process (or a status process) loop through all the queues each 0.1 second to grab any updates. If you send the absolute numbers, you can just run Counter.update() on the last one in the queue.

There's also the possibility of sharing the enlighten manager or counter objects through a customized implementation of multiprocessing.managers.BaseManager, but I haven't tried it.

If I get some time I can try to create some example code. It would probably be useful to have in the documentation anyway. For now, just think in terms that you should only call get_manager() once within your program for all processes.

avylove commented 4 years ago

I added an example using queues. There are a lot of ways to do this, really depends on the design of your program.

avylove commented 4 years ago

Going to close this. Let me know if you run into any more issues or have suggestions on how we can make it easier.