search

IntelLabs / coach

Reinforcement Learning Coach by Intel AI Lab enables easy experimentation with state of the art Reinforcement Learning algorithms
https://intellabs.github.io/coach/
Apache License 2.0
2.32k stars 460 forks source link

Dashboard crashes: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes #329

Closed timokau closed 5 years ago

timokau commented 5 years ago

I'm trying to monitor the training of CartPole with

coach -d -p CartPole_DQN

The dashboard opens as it should and works for a while. At some point however it will stop responding and the console will show this error:

2019-05-27 18:03:21,502 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x7fabf6e0dba8>: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes
File "session.py", line 214, in _document_patched:
raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes") Traceback (most recent call last):
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/application/handlers/code_runner.py", line 179, in run
    exec(self._code, module.__dict__)
  File "/nix/store/bj9cr61kilf9r1n8047vs37lmfxk12bk-python3.7-rl-coach-0.12.0/lib/python3.7/site-packages/rl_coach/dashboard.py", line 32, in <module>
    doc.add_root(landing_page)
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/document/document.py", line 305, in add_root
    self._trigger_on_change(RootAddedEvent(self, model, setter))
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/document/document.py", line 1099, in _trigger_on_change
    self._with_self_as_curdoc(invoke_callbacks)
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/document/document.py", line 1112, in _with_self_as_curdoc
    return f()
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/document/document.py", line 1098, in invoke_callbacks
    cb(event)
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/document/document.py", line 668, in <lambda>
    self._callbacks[receiver] = lambda event: event.dispatch(receiver)
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/document/events.py", line 126, in dispatch
    receiver._document_patched(self)
  File "/nix/store/h1h4wqz7km721kb13m17q9fdmzzv503h-python3-3.7.3-env/lib/python3.7/site-packages/bokeh/server/session.py", line 214, in _document_patched
    raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes")
RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes

I am using tornado 5.1 and bokeh 1.0.4. Is there some known-good version combination I should try?

I found a similar error reported here, where it was dismissed as "some sort of usage error".

gal-leibovich commented 5 years ago

I am using Bokeh 1.0.4 and tornado 5.1.1 and have never seen this error. Also, currently, Coach is only tested with Python 3.5. Although, we do see it working well with Python 3.6 (but do not test for it). I did not try it with Python 3.7. Could you please try Python3.5/3.6 with the above listed tornado and bokeh versions?

timokau commented 5 years ago

I cannot reproduce this anymore (not even with python 3.7 and tornado 5.1). Thanks for taking a look.