search

torrvision / crayon

A language-agnostic interface to TensorBoard
MIT License
779 stars 59 forks source link

Timeout threshold might be too low ? #43

Open mbanani opened 7 years ago

mbanani commented 7 years ago

I keep getting this error:

Traceback (most recent call last): File "train_viewpoint.py", line 495, in <module> main(args) File "train_viewpoint.py", line 23, in main cc = CrayonClient("focus.eecs.umich.edu") File "/z/home/mbanani/PyTorch2Env/local/lib/python2.7/site-packages/pycrayon/crayon.py", line 29, in __init__ " Server sent: {}.".format(r.text)) RuntimeError: Something went wrong! Server sent: Server: TensorBoard failed to answer request 'logdir'. Done

and I think the reason is that the timeout, specified here is too low when people are training on clusters and trying to connect to a docker on their personal machines. Maybe consider increasing the timeout ?

mbanani commented 7 years ago

Issue was caused by something else.

vakker commented 6 years ago

I have the same error. Could you elaborate a bit more on what was causing this for you?

mbanani commented 6 years ago

So I closed the issue before because I thought it was caused by running within a virtual environment, but the issue reappeared and I forgot to reopen the issue.

I am not really sure what's causing the issue, and I moved to using tensorboard directly, as shown here, which proved simpler to use for my purposes.

As for the error, I found that having a firewall that blocks some of the more common network ports or having many experiments running at the same time resulted in the error being produced more often. It would still run from time to time thought, which I found pretty weird, so my guess is that it's caused by the network taking too long to respond. Sorry this isn't more helpful

vakker commented 6 years ago

Thanks for the info, I'll investigate a bit further. I'm using Crayon in a Docker container, not sure if it has an effect.

motiwari commented 6 years ago

I had the same issue; for a quick and dirty workaround I simply commented out https://github.com/torrvision/crayon/blob/master/client/python/pycrayon/crayon.py#L32-L44 and it's working now.