Open ozziejin opened 4 years ago
Thanks for the report. I'm not 100% sure, but it sounds like there are 2 issues?
(1) Not being able to launch TensorBoard from a Jupyter notebook, using %tensorboard --logdir={dir}
One of the TensorBoard guides has a note for Jupyter users:
If you’re using a more complicated setup, like a global Jupyter installation and kernels for different Conda/virtualenv environments, then you must ensure that the tensorboard binary is on your PATH inside the Jupyter notebook context. One way to do this is to modify the kernel_spec to prepend the environment’s bin directory to PATH, as described here.
Could you please let us know if that works?
(2) When launching TensorBoard using tensorboard --logdir {dir}'
, it properly opens in a web browser, but no data is found. If the logdir provided is supposed to have data, could you please try the items in this FAQ:
https://github.com/tensorflow/tensorboard/blob/master/README.md#frequently-asked-questions ?
On my situation, Windows 10 64bits, tensorflow V2.1.1 (install with pip ), tensorboard (v2.1.1 installed maybe with anaconda... why ? I don't remember maybe it is the problem ?), I have noticed that the tensorboard process is not launch by %tensorboard command into jupyter notebook (chrome or firefox)
And we have to wait around 30 seconds for the process to be ready.
My temporary solution on Windows 10 to display tensorboard into notebook :
import time
import subprocess
logs_base_dir = #path to your run logs#
ls_output = subprocess.Popen(["tensorboard", "--logdir", logs_base_dir,
"--port", "6006"])
# patch to wait process
time.sleep(30)
# Load the TensorBoard notebook extension
%load_ext tensorboard
%tensorboard --logdir {logs_base_dir} --port 6006
My longer solution but it is a little dirty (about platform because done without an external library) I checked this code only on Windows... One part is adapted from https://stackoverflow.com/questions/7787120/python-check-if-a-process-is-running-or-not Other part is for Linux/Mac OS" : "I don t remember where I found it".
import tensorflow as tf
import subprocess
import platform
import os
def process_exists(process_name):
'''
Check if process currently exists in OS System Takslist
'''
MY_PLATFORM = platform.system()
if MY_PLATFORM == "Windows":
call = 'TASKLIST /FI "IMAGENAME eq ' + process_name + '"'
run_obj = subprocess.run(call, capture_output=True)
if re.search(process_name,
run_obj.stdout.decode('utf-8', 'backslashreplace')):
return True
else:
return False
else:
p = subprocess.Popen(['ps', '-A'], stdout=subprocess.PIPE)
out, err = p.communicate()
out = out.decode('utf-8', 'backslashreplace')
for line in out.splitlines():
if process_name in line:
return True
return False
logs_base_dir = #YOUR PATH TO RUNS LOGS#
if process_exists('tensorboard.exe'):
pass
elif process_exists('tensorboard'):
pass
else:
print("launch tensorboard process...")
popen_obj = subprocess.Popen(["tensorboard", "--logdir", logs_base_dir,
"--port", "6006"])
# patch to wait process
time.sleep(30)
# Load the TensorBoard notebook extension
%load_ext tensorboard
%tensorboard --logdir {logs_base_dir} --port 6006
The %tensorboard
magic already waits 60 seconds for the process to be
ready, so waiting 30 seconds manually shouldn’t make a difference if the
%tensorboard
command launches promptly.
If it really takes a total of 90 seconds for your TensorBoard to
initialize, let us know. Note that this initialization doesn’t entail
parsing all log files, just getting everything imported and the server
accepting requests. That seems kind of unlikely to me, but it’s not
impossible, so if you’re really hitting that perhaps we could add an
optional timeout argument to tensorboard.notebook.start
.
Actually, on my install, %tensorboard
tries to display instantly web page from server... but process didn't start. it is weird because process can be start just by executing !tensorboard.exe
into a jupyter cell
output :
`2020-05-12 02:28:05.314440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Error: A logdir or db must be specified. For example `tensorboard --logdir mylogdir` or `tensorboard --db sqlite:~/.tensorboard.db`. Run `tensorboard --helpfull` for details and examples.`
Environment information (required)
windows10 pro 64bit
Please run
diagnose_tensorboard.py
(link below) in the same environment from which you normally run TensorFlow/TensorBoard, and paste the output here:'Self-diagnosis script for TensorBoard.\nInstructions: Save this script to your local machine, then execute it in\nthe same environment (virtualenv, Conda, etc.) from which you normally\nrun TensorBoard. Read the output and follow the directions.\n'
Diagnostics
Diagnostics output
`````` --- check: autoidentify INFO: diagnose_tensorboard.py source unavailable INFO: diagnose_tensorboard.py source unavailable INFO: diagnose_tensorboard.py source unavailable --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0) INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0) INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0) INFO: os.name: nt INFO: os.name: nt INFO: os.name: nt INFO: os.uname(): N/A INFO: os.uname(): N/A INFO: os.uname(): N/A INFO: sys.getwindowsversion(): sys.getwindowsversion(major=10, minor=0, build=18363, platform=2, service_pack='') INFO: sys.getwindowsversion(): sys.getwindowsversion(major=10, minor=0, build=18363, platform=2, service_pack='') INFO: sys.getwindowsversion(): sys.getwindowsversion(major=10, minor=0, build=18363, platform=2, service_pack='') --- check: package_management INFO: has conda-meta: True INFO: has conda-meta: True INFO: has conda-meta: True INFO: $VIRTUAL_ENV: None INFO: $VIRTUAL_ENV: None INFO: $VIRTUAL_ENV: None --- check: installed_packages INFO: installed: tensorboard==2.1.0 INFO: installed: tensorboard==2.1.0 INFO: installed: tensorboard==2.1.0 INFO: installed: tensorflow==2.1.0 INFO: installed: tensorflow==2.1.0 INFO: installed: tensorflow==2.1.0 INFO: installed: tensorflow-estimator==2.1.0 INFO: installed: tensorflow-estimator==2.1.0 INFO: installed: tensorflow-estimator==2.1.0 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.1.0' INFO: tensorboard.version.VERSION: '2.1.0' INFO: tensorboard.version.VERSION: '2.1.0' --- check: tensorflow_python_version INFO: tensorflow.__version__: '2.1.0' INFO: tensorflow.__version__: '2.1.0' INFO: tensorflow.__version__: '2.1.0' INFO: tensorflow.__git_version__: 'unknown' INFO: tensorflow.__git_version__: 'unknown' INFO: tensorflow.__git_version__: 'unknown' --- check: tensorboard_binary_path INFO: which tensorboard: b'C:\\ProgramData\\Anaconda3\\Scripts\\tensorboard.exe\r\n' INFO: which tensorboard: b'C:\\ProgramData\\Anaconda3\\Scripts\\tensorboard.exe\r\n' INFO: which tensorboard: b'C:\\ProgramData\\Anaconda3\\Scripts\\tensorboard.exe\r\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC =Next steps
No action items identified. Please copy ALL of the above output, including the lines containing only backticks, into your GitHub issue or comment. Be sure to redact any sensitive information.
Issue description
Please describe the bug as clearly as possible. How can we reproduce the problem without additional resources (including external data files and proprietary Python modules)?
I've been having problems with tensorboard probably due to a unclean exit in windows10. I use the below code to launch it in Jupyter:
%load_ext tensorboard
%tensorboard --logdir={dir}
this is what I got: 'ERROR: Timed out waiting for TensorBoard to start. It may still be running as pid 24472.'
and below code in cmd window to launch TB in Chrome:
$ tensorboard --logdir {dir}'
this is what I got:
I've done some research and tried the following things: -"taskkill /im tensorboard.exe /f" to kill all live pids in command -deleted all the pid-xxxx.info files in the "%TMP%.tensorboard-info" directory. -deleted the whole "%TMP%.tensorboard-info" directly
Then I tried to change the port to 6005 by adding "--port 6005" in command window and TB could launch in Chrome under "localhost:6005. but it won't work in Jupyter.