tgolsson
commented
3 years ago Hello!
Very much interested in this, as we currently maintain a custom entrypoint to make Tensorboard work at all with our data sizes. Unfortunately, I can't get this to work anywhere. Using the latest nightly docker image I get the following error:
root@15bc33cc211f:/# tensorboard --logdir foobar --load_fast=true
Error: Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" }
Traceback (most recent call last):
File "/usr/local/bin/tensorboard", line 8, in <module>
sys.exit(run_main())
File "/usr/local/lib/python3.6/dist-packages/tensorboard/main.py", line 46, in run_main
app.run(tensorboard.main, flags_parser=tensorboard.configure)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 267, in main
return runner(self.flags) or 0
File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 283, in _run_serve_subcommand
server = self._make_server()
File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 433, in _make_server
(data_provider, deprecated_multiplexer) = self._make_data_provider()
File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 425, in _make_data_provider
ingester.start()
File "/usr/local/lib/python3.6/dist-packages/tensorboard/data/server_ingester.py", line 150, in start
% popen.poll()
RuntimeError: Data server exited with 1; check stderr for detailsPresumably it tries to bind some port that's already in use by another process; unfortunately it doesn't say which one.
Also, it doesn't seem to work with logdir_spec, only logdir. This isn't a huge pain, but the error message just states that I didn't pass logdir -- it should probably explicitly state that load_fast and logdir_spec are incompatible.
wchargin
brychcy
Raphtor
sjincho
8bitmp3
yisitu
jstremme
stephanwlee
yoshipon
an-ivanov
GeorgePearse
Jiayuan-Gu
lemairecarl
mhdadk
drmeerkat
Corwinpro
bmd3k
samos123
mueller91
wookayin
profPlum
Frn1nd0
davidxia
This thread is for tracking feedback about TensorBoard’s experimental mode for fast data loading. Typical speedups range from 100× to 400×.
Who should try this: Anyone who’s found TensorBoard’s data loading to be slower than they’d like.
Who shouldn’t try this: Windows users (for now).
Feedback: Feedback form, or reply on this thread.
Try it out
To try this out, please uninstall all copies of TensorBoard and then install the latest version of
tb-nightly:Then, invoke TensorBoard with the
--load_fast=trueflag:Use TensorBoard as you usually would. It should work the same way, just faster.
Feedback
You can respond to this anonymous Google Form, or reply on this thread, or open a new issue. Let us know: did it work? how much faster was it? any suggestions or requests?
Known issues
We know about these, but please let us know if they matter for you, so that we can prioritize working on them:
FAQ
What does “data loading” include?
It includes time spent reading files in your logdir. It does not include time spent painting charts on the frontend.
What is the
--load_fastflag?Pass
--load_fast=trueto tell TensorBoard to use a new data loading mechanism, which is generally hundreds of times faster.Is
--load_fast=trueright for me?Currently, this mode is supported on Linux and macOS. If you are interested in using it on other platforms, ping @wchargin and I’ll show you how to build it.
Most features of TensorBoard are expected to work with the new data loading mechanism. All standard TensorBoard dashboards (scalars, images, etc.) should work, and flags like
--reload_intervalshould work, too. You can use logdirs on local disk or on GCS buckets (public or private).Do I need to have TensorFlow installed?
No.
What’s happening under the hood?
Instead of crawling your logdir in a mixture of Python and C++ code with a lot of locking, cross-language marshalling, and slow data manipulation in Python, we read the data in a dedicated subprocess. This program is written in Rust and is optimized for concurrent reading and serving. More design details here.