Tensorboard does not load all data at once, have to refresh multiple times

JACKHAHA363 commented 3 years ago

Environment information (required)

Diagnostics

Diagnostics output

`````` --- check: autoidentify INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='$USER', release='5.4.0-70-generic', version='#78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: $VIRTUAL_ENV: '/home/$USER/py38' --- check: installed_packages INFO: installed: tensorboard==2.4.1 WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview'] WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly'] --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.4.1' --- check: tensorflow_python_version Traceback (most recent call last): File "diagnose_tensorboard.py", line 522, in main suggestions.extend(check()) File "diagnose_tensorboard.py", line 75, in wrapper result = fn() File "diagnose_tensorboard.py", line 278, in tensorflow_python_version import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow' --- check: tensorboard_data_server_version INFO: no data server installed --- check: tensorboard_binary_path INFO: which tensorboard: b'/home/$USER/py38/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC = socket.SOCK_STREAM = socket.AI_ADDRCONFIG = socket.AI_PASSIVE = Loopback flags: Loopback infos: [(, , 6, '', ('::1', 0, 0, 0)), (, , 6, '', ('127.0.0.1', 0))] Wildcard flags: Wildcard infos: [(, , 6, '', ('0.0.0.0', 0)), (, , 6, '', ('::', 0, 0, 0))] --- check: readable_fqdn INFO: socket.getfqdn(): '$USER' --- check: stat_tensorboardinfo INFO: directory: /tmp/.tensorboard-info INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=9045394, st_dev=66309, st_nlink=2, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1617415716, st_mtime=1617943517, st_ctime=1617943517) INFO: mode: 0o40777 --- check: source_trees_without_genfiles INFO: tensorboard_roots (1): ['/home/$USER/py38/lib/python3.8/site-packages']; bad_roots (0): [] --- check: full_pip_freeze INFO: pip freeze --all: absl-py==0.11.0 appdirs==1.4.3 argon2-cffi==20.1.0 astunparse==1.6.3 async-generator==1.10 attrs==20.2.0 backcall==0.2.0 black==19.10b0 bleach==3.2.1 CacheControl==0.12.6 cachetools==4.1.1 certifi==2019.11.28 cffi==1.14.3 cfgv==3.2.0 chardet==3.0.4 click==7.1.2 cloudpickle==1.6.0 colorama==0.4.3 contextlib2==0.6.0 cycler==0.10.0 decorator==4.4.2 defusedxml==0.6.0 distlib==0.3.1 distro==1.4.0 entrypoints==0.3 et-xmlfile==1.0.1 filelock==3.0.12 flake8==3.7.9 flatbuffers==1.12 future==0.18.2 gast==0.3.3 gitdb==4.0.5 GitPython==3.1.11 google-auth==1.23.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 grpcio==1.32.0 gym==0.18.0 -e git+git@github.com:JACKHAHA363/gym_psketch.git@d4519588b9e0806f0f62af393e2774e3eda2f853#egg=gym_psketch h5py==2.10.0 html5lib==1.0.1 HyperPyYAML==0.0.1 identify==1.5.13 idna==2.8 imageio==2.9.0 ipaddr==2.2.0 ipdb==0.13.4 ipykernel==5.3.4 ipython==7.18.1 ipython-genutils==0.2.0 ipywidgets==7.5.1 -e git+git@github.com:JACKHAHA363/iterative_representation.git@2e85a3f41dd67e3c0f1197d06ad02715994cb896#egg=iterative_repr jdcal==1.4.1 jedi==0.17.2 Jinja2==2.11.2 joblib==1.0.0 jsonschema==3.2.0 jupyter==1.0.0 jupyter-client==6.1.7 jupyter-console==6.2.0 jupyter-core==4.6.3 jupyterlab-pygments==0.1.2 Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 lightgbm==3.1.1 lockfile==0.12.2 Markdown==3.3.3 MarkupSafe==1.1.1 matplotlib==3.3.3 mccabe==0.6.1 mistune==0.8.4 more-itertools==8.7.0 msgpack==0.6.2 natsort==7.1.1 nbclient==0.5.1 nbconvert==6.0.7 nbformat==5.0.8 nest-asyncio==1.4.1 networkx==2.5 nodeenv==1.5.0 notebook==6.1.4 numpy==1.19.2 oauthlib==3.1.0 opencv-python==4.5.1.48 openpyxl==3.0.6 opt-einsum==3.3.0 packaging==20.3 pandas==1.1.3 pandocfilters==1.4.2 parso==0.7.1 pathspec==0.8.1 pep517==0.8.2 pexpect==4.8.0 pickleshare==0.7.5 Pillow==7.2.0 pip==20.0.2 pkg-resources==0.0.0 pluggy==0.13.1 pre-commit==2.10.1 progress==1.5 prometheus-client==0.8.0 prompt-toolkit==3.0.8 protobuf==3.14.0 ptyprocess==0.6.0 py==1.10.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycodestyle==2.5.0 pycparser==2.20 pyflakes==2.1.1 pyglet==1.5.0 Pygments==2.7.1 PyJWT==1.7.1 pyparsing==2.4.6 pyrsistent==0.17.3 pytest==5.4.1 python-dateutil==2.8.1 python-git==2018.2.1 pytoml==0.1.21 pytz==2020.1 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==19.0.2 qtconsole==4.7.7 QtPy==1.9.0 regex==2020.11.13 requests==2.22.0 requests-oauthlib==1.3.0 retrying==1.3.3 rsa==4.6 ruamel.yaml==0.16.12 ruamel.yaml.clib==0.2.2 scikit-image==0.18.1 scikit-learn==0.24.0 scipy==1.6.0 seaborn==0.11.1 Send2Trash==1.5.0 sentencepiece==0.1.95 setuptools==44.0.0 sexpdata==0.0.3 six==1.15.0 sklearn==0.0 smmap==3.0.4 snakeviz==2.1.0 -e git+git@github.com:speechbrain/speechbrain.git@140d2d76b2480b550a7a5c77dbc6a1bed6673ccd#egg=SpeechBrain tensorboard==2.4.1 tensorboard-plugin-wit==1.8.0 tensorboardX==2.1 termcolor==1.1.0 terminado==0.9.1 testpath==0.4.4 threadpoolctl==2.1.0 tifffile==2021.1.14 toml==0.10.2 torch==1.7.1 torchaudio==0.7.2 torchtext==0.4.0 torchvision==0.7.0 tornado==6.0.4 tqdm==4.51.0 traitlets==5.0.5 twilio==6.51.0 typed-ast==1.4.2 typing-extensions==3.7.4.3 urllib3==1.25.8 virtualenv==20.4.2 wcwidth==0.2.5 webencodings==0.5.1 Werkzeug==1.0.1 wheel==0.36.2 widgetsnbextension==3.5.1 wrapt==1.12.1 yamllint==1.23.0 ``````

Issue description

I try to visualize the two run logs with the following folder structure. Please download here

├── run1
│   └── logs
│       └── events.out.tfevents.1617838872.cn-b003
└── run2
    └── logs
        └── events.out.tfevents.1617836239.cn-b003

with tensorboard --logdir .. After I open it, only partial log data is shown. I have to refresh multiple times to get the full data. I am using google chrome Version 89.0.4389.90 (Official Build) (64-bit).

Here is the result of initial refresh

Screenshot from 2021-04-09 00-52-52

second refresh Screenshot from 2021-04-09 00-53-29

and third refresh. Screenshot from 2021-04-09 00-53-37

stephanwlee commented 3 years ago

Hi @JACKHAHA363, thanks for the issue.

This is a known issue; our Python based log ingestor is quite slow and you can remediate your issue in one of two ways:

Install TensorFlow (I see that you don't have any TensorFlow installed in your pip freeze) to leverage C++ backed ingestor
Install our new data ingestor (still beta) by installing tb-nightly and using --load_fast=true flag. To learn more:
- https://github.com/tensorflow/tensorboard/issues/4784
- https://twitter.com/TensorFlow/status/1372654563795292161

If none of the solutions above help, please let us know (by reopening the issue). Especially, if you find the new data ingestor to be slow, please give the feedback directly at #4784. Thanks!

stephanwlee commented 3 years ago

Duplicate of #2411

anishazaveri commented 7 months ago

I'm still having this issue.

I'm on MacOS Ventura 13.6.
--load_fast=true is not supported on my machine
I have tensorflow==2.15.0 installed
My logdir is ~50 MB

Here's my pip freeze:

absl-py==2.0.0
astunparse==1.6.3
cachetools==5.3.2
certifi==2023.11.17
charset-normalizer==3.3.2
flatbuffers==23.5.26
gast==0.5.4
google-auth==2.25.2
google-auth-oauthlib==1.2.0
google-pasta==0.2.0
grpcio==1.60.0
h5py==3.10.0
idna==3.6
keras==2.15.0
libclang==16.0.6
Markdown==3.5.1
MarkupSafe==2.1.3
ml-dtypes==0.2.0
numpy==1.26.2
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==23.2
protobuf==4.23.4
pyasn1==0.5.1
pyasn1-modules==0.3.0
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
six==1.16.0
tensorboard==2.15.1
tensorboard-data-server==0.7.2
tensorflow==2.15.0
tensorflow-estimator==2.15.0
tensorflow-io-gcs-filesystem==0.34.0
tensorflow-macos==2.15.0
termcolor==2.4.0
typing_extensions==4.9.0
urllib3==2.1.0
Werkzeug==3.0.1
wrapt==1.14.1

tensorflow / tensorboard