tensorboard profile Step-time Graph is empty

[pedro-abundio-wang]

Consider Stack Overflow for getting support using TensorBoard—they have a larger community with better searchability:

https://stackoverflow.com/questions/tagged/tensorboard

Do not use this template for for setup, installation, or configuration issues. Instead, use the “installation problem” issue template:

https://github.com/tensorflow/tensorboard/issues/new?template=installation_problem.md

To report a problem with TensorBoard itself, please fill out the remainder of this template.

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same environment from which you normally run TensorFlow/TensorBoard, and paste the output here:

https://raw.githubusercontent.com/tensorflow/tensorboard/master/tensorboard/tools/diagnose_tensorboard.py

Diagnostics

Diagnostics output

`````` --- check: autoidentify INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='sifu-GPU-SERVER', release='5.4.0-51-generic', version='#56~18.04.1-Ubuntu SMP Tue Oct 6 09:47:18 UTC 2020', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: True INFO: $VIRTUAL_ENV: None --- check: installed_packages WARNING: no installation among: ['tb-nightly', 'tensorboard', 'tensorflow-tensorboard'] INFO: installed: tensorflow==2.2.0 INFO: installed: tensorflow-estimator==2.2.0 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.4.0' --- check: tensorflow_python_version INFO: tensorflow.__version__: '2.2.0' INFO: tensorflow.__git_version__: 'unknown' --- check: tensorboard_data_server_version INFO: no data server installed --- check: tensorboard_binary_path INFO: which tensorboard: b'/home/pedro/anaconda3/envs/atom/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC = socket.SOCK_STREAM = socket.AI_ADDRCONFIG = socket.AI_PASSIVE = Loopback flags: Loopback infos: [(, , 6, '', ('::1', 0, 0, 0)), (, , 6, '', ('127.0.0.1', 0))] Wildcard flags: Wildcard infos: [(, , 6, '', ('0.0.0.0', 0)), (, , 6, '', ('::', 0, 0, 0))] --- check: readable_fqdn INFO: socket.getfqdn(): 'sifu-GPU-SERVER' --- check: stat_tensorboardinfo INFO: directory: /tmp/.tensorboard-info INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=56950928, st_dev=2050, st_nlink=2, st_uid=1002, st_gid=1002, st_size=4096, st_atime=1619571241, st_mtime=1619574568, st_ctime=1619574568) INFO: mode: 0o40777 --- check: source_trees_without_genfiles INFO: tensorboard_roots (1): ['/home/pedro/anaconda3/envs/atom/lib/python3.8/site-packages']; bad_roots (0): [] --- check: full_pip_freeze INFO: pip freeze --all: absl-py @ file:///tmp/build/80754af9/absl-py_1615411202722/work aiohttp @ file:///tmp/build/80754af9/aiohttp_1614360992924/work astunparse==1.6.3 async-timeout==3.0.1 attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work blinker==1.4 brotlipy==0.7.0 cachetools @ file:///tmp/build/80754af9/cachetools_1611600262290/work certifi==2020.12.5 cffi @ file:///tmp/build/80754af9/cffi_1613246945912/work chardet @ file:///tmp/build/80754af9/chardet_1605303185383/work click @ file:///home/linux1/recipes/ci/click_1610990599742/work coverage @ file:///tmp/build/80754af9/coverage_1614613670853/work cryptography @ file:///tmp/build/80754af9/cryptography_1613400861696/work cycler==0.10.0 Cython @ file:///tmp/build/80754af9/cython_1614014838717/work dill @ file:///tmp/build/80754af9/dill_1605066453875/work dm-tree==0.1.5 future==0.18.2 gast==0.3.3 google-auth @ file:///tmp/build/80754af9/google-auth_1614883971544/work google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1614894617465/work google-pasta==0.2.0 googleapis-common-protos @ file:///tmp/build/80754af9/googleapis-common-protos-feedstock_1613732354958/work grpcio @ file:///tmp/build/80754af9/grpcio_1614884175859/work gviz-api==1.9.0 h5py @ file:///tmp/build/80754af9/h5py_1593454122442/work idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/work Markdown @ file:///tmp/build/80754af9/markdown_1614363528767/work matplotlib @ file:///tmp/build/80754af9/matplotlib-suite_1613407855456/work mkl-fft==1.3.0 mkl-random==1.1.1 mkl-service==2.3.0 mock @ file:///tmp/build/80754af9/mock_1607622725907/work multidict @ file:///tmp/build/80754af9/multidict_1607367757617/work numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603570489231/work oauthlib==3.1.0 olefile==0.46 opt-einsum==3.1.0 Pillow @ file:///tmp/build/80754af9/pillow_1617383569452/work pip==21.0.1 promise @ file:///tmp/build/80754af9/promise_1614011636525/work protobuf==3.14.0 psutil @ file:///tmp/build/80754af9/psutil_1612298023621/work pyasn1==0.4.8 pyasn1-modules==0.2.8 pycocotools @ git+https://github.com/cocodataset/cocoapi@8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9#subdirectory=PythonAPI pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work PyJWT==1.7.1 pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work PyYAML==5.4.1 requests @ file:///tmp/build/80754af9/requests_1608241421344/work requests-oauthlib==1.3.0 rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work scipy @ file:///tmp/build/80754af9/scipy_1614022789989/work setuptools==52.0.0.post20210125 sip==4.19.13 six @ file:///home/linux1/recipes/ci/six_1610970791821/work tensorboard @ file:///home/builder/ktietz/aggregate/tensorflow_recipes/ci_te/tensorboard_1614593728657/work/tmp_pip_dir tensorboard-plugin-profile==2.2.0 tensorboard-plugin-wit==1.6.0 tensorflow==2.2.0 tensorflow-addons==0.10.0 tensorflow-datasets @ file:///tmp/build/80754af9/tensorflow-datasets_1615394951098/work tensorflow-estimator==2.2.0 tensorflow-metadata==0.14.0 tensorflow-model-optimization==0.5.0 termcolor==1.1.0 tornado @ file:///tmp/build/80754af9/tornado_1606942300299/work tqdm @ file:///tmp/build/80754af9/tqdm_1611857934208/work typeguard==2.11.1 typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1611751222202/work urllib3 @ file:///tmp/build/80754af9/urllib3_1611694770489/work Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work wheel==0.36.2 wrapt==1.12.1 yarl @ file:///tmp/build/80754af9/yarl_1606939922162/work zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work ``````

Next steps

No action items identified. Please copy ALL of the above output, including the lines containing only backticks, into your GitHub issue or comment. Be sure to redact any sensitive information.

For browser-related issues, please additionally specify:

• Browser type and version (e.g., Chrome 64.0.3282.140): Google Chrome Version 87.0.4280.88 (Official Build) (64 bit)
• Screenshot, if it’s a visual issue:

Issue description

Please describe the bug as clearly as possible. How can we reproduce the problem without additional resources (including external data files and proprietary Python modules)?

After running classifier_trainer.py to train models about 90 epochs, the tensorboard profiler Step-time Graph is empty, how can i fix it?

the model is saved in $HOME/models/resnet18v2/imagenet/ and i run the following command

tensorboard --logdir=$HOME/models/resnet18v2/imagenet/ then everything in tensorborad seem works fine, but only the profile Step-time Graph is empty.

the model profiler file is in the following

(atom) pedro@sifu-GPU-SERVER:/home/pedro/models/resnet18v2/imagenet/train/plugins/profile/2021_04_27_16_52_12$ ls -l
total 96
-rw-r--r-- 1 root root  2413 April  27 16:52 sifu-GPU-SERVER.input_pipeline.pb
-rw-r--r-- 1 root root 28994 April  27 16:52 sifu-GPU-SERVER.kernel_stats.pb
-rw-r--r-- 1 root root  3828 April  27 16:52 sifu-GPU-SERVER.overview_page.pb
-rw-r--r-- 1 root root  7076 April  27 16:52 sifu-GPU-SERVER.tensorflow_stats.pb
-rw-r--r-- 1 root root 47699 April  27 16:52 sifu-GPU-SERVER.trace.json.gz
(atom) pedro@sifu-GPU-SERVER:/home/pedro/models/resnet18v2/imagenet/train/plugins/profile/2021_04_27_16_52_12$

[wchargin]

Hi @pedro-abundio-wang! Thanks for the report. The profile dashboard is a separate plugin; @yisitu, could you take a look?

[yisitu]

Hi @pedro-abundio-wang

Would it be possible to update to TF 2.4 instead to see if the problem persists?

[arghyaganguly]

Closing this as there has been no update for sometime.Please feel free to reopen with information/insight from the above comment trace.Thanks.