psybuzz
commented
3 years ago Thanks for the report. Summaries written by TensorFlow's tf.summary.* APIs are supposed to produce event files with proper checksums. The error could indicate that either the event file in question has been modified, or perhaps was written in an unexpected way. Could you share how is machine B logging events?
If the machine in question is not using something like
import tensorflow as tf
writer = tf.summary.create_file_writer('test/logdir')
with writer.as_default():
tf.summary.scalar('loss', 0.345, step=1)would it be possible to share the summary writing code for us to investigate?
If you trust the source that logs your summary data, and do not care about this checksum warning, it is also possible to suppress the check by passing extra flags to tensorboard:
tensorboard --logdir my_logdir --extra_data_server_flags=--no-checksum
arghyaganguly
vadimcn
svobora
HannesStark
zhixuanli
siyi-wind
rkechols
nfelt
B217040
qazi0
Environment TensorBoard 2.5.0; Tensorflow 2.5.0
Please run
diagnose_tensorboard.py(link below) in the same environment from which you normally run TensorFlow/TensorBoard, and paste the output here:Diagnostics
Diagnostics output
`````` --- check: autoidentify INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='3e52c33c1851', release='5.4.0-74-generic', version='#83~18.04.1-Ubuntu SMP Tue May 11 16:01:00 UTC 2021', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: $VIRTUAL_ENV: None WARNING: The directory '/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. --- check: installed_packages INFO: installed: tensorboard==2.5.0 INFO: installed: tensorflow==2.5.0 INFO: installed: tensorflow-estimator==2.5.0rc0 INFO: installed: tensorboard-data-server==0.6.1 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.5.0' 2021-07-08 20:57:09.014915: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 --- check: tensorflow_python_version INFO: tensorflow.__version__: '2.5.0' INFO: tensorflow.__git_version__: 'v2.5.0-rc3-213-ga4dfb8d1a71' --- check: tensorboard_data_server_version INFO: data server binary: '/usr/local/lib/python3.6/dist-packages/tensorboard_data_server/bin/server' Traceback (most recent call last): File "/workspace/diagnose_tensorboard.py", line 522, in main suggestions.extend(check()) File "/workspace/diagnose_tensorboard.py", line 75, in wrapper result = fn() File "/workspace/diagnose_tensorboard.py", line 301, in tensorboard_data_server_version check=True, File "/usr/lib/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: TypeError: __init__() got an unexpected keyword argument 'capture_output' --- check: tensorboard_binary_path INFO: which tensorboard: b'/usr/local/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC =Next steps
No action items identified. Please copy ALL of the above output, including the lines containing only backticks, into your GitHub issue or comment. Be sure to redact any sensitive information.
Issue I see the following error when trying to run Tensorboard to visualize logs:
[ WARN rustboard_core::run] Read error in /workspace/log/events.out.tfevents.1624569739.8a1c50246dae.30295.276717.v2: ReadRecordError(BadLengthCrc(ChecksumError { got: MaskedCrc(0x07980329), want: MaskedCrc(0x00000000) }))
For some context, I am using a shared file system between several machines and this error comes up when trying to run Tensorboard on machine A, pointing to a log generated by machine B.