Closed tradingjunkie closed 3 years ago
Thanks for the report. Summaries written by TensorFlow's tf.summary.*
APIs are supposed to produce event files with proper checksums. The error could indicate that either the event file in question has been modified, or perhaps was written in an unexpected way. Could you share how is machine B logging events?
If the machine in question is not using something like
import tensorflow as tf
writer = tf.summary.create_file_writer('test/logdir')
with writer.as_default():
tf.summary.scalar('loss', 0.345, step=1)
would it be possible to share the summary writing code for us to investigate?
If you trust the source that logs your summary data, and do not care about this checksum warning, it is also possible to suppress the check by passing extra flags to tensorboard:
tensorboard --logdir my_logdir --extra_data_server_flags=--no-checksum
@tradingjunkie , please share the information (machine B logging events, code used to write summary) as asked by @psybuzz in his last comment.
@tradingjunkie , closing this as this ticket has been inactive awaiting your response for some time.Please feel free to reopen if required.Thanks.
Had this exact problem when TB was reading logs exported from another machine via a NFS share.
I've instrumented RustBoard to print out the contents of the last read block on CRC errors, and sure enough, it was all zeros!
Probably not RustBoard's fault though, I would be inclined to blame this on a faulty NFS driver, especially since similar bugs had happened in the past.
Interestingly enough, this problem does not occur when the log files are read from Python (which I tried to create a repro case). Not sure why, maybe this is just due to the difference in speed. But this allowed me to work around it by writing a Python script that replicates logs to a local file system from where they are read by TB.
This is still an issue. When multiple machines are writing logs to a shared samba storage, the logs are not processes correctly. Sometimes tensorboard reads several epochs before the failure. Perhaps the read operation is performed during writing of the log file, and on first sign of error, tensorboard blocks the offending file and is not trying to re-read the log file later.
[2022-05-23T09:08:56Z WARN rustboard_core::run] Read error in ./logs/20220520-110941/events.out.tfevents.1653045009.g12.3001.0.v2: ReadRecordError(BadLengthCrc(ChecksumError { got: MaskedCrc(0x07980329), want: MaskedCrc(0x00000000) }))
To get around the issue, I need to restart tensorboard every time I want to look at the results.
@svobora did you find a solution to this issue?
Perhaps the read operation is performed during writing of the log file...
Log records are length-prefixed and TB log reader correctly waits till it has read the requisite number of bytes before processing the record (I've studied log reader source). I can't imagine how this error can occur without a screw-up somewhere in the filesystem stack. So technically TB is not at fault here.
Still, it would be nice if on a checksum error, it would try to recover by re-reading all bytes after the last successfully processed record. Unfortunately, I could not find an easy way to do this, since TB reader is built around non seek-able streams, so it would require way too much change for a drive-by pull request.
@arghyaganguly Any chance you could re-open this issue and have TB devs look into implementing a fix?
This is still an issue. When multiple machines are writing logs to a shared samba storage, the logs are not processes correctly. Sometimes tensorboard reads several epochs before the failure. Perhaps the read operation is performed during writing of the log file, and on first sign of error, tensorboard blocks the offending file and is not trying to re-read the log file later.
[2022-05-23T09:08:56Z WARN rustboard_core::run] Read error in ./logs/20220520-110941/events.out.tfevents.1653045009.g12.3001.0.v2: ReadRecordError(BadLengthCrc(ChecksumError { got: MaskedCrc(0x07980329), want: MaskedCrc(0x00000000) }))
To get around the issue, I need to restart tensorboard every time I want to look at the results.
Still having this problem.
Perhaps the read operation is performed during writing of the log file...
Log records are length-prefixed and TB log reader correctly waits till it has read the requisite number of bytes before processing the record (I've studied log reader source). I can't imagine how this error can occur without a screw-up somewhere in the filesystem stack. So technically TB is not at fault here.
Still, it would be nice if on a checksum error, it would try to recover by re-reading all bytes after the last successfully processed record. Unfortunately, I could not find an easy way to do this, since TB reader is built around non seek-able streams, so it would require way too much change for a drive-by pull request.
@arghyaganguly Any chance you could re-open this issue and have TB devs look into implementing a fix?
Sorry but it is typical for bugs that the devs "can't imagine" how it can happen... The obvious fix is to not add the CRC-failed log files to ignore list, possibly trying to read the files again later, with errors silenced. Another solution is to write log files to tmp files and only move them to final location (atomic) once they are written to drive.
Using fast data loading tensorboard --logdir /path/to/logs --load_fast true
solved this issue. https://github.com/tensorflow/tensorboard/issues/4784
Using fast data loading
tensorboard --logdir /path/to/logs --load_fast true
solved this issue. #4784
Thanks for your suggestion!
Using fast data loading
tensorboard --logdir /path/to/logs --load_fast true
solved this issue. #4784
The problem persists.
I'm getting the same problem after copying the log files from a remote machine (Linux) to my local machine (MacOS). Does this mean that TB logs can only be read on the machine where they were created? Copying the file somehow corrupts it?
@rkechols There shouldn't be any constraint about TB logs only being readable on certain machines; the file format itself should be portable. Most likely the file itself is being corrupted during the copy somehow. I'd suggest taking a checksum of the file contents before and after copying it (e.g. the SHA-256 hash or something) to confirm if that's happening.
@nfelt it turns out you were right that the file was corrupted "in transit". I re-copied the file a different way and had no issue.
@rkechols Would you mind not spamming a topic irrelevant to your issue?
@svobora That kind of remark is uncalled for, please be polite if you're going to participate in the issue thread. I agree it's not ultimately the exact same issue, but it's reasonable that @rkechols was confused.
If your concern is that this issue should be re-opened, I can re-open and re-title it accordingly, but there's no guarantee that we'll address it right away, especially since this issue only affects cases where the file transiently appears to be corrupted.
@nfelt it turns out you were right that the file was corrupted "in transit". I re-copied the file a different way and had no issue.
I'm trying to do the same thing and having the same issue - I'm transferring the log file through git. Which way did you get it to work ?
Still having the issue, I was drinking tea when the issue happened. Do you know how high water temperature is optimal for green tea?
Just kidding, I solved the issue by switching from EXT4 to BTRFS on the target network storage (Synology device).
Still having the issue, I was drinking tea when the issue happened. Do you know how high water temperature is optimal for green tea?
Just kidding, I solved the issue by switching from EXT4 to BTRFS on the target network storage (Synology device).
@svobora can you please explain? While running in an Azure Linux amd64 VM (ext4) I'm running into this error while running parallel training trials with Ray Tune. Are you suggesting there's no way to get to work in ext4?
@svobora can you please explain? While running in an Azure Linux amd64 VM (ext4) I'm running into this error while running parallel training trials with Ray Tune. Are you suggesting there's no way to get to work in ext4?
As I said, I never managed to fix it until I switched the filesystem. Then it finally started working.
Environment TensorBoard 2.5.0; Tensorflow 2.5.0
Please run
diagnose_tensorboard.py
(link below) in the same environment from which you normally run TensorFlow/TensorBoard, and paste the output here:Diagnostics
Diagnostics output
`````` --- check: autoidentify INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='3e52c33c1851', release='5.4.0-74-generic', version='#83~18.04.1-Ubuntu SMP Tue May 11 16:01:00 UTC 2021', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: $VIRTUAL_ENV: None WARNING: The directory '/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. --- check: installed_packages INFO: installed: tensorboard==2.5.0 INFO: installed: tensorflow==2.5.0 INFO: installed: tensorflow-estimator==2.5.0rc0 INFO: installed: tensorboard-data-server==0.6.1 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.5.0' 2021-07-08 20:57:09.014915: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 --- check: tensorflow_python_version INFO: tensorflow.__version__: '2.5.0' INFO: tensorflow.__git_version__: 'v2.5.0-rc3-213-ga4dfb8d1a71' --- check: tensorboard_data_server_version INFO: data server binary: '/usr/local/lib/python3.6/dist-packages/tensorboard_data_server/bin/server' Traceback (most recent call last): File "/workspace/diagnose_tensorboard.py", line 522, in main suggestions.extend(check()) File "/workspace/diagnose_tensorboard.py", line 75, in wrapper result = fn() File "/workspace/diagnose_tensorboard.py", line 301, in tensorboard_data_server_version check=True, File "/usr/lib/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: TypeError: __init__() got an unexpected keyword argument 'capture_output' --- check: tensorboard_binary_path INFO: which tensorboard: b'/usr/local/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC =Next steps
No action items identified. Please copy ALL of the above output, including the lines containing only backticks, into your GitHub issue or comment. Be sure to redact any sensitive information.
Issue I see the following error when trying to run Tensorboard to visualize logs:
[ WARN rustboard_core::run] Read error in /workspace/log/events.out.tfevents.1624569739.8a1c50246dae.30295.276717.v2: ReadRecordError(BadLengthCrc(ChecksumError { got: MaskedCrc(0x07980329), want: MaskedCrc(0x00000000) }))
For some context, I am using a shared file system between several machines and this error comes up when trying to run Tensorboard on machine A, pointing to a log generated by machine B.