Open AlonKellner opened 8 months ago
arcra
commented
8 months ago Can you clarify what you mean by "all experiments"? Are you referring to the runs? Could you share a screenshot, to make that clearer?
Do you see any errors in the console logs?
Could it be something related or similar to what is reported in #6713?
AlonKellner
commented
8 months ago Can you clarify what you mean by "all experiments"? Are you referring to the runs? Could you share a screenshot, to make that clearer?
Yes, sorry for using the wrong terminology, when I wrote "experiments" I was referring to "runs".
As for screenshots, I've added screenshots to the reproduction repo, there are 6 of them that explain the full problem, so I won't share all of them here, but here is the one from step-3, when there are 2 runs in minio, but only 1 run is visible in tensorboard:
Do you see any errors in the console logs?
No, the only console logs are:
TensorFlow installation not found - running with reduced feature set.
TensorBoard 2.15.1 at http://2666274e9da3:6006/ (Press CTRL+C to quit)Could it be something related or similar to what is reported in #6713?
It does not seem like it, since I do not see any errors.
arcra
commented
8 months ago Yes, sorry for using the wrong terminology, when I wrote "experiments" I was referring to "runs".
No worries, I just wanted to make sure I understand what the issue is correctly.
I believe it's (similarly to #6713) an issue with our "no-TF compatibility" implementation of the GFile interface, particularly the support for the S3 files. I believe a workaround might be to install tensorflow, so it would use the TF implementation. If you do, please confirm whether that solves the problem for you.
Unfortunately, we don't have the bandwidth to investigate this with more detail. Our compat support for the S3 filesystem is done as best-effort.
AlonKellner
commented
7 months ago I tried to install tensorflow, but then I get the error:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/lightning/fabric/loggers/tensorboard.py", line 208, in log_metrics
self.experiment.add_scalar(k, v, step)
File "/usr/local/lib/python3.10/site-packages/lightning/fabric/loggers/logger.py", line 118, in experiment
return fn(self)
File "/usr/local/lib/python3.10/site-packages/lightning/fabric/loggers/tensorboard.py", line 191, in experiment
self._experiment = SummaryWriter(log_dir=self.log_dir, **self._kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/utils/tensorboard/writer.py", line 243, in __init__
self._get_file_writer()
File "/usr/local/lib/python3.10/site-packages/torch/utils/tensorboard/writer.py", line 273, in _get_file_writer
self.file_writer = FileWriter(
File "/usr/local/lib/python3.10/site-packages/torch/utils/tensorboard/writer.py", line 72, in __init__
self.event_writer = EventFileWriter(
File "/usr/local/lib/python3.10/site-packages/tensorboard/summary/writer/event_file_writer.py", line 72, in __init__
tf.io.gfile.makedirs(logdir)
File "/usr/local/lib/python3.10/site-packages/tensorflow/python/lib/io/file_io.py", line 513, in recursive_create_dir_v2
_pywrap_file_io.RecursivelyCreateDir(compat.path_to_bytes(path))
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented (file: 's3://tensorboard/test-test/version_0')My workaround is a bad one, I wrote a simple bash script that restarts tensorboard every minute, that way it reloads all runs every minute, which works for my use-case.
arcra
commented
7 months ago Looks like there's a separate package that might provide support for that filesystem. Can you try installing tensorflow-io and see if that solves your problem?
Sources: https://discuss.tensorflow.org/t/access-s3-on-tensorflow/8633 https://blog.ukjae.io/posts/enabling-s3-filesystem-support-for-tensorflow-serving/
Im addition to this issue, I posted a question in stackoverflow.
Environment information
Diagnostics
Diagnostics output
`````` --- check: autoidentify INFO: diagnose_tensorboard.py version df7af2c6fc0e4c4a5b47aeae078bc7ad95777ffa --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=10, micro=13, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='c5ff1db54ce4', release='5.15.133.1-microsoft-standard-WSL2', version='#1 SMP Thu Oct 5 21:02:42 UTC 2023', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: $VIRTUAL_ENV: None --- check: installed_packages INFO: installed: tensorboard==2.15.1 WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview'] WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly'] INFO: installed: tensorboard-data-server==0.7.2 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.15.1' --- check: tensorflow_python_version Traceback (most recent call last): File "//diagnose_tensorboard.py", line 511, in main suggestions.extend(check()) File "//diagnose_tensorboard.py", line 81, in wrapper result = fn() File "//diagnose_tensorboard.py", line 267, in tensorflow_python_version import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow' --- check: tensorboard_data_server_version INFO: data server binary: '/usr/local/lib/python3.10/site-packages/tensorboard_data_server/bin/server' INFO: data server binary version: b'rustboard 0.7.2' --- check: tensorboard_binary_path INFO: which tensorboard: b'/usr/local/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC =Issue description
Here is a repository with a full reproduction of my issue: https://github.com/AlonKellner/s3-tensorboard-issue-reproduction
When using tensorboard with an s3 compatible storage, only the first experiments that the server comes across are shown in the UI.
All experiments that are present during startup are shown fully, if no experiment is present during start up, the first detected experiment will be shown partially.
After an experiment is detected and shown, no further steps and experiments will be reloaded and shown.
When using the
--reload_task processoption, no experiment is shown whatsoever.I have personally reproduced this unexpected behavior with both ceph (with an on-prem instance) and minio (with a local docker image, see reproduction repo).
The expected behavior is that any new experiment that is written to the s3 compatible storage should be reloaded in the UI when pressing the
reloadbutton, as well as new steps in that new experiment.Also, I expect this behavior to work correctly with the
--reload_multifile=trueoption.Workarounds are also welcome, thanks :)