search

tensorflow / tensorboard

TensorFlow's Visualization Toolkit
Apache License 2.0
6.6k stars 1.64k forks source link

Embedding metadata is still not read or displayed in TensorBoard #5247

Open ldorigo opened 2 years ago

ldorigo commented 2 years ago

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same environment from which you normally run TensorFlow/TensorBoard, and paste the output here:

Diagnostics

Diagnostics output `````` --- check: autoidentify INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='lenz-ubuntu', release='5.11.0-25-generic', version='#27~20.04.1-Ubuntu SMP Tue Jul 13 17:41:23 UTC 2021', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: $VIRTUAL_ENV: '/home/luca/medrel/medrel_venv' --- check: installed_packages INFO: installed: tensorboard==2.6.0 INFO: installed: tensorflow==2.6.0 INFO: installed: tensorflow-estimator==2.6.0 INFO: installed: tensorboard-data-server==0.6.1 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.6.0' --- check: tensorflow_python_version 2021-08-15 14:57:17.271237: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2021-08-15 14:57:17.271258: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. INFO: tensorflow.__version__: '2.6.0' INFO: tensorflow.__git_version__: 'v2.6.0-rc2-32-g919f693420e' --- check: tensorboard_data_server_version INFO: data server binary: '/home/luca/medrel/medrel_venv/lib/python3.8/site-packages/tensorboard_data_server/bin/server' INFO: data server binary version: b'rustboard 0.6.1' --- check: tensorboard_binary_path INFO: which tensorboard: b'/home/luca/medrel/medrel_venv/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC = socket.SOCK_STREAM = socket.AI_ADDRCONFIG = socket.AI_PASSIVE = Loopback flags: Loopback infos: [(, , 6, '', ('::1', 0, 0, 0)), (, , 6, '', ('127.0.0.1', 0))] Wildcard flags: Wildcard infos: [(, , 6, '', ('0.0.0.0', 0)), (, , 6, '', ('::', 0, 0, 0))] --- check: readable_fqdn INFO: socket.getfqdn(): 'lenz-ubuntu' --- check: stat_tensorboardinfo INFO: directory: /tmp/.tensorboard-info INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=1458200, st_dev=2050, st_nlink=2, st_uid=1004, st_gid=1004, st_size=4096, st_atime=1629031205, st_mtime=1629032212, st_ctime=1629032212) INFO: mode: 0o40777 --- check: source_trees_without_genfiles INFO: tensorboard_roots (1): ['/home/luca/medrel/medrel_venv/lib/python3.8/site-packages']; bad_roots (0): [] --- check: full_pip_freeze INFO: pip freeze --all: abbreviations==0.2.5 absl-py==0.13.0 anyio==3.3.0 appdirs==1.4.4 argon2-cffi==20.1.0 astunparse==1.6.3 async-generator==1.10 attrs==21.2.0 Babel==2.9.1 backcall==0.2.0 biopython==1.79 black==21.6b0 bleach==3.3.1 blis==0.7.4 cachetools==4.2.2 catalogue==2.0.4 certifi==2021.5.30 cffi==1.14.6 chardet==4.0.0 charset-normalizer==2.0.4 clang==5.0 click==7.1.2 conllu==4.4 cupy-cuda101==9.2.0 cymem==2.0.5 data-science-types==0.2.23 debugpy==1.4.1 decorator==5.0.9 defusedxml==0.7.1 deprecation==2.1.0 en-core-sci-lg==0.4.0 en-core-sci-md==0.4.0 entrypoints==0.3 fastrlock==0.6 flatbuffers==1.12 gast==0.4.0 gensim==4.0.1 google-auth==1.34.0 google-auth-oauthlib==0.4.5 google-pasta==0.2.0 grpcio==1.39.0 gviz-api==1.9.0 h5py==3.1.0 idna==3.2 ipykernel==6.0.3 ipython==7.25.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 jedi==0.18.0 Jinja2==3.0.1 joblib==1.0.1 json5==0.9.6 jsonschema==3.2.0 jupyter-client==6.1.12 jupyter-core==4.7.1 jupyter-packaging==0.10.4 jupyter-server==1.10.1 jupyterlab==3.0.14 jupyterlab-pygments==0.1.2 jupyterlab-server==2.6.1 jupyterlab-widgets==1.0.0 keras==2.6.0 keras-nightly==2.5.0.dev2021032900 Keras-Preprocessing==1.1.2 Lasagne==0.1 Markdown==3.3.4 MarkupSafe==2.0.1 matplotlib-inline==0.1.2 mistune==0.8.4 more-itertools==8.8.0 murmurhash==1.0.5 mypy==0.902 mypy-extensions==0.4.3 nbclassic==0.3.1 nbclient==0.5.3 nbconvert==6.1.0 nbformat==5.1.3 nest-asyncio==1.5.1 networkx==2.6.2 nltk==3.6.2 nmslib==2.1.1 notebook==6.4.0 numpy==1.21.1 oauthlib==3.1.1 obonet==0.3.0 opt-einsum==3.3.0 packaging==21.0 pandocfilters==1.4.3 parso==0.8.2 pathspec==0.8.1 pathy==0.6.0 pexpect==4.8.0 pickleshare==0.7.5 pip==20.0.2 pkg-resources==0.0.0 preshed==3.0.5 prometheus-client==0.11.0 prompt-toolkit==3.0.19 protobuf==3.17.3 psutil==5.8.0 ptyprocess==0.7.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.6.1 pycparser==2.20 pydantic==1.8.2 pydot==1.4.2 Pygments==2.9.0 pyparsing==2.4.7 pyrsistent==0.18.0 pysbd==0.3.4 python-dateutil==2.8.1 python-Levenshtein==0.12.2 pytz==2021.1 PyYAML==5.4.1 pyzmq==22.1.0 regex==2021.4.4 requests==2.26.0 requests-oauthlib==1.3.0 requests-unixsocket==0.2.0 rsa==4.7.2 scikit-learn==0.24.2 scipy==1.7.1 scispacy==0.4.0 Send2Trash==1.7.1 setuptools==57.4.0 six==1.15.0 smart-open==5.1.0 sniffio==1.2.0 spacy==3.0.7 spacy-legacy==3.0.8 sqlparse==0.4.1 srsly==2.4.1 tensorboard==2.6.0 tensorboard-data-server==0.6.1 tensorboard-plugin-profile==2.5.0 tensorboard-plugin-wit==1.8.0 tensorflow==2.6.0 tensorflow-estimator==2.6.0 termcolor==1.1.0 terminado==0.10.1 testpath==0.5.0 Theano==1.0.5 thinc==8.0.8 threadpoolctl==2.2.0 toml==0.10.2 tomlkit==0.7.2 tornado==6.1 tqdm==4.62.0 traitlets==5.0.5 typer==0.3.2 types-python-dateutil==0.1.4 typing-extensions==3.7.4.3 Unidecode==1.2.0 urllib3==1.26.6 vprof==0.38 wasabi==0.8.2 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.1.0 Werkzeug==2.0.1 wheel==0.36.2 widgetsnbextension==3.5.1 wrapt==1.12.1 ``````

Issue description

Specifying the metadata file for the embeddings projector does not work - it fails silently and never gets loaded.

Sample code:

    tensorboard_callback = tf.keras.callbacks.TensorBoard(
        log_dir="/hdd/logs",
        profile_batch=(2, 5),
        histogram_freq=1,
        embeddings_freq=1,
        embeddings_metadata="/hdd/logs/metadata.tsv",
    )

This is the exact same issue as #4246 (https://github.com/tensorflow/tensorboard/issues/4246). It was closed despite the "solution" being a manual workaround that only fixes part of the problem.

Reopening to get a proper fix.

bileschi commented 2 years ago

Thanks for reporting this issue, @ldorigo ! Sorry you are having trouble?

Would it be too much to ask for a minimal reproduction? The embedding projector was inherited by and unfamiliar to much the TB team, and you may be able to save us a great deal of time and get a much faster fix.

Thanks! Sincerely, Stan on behalf on the TensorBoard team.

ldorigo commented 2 years ago

Hi, unfortunately I'm away from my workstation for the next week, so won't be able to provide an example very soon. If still needed when I'm back I'll be happy to oblige.

Note that the issue I linked to describes how to reproduce it in some more detail, maybe that is enough for you.