Tensorboard does not recognize absolute GCP storage path in projector_config

[RobbeSneyders]

System specifications

• TensorBoard version: 0.1.8
  • TensorFlow version: 1.3.0
  • OS Platform and version: Ubuntu 16.04 subsystem on Windows 10 wsl
  • Python version: 2.7

Problem specification

I am using a projector_config.pbtxt file to link my embedding tensors to their metadata. The config file is structured like this:

embeddings {
  tensor_name: "scope/first_tensor_name:0"
  metadata_path: "gs://bucket/not-logdir/first_dict"
}
embeddings {
  tensor_name: "scope/second_tensor_name:0"
  metadata_path: "gs://bucket/not-logdir/second_dict"
}
...

When I load the projector in tensorboard, I get the following error message:

Error fetching metadata "gs://bucket/logdir/gs://bucket/not-logdir/first_dict" not found, or is not a file

Tensorboard uses my absolute path as a relative path.

Source code

The metadata path is fetched here and constructed to an absolute path here:

def _rel_to_abs_asset_path(fpath, config_fpath):
  fpath = os.path.expanduser(fpath)
  if not os.path.isabs(fpath):
    return os.path.join(os.path.dirname(config_fpath), fpath)
  return fpath

This snippet shows that the path should only be made absolute if it is not an absolute path already. The os.path.isabs() function however fails to detect if a GCP storage path is absolute.

Is there another way to specify a metadata path outside of the logdir on GCP storage or should this be fixed?

The relevant code seems to be unchanged between my version 0.1.8 and the current version 1.6.0.

[chihuahua]

The metadata_path property should be relative to the log directory. Does specifying a path such as ../not-logdir/first_dict work?

[RobbeSneyders]

Specifying a relative path does not work either.

With the following config:

embeddings {
  tensor_name: "scope/first_tensor_name:0"
  metadata_path: "../not-logdir/first_dict"
}

Tensorboard shows the following error message:

Error fetching metadata "gs://bucket/logdir/../not-logdir/first_dict" not found, or is not a file

Which can be expected, since os.path.join() can't handle relative paths. I was looking into os.path.normpath() to collapse the relative path, but unfortunately it also removes double slashes:

>>> dict_path = os.path.join('gs://bucket/logdir', '../not-logdir/first_dict')
# 'gs://bucket/logdir/../not-logdir/first_dict'
>>> os.path.normpath(dict_path)
# 'gs:/bucket/not-logdir/first_dict'

[chihuahua]

Hmm, is there a chance you could move your mappings into the log directory? Or within a subdirectory of the log directory?

[RobbeSneyders]

It is possible to copy the mappings into the log directory, however I think it would be valuable for Tensorboard to support other cases. Especially since this only seems to be a requirement for GCS, which I suspect is one of the main file systems used with tensorboard.

[suiluj]

Does someone have a solution for metadata and embedding checkpoints in subfolders in the meantime?

I want to do the same thing because i want to compare embeddings of different training runs which are stored in subfolders of course.

Even with the TensorBoard callback i was not able to specify a correct metadata path.

The documentation is not clear and a link for details in the documation is not reachable anymore.

I really would like to use the embedding projector with metadata and multiple runs.

Here is an issue i created a few days ago with all the links to the tensorflow documentation:

https://github.com/tensorflow/tensorflow/issues/47565 ( TensorBoard Embedding Projector Metadata filepath format not clear )

I would really appreciate any small hint. Thank you.