microsoft / vscode-tensorboard

Launch and view Tensorboards in VS Code
MIT License
4 stars 3 forks source link

VSCode Tensorboard extension not finding events data #9

Closed TylerAnderton closed 8 months ago

TylerAnderton commented 1 year ago

Tensorboard says "No scalar[/image/etc] data was found."

I have verified my data is logging in the correct place by opening Tensorboard manually through the terminal/browser. I can also see the events files where they should be.

I have tried both A) putting my event files in the working directory and selecting "Use current working directory" B) leaving my files in my log folder and selecting "Select another folder"

I have gone back to previous projects that used to work just fine, and changing nothing, I now experience the same issue.

I noticed I had to switch to the Pre-Release version of the extension, because attempting to install the Release version returned this error message "Can't install release version of 'ms-toolsai.tensorboard' extension because it has no release version." If I had to guess, I would say the Pre-Release version is not pointing to the correct log folder when using either of the above options.

I am on a 2021 Macbook pro for reference, macos 12.6.3.

karrtikr commented 1 year ago

cc/ @DonJayamanne

DonJayamanne commented 1 year ago

@TylerAnderton sorry you are running into this issue

Tensorboard says "No scalar[/image/etc] data was found."

Does it work for you when you manually launch tensorboard from the terminal?

TylerAnderton commented 1 year ago

I followed your instructions, but I'm not seeing any difference in the Tensorboard output panel.

As you can see in the screenshot, Tensorboard is being pointed to the correct log folder, but it can't see any of the events that are clearly in there. I have also tried removing the events from the subfolders, placing them directly in /log/, but that didn't work either.

Here's my settings.json file contents:

{
    "tensorBoard.log": "debug"
}

Screen Shot 2023-10-25 at 11 57 11 AM

Tensorboard does work when I launch with the terminal and view in the browser.

DonJayamanne commented 1 year ago

Tensorboard does work when I launch with the terminal and view in the browser.

Thank you for getting back. At this point its not a problem with the extension As you cannot get this working outside VS Code/extension, its more to do with some confirmation or the like at your end. I'm sorry, but there's not mure more I can help with as I'm not really an expert with Tensorboard pyton package and the CLI.

I'd suggest trying their repo or Stackoverflow

TylerAnderton commented 1 year ago

I said Tensorboard does work when I launch with the terminal and view in the browser (ie outside VS Code/extension).

The problem is clearly with the VSCode extension.

DonJayamanne commented 1 year ago

I said Tensorboard does work when I launch with

so sorry about that

DonJayamanne commented 1 year ago

Please could you open the output panel in vscode

TylerAnderton commented 1 year ago

This? I'm not sure what "output" you're referring to that has any new/different information.

Sharing a screenshot because I literally cannot copy/paste the text from Tensorboard. Screen Shot 2023-10-25 at 4 57 25 PM

DonJayamanne commented 1 year ago

Screenshot 2023-10-26 at 10 59 03

TylerAnderton commented 1 year ago

From my VSCode project:

16:52:13.558 [debug] Spawning process: /Users/tyler/miniconda3/envs/cs394d/bin/python /Users/tyler/.vscode/extensions/ms-toolsai.tensorboard-2023.10.1002931100/pythonFiles/tensorboard_launcher.py "/Users/tyler/GitHub Repositories/UT Austin/2023 Fall/CS394D Deep Learning/Homework 3/homework3/log" in /Users/tyler/GitHub Repositories/UT Austin/2023 Fall/CS394D Deep Learning/Homework 3/homework3/log
16:52:16.865 [error] 
NOTE: Using experimental fast data loading logic. To disable, pass
    "--load_fast=false" and report issues on GitHub. More details:
    https://github.com/tensorflow/tensorboard/issues/4784

16:52:16.960 [error] INFO:pytorch_profiler:Monitor runs begin

From VSCode Insiders, following your instructions above:

21:07:56.244 [debug] Spawning process: /usr/local/bin/python3 /Users/tyler/.vscode-insiders/extensions/ms-toolsai.tensorboard-2023.10.1002931100/pythonFiles/tensorboard_launcher.py "/Users/tyler/GitHub Repositories/UT Austin/2023 Fall/CS394D Deep Learning/Homework 3/homework3/log" in /Users/tyler/GitHub Repositories/UT Austin/2023 Fall/CS394D Deep Learning/Homework 3/homework3/log
TylerAnderton commented 1 year ago

Any updates on this?

DonJayamanne commented 1 year ago

said Tensorboard does work when I launch with the terminal and view in the browser (ie outside VS Code/extension).

which python environment are you using in the terminal? in vscode you seem to be using miniconda, and vscode insiders you are using another version

TylerAnderton commented 1 year ago

My apologies for the delayed response and the confusion. I had not properly set my python interpreter in VSCode Insiders.

I set up a fresh miniconda environment and confirmed the issue still persists in both VSCode and VSCode insiders, but I am still able to view my logged data using the terminal/browser within the same miniconda environment.

vscode cs394d_test vscode_insiders cs394d_test
gabrielbcn commented 11 months ago

I'm in the same situation, working in terminal, not working in VSCode (extension), my logs:

00:05:39.150 [debug] Spawning process: /opt/homebrew/Caskroom/miniforge/base/envs/XCS234_A2/bin/python /Users/gabriel/.vscode/extensions/ms-toolsai.tensorboard-2023.10.1002992421/pythonFiles/tensorboard_launcher.py "/Users/gabriel/Documents/learning/stanford/XCS234 RL/XCS234-A2/src/results/q7_dqn" in /Users/gabriel/Documents/learning/stanford/XCS234 RL/XCS234-A2/src/results/q7_dqn
00:05:41.492 [error] 
NOTE: Using experimental fast data loading logic. To disable, pass
    "--load_fast=false" and report issues on GitHub. More details:
    https://github.com/tensorflow/tensorboard/issues/4784

00:05:41.493 [error] WARNING:tensorboard-plugin-profile:Unable to load profiler plugin. Import error: No module named 'tensorflow'

00:05:41.542 [error] INFO:pytorch_profiler:Monitor runs begin
pavelacamposp commented 11 months ago

I recently encountered the same problem with the VS Code Tensorboard extension, where it displayed "No scalar data was found", despite the data being correctly logged and visible when using Tensorboard through a terminal.

I found out that this issue occurred only with logdir paths that contained spaces. After removing them, the Tensorboard extension functioned without any problems. This behavior was consistent on both Windows and Linux (Ubuntu) systems.

It might be worth checking the path for spaces or special characters that could cause issues in the path recognition of the Tensorboard extension.

I hope this helps!

paprikodlak commented 10 months ago

I have the same problem, when I launch tensorboard from command line, it works as expected. When I launch it in VS Code, it does not find the logs. Here is the tensorboard output

08:09:39.707 [debug] Spawning process: xx\.venv\Scripts\python.exe d:/Users/pawlik/.vscode/extensions/ms-toolsai.tensorboard-2023.10.1002992421/pythonFiles/tensorboard_launcher.py "%homepath%/project/logs" in %homepath%\project\logs
08:09:40.333 [error] 2024-01-25 08:09:40.333598: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variab
08:09:40.334 [error] le `TF_ENABLE_ONEDNN_OPTS=0`.

08:09:42.277 [error] WARNING:tensorflow:From %homepath%\project\.venv\Lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
paprikodlak commented 10 months ago

The problem seems to be with the folder selection and passing to the script.

I suspect that the path passed as logdir variable is fully qualified, but the tensorboard treats it as relative.

When I hardcode the path, it works fine.

# file: .vscode\extensions\ms-toolsai.tensorboard-2023.10.1002992421\pythonFiles\tensorboard_launcher.py 
import time
import sys
import os
import mimetypes
from tensorboard import program

def main(logdir):
    # Environment variable for PyTorch profiler TensorBoard plugin
    # to detect when it's running inside VS Code
    os.environ["VSCODE_TENSORBOARD_LAUNCH"] = "1"

    # Work around incorrectly configured MIME types on Windows
    mimetypes.add_type("application/javascript", ".js")

    # Start TensorBoard using their Python API
    tb = program.TensorBoard()

    """  !! hard coding logs path !!       """
    logdir = "./logs"
    """ !!! """

    tb.configure(bind_all=False, logdir=logdir)
    url = tb.launch()
    sys.stdout.write("TensorBoard started at %s\n" % (url))
    sys.stdout.flush()

    while True:
        try:
            time.sleep(60)
        except KeyboardInterrupt:
            break
    sys.stdout.write("TensorBoard is shutting down")
    sys.stdout.flush()

if __name__ == "__main__":
    if len(sys.argv) == 2:
        logdir = str(sys.argv[1])
        sys.stdout.write("Starting TensorBoard with logdir %s" % (logdir))
        main(logdir)
DonJayamanne commented 9 months ago

Oh thank you for investigating this issue and proving the information. so, it seems tensorboard code requires relative paths,

DonJayamanne commented 9 months ago

@paprikodlak

ms-toolsai-tensorboard-insiders.vsix.zip

paprikodlak commented 9 months ago

Seems to be working as expected.

Ksorz commented 8 months ago

Windows VSCode: Kill old process, try this: 1) taskkill /F /PID PID number 2) taskkill /IM "tensorboard.exe" /F 3) delete files from: "\AppData\Local\Temp\.tensorboard-info"

Start tb again: log_dir = 'GLOBAL PATH TO LOG FOLDER' %load_ext tensorboard %tensorboard --logdir "{log_dir}"