search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.06k stars 1.82k forks source link

Tensorboard logging #5609

Open ferqui opened 1 year ago

ferqui commented 1 year ago

Describe the issue: Hello, I'm using NNI v3.0 and trying to log using tensorboard. On the code I create the logging directory as log_dir = os.path.join(os.environ["NNI_OUTPUT_DIR"], "tensorboard") but when I try to open tensorboard on the webgui doesn't appear any logging. Looking at the output directory I noticed that the NNI_OUTPUT_DIR points towards to

nni-experiments/dh1ou4w7/environments/local-env/trials/dqOLE/tensorboard

but tensorboard is trying to read

nni-experiments/dh1ou4w7/trials/QlXHm/output/tensorboard

How can I get the correct one and what's the difference of both?

ferqui commented 1 year ago

Hello, is there any news regarding this?

Itomigna2 commented 10 months ago

I got same problem. I guess it is occurred from training service(local).

I fix the problem by the code below,

main .py code

log_dir = os.path.join(os.environ["PWD"], 'nni-experiments', os.environ["NNI_EXP_ID"], 'trials', os.environ["NNI_TRIAL_JOB_ID"], 'output/tensorboard')

.yml file

experimentWorkingDirectory: ./nni-experiments

I think it can be a problem when experiment setting is changed. I hope to know safe solution or the bug fixed.

Thanks for every contributors! NNI is really good and useful.