search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.03k stars 1.81k forks source link

show image on webUI #2446

Closed kingaza closed 3 years ago

kingaza commented 4 years ago

What would you like to be added: I want to see the image from model output, besides with metrics. But it seems only numbers can be seen in webUI

Why is this needed: metrics are not enough to evaluate the model performance, therefore I have to check the output image manually

Without this feature, how does current nni work： no idea

Components that may involve changes: I don't know

Brief description of your proposal if any: n.a.

chicm-ms commented 4 years ago

Hi @kingaza , Thanks for creating this issue. In tensorboard, we can use tf.summary to write different types of data ( such as input/output images, embeddings) into log file and then display them on UI. Is this feature similar as your requirement? Can you provide a little bit more details? such as what is the type of your model/task? Is there any specific requirements to display the images? Currently nni has not yet implemented this feature by now.

kingaza commented 4 years ago

thx for your reply. tensorboard is very useful for visualization. However, there could be hundreds of trials in one experiment due to many options in search space, and I don't want to open different tensorboard time and time again. Currently I make a specific folder and save the output images to this folder manually. So it does help me much if NNI can display even one image for every trial.

kingaza commented 4 years ago

In some cases, i.e. medical image generation, metrics like ssim/nrmse/psnr are not golden standard, so that reviewing output images is very important for performance evaluation, even though they are not comparable.

SparkSnail commented 4 years ago

Hi @kingaza , NNI support starting a tensorboard process using all trial's logpath, try nnictl tensorboard start --trial_id all command. refer

scarlett2018 commented 3 years ago

@kvartet - this might be a good reference scenariorequirement for "all experiments management" feature, FYI.