search

allenai / allenact

An open source framework for research in Embodied-AI from AI2.
https://www.allenact.org
Other
316 stars 52 forks source link

Enable to log metrics regardless of train/val/test #292

Open qiwu57kevin opened 3 years ago

qiwu57kevin commented 3 years ago

Problem

I would like to log metrics in a document during training and validation as what you have done for testing. Currently I have to open tensorbard and get

Desired solution

Is it possible to use a flag, like the tensorboard logging, to enable or disable metric logging regardless of stages?

Lucaweihs commented 3 years ago

Hi @KevinWu57,

PR #253 actually introduces an update to allow for saving validation metrics via a --collect_valid_results flag, this should be merged in the next few days (I just need to find time for a final review) but you should be able to install this version of AllenAct ahead of time if you'd like:

pip install -e "git+https://github.com/allenai/allenact.git@b6c4a2f7ae1759c3e40eb0b849f9778093b79ddc#egg=allenact&subdirectory=allenact"
pip install -e "git+https://github.com/allenai/allenact.git@b6c4a2f7ae1759c3e40eb0b849f9778093b79ddc#egg=allenact_plugins&subdirectory=allenact_plugins"

Saving the training metrics to a metric file is something we hadn't thought of so might take some more effort. Note that you can always run testing but update the relevant experiment config file to use training dataset instead of the test dataset as a temporary workaround. If we built a function that let you extract the training metrics from the tensorboard file, would that be sufficient for you? I.e. are you just looking for a way to get these metrics without having to run tensorboard explicitly?

qiwu57kevin commented 3 years ago

Hi @Lucaweihs ,

Thanks for your response! Yes, I think a function that can extract the training metrics from the tensorboard file would be good for me so that I don't have to run tensorboard.