Resolved: Feature/#48

[jieonh]

Motivation 🤔

Inference logging advancement

• Save additional informations with test score (model, dataset, optimizer, etc.)
• Save it to csv table form for convenient experimental analysis

Key Changes 🔑

1. Inference logging : Additional informations are retrieved in two ways (using try-except syntax)
   • hydra yaml files
   • General trial informations are stored in .hydra automatically when executing main.py.
   • For now, log_dir, test_roc_auc, Dataset, Model, Optimizer, loss_func, hparams_search, Multirun, epoch are extracted among those informations. (log_dir is the index of the csv)
   • function: _load_hydra_config_
   • model checkpoint
   • This will probably be needed when user wants to brings their own model checkpoints for inference. (=For users who don't have hydra config file)
   • Previously, I modified the code(train.py) to store various informations(model name, used data set , etc.) when saving best model checkpoint. So here, we use this information instead of the hydra yaml file.

• Inference logging would look like this time stamp directory made by hydra └─ inference_result.csv └─test_accuracy.yaml

2. Minor change Thanks to @chrstnkgn , I found that there was a timestamp key value overlapping problem in the case of multirun when executing inference.py. So the key was changed to include multirun informations. ex) 2023-01-02_05-23-02 : ./logs/train/2023-01-02_01-15-06/model=densenet/best_saved.pth -> 2023-01-02_05-23-02/model=resnet : ./logs/train/2023-01-02_01-15-06/model=densenet/best_saved.pth

To Reviewers 🙏

• There was an error when executing with raytune, but I think that parts I worked on are clearly not related to train and raytune, so I opened a pull request.
• If necessary, it is possible to add more configs other than current csv columns.
• Any suggestion and opinions are very welcomed

resolves: #48

[jieonh]

@seoulsky-field
I'm sorry for the delay in answering, and thank you for the good question! 👍 I double checked that part and those empty values above mean null because those train logs were executed before the hparams_search default option was changed to none (= when default was set to null).

[jieonh]

@kdg1993 Could I ask more details about motivation and key points of the inference structure that you mentioned above? I'd be happy to explain !