Feature: Refine Hyperparameter Tuning

[jieonh]

What

Overall parameter tuning is required when finalizing the benchmark design. In order to provide detailed optimized tuning results for each tasks like retina benchmark, it is necessary to refine current hyper parameter tuning structure.

Why

If hyperparameter tuning is going to performed throughout the code in addition to current basic config tuning (lr, batch_size, etc.), there are some parts that need to be changed in the current structure. The following areas might be considered:

1. Parameters that included only in specific cases

   • ex) gamma_neg, gamma_pos in AssymetricLoss

2. Some tuning results might vary depending on the combination

   • ex) best learning rate for each model architecture
     • DenseNet : 1e-4, ResNet: 1e-5

3. Currently, all parameters are included in the ray tune config - param_space, but this part needs to be divided in detail. ex)

   • gamma_neg, gamma_pos -> AssymetricLoss config
   • lr, weight_decay, betas, eps -> Optimizer config
   • batch_size, seed -> Experimental setting config

ref: retina_benchmark

How

• [ ] Include ASL configs in searchspace -> To work only when using ASLoss

(The part below is still in the process of planning)

• [ ] [python code] Modify hyper parameter selecting structure (in train.py- trainval, main.py-defaul,raytune )
• [ ] [hydra yaml config] Refinement search space structure

[kdg1993]

While I'm trying to follow up the feature/#76 since the updated codes on the commit seems very nice, I saw a tiny part that looked like a typo.

I believe that you already know about this, but I decided to mention it just in case.