Question about adam_onecycle optimizer

[junha-l]

Hi,

I'm currently using a custom adam_onecycle optimizer implemented in this codebase, with true_wd=True and bn_wd=True, along with its associated scheduler. However, I've noticed that PyTorch provides AdamW optimizer and OneCycleLR scheduler, which seem to have similar functionality.

I ran some experiments switching between custom adam_onecycle and PyTorch's implementation, and I noticed significant performance differences.

Could you help me in understanding the main differences between:

1. custom adam_onecycle optimizer (with true_wd=True, bn_wd=True) + its scheduler
2. PyTorch's AdamW + OneCycleLR scheduler

Specifically, I'm interested in understanding what might be causing the performance differences I observed.

Thanks!

[bollossom]

Hi,

I'm currently using a custom adam_onecycle optimizer implemented in this codebase, with true_wd=True and bn_wd=True, along with its associated scheduler. However, I've noticed that PyTorch provides AdamW optimizer and OneCycleLR scheduler, which seem to have similar functionality.

I ran some experiments switching between custom adam_onecycle and PyTorch's implementation, and I noticed significant performance differences.

Could you help me in understanding the main differences between:

1. custom adam_onecycle optimizer (with true_wd=True, bn_wd=True) + its scheduler
2. PyTorch's AdamW + OneCycleLR scheduler

Specifically, I'm interested in understanding what might be causing the performance differences I observed.

Thanks!

Does adamw perform better than adam? How much better?

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.