Customer optimizer implementation

[gokceuludogan]

The optimizer implementation has been updated to allow the use of custom optimizers and schedulers. You can provide the necessary parameters to training_params.

To use the predefined adafactor optimizers, include the following parameters:

optimizer_params:
  optimizer_type: adafactor
  scheduler: False

Training with the default trainer optimizer and scheduler can be done as follows:

optimizer_params:
  optimizer_type: default
  scheduler: False

Here's an example of how to use a custom optimizer in training arguments:

training_params:
  learning_rate: 5e-5
  optim: adamw_torch
  weight_decay: 0
  adam_beta1: 0.9
  adam_beta2: 0.999
  adam_epsilon: 1e-8
  lr_scheduler_type: linear
  lr_scheduler_kwargs: {}
  warmup_ratio: 0.0
  warmup_steps: 0

Note that custom parameters in training arguments and optimizer_params shouldn't be used simultaneously.