Adafactor - IndexError: list index out of range

[rayshan]

When using this optimizer config:

I get this error:

I think when I hit this line, name === 'adafactor', so it cannot be split. I think learning rate was supposed to be appended to the string in this line but it didn't.

Recommendation: store values in variables instead of manipulating strings.

[Linaqruf]

You can't use adafactor scheduler if relative_step is False

I suggest to stick with constant with warmup

[rayshan]

Got it, thank you!