FluxML / FluxTraining.jl

A flexible neural net training library inspired by fast.ai
https://fluxml.ai/FluxTraining.jl
MIT License
118 stars 25 forks source link

Switch to ParameterSchedulers.jl #107

Closed rejuvyesh closed 2 years ago

rejuvyesh commented 2 years ago

Fixes #106.

Not sure if you want to keep an equivalent of onecycle.

rejuvyesh commented 2 years ago

I tried adding back an approximate version of onecycle based on existing schedulers in ParameterSchedulers but it's not exactly the same.

darsnack commented 2 years ago

I guess you want the ability to have a sequence of two sine waves with different periods, and the second starting at an offset t == half period of first sine?

darsnack commented 2 years ago

With ParameterScheduler@0.3.1, you should be able to do this like:

function onecycle(
        nsteps, max_val;
        pct_start=0.25,
        div=25, divfinal=1e5,
        start_val=max_val / div, end_val=max_val / divfinal)
    warmup = ceil(Int, nsteps * pct_start)
    warmdown = nsteps - warmup

    Sequence(Sin(λ0=max_val, λ1=start_val, period=2*warmup) => warmup,
             Shifted(Sin(λ0=max_val, λ1=end_val, period=2*warmdown), warmdown + 1) => warmdown)
end
darsnack commented 2 years ago

I think the docstring should be updated to:

Callback for hyperparameter scheduling. Takes pairs of [`HyperParameter`](#)
types and [ParameterSchedulers.jl schedules](https://darsnack.github.io/ParameterSchedulers.jl/dev/README.html).
darsnack commented 2 years ago

@lorenzoh what do you think about this? Otherwise, I think this is good to go?