Feature: allow picking whatever pytorch lr_scheduler

[emanjavacas]

Here is a draft for the lr_scheduler params (related to #78) and how to avoid breaking backwards compatibility. Could you use it to test the results of the cosineannealing that you got?

[emanjavacas]

From the last discussion,t we concluded to leave out the stuff coming from 3D party libraries (including the delayed stuff)

[PonteIneptique]

Delayer is not third party, I coded it :) And it's only with Delayer that Cosine outperforms significantly Adam :)

[emanjavacas]

Could you pin point what is it that the Delayer class is achieving? I see a lot of new code involved in printing stuff and so on. Perhaps there is an easier way to add what the Delayer class contributes.

[PonteIneptique]

There is definitely. May I pr your PR ?

[emanjavacas]

Ye, go ahead. Only I'd rather really keep it simple. For instance if there is that class that delays the step, we don't need to add a bunch of new functions to adapt the printing code for that class.

[emanjavacas]

Check the last commit, that might solve the issues

[PonteIneptique]

Great !

[emanjavacas]

Any chance you could quickly replicate the results for the combination that was on par with ReduceLROnPlateau but was faster? (No rush here)

On Mon, Dec 7, 2020 at 6:25 PM Thibault Clérice notifications@github.com wrote:

Great !

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/emanjavacas/pie/pull/79#issuecomment-740062144, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPIPI4DR6VURKQ7RURLHH3STUFYFANCNFSM4UQORVYQ .

-- Enrique Manjavacas

[PonteIneptique]

Running:

   "lr_scheduler": "CosineAnnealingLR", // LR Scheduler to use: ReduceLROnPlateau,
   "lr_scheduler_delay": 10,
   "lr_scheduler_params": {
      "T_max": 40,
      "min_lr": 0.000001, // minimum learning rate
      "lr_patience":7,
      "factor":0.6,
   },

[PonteIneptique]

You don't want me to rerun the 2x10 runs right ?

[PonteIneptique]

It works: ran once on my personal GPU for 67 epochs (higher parts of the box in https://github.com/emanjavacas/pie/issues/76#issue-757884805) with score within the box 97.61. (previous version of comment was using dev score, because I am tired and can't read)