Open MLfreakPy opened 1 month ago
Maybe @JQGoh, @jmorales, @BrunoBelucci you have some idea of how to solve it?
Hello! Can you send a code snippet of OneCycle that works and one for ReduceLROnPlateau to reproduce on my end? Thanks!
Hello,
To the best of my knowledge, it is not currently possible to use ReduceLROnPlateau out-of-the-box with the existing implementation of neuralforecast. As mentioned in this comment, this would require the user to provide a "lr_scheduler_config," which is not directly supported at this time. This "lr_scheduler_config" should be returned by the configure_optimizers method in Lightning (refer to this documentation for more details).
In the "lr_scheduler_config," you need to specify the "monitor" parameter with the metric you're tracking to detect the plateau. Typically, the "interval" is also set to "epoch" because the plateau is usually identified per epoch rather than per step, though this may vary depending on your model.
Currently, I am not using neuralforecast with ReduceLROnPlateau, so I cannot offer an immediate solution. However, based on our previous discussion in #998, I believe the simplest workaround for now is to monkey-patch the configure_optimizers function of your model to return a dictionary as expected by Lightning. This dictionary should include an "optimizer" key and, optionally, a "lr_scheduler" key, which can be a single LR scheduler or a lr_scheduler_config (as specified in the documentation).
If you do not manage to solve this, you can provide me a minimum (not) working example and I can see if I can fix it for you during my free time.
Regards, Bruno
Thank you so much for your explanations and looking into how to resolve the issue!!
Below a code snippet that (a) works with OneCycleLR and (b) that reproduces the error with ReduceLRonPlateau.
# GENERAL + DATA-PROCESSING
!pip install neuralforecast
from neuralforecast import NeuralForecast
from neuralforecast.models import NHITS
from neuralforecast.utils import AirPassengersDF
Y_df = AirPassengersDF # Defined in neuralforecast.utils
Y_df.head()
a. - OneCycleLR
from torch.optim.lr_scheduler import OneCycleLR
lr_scheduler_cls = OneCycleLR
max_lr = 1e-2
lr_scheduler_kwargs = {
'max_lr': max_lr,
'total_steps': 50}
horizon = 12
models = [ NHITS(h=horizon,
input_size=2 * horizon,
max_steps=50,
lr_scheduler = lr_scheduler_cls,
lr_scheduler_kwargs = lr_scheduler_kwargs) ]
nf = NeuralForecast(models=models, freq='M')
nf.fit(df=Y_df)
b. -ReduceLROnPlateau
from torch.optim.lr_scheduler import ReduceLROnPlateau
lr_scheduler_cls = ReduceLROnPlateau
lr_scheduler_kwargs = {
'mode': 'min',
'factor': 0.5,
'patience': 2}
horizon = 12
models = [ NHITS(h=horizon,
input_size=2 * horizon,
max_steps=50,
lr_scheduler = lr_scheduler_cls,
lr_scheduler_kwargs = lr_scheduler_kwargs) ]
nf = NeuralForecast(models=models, freq='M')
nf.fit(df=Y_df)
ERROR MESSAGE
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/core/optimizer.py in _configure_schedulers_automatic_opt(schedulers, monitor) 275 ) 276 if scheduler["reduce_on_plateau"] and scheduler.get("monitor", None) is None: --> 277 raise MisconfigurationException( 278 "The lr scheduler dict must include a monitor when a
ReduceLROnPlateau
scheduler is used." 279 ' For example: {"optimizer": optimizer, "lr_scheduler":'MisconfigurationException: The lr scheduler dict must include a monitor when a
ReduceLROnPlateau
scheduler is used. For example: {"optimizer": optimizer, "lr_scheduler": {"scheduler": scheduler, "monitor": "your_loss"}}
@MLfreakPy
The current implementation does not work for ReduceLROnPlateau
which requires monitor
to be specified.
Nevertheless, you could check out this branch https://github.com/Nixtla/neuralforecast/pull/1015 and install the the dev-version of neuralforecast. This allows you to call set_configure_optimizers
function to have a full control of configure_optimizers
related settings.
Note that I provided an example of ReduceLROnPlateau
in
https://app.reviewnb.com/Nixtla/neuralforecast/pull/1015/
Thank you very much @JQGoh!! I will use this dev-version and your example for implementation 👍 This flexibility in configuring the optimizers is amazing :)
@JQGoh I tried to manipulate the set_configure_optimizers
as you described. For that reason I (tried to) install the developer version of nixtla as per below. Maybe I do this wrongly. I cannot see the set_configure_optimizers
parameter in the imported neuralforecast models nor use it to implement LR on plateau. Do you have an idea where the problem resides?
# install DEV-version of nixtla (yet not sucessfull?)
!git clone https://github.com/Nixtla/neuralforecast.git
%cd neuralforecast
!pip install -e
#
from torch.optim.lr_scheduler import ReduceLROnPlateau
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer = optimizer, mode='min', factor=0.5, patience=2)
model.set_configure_optimizers(optimizer=optimizer,scheduler=scheduler,monitor="train_loss")
@JQGoh I tried to manipulate the
set_configure_optimizers
as you described. For that reason I (tried to) install the developer version of nixtla as per below. Maybe I do this wrongly. I cannot see theset_configure_optimizers
parameter in the imported neuralforecast models nor use it to implement LR on plateau. Do you have an idea where the problem resides?# install DEV-version of nixtla (yet not sucessfull?) !git clone https://github.com/Nixtla/neuralforecast.git %cd neuralforecast !pip install -e # from torch.optim.lr_scheduler import ReduceLROnPlateau optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate) scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer = optimizer, mode='min', factor=0.5, patience=2) model.set_configure_optimizers(optimizer=optimizer,scheduler=scheduler,monitor="train_loss")
@MLfreakPy
I just realized that I missed a few details.
That PR was my attempt to support a more complete configurable optimizers, which you will need to clone from my forked repository https://github.com/JQGoh/neuralforecast and checkout the branch feat/modify-config-optimizers
.
But do take note that because my forked repository does not always sync with the original repository master, it is better you could copy the key changes as detailed in https://github.com/Nixtla/neuralforecast/pull/1015 to the local copy of your source code, if you want to use the latest updated neuralforecast library.
@jmoralez
FYI, last we spoke about revisiting the PR once we get to work towards v2.0 version. I am re-considering whether that work is still valuable for the community prior to v2.0 version. I admit that the interface/user experience is questionable since we have four parameters (e.g. optimizer
) specified for the model and on top of that we could overwrite that behavior via
set_configure_optimizers
function. But I shall leave this to the Nixtla team to decide when/how should we support this in the future.
Hey. I think we can introduce that method and deprecate the arguments
@jmoralez That sounds great😊
What happened + What you expected to happen
Hi together,
first, thank you so much for the implementation of a custom lr scheduler!
I ran into problems employing a
ReduceLROnPlateau
scheduler.Currently I am passing through the
lr_scheduler
andlr_scheduler_kwargs
parameters. However, I get the below error when implementing ReduceLROnPlateau. Other schedulers, like OneCycle, work perfectly though! I think some of your earlier discussion evolved around it, however, I couldn't make it work yet. [(https://github.com/Nixtla/neuralforecast/pull/998)]MisconfigurationException('The lr scheduler dict must include a monitor when a ReduceLROnPlateau scheduler is used. For example: {"optimizer": optimizer, "lr_scheduler": {"scheduler": scheduler, "monitor": "your_loss"}}').
Versions / Dependencies
GoogleColab neuralforecast Version: 1.7.4
Reproduction script
See above - if needed happy to supply more details.
Issue Severity
Medium: It is a significant difficulty but I can work around it.