search

MilaNLProc / contextualized-topic-models

A python package to run contextualized topic modeling. CTMs combine contextualized embeddings (e.g., BERT) with topic models to get coherent topics. Published at EACL and ACL 2021 (Bianchi et al.).
MIT License
1.18k stars 142 forks source link

Add patience to reduce LR as CTM argument #141

Open drodrigues0 opened 10 months ago

drodrigues0 commented 10 months ago

Description

Currently, ReduceLROnPlateau function for CTM has a high hard-coded value for patience. So, whenever you want the LR to decrease during training we need to set the EarlyStopping patience to some value higher than 10, otherwise the train stops before the LR is actually decreased.

This is impractical because, waiting 10 epochs for the LR to decrease is a lot.

Code line

Propose to fix it

class CTM:

    # ...

    def __init__(
        self,
        # ...
        reduce_on_plateau=False,
        lr_patience: int = 3
        # ...
# init lr scheduler
if self.reduce_on_plateau:
    self.scheduler = ReduceLROnPlateau(self.optimizer, patience=sefl.lr_patience)
vinid commented 10 months ago

Hello!

do you want to send a PR to add this?

On Mon, Aug 28, 2023 at 8:40 AM Diogo Rodrigues @.***> wrote:

  • Contextualized Topic Models version: 2.5.0
  • Python version: 3.9.16
  • Operating System: Ubuntu 20.04.6 LTS

Description

Currently, ReduceLROnPlateau function for CTM has a high hard-coded value for patience. So, whenever you want the LR to decrease during training we need to set the EarlyStopping patience to some value higher than 10, otherwise the train stops before the LR is actually decreased.

This is impractical because, waiting 10 epochs for the LR to decrease is a lot.

Code line https://github.com/MilaNLProc/contextualized-topic-models/blob/5dd3238dd3a49e8a792163b02d4c3f3b5deaae9f/contextualized_topic_models/models/ctm.py#L154C13-L154C76 Propose to fix it

class CTM:

# ...

def __init__(
    self,
    # ...
    reduce_on_plateau=False,
    lr_patience: int = 3
    # ...

— Reply to this email directly, view it on GitHub https://github.com/MilaNLProc/contextualized-topic-models/issues/141, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARBSSZZGUSBZREYG5VLTY3XXQ4PJANCNFSM6AAAAAA4A57AEA . You are receiving this because you are subscribed to this thread.Message ID: @.***>