Question about hyper-parameter search in GCGL

[WeiWeic6222848]

In the appendix, I found the following grid-search for hyper-parameters, which I think is used in GCGL tasks too.

However, the GCGL datasets only contains less than 1000 samples per task, this means GEM is effectively storing the entire history of data for regularization purposes when n_memory is set to 1000 during hyper-param search, which I assume to have the best performance. However, this doesn't seem fair as it destroys the purpose of continual learning, but maybe you used different parameters for GCGL? I want to know the exact hyperparameter which is used for the evaluation of GEM-GCGL if possible.

Thank you Wei

[QueuQ]

In the appendix, I found the following grid-search for hyper-parameters, which I think is used in GCGL tasks too.

However, the GCGL datasets only contains less than 1000 samples per task, this means GEM is effectively storing the entire history of data for regularization purposes when n_memory is set to 1000 during hyper-param search, which I assume to have the best performance. However, this doesn't seem fair as it destroys the purpose of continual learning, but maybe you used different parameters for GCGL? I want to know the exact hyperparameter which is used for the evaluation of GEM-GCGL if possible.

Thank you Wei

Hi Wei,

Sorry for the misunderstanding. In our experiments, the memory size is set as 100.

One thing to mention is that larger memory may not always good for methods like GEM. Since GEM clips the gradient of the new tasks with the gradients calculated with data stored in memory, the larger the memory is, the less flexibility the model has for adapting to the new tasks, which may influence the performance on new tasks and the overall performance. When the buffered data contains some noisy examples or outliers, this adverse effect may be more significant. Therefore, there is a trade-off between the buffer size and the flexibility (capability to adapt to new tasks).

[WeiWeic6222848]

Thank you for the clarification about the hyperparameters, and yeah indeed, larger memory indeed makes the model less flexible. I think the effect is also more severe if the tasks are small. Thank you for the additional insight.