Liuhong99 / Sophia

The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”
MIT License
931 stars 52 forks source link

A new configurator? #13

Open arman-hk opened 1 year ago

arman-hk commented 1 year ago

@Liuhong99 Hi 👋🏻,

I might have a simple solution for configurator.py: a shift from global configuration management to a class-based approach

main goals:

  1. separation of concerns: By using a Config class to encapsulate configuration management, we separate this concern from the command-line argument parsing. This makes code more maintainable.
  2. selective import: Instead of importing every variable globally, we can selectively import frequently used or important variables, without needing to prepend config. to every single variable.

and then in the scripts where these configs are needed, could do this:

from configurator import config

batch_size = config.batch_size
learning_rate = config.learning_rate

"""
now we use 'batch_size' and 'learning_rate' directly in the script
and avoid typing 'config.' before each variable.
"""