AnaLogState, by its design, is for tracking/storing global statistics such as hessian (or covariance) and mean gradient. However, log_state is only related to the current batch. Therefore, it would be nice to separate log_state from AnaLogState, and create a new class like BatchInfo that handles batch-specific information including log_state, data_id, and mask.
sangkeun00
commented
8 months ago
AnaLogState, by its design, is for tracking/storing global statistics such as hessian (or covariance) and mean gradient. However,log_stateis only related to the current batch. Therefore, it would be nice to separatelog_statefromAnaLogState, and create a new class likeBatchInfothat handles batch-specific information includinglog_state,data_id, andmask.