Open ZhenYangIACAS opened 10 months ago
By design, LLaMA2-Accessory should not behave like this. Please check the log, where the composition of the actually used dataset, as well as the total number of data items in use, are printed. You may search by the key word "total length" in the log file.
@ChrisLiu6 The total length printed is right, but the actual batch numbers (2208 in the above log) seems not to change with the total length. I am so confused by this.
I consider multiple meta_type datasets, but the log shows that the total batch number in one epoch is decided by the smallest dataset. As the log shows below, when I add more datasets, the total number of batches in one epoch is still 2208, which is the batch number of the smallest dataset. Why is this ?
log:
[18:41:11.182261] Epoch: [2] [2120/2208] lr: 0.000098 closs: 3.0516 (4.3038) grad_norm: 8.2056 (6.4250) time: 0.8935 data: 0.0003 max mem: 27193 [18:41:19.053091] Epoch: [2] [2130/2208] lr: 0.000098 closs: 2.6504 (4.2980) grad_norm: 8.2056 (6.4250) time: 0.8218 data: 0.0003 max mem: 27193 [18:41:28.146424] Epoch: [2] [2140/2208] lr: 0.000098 closs: 2.9470 (4.2958) grad_norm: 8.2056 (6.4250) time: 0.8480 data: 0.0003 max mem: 27193 [18:41:37.381831] Epoch: [2] [2150/2208] lr: 0.000098 closs: 3.6000 (4.2927)