Open ShoufaChen opened 1 month ago
Hi @ShoufaChen thanks for the issue, we should update the documentation to explain this better.
To answer your questions: it depends mainly on the size and composition of your state. If you're storing eg an int representing index or file-offset, then it shouldn't be an issue. If your state is very large, involving eg buffers of data for shuffling, then the overhead of creating a checkpoint and passing it through multiprocessing queue may slow down training, this variable lets you decrease the frequency of checkpointing. If eg you know you're checkpointing every 1000 steps, you can set this value to 1000.
Hello,
Thank you for your awesome implementation of StatefulDataloader.
I have a question about
snapshot_every_n_steps
. It seems there is not much detailed explanation about this argument.snapshot_every_n_steps=1
) cause a data loading burden?cc @andrewkho