As the run progresses the save00 directory becomes excessively file heavy and increases the file copy time. I have done zero research on this subject, but it feels like we could:
Get the recursive file list
Split the list by the number of workers and give each worker a slice of files to copy
As the run progresses the
save00
directory becomes excessively file heavy and increases the file copy time. I have done zero research on this subject, but it feels like we could: