I'm starting to look into parallel transfer of one file. The function mcachefs_transfer_file seems to already do the backup in chunks (window_size), so I'm thinking about re-factor it so it can transfer the chunks in parallel using multiple threads.
Since I'm using mcachefs over the internet, parallel transfer of chunks of a single file will be extremely beneficial, specially for big files of tens of Gigabytes.
The goal is to use the maximum bandwidth available when transferring such files, something we can only achieve by splitting and transferring the file in parallel.
If you have any considerations about this, or any suggestions about implementation, I would love to hear!
I'm starting to look into parallel transfer of one file. The function
mcachefs_transfer_fileseems to already do the backup in chunks (window_size), so I'm thinking about re-factor it so it can transfer the chunks in parallel using multiple threads.Since I'm using mcachefs over the internet, parallel transfer of chunks of a single file will be extremely beneficial, specially for big files of tens of Gigabytes.
The goal is to use the maximum bandwidth available when transferring such files, something we can only achieve by splitting and transferring the file in parallel.
If you have any considerations about this, or any suggestions about implementation, I would love to hear!
cheers... -H