RoboCopy Benchmarking

[RFBomb]

I'm currently writing a custom IRoboCommand to orchestrate building a complex file batch that pulls from various folders and singular files to a final destination. As such, I've written an interface that has a method for CopyToAsync() to copy a file from source to destination asynchronously. I'm creating this thread to document some benchmarks, as RoboCopy is great at what it does.

I used ChatGPT and the microsoft documentation to create some async methods to handle the copying that reports via an IProgress<long> object. As well as a project developed to use CopyFileEx to perform the copy operation.

The benchmark for these tests was run in a console app in release mode, using a 334MB file. Each copy operation was performed 10x and the time report was the average result. My goal was to determine if a comparable time-to-copy could be achieve for a single file that falls within the current documentation recommendations I've been able to track down over a few days.

Note : This test was done on a console application targeting Net5.0 (as my work computer does not have VS2022 yet). Net6.0 benchmarks will happen in a followup comment.

CopyFileEx : https://github.com/RFBCodeWorks/CachedRoboCopy/blob/master/CachedRoboCopy/FileCopier.cs

static async Task NoProgressAsync(string source, string destination, int bufferSize = 81920)
{
    using (var reader = File.OpenRead(source))
    {
        using (var writer = File.OpenWrite(destination))
        {
            await reader.CopyToAsync(writer, bufferSize);
            writer.Dispose();
            reader.Dispose();
        }
    }
}

public static async Task ProgressAsync(string source, string destination, IProgress<long> progress, bool overwrite = false, CancellationToken token = default, int bufferSize = 81920)
{
    token.ThrowIfCancellationRequested();
    if (!File.Exists(source)) throw new FileNotFoundException("File not found : " + source);
    if (!overwrite && File.Exists(destination)) throw new IOException("Destination file already exists");
    if (progress is null) throw new ArgumentNullException(nameof(progress));
    int reportingInterval = 250;
    Stopwatch reportTimer = new Stopwatch();
    try
    {
        using (FileStream sourceStream = new FileStream(source, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize, true))
        using (FileStream destinationStream = new FileStream(destination, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize, true))
        {
            byte[] buffer = new byte[bufferSize];
            long totalBytesRead = 0;
            int bytesRead;

            reportTimer.Start();
            while ((bytesRead = await sourceStream.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false)) > 0)
            {
                token.ThrowIfCancellationRequested();
                await destinationStream.WriteAsync(buffer, 0, bytesRead, token).ConfigureAwait(false);
                totalBytesRead += bytesRead;
                if (reportTimer.ElapsedMilliseconds >= reportingInterval)
                {
                    progress?.Report(totalBytesRead);
                    reportTimer.Restart();
                }
            }
            reportTimer.Stop();
            progress?.Report(totalBytesRead);
        }
    }
    catch (OperationCanceledException)
    {
        // If the operation was canceled, delete the destination file
        if (File.Exists(destination))
        {
            File.Delete(destination);
            Console.WriteLine($"File copy operation canceled. Destination file '{destination}' deleted.");
        }

        throw; // Re-throw the OperationCanceledException after handling
    }
}

[RFBomb]

Net5.0 Tests

Note : to reiterate, each result set is an average of 10 copy operations using the same source/destination. The RoboCommand had default settings generated by the library.

USB 2.0 Copy Operation: (only ran this once as the USB i was using took like an hour to do this test)

File Size : 334MB
              RoboCommand Test :     137961.0ms
           Progress Async Test :     178670.9ms
        No Progress Async Test :     138711.8ms
    FileCopyEx - With Progress :     134373.3ms
 FileCopyEx - Without Progress :     134766.5ms
FileCopyEx - Periodic Progress :     135742.2ms

USB 3.0 Copy Operation

File Size : 334MB
              RoboCommand Test :       7450.7ms
           Progress Async Test :     120037.6ms -- Speed compared to RoboCopy : +1,511.1%
        No Progress Async Test :      13266.2ms -- Speed compared to RoboCopy : +78.1%
    FileCopyEx - With Progress :      11695.2ms -- Speed compared to RoboCopy : +57.0%
 FileCopyEx - Without Progress :      13296.4ms -- Speed compared to RoboCopy : +78.5%
FileCopyEx - Periodic Progress :      14549.1ms -- Speed compared to RoboCopy : +95.3%
   Task.Run(() => File.Copy()) :      13705.2ms -- Speed compared to RoboCopy : +83.9%

Same Drive Copy Operation:

Result Set 1 : 
File Size : 334MB
              RoboCommand Test :       2452.7ms
           Progress Async Test :       2842.1ms -- Speed compared to RoboCopy : +15.9%
        No Progress Async Test :       1345.4ms -- Speed compared to RoboCopy : -45.1%
    FileCopyEx - With Progress :        874.7ms -- Speed compared to RoboCopy : -64.3%
 FileCopyEx - Without Progress :        752.4ms -- Speed compared to RoboCopy : -69.3%
FileCopyEx - Periodic Progress :        711.8ms -- Speed compared to RoboCopy : -71.0%

Result Set 2 : 
File Size : 334MB
              RoboCommand Test :       1046.1ms
           Progress Async Test :       2631.3ms -- Speed compared to RoboCopy : +151.5%
        No Progress Async Test :        537.1ms -- Speed compared to RoboCopy : -48.7%
    FileCopyEx - With Progress :        254.6ms -- Speed compared to RoboCopy : -75.7%
 FileCopyEx - Without Progress :        193.3ms -- Speed compared to RoboCopy : -81.5%
FileCopyEx - Periodic Progress :        229.6ms -- Speed compared to RoboCopy : -78.1%
   Task.Run(() => File.Copy()) :        168.5ms -- Speed compared to RoboCopy : -83.9%

Result Set 3 : 
File Size : 334MB
              RoboCommand Test :        700.4ms
           Progress Async Test :       3010.0ms -- Speed compared to RoboCopy : +329.8%
        No Progress Async Test :        365.7ms -- Speed compared to RoboCopy : -47.8%
    FileCopyEx - With Progress :        198.4ms -- Speed compared to RoboCopy : -71.7%
 FileCopyEx - Without Progress :        216.3ms -- Speed compared to RoboCopy : -69.1%
FileCopyEx - Periodic Progress :        260.6ms -- Speed compared to RoboCopy : -62.8%
   Task.Run(() => File.Copy()) :          183ms -- Speed compared to RoboCopy : -73.9%

Network Drive ( For reference, windows copy averaged 50-60MB/s copying this file when dragged-and-dropped from the network to the C:\ drive )

Result Set 1 : 
File Size : 334MB
              RoboCommand Test :       4596.0ms
           Progress Async Test :       2485.5ms -- Speed compared to RoboCopy : -45.9%
        No Progress Async Test :        368.5ms -- Speed compared to RoboCopy : -92.0%
    FileCopyEx - With Progress :       4161.1ms -- Speed compared to RoboCopy : -9.5%
 FileCopyEx - Without Progress :       4574.8ms -- Speed compared to RoboCopy : -0.5%
FileCopyEx - Periodic Progress :       3907.3ms -- Speed compared to RoboCopy : -15.0%

Result Set 2: 
File Size : 334MB
              RoboCommand Test :       4037.2ms
           Progress Async Test :       2632.9ms -- Speed compared to RoboCopy : -34.8%
        No Progress Async Test :        392.0ms -- Speed compared to RoboCopy : -90.3%
    FileCopyEx - With Progress :       4443.8ms -- Speed compared to RoboCopy : +10.1%
 FileCopyEx - Without Progress :       4153.1ms -- Speed compared to RoboCopy : +2.9%
FileCopyEx - Periodic Progress :       4587.4ms -- Speed compared to RoboCopy : +13.6%
   Task.Run(() => File.Copy()) :       4167.7ms -- Speed compared to RoboCopy : +3.2%

Conclusions:

• USB 2.0 : How things are copied over does not make a big difference at all. ( big surprise, usb2.0 speeds are slow )
• USB 3.0 : robocopy all the way.
• Same Drive : Avoid FileStream.WriteAsync() as written in the original post, as its substantially slower (though it does offer progress reporting). CopyFileEx is substantially better for progress reporting.
• Network Drive : depends on network conditions. If using a VPN then robocopy is probably preferred, but otherwise a local network may see speed gains manually calling the operation.

If progress reporting and cancellation is not required, Task.Run(() => File.Copy()) will yield best results. This is somewhat expected as this runs the copy operating consuming a thread on its own the entire time, just as a background worker.

If cancellation is required, without progress reporting, CopyFileEx or the FileStream.CopyToAsync() works well.

If async and progress reporting is required, CopyFileEx is fantastic. RoboCopy is good enough though.

Im curious what Microsoft uses in their drag and drop dialogue to report copy progress.

[SchreinerK]

When I was programming in windows 7 days, this was the SHFileOperation (shell32). but I don't know whether this has changed in the meantime. https://learn.microsoft.com/de-de/windows/win32/api/shellapi/nf-shellapi-shfileoperationw

[RFBomb]

As far as I can, tell that entire functions that has been deprecated with the release of Windows Vista.

I was using CopyFileEx because it has been reported since Windows XP and they were keeping it current, with even a new feature being added for windows, which was SMB compression.