System.Threading.Tasks.TaskCanceledException : Intermittent Error while Copying Content to a SharePoint Embedded

[nkraman0]

Describe the bug

We are encountering an issue intermittently with System.Net.Http.HttpRequestException while copying content to a stream. Below are the details:

System.Threading.Tasks.TaskCanceledException: The operation was canceled. —> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. —> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request.

Expected behavior

The file with more than 1.5 GB should be uploaded to SharePoint embedded with out error.

The following program has been used to upload the files in batches.

async public Task WhenValidIDPassed_BulkUploadLargeFileUsingSDK() { GraphServiceClient graphServiceClient = GraphServiceClientFactory.GetGraphServiceClientAsync();
IConfigurationRoot config = ConfigProvider.GetConfigRoot(); string driveId = config["DriveId"];

 int bulkuploadsize = 0;

 Int32.TryParse(config["BulkUploadSize"], out bulkuploadsize);
 int index = 0;
 string filePathToUpload = config["FilePathToUpload"];
 try
 {
     while (index < bulkuploadsize) // condition
     {
         filePathToUpload = Path.Combine(FileUtils.GetRandomFile());

         using var fileStream = File.OpenRead(filePathToUpload);
         Console.WriteLine($"Upload started for a file {index}  {filePathToUpload}  {DateTime.Now}");

         string newFolderId = Guid.NewGuid().ToString();
         var newDriveItem = new DriveItem()
         {
             Name = newFolderId,
             Folder = new Folder(),
             AdditionalData = new Dictionary<string, object>
     {
         {
             "@microsoft.graph.conflictBehavior" , "rename"
         },
     }
         };
         var createdDriveItem = await graphServiceClient.Drives[driveId].Items["root"].Children.PostAsync(newDriveItem);
         var folderGraphId = createdDriveItem.Id;

         var uploadSessionRequestBody = new Microsoft.Graph.Drives.Item.Items.Item.CreateUploadSession.CreateUploadSessionPostRequestBody
         {
             Item = new DriveItemUploadableProperties
             {
                 AdditionalData = new Dictionary<string, object>
     {
         { "@microsoft.graph.conflictBehavior", "replace" },
     },
             },
         };

         var uploadSession = await graphServiceClient.Drives[driveId]
          .Items[folderGraphId]                 
        .ItemWithPath(filePathToUpload)
        .CreateUploadSession
        .PostAsync(uploadSessionRequestBody);

         int maxSliceSize = 320 * 50 * 1024;
         var fileUploadTask = new LargeFileUploadTask<DriveItem>(
             uploadSession, fileStream, maxSliceSize, graphServiceClient.RequestAdapter);

         var totalLength = fileStream.Length;
         // Create a callback that is invoked after each slice is uploaded
         IProgress<long> progress = new Progress<long>(prog =>
         {
             Console.WriteLine($"Uploaded {prog} bytes of {totalLength} bytes");
         });

         var retries = 0;
         var itemResponse = await UploadFileAsync(fileUploadTask, progress);

         while (retries < 3 && itemResponse == null)
         {
             var uploadResult = await UploadFileAsync(fileUploadTask, progress, true);
             retries++;
         }
         Console.WriteLine($"Upload Completed for a file {index} at {DateTime.Now} -  {filePathToUpload} {itemResponse.WebUrl}  ");
         index++;
     }

 }
 catch (ODataError ex)
 {
     Console.WriteLine($"Error uploading: {ex}");

 }

}

How to reproduce

Upload a files larger than 1.5 GB in a batch for size 1000.

SDK Version

5.34.0

Latest version known to work for scenario above?

No response

Known Workarounds

We are able to upload the same set of files in batches using Graph API. The code snippet has been enclosed below.

async public Task WhenValidIDPassed_BulkUploadLargeFileUsingAPI2() { AccessToken tokenCredential = GraphServiceClientFactory.GetGraphServiceTokenAsync(); IConfigurationRoot config = ConfigProvider.GetConfigRoot(); var driveId = config["DriveId"]; var authToken = config["AuthToken"]; int bulkuploadsize = 0; Int32.TryParse(config["BulkUploadSize"], out bulkuploadsize); int index = 0; string filePathToUpload = config["FilePathToUpload"]; try { while (index < bulkuploadsize) // condition { var filePath = Path.Combine(FileUtils.GetRandomFile());

         var CHUNK_SIZE = 10485760; // 10 MB
         using (var httpClient = new HttpClient())
         {
             httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", tokenCredential.Token);
                                 var folderName = Guid.NewGuid().ToString();
             //Create a folder:
             var createFolderResponse = await CreateFolder(httpClient, driveId, folderName);

             if (!createFolderResponse.IsSuccessStatusCode)
             {
                 Console.WriteLine("Failed to create folder.");
                 Console.WriteLine(await createFolderResponse.Content.ReadAsStringAsync());
                 return;
             }
               // Create upload session
             var uploadSessionUri = $"https://graph.microsoft.com/v1.0/drives/{driveId}/items/root:/{folderName}/{Path.GetFileName(filePath)}:/createUploadSession";
             var uploadSessionResponse = await httpClient.PostAsync(uploadSessionUri, null);
             uploadSessionResponse.EnsureSuccessStatusCode();

             var uploadSessionContent = uploadSessionResponse.Content.ReadAsStringAsync().Result;
             var uploadUrl = JObject.Parse(uploadSessionContent)["uploadUrl"].ToString();
             Console.WriteLine(uploadUrl);
             FileInfo fileInfo = new FileInfo(filePath);
             long size = fileInfo.Length;
             int chunks = (int)(size / CHUNK_SIZE) + (size % CHUNK_SIZE > 0 ? 1 : 0);

             using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))
             {
                 int start = 0;

                 for (int chunkNum = 0; chunkNum < chunks; chunkNum++)
                 {
                     byte[] buffer = new byte[CHUNK_SIZE];
                     int bytesRead = await fs.ReadAsync(buffer, 0, CHUNK_SIZE);

                     string uploadRange = $"bytes {start}-{start + bytesRead - 1}/{size}";
                     Console.WriteLine($"chunk: {chunkNum} bytes read: {bytesRead} upload range: {uploadRange}");

                     using (var content = new ByteArrayContent(buffer, 0, bytesRead))
                     {
                         content.Headers.Add("Content-Length", bytesRead.ToString());
                         content.Headers.Add("Content-Range", uploadRange);

                         HttpResponseMessage result = await httpClient.PutAsync(uploadUrl, content);
                         result.EnsureSuccessStatusCode();
                     }

                     start += bytesRead;
                 }

             }

         }
         Console.WriteLine($"Upload Completed for a file {index} at {DateTime.Now} -  {filePathToUpload} {filePath}  ");
         index++;
     }
 }
 catch (Exception ex) { }

}

Debug output

Click to expand log

``` ```

Configuration

No response

Other information

No response

[andrueastman]

Thanks for raising this @nkraman0

Any chance you can share a details of your .NET runtime? As the main difference between the two samples is the httpClient used, Are you able to share how the graphClient is initialized in the first example?

Are you also able to confirm if the scenario re-occurs if you pass a similar ehttpClient to LargeFileUploadTask as below?

            var graphServiceClient = new GraphServiceClient(new HttpClient()); // pass a different custom HttpClient to the client for making requests...
            var fileUploadTask = new LargeFileUploadTask<DriveItem>(uploadSession, fileStream, maxSliceSize, graphServiceClient.RequestAdapter);

Similar to #2608

[nkraman0]

Target framework version is : .NET 6.0

Enclosed, you’ll find the code snippet demonstrating how the GraphClient is initialized.

internal static GraphServiceClient GetGraphServiceClientAsync() { IConfigurationRoot config = ConfigProvider.GetConfigRoot();

X509Certificate2 certificate = new X509Certificate2(config["Azure:ClientCertificatePath"], config["Azure:ClientCertificatePassword"], X509KeyStorageFlags.MachineKeySet); TokenCredential tokenprovider = new ClientCertificateCredential( config["Azure:TenantId"], config["Azure:ClientId"], certificate); var result = new GraphServiceClient(tokenprovider); return result; }

[andrueastman]

Thanks for the response @nkraman0.

Are you able to confirm any difference if you use a different httpClient as suggested earlier?

Are you also able to confirm if the scenario re-occurs if you pass a similar ehttpClient to LargeFileUploadTask as below?

            var graphServiceClient = new GraphServiceClient(new HttpClient()); // pass a different custom HttpClient to the client for making requests...
            var fileUploadTask = new LargeFileUploadTask<DriveItem>(uploadSession, fileStream, maxSliceSize, graphServiceClient.RequestAdapter);

Finally, as your example shows logging taking place, are you able to share the logs captured during the upload to help understand at what point the upload throws the error?

Console.WriteLine($"Uploaded {prog} bytes of {totalLength} bytes");

[nkraman0]

@andrueastman , We tried the suggested solution we do not see any difference in the behavior.

[andrueastman]

Finally, as your example shows logging taking place, are you able to share the logs captured during the upload to help understand at what point the upload throws the error?

Console.WriteLine($"Uploaded {prog} bytes of {totalLength} bytes");

[nkraman0]

LargeFileUploadFailureWithSDK.txt Hello @andrueastman, I’ve included the log file in the link

The GraphServiceClient object was created following your instructions. Please refer to the code snippet for details

internal static GraphServiceClient GetGraphServiceHttpClientAsync() { IConfigurationRoot config = ConfigProvider.GetConfigRoot();

X509Certificate2 certificate = new X509Certificate2(config["Azure:ClientCertificatePath"], config["Azure:ClientCertificatePassword"], X509KeyStorageFlags.MachineKeySet);
TokenCredential tokenprovider = new ClientCertificateCredential(
    config["Azure:TenantId"],
    config["Azure:ClientId"],
    certificate);
var result = new GraphServiceClient(new HttpClient(), tokenprovider);
return result;

}

[andrueastman]

Thanks for the extra information here @nkraman0

Taking a look at the log shared, the following error is thrown,

System.Net.Http.HttpRequestException: No such host is known. (kxxx.xxx.com:443) ---> System.Net.Sockets.SocketException: No such host is known.

This error is thrown in the event that there's a network drop/issues on the host maichine so the host cannot be resolved. Similarly, a TaskCanceledException will be thrown if there's a network problem that causes the client to wait for more than 100 seconds for the server response. Any chance you can confirm if this is something you expect to happen in your scenario? If so, you can probably catch the exception and call resumeAsync() for the task to complete it's job.

var fileUploadTask = new LargeFileUploadTask<DriveItem>(uploadSession, fileStream, maxSliceSize, graphServiceClient.RequestAdapter);

try
{
    await fileUploadTask.UploadAsync();
}
catch (HttpRequestException ex) // or TaskCancelledException for a temporary network failure?
{
    Console.WriteLine(ex.Message); // validate if is an expected network failure...
    await fileUploadTask.ResumeAsync();// resume from how far we got..
}

[nkraman0]

Thanks for the explanation of the System.Net.Http.HttpRequestException and the potential causes. I appreciate the info on TaskCanceledException as well.

I can implement a try-catch block to capture the System.Net.Http.HttpRequestException and potentially other network-related exceptions.I'll share a feedback after execution of large file upload.