A task was canceled after 60 seconds when Internet connection is slow

[LucaCode92]

Hey!

I have question about error which I get. I'm trying to send files to my Test application (Blazor Server Side). When Internet connection is good then everything works fine, but when my Internet connection is slow then I get error "A task was canceled" after 60 seconds. Do you have any advice how to solve my problem? :)

Here's error StackTrace: A task was canceled. at Microsoft.JSInterop.JSRuntime.InvokeWithDefaultCancellation[T](String identifier, Object[] args) at Tewr.Blazor.FileReader.FileReaderJsInterop.ReadFileMarshalledBase64Async(Int32 fileRef, Int64 position, Int32 count, CancellationToken cancellationToken) at Tewr.Blazor.FileReader.FileReaderJsInterop.ReadFileMarshalledAsync(Int32 fileRef, Byte[] buffer, Int64 position, Int64 bufferOffset, Int32 count, CancellationToken cancellationToken) at Tewr.Blazor.FileReader.FileReaderJsInterop.ReadFileAsync(Int32 fileRef, Byte[] buffer, Int64 position, Int64 bufferOffset, Int32 count, CancellationToken cancellationToken) at Tewr.Blazor.FileReader.FileReaderJsInterop.InteropFileStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) at System.Net.Http.MultipartContent.SerializeToStreamAsyncCore(Stream stream, TransportContext context, CancellationToken cancellationToken) at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at TestApp.Components.Tables.IngredientsTable.SaveFileAsyncc(Stream inputFileTest, String inputFileNameTest, String inputFileOrginalNameTest) in C:\Users\User\source\repos\TestApp\TestApp\Components\Tables\IngredientsTable.razor:line 666 at TestApp.Components.Tables.IngredientsTable.Save() in C:\Users\User\source\repos\TestApp\TestApp\Components\Tables\IngredientsTable.razor:line 843

[Tewr]

Hello.

Post the code you are using to interact with the library. I'm not entirely sure when this happens, but my guess i you could use a cancellationTokenSource and configure the timeout differently.

[LucaCode92]

I'm not sure it's good way what I'm doing but anyway: I want allow user to add files to the list separately, then after accept form send all to the server one by one.

I hope it will be understandable :D Code under file input onchange method:

    List<AdditionalIngredientFile> additionalIngredientFiles = new List<AdditionalIngredientFile>();
    ElementReference additionalInputRef;
    IFileReference additionalInputFile;
    string additionalFileError = string.Empty;
    int additionalFileOrderNum = 0;

    private async Task OpenAdditionalFileAsync()
    {
        additionalInputFile = (await filereader.CreateReference(additionalInputRef).EnumerateFilesAsync()).FirstOrDefault();

        var additionalInputFileInfo = await additionalInputFile.ReadFileInfoAsync();

        if(additionalInputFileInfo.Size > 20 * 1024 * 1024)
        {
            ingredient.isAdditionalFileSizeProper = false;
            additionalFileError = $"Size of file {additionalInputFileInfo.Name} exceeds 20MB. The file will not be added.";
        }
        else
        {
            additionalFileOrderNum++;
            ingredient.isAdditionalFileSizeProper = true;

            var newAdditionalFile = new AdditionalIngredientFile()
            {
                ingredientId = ingredient.id,
                Name = $"{Path.GetFileNameWithoutExtension(additionalInputFileInfo.Name)}_{DateTime.Now.ToString("yyyyMMddHHmmssffff")}{Path.GetExtension(additionalInputFileInfo.Name)}",
                orginalName = additionalInputFileInfo.Name,
                orderNum = additionalFileOrderNum,
                inputFile = await additionalInputFile.OpenReadAsync()

            };

            additionalIngredientFiles.Add(newAdditionalFile);
        }
    }

It's how AdditionalIngredientFile class looks like:

public class AdditionalIngredientFile
    {
        [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
        [Key]
        public long id { get; set; }

        public long ingredientId { get; set; }

        public string Name { get; set; }

        public string orginalName { get; set; }
        public int orderNum { get; set; }

        [NotMapped]
        public Stream inputFile { get; set; }
    }

Then after submit I'm using foreach (SaveFileAsyncc is a function that sends files ):

if(additionalIngredientFiles.Count > 0)
{
    foreach(var file in additionalIngredientFiles)
    {
        if(await SaveFileAsyncc(file.inputFile, file.Name, file.orginalName))
        {
            await aifServ.Create(file);
        }
    }
}

Here is SaveFileAsyncc:

private async Task<bool> SaveFileAsyncc(Stream inputFileTest, string inputFileNameTest, string inputFileOrginalNameTest)
{
    var content = new MultipartFormDataContent();
    content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("form-data");
    var response = new HttpResponseMessage();

    var cts = new CancellationTokenSource(TimeSpan.FromSeconds(360)).Token;

    content.Add(new StreamContent(inputFileTest, (int)inputFileTest.Length), "file", inputFileOrginalNameTest);
    content.Add(new StringContent(inputFileNameTest), "fileName");
    response = await client.PostAsync($"{nm.BaseUri}api/Upload", content, cts).ConfigureAwait(false);

    //response.EnsureSuccessStatusCode();

    if (response.IsSuccessStatusCode)
    {
        return true;
    }
    else
    {
        return false;
    }
}

And api Post:

[Route("api/[controller]")]
[ApiController]

public class UploadController : Controller
{
    private readonly IWebHostEnvironment environment;
    public UploadController(IWebHostEnvironment environment)
    {
        this.environment = environment;
    }

    [HttpPost]
    //[Consumes("multipart/form-data")]
    //[DisableFormValueModelBinding]
    //[FromForm(Name = "fileName")]
    public async Task<IActionResult> Post([FromForm(Name = "file")]IFormFile file, [FromForm(Name = "fileName")] string fileName)
    {
        //return BadRequest(fileName);

        if (file == null || file.Length == 0)
            return BadRequest("Upload a file");

        string filepath = Path.Combine(environment.ContentRootPath, "attachments", fileName);

        using (var filestream = new FileStream(filepath, FileMode.Create, FileAccess.Write))
        {
            var ct = new CancellationTokenSource(TimeSpan.FromSeconds(360)).Token;
            await file.CopyToAsync(filestream, ct);
        }

        return Ok(filepath);
    }
}

[Tewr]

I'm assuming you've set up a descent signalR buffer size/max message size as per the readme, as this is an intermittent problem?

I can't test your code atm, but I suspect that you cancellation token you specified does not reach the ReadAsync method. I'm not entirely sure of the implementation on httpclient and StreamContent.

You could try copying to the target stream explicitly and see if that helps, at least it would give some clues. You could do this by implementing you own httpcontent and overriding SerializeToStreamAsync. That way you can call ReadAsync by yourself and pass the appropriate cancellation token. Here is a short article on the subject: https://thomaslevesque.com/2013/11/30/uploading-data-with-httpclient-using-a-push-model/

[LucaCode92]

I'm not sure if I understood correctly, but I did something like this:

Found StreamContent.cs source code and prepare class:

public class OwnStreamContent : HttpContent
    {
        private Stream _content;
        private int _bufferSize;
        private bool _contentConsumed;
        private long _start;
        private CancellationTokenSource _cts;

        public OwnStreamContent(Stream content, int bufferSize)
        {
            if(content == null)
            {
                throw new ArgumentNullException(nameof(content));
            }
            if(bufferSize <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(bufferSize));
            }

            InitializeContent(content, bufferSize);
        }

        private void InitializeContent(Stream content, int bufferSize)
        {
            _content = content;
            _bufferSize = bufferSize;
            _cts = new CancellationTokenSource();

            _cts.CancelAfter(TimeSpan.FromSeconds(360));

            if(content.CanSeek)
            {
                _start = content.Position;
            }
        }

        protected override Task SerializeToStreamAsync(Stream stream, TransportContext? context) =>
            SerializeToStreamAsyncCore(stream, _cts.Token);

        private Task SerializeToStreamAsyncCore(Stream stream, CancellationToken cancellationToken)
        {
            Debug.Assert(stream != null);
            PrepareContent();
            return StreamToStreamCopy.CopyAsync(
                _content,
                stream,
                _bufferSize,
                !_content.CanSeek, // If the stream can't be re-read, make sure that it gets disposed once it is consumed.
                cancellationToken);
        }

        private void PrepareContent()
        {
            if (_contentConsumed)
            {
                // If the content needs to be written to a target stream a 2nd time, then the stream must support
                // seeking (e.g. a FileStream), otherwise the stream can't be copied a second time to a target
                // stream (e.g. a NetworkStream).
                if (_content.CanSeek)
                {
                    _content.Position = _start;
                }
                else
                {
                    throw new InvalidOperationException();
                }
            }

            _contentConsumed = true;
        }

        protected override bool TryComputeLength(out long length)
        {
            if (_content.CanSeek)
            {
                length = _content.Length - _start;
                return true;
            }
            else
            {
                length = 0;
                return false;
            }
        }

        protected override void Dispose(bool disposing)
        {
            if (disposing)
            {
                _content.Dispose();
            }
            base.Dispose(disposing);
        }
    }

And copy internal static class:

internal static class StreamToStreamCopy
    {
        /// <summary>Copies the source stream from its current position to the destination stream at its current position.</summary>
        /// <param name="source">The source stream from which to copy.</param>
        /// <param name="destination">The destination stream to which to copy.</param>
        /// <param name="bufferSize">The size of the buffer to allocate if one needs to be allocated. If zero, use the default buffer size.</param>
        /// <param name="disposeSource">Whether to dispose of the source stream after the copy has finished successfully.</param>
        public static void Copy(Stream source, Stream destination, int bufferSize, bool disposeSource)
        {
            Debug.Assert(source != null);
            Debug.Assert(destination != null);
            Debug.Assert(bufferSize >= 0);

            if (bufferSize == 0)
            {
                source.CopyTo(destination);
            }
            else
            {
                source.CopyTo(destination, bufferSize);
            }

            if (disposeSource)
            {
                DisposeSource(source);
            }
        }

        /// <summary>Copies the source stream from its current position to the destination stream at its current position.</summary>
        /// <param name="source">The source stream from which to copy.</param>
        /// <param name="destination">The destination stream to which to copy.</param>
        /// <param name="bufferSize">The size of the buffer to allocate if one needs to be allocated. If zero, use the default buffer size.</param>
        /// <param name="disposeSource">Whether to dispose of the source stream after the copy has finished successfully.</param>
        /// <param name="cancellationToken">CancellationToken used to cancel the copy operation.</param>
        public static Task CopyAsync(Stream source, Stream destination, int bufferSize, bool disposeSource, CancellationToken cancellationToken = default(CancellationToken))
        {
            Debug.Assert(source != null);
            Debug.Assert(destination != null);
            Debug.Assert(bufferSize >= 0);

            try
            {
                Task copyTask = bufferSize == 0 ?
                    source.CopyToAsync(destination, cancellationToken) :
                    source.CopyToAsync(destination, bufferSize, cancellationToken);

                if (!disposeSource)
                {
                    return copyTask;
                }

                switch (copyTask.Status)
                {
                    case TaskStatus.RanToCompletion:
                        DisposeSource(source);
                        return Task.CompletedTask;

                    case TaskStatus.Faulted:
                    case TaskStatus.Canceled:
                        return copyTask;

                    default:
                        return DisposeSourceAsync(copyTask, source);

                        static async Task DisposeSourceAsync(Task copyTask, Stream source)
                        {
                            await copyTask.ConfigureAwait(false);
                            DisposeSource(source);
                        }
                }
            }
            catch (Exception e)
            {
                // For compatibility with the previous implementation, catch everything (including arg exceptions) and
                // store errors into the task rather than letting them propagate to the synchronous caller.
                return Task.FromException(e);
            }
        }

        /// <summary>Disposes the source stream.</summary>
        private static void DisposeSource(Stream source)
        {
            try
            {
                source.Dispose();
            }
            catch (Exception e)
            {
                // Dispose() should never throw, but since we're on an async codepath, make sure to catch the exception.
                //if (NetEventSource.Log.IsEnabled()) NetEventSource.Error(null, e);
            }
        }
    }

but effect is still the same - cancel after 60 seconds when Internet connection is slow. If it's not what you had in mind then please let me know more details :)

[Tewr]

This is exactly what I had in mind actually. Could you just try to replace the two calls to Stream.CopyToAsync with a read and write loop? Like this:

byte[] buffer = new byte[bufferSize]; int bytesRead; while ((bytesRead = await ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false)) != 0) { await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken).ConfigureAwait(false); }

That should enable you to pin down the call to ReadAsync with the correct cancellation token without any middle men - like you can see in #156, CopyToAsync is super optimized and does not necessarily play well with blazor server side. I know it uses its own buffer size, could it be using some other cancellation token as well?

[LucaCode92]

Could you please let me know which parts of code exactly should I change?

I changed:

Task copyTask = bufferSize == 0 ?
                    source.CopyToAsync(destination, cancellationToken) :
                    source.CopyToAsync(destination, bufferSize, cancellationToken);

to:

Task copyTask = null;

                if (bufferSize > 0)
                {
                    copyTask = Task.Run(async () => {

                        byte[] buffer = new byte[bufferSize];
                        int bytesRead;
                        while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false)) != 0)
                        {
                            await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken).ConfigureAwait(false);
                        }

                    });
                }

But still the same :( 60 seconds and task is canceled

[Tewr]

Ok. Guess that was wild goose chase. Buffersize might be relevant. If you debug that method where CopyToAsync is called, what is the buffer size?

Also, did you set MaximumReceiveMessageSize and to what value?

[LucaCode92]

Application works fine in debug mode when sending files locally. But the problem is through Internet. Of course if I'm using my mobile Internet with good signal then files are sent within 60 seconds and everything works perfectly :) Is there any Blazor or IIS setting that cancel request/task?

[LucaCode92]

@Tewr I think I found the 60 seconds limitation:

Now it's 6 minutes :) but anyway still have some problem with larger files:

Any idea? :) I back to standard StreamContent because my own cause another error :)

[Tewr]

It's the buffer size that is too large. What you need to accomplish is transfer a chunk of data big as the buffer size during the jsinteropdefault timeout.

But without a custom httpcontent class I'm not sure how to change it. If you had errors implementing it yourself maybe try the PushStreamContent class of web API client 2 (or is that what you did?)

Also some observations: You are setting the default timeout, but the cancellation token timeout is "ignored" by blazor (unless it's shorter than default). Timeout has to be set explicitly when calling interop to override default, this is stated in the blazor documentation. My library does not offer the possibility to change that timeout, but it's an easy feature to add.

[LucaCode92]

So if I will use own httpcontent class then how big value for buffer size should I set?

It would be great if you can add it. I don't think that changing default timeout for all js interop is a good idea :)

Error I mentioned - when using own httpcontent class:

09/09/2020 15:52:13 System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.ExposedSocketNetworkStream'.
   at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source)
   at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
   at TestApp.Classes.StreamToStreamCopy.OwnCopyAsync(Stream source, Stream destination, Int32 bufferSize, CancellationToken ct) in C:\Users\User\source\repos\TestApp\TestApp\Classes\StreamToStreamCopy.cs:line 45
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   at System.Net.Http.MultipartContent.SerializeToStreamAsyncCore(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
The operation was canceled.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at TestApp.Components.Tables.IngredientsTable.SaveFileAsyncc(Stream inputFileTest, String inputFileNameTest, String inputFileOrginalNameTest) in C:\Users\User\source\repos\TestApp\TestApp\Components\Tables\IngredientsTable.razor:line 678
   at TestApp.Components.Tables.IngredientsTable.Save() in C:\Users\User\source\repos\TestApp\TestApp\Components\Tables\IngredientsTable.razor:line 861

[Tewr]

You should set a buffer size that you know for sure can be transferred in the time of the default timeout. Something like 80KB.

[LucaCode92]

Ok, it's definitely better :) So in fact one error left. "Cannot access a disposed object" occurs after almost 2 minutes. Do you know how to avoid that?

[Tewr]

I don't, really. You stack trace looks like as if it's the target stream being disposed at one point.

I gotta get on a computer (I'm on my phone) but there is a second solution that came to me yesterday: an intermediate buffering stream. A custom httpcontent class is way overkill now that we know that the problem is the buffersize passed to ReadAsync

Edit: I don't think the native buffered stream can help us here, according to docs it's may do nothing should the buffer size be larger than the inner. Still, the approach is viable: wrapping the stream and splitting up the read in smaller chunks in the wrapping ReadAsync. I could provide some options to work around this scenario for a future release

[Tewr]

spawned #158

[LucaCode92]

@Tewr nice to know :) I will try to check your changes :)

About disposed object - I decided to use try catch.. in most cases with quite good Internet connection everything works fine.

[Tewr]

The default StreamContent class accepts buffersize as an argument. as we were busy looking at timeouts I totally oversaw that, sorry. So no need for a custom httpcontent

You are setting the buffersize here to the size of the file: new StreamContent(inputFileTest, (int)inputFileTest.Length) thus, try the following instead

var bufferSize = 81920; // 80Kb
new StreamContent(inputFileTest, bufferSize);

I have committed some changes that should fix most of your problems, but still, explicitly setting the buffersize can be neccessary for slow connections.

[Tewr]

I'm closing this as it's been two months without feedback. Feel free to reopen or create a new issue if youre still having issues.