The request was canceled due to the configured HttpClient.Timeout of 120 seconds elapsing

[FreePhoenix888]

Function which fails

    public async Task SaveCandles(IList<Candle> candles)
    {
        using var bulkCopyInterface = new ClickHouseBulkCopy(ClickHouseConnection)
        {
            DestinationTableName = "candles",
            BatchSize = candles.Count
        };
        var candleRows = candles.Select((candle) => new object[] { candle.StartingTime, candle.OpeningPrice, candle.ClosingPrice, candle.LowestPrice, candle.HighestPrice, candle.Volume });
        await bulkCopyInterface.WriteToServerAsync(candleRows);
    }

Error

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 120 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ClickHouse.Client.ADO.ClickHouseConnection.OpenAsync(CancellationToken cancellationToken)
   at ClickHouse.Client.ADO.ClickHouseCommand.PostSqlQueryAsync(String sqlQuery, CancellationToken token)
   at ClickHouse.Client.ADO.ClickHouseCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
   at DoubletsVsClickHouseBenchmarks.ClickHouseAdapter.RemoveCandles() in /workspace/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks/ClickHouseAdapter.cs:line 73
   at Platform.Data.Doublets.Benchmarks.DoubletsVsClickHouseBenchmarks.LinksPlatformBenchmark() in /workspace/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks.cs:line 61
   at BenchmarkDotNet.Autogenerated.Runnable_0.<.ctor>b__3_3() in /workspace/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks/bin/Release/net7.0/ae56c136-45db-472c-8c61-5a3d9f17d97e/ae56c136-45db-472c-8c61-5a3d9f17d97e.notcs:line 197
   at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount) in /workspace/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks/bin/Release/net7.0/ae56c136-45db-472c-8c61-5a3d9f17d97e/ae56c136-45db-472c-8c61-5a3d9f17d97e.notcs:line 319
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName) in /workspace/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks/bin/Release/net7.0/ae56c136-45db-472c-8c61-5a3d9f17d97e/ae56c136-45db-472c-8c61-5a3d9f17d97e.notcs:line 176
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /workspace/DoubletsVsClickHouseBenchmarks/DoubletsVsClickHouseBenchmarks/bin/Release/net7.0/ae56c136-45db-472c-8c61-5a3d9f17d97e/ae56c136-45db-472c-8c61-5a3d9f17d97e.notcs:line 57

[FreePhoenix888]

@DarkWanderer

[FreePhoenix888]

https://github.com/DarkWanderer/ClickHouse.Client/commit/0773e79574a063c0d6bab27cfe2ddf192f2ddc35 By looking at the commit message I guess it is possible to change this timeout

I am not sure if this error was thrown by server or client

[DarkWanderer]

You are trying to write full data set as a single batch: BatchSize = candles.Count - this will not work for very large amounts of data. Recommended batch size is 100'000-1'000'000 (which will also make use of parallel insertion feature)

[DarkWanderer]

You can set Timeout in connection string, or provide a pre-configure HttpClient. Either way, inserting in really large batches does not allow ClickHouse to work well and it feels better if there is a reasonable batch size (as mentioned above), from perspective of part size on disk and cluster replication