search

beamable / BeamableProduct

The beamable product suite including com.beamable, com.beamable.server, microservice base image, portal, the installer, and build scripts
Other
5 stars 0 forks source link

CLI update error in 2.0.1 #3619

Open cdhanna opened 2 weeks ago

cdhanna commented 2 weeks ago

If you have a C#MS, and you do a 

beam services deploy

Then sometimes you get this strange timeout-ish style error

Internal slack convo; https://disruptorbeam.slack.com/archives/C03P1J5NR97/p1724793009195179

[Exception] System.Net.Http.HttpRequestException Error while copying content to a stream.    at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource 
pendingRequestsCts, CancellationToken originalCancellationToken)
   at cli.Services.BeamoLocalSystem.UploadChunk(ContainerUploadData data, FileChunk chunk, Uri location, CancellationToken token) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 437
   at cli.Services.BeamoLocalSystem.UploadFileBlob(ContainerUploadData data, String filename, CancellationToken token) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 378
   at cli.Services.BeamoLocalSystem.UploadLayer(ContainerUploadData data, Action`2 onContainerUploadProgress, String layerPath, CancellationToken token) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 337
   at cli.Services.BeamoLocalSystem.Upload(ContainerUploadData data, String folder, CancellationToken token, Action`2 onContainerUploadProgress) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 319
   at cli.Services.BeamoLocalSystem.<>c__DisplayClass84_0.<<UploadContainers>b__2>d.MoveNext() in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 264

[Exception] System.Net.Http.HttpRequestException An error occurred while sending the request.    at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, 
CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource 
pendingRequestsCts, CancellationToken originalCancellationToken)
   at cli.Services.BeamoLocalSystem.UploadChunk(ContainerUploadData data, FileChunk chunk, Uri location, CancellationToken token) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 437
   at cli.Services.BeamoLocalSystem.UploadFileBlob(ContainerUploadData data, String filename, CancellationToken token) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 378
   at cli.Services.BeamoLocalSystem.UploadLayer(ContainerUploadData data, Action`2 onContainerUploadProgress, String layerPath, CancellationToken token) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 337
   at cli.Services.BeamoLocalSystem.Upload(ContainerUploadData data, String folder, CancellationToken token, Action`2 onContainerUploadProgress) in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 319
   at cli.Services.BeamoLocalSystem.<>c__DisplayClass84_0.<<UploadContainers>b__2>d.MoveNext() in 
/home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 264

Error while copying content to a stream.
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at cli.Services.BeamoLocalSystem.UploadChunk(ContainerUploadData data, FileChunk chunk, Uri location, CancellationToken token) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 437
   at cli.Services.BeamoLocalSystem.UploadFileBlob(ContainerUploadData data, String filename, CancellationToken token) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 378
   at cli.Services.BeamoLocalSystem.UploadLayer(ContainerUploadData data, Action`2 onContainerUploadProgress, String layerPath, CancellationToken token) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 337
   at cli.Services.BeamoLocalSystem.Upload(ContainerUploadData data, String folder, CancellationToken token, Action`2 onContainerUploadProgress) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 319
   at cli.Services.BeamoLocalSystem.<>c__DisplayClass84_0.<<UploadContainers>b__2>d.MoveNext() in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 264
--- End of stack trace from previous location ---
   at cli.Services.BeamoLocalSystem.UploadContainers(String[] beamoIds, String[] folders, String dockerRegistryUrl, CancellationToken cancellationToken, Action`2 onContainerUploadProgress, Action`2 onContainerUploadCompleted) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 279
   at cli.Services.BeamoLocalSystem.DeployToRemote(BeamoLocalSystem localSystem, String dockerRegistryUrl, String comments, Dictionary`2 perServiceComments, Action`2 buildPullImageProgress, Action`1 onServiceDeployCompleted, Action`2 onContainerUploadProgress, Action`2 onContainerUploadCompleted) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Services/BeamoLocalSystem_RemoteCommunication.cs:line 84
   at cli.ServicesDeployCommand.<>c__DisplayClass7_0.<<Handle>b__0>d.MoveNext() in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Commands/Services/ServicesDeployCommand.cs:line 213
--- End of stack trace from previous location ---
   at Spectre.Console.Progress.<>c__DisplayClass31_0.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:line 103
--- End of stack trace from previous location ---
   at Spectre.Console.Progress.<>c__DisplayClass32_0`1.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:line 138
--- End of stack trace from previous location ---
   at Spectre.Console.Internal.DefaultExclusivityMode.RunAsync[T](Func`1 func) in /_/src/Spectre.Console/Internal/DefaultExclusivityMode.cs:line 40
   at Spectre.Console.Progress.StartAsync[T](Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:line 121
   at Spectre.Console.Progress.StartAsync(Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:line 101
   at cli.ServicesDeployCommand.Handle(ServicesDeployCommandArgs args) in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/Commands/Services/ServicesDeployCommand.cs:line 186
   at cli.DependencyInjectionExtensions.<>c__DisplayClass3_0`3.<<AddSubCommandWithHandler>b__1>d.MoveNext() in /home/runner/work/BeamableProduct/BeamableProduct/cli/cli/DependencyInjectionExtensions.cs:line 78
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<<AddMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

This happens after a few timeouts while uploading an image

cdhanna commented 2 weeks ago

The fix here should be to add some retry logic and error logging that the retry is happening.

cdhanna commented 2 weeks ago

We should validate that upload works with CLI 2.1RC2; if not, we'll pull this ticket into the CLI 2.1 milestone; otherwise it can go out in the 2.1.1