Umbraco.Deploy.OnPrem fails to restore/partial restore while 'queue for transfer' works

[rogalmic]

After upgrading from Umbraco.Deploy.Onprem 10.1.3 to 10.2.0 all restore, partial restore, compare actions performed on Development environment from Test environment give this exception:

d22f555a - Executing, 10% - Get remote session
d22f555a - Executing, 80% - Exception!
d22f555a - Executing, 80% - Abort remote session
d22f555a - Executing, 80% - Restore caches and indexes
d22f555a - Failed, 80% - Total time 00:00:00.322.

-- EXCEPTION ---------------------------------------------------
 (proper exception traces at the end of this log)
Umbraco.Deploy.Infrastructure.Exceptions.RemoteApiException: The remote API was not found, or args were invalid, or request was too long.
----------------------------------------------------------------

EXCEPTION:

Umbraco.Deploy.Infrastructure.Exceptions.RemoteApiException: The remote API was not found, or args were invalid, or request was too long.
   at Umbraco.Deploy.Infrastructure.Http.DeployHttpClient.SucceedOrThrow(HttpResponseMessage result, CancellationToken token)
   at Umbraco.Deploy.Infrastructure.Http.DeployHttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken token)
   at Umbraco.Deploy.Infrastructure.Http.DeployHttpClient.SendAsync(HttpRequestMessage request, CancellationToken token)
   at Umbraco.Deploy.Infrastructure.Environments.RemoteUmbracoEnvironment.BeginSessionAsync(Guid sessionId, CancellationToken token)
   at Umbraco.Deploy.Infrastructure.Work.WorkItems.DeployRestoreWorkItemBase.GetSessionOnWithBatchInfo(IUmbracoEnvironment env, Nullable`1 batchIndex, Nullable`1 numberOfBatches)
   at Umbraco.Deploy.Infrastructure.Work.WorkItems.DeployRestoreWorkItemBase.GetSessionOn(IUmbracoEnvironment env)
   at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.ExecuteAsyncSub(IWorkContext context, List`1 resume, CancellationToken token)
   at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.ExecuteAsyncSub(IWorkContext context, List`1 resume, CancellationToken token)
   at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.ExecuteAsync(IWorkContext context, CancellationToken token)

REMOTE:

The remote section of exception log is totally empty (see above).

At the same time, when using option 'queue for transfer' on dev, test receives the node - which indicates that the connection is established fine. Also when i change an api key, i get different error.

Umbraco version does not seem to matter, tried on 10.4.2 and 10.5.0. The issue is 100% reproducible on 10.2.0. The restore from Prod seems to be working with same scenario.

[AndyButland]

This one is a bit of a mystery I'm afraid. It all points to an issue with connectivity to your upstream environment (the error message you see is triggered when a 404 response is found, and that also would explain why there's no "remote" error to report). But of course you say transfers to the upstream environment work as expected, which we wouldn't expect if there was a connectivity issue.

Would be worth making sure you have the same version on both environments, though even that I can't immediately see why would lead to the issue you are reporting.

I also can't see what's in the upgrade you've made that might have caused the problem, but to verify that it might be worth downgrading again just to be sure. To downgrade you will first need to reset the migration state in the database with:

update umbracoKeyValue
set value = '{c757ab7e-a3d0-4533-8cd9-9ce457927a81}'
where [key] = 'Umbraco.Core.Upgrader.State+Deploy.Migration'

[rogalmic]

I created a vanilla Umbraco setup that reproduces this with all environments using sqlite (attached). TestProjectsUmbraco10.zip

This is pretty weird, after downgrade it still does not work.

Also, collected logs on debug level from server side:

[11:24:27 INF] Request starting HTTP/1.1 POST http://localhost:43478//umbraco/backoffice/deploy/environment/BeginSession application/json;+charset=utf-8 94
[11:24:27 INF] Registered with MainDom, localContentDbExists? True, localMediaDbExists? True
[11:24:27 INF] Creating the content store, localContentDbExists? True
[11:24:27 INF] Creating the media store, localMediaDbExists? True
[11:24:27 INF] Loading content from local cache file [Timing e5f658b]
[11:24:27 INF] Completed. (31ms) [Timing e5f658b]
[11:24:27 INF] Loading media from local cache file [Timing fe70962]
[11:24:27 INF] Completed. (0ms) [Timing fe70962]
[11:24:27 INF] Executing endpoint 'Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)'
[11:24:27 INF] Route matched with {action = "Index", controller = "Render", area = ""}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Index() on controller Umbraco.Cms.Web.Common.Controllers.RenderController (Umbraco.Web.Common).
[11:24:27 INF] Executing ViewResult, running view ~/umbraco/UmbracoWebsite/NotFound.cshtml.
[11:24:27 INF] Executed ViewResult - view ~/umbraco/UmbracoWebsite/NotFound.cshtml executed in 11.4417ms.
[11:24:27 INF] Executed action Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common) in 21.6239ms
[11:24:27 INF] Executed endpoint 'Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)'
[11:24:27 INF] Request finished HTTP/1.1 POST http://localhost:43478//umbraco/backoffice/deploy/environment/BeginSession application/json;+charset=utf-8 94 - 404 - text/html;+charset=utf-8 633.3092ms
[11:24:27 INF] Request starting HTTP/1.1 POST http://localhost:43478//umbraco/backoffice/deploy/environment/AbortSessionWithInfos application/json;+charset=utf-8 356
[11:24:27 INF] Executing endpoint 'Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)'
[11:24:27 INF] Route matched with {action = "Index", controller = "Render", area = ""}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Index() on controller Umbraco.Cms.Web.Common.Controllers.RenderController (Umbraco.Web.Common).
[11:24:27 INF] Executing ViewResult, running view ~/umbraco/UmbracoWebsite/NotFound.cshtml.
[11:24:27 INF] Executed ViewResult - view ~/umbraco/UmbracoWebsite/NotFound.cshtml executed in 1.1213ms.
[11:24:27 INF] Executed action Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common) in 1.3019ms
[11:24:27 INF] Executed endpoint 'Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)'
[11:24:27 INF] Request finished HTTP/1.1 POST http://localhost:43478//umbraco/backoffice/deploy/environment/AbortSessionWithInfos application/json;+charset=utf-8 356 - 404 - text/html;+charset=utf-8 4.0204ms

We are using headless approach in umbraco, there are no templates - maybe that is the issue?

[rogalmic]

@AndyButland So the issue is the url in configuration cannot have trailing slash, because it will create wrong url: http://localhost:43478//umbraco/backoffi...

Maybe the plugin could combine host and path using:

Uri baseUri = new Uri("http://www.contoso.com/");
Uri myUri = new Uri(baseUri, "/catalog/shownew.htm");

to prevent future issues?

[AndyButland]

OK, glad you've sorted it and good to know it wasn't something inadvertently introduced in the update, but you are right in that we should certainly be more "forgiving" there depending on whether a trailing slash is included or not. We'll take a look for the next update.