search

SharePoint / sp-dev-docs

SharePoint & Viva Connections Developer Documentation
https://docs.microsoft.com/en-us/sharepoint/dev/
Creative Commons Attribution 4.0 International
1.25k stars 1.01k forks source link

Random 500 internal server errors with different APIs and operations at SharePoint Online #4924

Closed advdberg closed 4 years ago

advdberg commented 5 years ago

Category

[X] Bug [ ] Enhancement

Environment

[X] Office 365 / SharePoint Online [ ] SharePoint 2016 [ ] SharePoint 2013

Expected or Desired Behavior

Template is applied without errors (or at least with detailed errors)

Observed Behavior

We get an Intermittant 500 server error on applying a template with the following tracelog:

powershell.exe Information: 0 : 2019-10-08 14:40:40.6587    [OfficeDevPnP.Core]    [0]    [Information]    Adding field (382aae1d-7054-4b5d-85ca-ead7dfdd96f0) to content type (0x01010043443884AE06CF4D9F9C2D75C47BF75A).    0ms   
powershell.exe Information: 0 : 2019-10-08 14:40:42.7683    [OfficeDevPnP.Core]    [0]    [Information]    Adding field (eea8f2b4-9508-46a0-9f42-69008561a746) to content type (0x01010043443884AE06CF4D9F9C2D75C47BF75A).    0ms   
powershell.exe Error: 0 : 2019-10-08 14:40:42.8361    [OfficeDevPnP.Core]    [0]    [Error]    ExecuteQuery threw following exception: System.Net.WebException: De externe server heeft een fout geretourneerd: (500) Interne serverfout.
   bij System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   bij System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.SharePoint.Client.SPWebRequestExecutor.<ExecuteAsync>d__0.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryToServerAsync>d__6.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryAsync>d__0.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.SharePoint.Client.ClientRuntimeContext.<ExecuteQueryAsync>d__0.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.SharePoint.Client.ClientContext.<ExecuteQueryAsync>d__4.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__7.MoveNext().    0ms   
powershell.exe Information: 0 : 2019-10-08 14:40:42.8371    [Content Types]    [15]    [Debug]    Code execution scope ended    115033ms    57b414c0-2d01-471d-b956-b5bab9c3667f
powershell.exe Information: 0 : 2019-10-08 14:40:42.8371    [Provisioning]    [15]    [Debug]    Code execution scope ended    168283ms    57b414c0-2d01-471d-b956-b5bab9c3667f

Steps to Reproduce

execute the following PowerShell: Apply-PnPProvisioningTemplate -Path "C:\temp\SiteTemplate(1).xml" -Verbose

ghost commented 5 years ago

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

marcjornet commented 4 years ago

Similar error here but with c#. 2 days ago we have applied Templates to created sites without any problem.

2019-11-14 10:28:09,594 ERROR MSF_Meetig_Site_v2.Program - ERROR: The remote server returned an error: (500) Internal Server Error. 2019-11-14 10:28:09,595 ERROR MSF_Meetig_Site_v2.Program - STACK: 2019-11-14 10:28:09,630 ERROR MSF_Meetig_Site_v2.Program - at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.SPWebRequestExecutor.d0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientRequest.d6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientRequest.d0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientRuntimeContext.d0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientContext.d4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientContextExtensions.d7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.SharePoint.Client.ClientContextExtensions.d__7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientContextExtensions.ExecuteQueryRetry(ClientRuntimeContext clientContext, Int32 retryCount, Int32 delay, String userAgent) at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.ObjectListInstance.CreateView(Web web, View view, ViewCollection existingViews, List createdList, PnPMonitoredScope monitoredScope, TokenParser parser, Int32 currentViewIndex, Int32 total) at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.ObjectListInstance.ProcessViews(Web web, TokenParser parser, PnPMonitoredScope scope, ListInfo listInfo) at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.ObjectListInstance.ProvisionObjects(Web web, ProvisioningTemplate template, TokenParser parser, ProvisioningTemplateApplyingInformation applyingInformation) at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.SiteToTemplateConversion.ApplyRemoteTemplate(Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation provisioningInfo, Boolean calledFromHierarchy, TokenParser tokenParser) at Microsoft.SharePoint.Client.WebExtensions.ApplyProvisioningTemplate(Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation applyingInformation) at MSF_Meetig_Site_v2.Program.ApplyTemplate(String parent, String platformName, String userName, SecureString pwd)

Any solution?

Kind regards,

advdberg commented 4 years ago

We still see the same behaviour in our logging system. In our Provisioning solution we retry applying the template for 2 times (so 3 in total) this seems to cover most of the scenarios where we get these errors but that sometimes causes massive delays in provisoning a ready-to-use collaboration environment while some templates might take up to 30 minutes to apply. Depending on in what stage of the template the error is thrown it can take up to 2 to 3 times the normal time to get a fully operational site.

marcjornet commented 4 years ago

Same here, yesterday for some sites we had to retry more than 5 times, being a mess for our customer :(

richardb52 commented 4 years ago

We have been encountering the same issue over the last 2 months or so. From our logging it seems to happen across different stages of the provisioning, sometimes after configuring Search Settings, sometimes during configuring List Instances.

mblennegard commented 4 years ago

I can confirm I have the same issue as well. I had a provisioning script that worked just fine up until 1-2 weeks ago.

Now all of a sudden the provisioning engine stops at what I can only describe as random points in the XML execution, just like @richardb52 testifies as well. Sometimes it stops right away when applying security, sometimes after applying site settings and sometimes when updating/creating a list.

What is always happening though is that the code execution stops with the error "The remote server returned an error: (500) Internal Server Error."

JBestebreurtje commented 4 years ago

I can confirm I have the same issue as well.

"The remote server returned an error: (500) Internal Server Error."

advdberg commented 4 years ago

We see an increasing number of 500 errors on multiple environments since a couple of hours. The service health message points out it's a 'global' issue image

arunkumarperumal commented 4 years ago

we are also facing the same issue. At various stages of the site creation it gives the "500 Internal Server error". Also noted that it took 20 minutes to create a modern communication site using the provisioning engine.

virtualducttape commented 4 years ago

We are also experiencing this issue, we can stumble our way through by continually trying to apply the template again and again, but that's not really a feasible solution.

mbratukha commented 4 years ago

Since today we are also facing the same issue. The same template can be provisioned several times without any problem, but next several times comes this error.

vilmarci commented 4 years ago

Same issue here.

green-munkey commented 4 years ago

Same issue .

fzbm commented 4 years ago

Same issue on several tenants. Random Internal Server Errors when deploying a template.

jackpoz commented 4 years ago

same issue on multiple tenants

netti01 commented 4 years ago

During the last weeks/month the overall SPO infractructure is getting worse and worse. This is a really annoying situation for us and all of our customers. Again a great firework of new UI features Microsoft announced at Inspire but there is no single word about Microsoft's invest in hardening or performance improvements for the overall infrastructure... (Mostly all of our customer tenants have a SharePoint Healthscore greater 7 ;-( )

And what is even more frustrating is the overall Communication of Microsoft! The so called "Network Incident" they notified about yesterday (see above) was visible for 2 hours...

@VesaJuvonen @PaoloPia @

pnpgit-ds commented 4 years ago

same issue here on several tenants.

cleferman commented 4 years ago

Same issue, it fails at different steps on multiple tenants.

h3rd4 commented 4 years ago

Same issue for me on several tenants. 500 server errors randomly retrieve when try apply template. Situation went wrong couple of days ago.

We have lot of provisioning systems based on pnp templating and situation is frustrating for me when i must repeat and repeat provisioning manually for each individual fail.

tarjeil commented 4 years ago

Same issue here. Has anyone reported this one to support?

My experience is that they will point to PnP and close the issue. Maybe we could join forces - I am happy to escalate to our account manager at MSFT

rogerskauen commented 4 years ago

Same issue for me

TazzyMan commented 4 years ago

Same here. Multiple tenants

MattVandepitte commented 4 years ago

We have the issue while migrating with ShareGate and with the PnP engine on different tenants.

Moutah commented 4 years ago

Same issue here as well.

SharePoint server randomly returns 500 on various requests. While 500 should not happen, maybe retrying the request for a given amount of attempts before exiting could overcome this issue?

JBPuzzled commented 4 years ago

We also experience issue while running provisioning PowerShell scripts using PnP.

Next to that we also get similar internal server errors while migrating using ShareGate Desktop. They stated that those errors originate from SPO.

fzbm commented 4 years ago

@Moutah Sadly this is not the case. Our deployment process retries 10 times and every try fails with an Internal Server Error on a random position.

VesaJuvonen commented 4 years ago

Thanks everyone for reporting the issue, but to be able to investigate, we'd need to understand few things around what's happening...

Thanks for the additional details advance. These are critical to get started with any troubleshooting.

VesaJuvonen commented 4 years ago

Also - if someone has Premier support, would absolutely recommend reporting this issue through that channel as the issue is generated from the underlying out-of-the-box APIs (impacts also partner solutions), so getting investigation/resolution moving from multiple sides is always good.

ghost commented 4 years ago

i have this with the pnppowershell module version Name Version

SharePointPnPPowerShellOnline 3.15.1911.0

we use the following script te generate a template: `#Global variables # $siteURL = "sitetemplateurl" <#Fill in the URL to the Template Site#> $cred = Get-Credential $saveDir = "C:\pnpprov\" <#Fill in the Path where the Template will be saved, this path must exist#> $library = "Sitepagina's" #EN = Site Pages , NL = Sitepagina's

Change Direcory Path

pushd $saveDir -ErrorAction SilentlyContinue

Connecting To Template Site

Write-Host "Connecting to: $siteURL" -ForegroundColor Yellow Connect-PnPOnline -Url $siteURL -Credentials $cred Write-Host "Connected!" -ForegroundColor Green

Get Web Object

$web = Get-PnPWeb $sourceSite = $web.ServerRelativeUrl

Get all pages in the site pages library

$pages = Get-PnPListItem -List $library

Save current homepage

$currentHomePage = Get-PnPHomePage $pagesList = New-Object System.Collections.Generic.List[System.Object] $pageNumber = 1

foreach($page in $pages){ if($page.FileSystemObjectType -eq "File"){ $pagePath = $page.FieldValues["FileRef"] $pageFile = $pagePath -replace $sourceSite, "" $pageTemplate = $pageFile -replace "/SitePages/","" -replace ".aspx",".xml" $pagesList.Add($($pageTemplate -replace "./TemplateWithPages","")) 

    #set current page as home page
    Set-PnPHomePage -RootFolderRelativeUrl ($pagePath -replace ($sourceSite+"/"), "")

    Write-Host ("Saving page #" + $pageNumber + " - " + $pageTemplate)

    if($pageNumber -eq 1){
        Get-PnPProvisioningTemplate -Out $($saveDir + "\" + $pageTemplate) -PersistBrandingFiles
    }else{
        Get-PnPProvisioningTemplate -Out $($saveDir + $pageTemplate) -Handlers PageContents
    }

    $pageNumber++
}

}

$pagesList.ToArray()

apply default default homepage

Set-PnPHomePage -RootFolderRelativeUrl $currentHomePage

copy base template

Copy-item -path ($saveDir + $pagesList[0]) -destination ($saveDir + "\pnpprov-Template.xml")

open main xml

$mainFile = [xml][io.File]::ReadAllText($($saveDir + $("\pnpprov-Template.xml"))) $clientSidePages = $mainFile.Provisioning.Templates.ProvisioningTemplate.ClientSidePages

remove pages to avoid duplicates

$clientSidePages.RemoveChild($mainFile.Provisioning.Templates.ProvisioningTemplate.ClientSidePages.ClientSidePage)

foreach( $page in $pagesList ){

open page template xml

$xmlContents = [xml][io.File]::ReadAllText($saveDir + "\" + $page)
foreach($node in $xmlContents.Provisioning.Templates.ProvisioningTemplate.ClientSidePages.ClientSidePage)
{

    #copy nodes from page xml
    $importNode = $clientSidePages.OwnerDocument.ImportNode($node, $true);
    $clientSidePages.AppendChild($importNode) | Out-Null

    #save final tempate
    $mainFile.Save($($saveDir + $("\pnpprov-Template.xml")))

}

}

popd`

after template is generated we only need the .xml pnpprov-template.xml

and then we create a new teamsite/subsite under site collection and with follow code we apply the template ` $siteURL = "siteurl for the apply template" <#Fill in the URL to the Destionation Sub Site where the template will be applied#> $Importfile = "C:\pnpprov\pnpprov-Template.xml" <#Fill in the Provisioning Template xml file name #>

Connecting To Template Site

Write-Host "Connecting to: $siteURL" -ForegroundColor Yellow Connect-PnPOnline -Url $siteURL -Credentials $cred Write-Host "Connected!" -ForegroundColor Green $web = Get-PnPWeb

Apply Provision Template

Apply-PnPProvisioningTemplate -Path $Importfile after that we get the follow error message, based on sharepoint online Apply-PnPProvisioningTemplate : The remote server returned an error: (500) Internal Server Error. At line:1 char:1

PS C:\pnpprov> Get-PnPException

Message : The remote server returned an error: (500) Internal Server Error. Stacktrace : at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.SiteToTemplateConversion.ApplyRemoteTemplate(Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation provisioningInfo, Boolean calledFromHierarchy, TokenParser tokenParser) at SharePointPnP.PowerShell.Commands.Provisioning.Site.ApplyProvisioningTemplate.ExecuteCmdlet() at SharePointPnP.PowerShell.Commands.PnPCmdlet.ProcessRecord() ScriptLineNumber : 1 `

h3rd4 commented 4 years ago

I thing the behaviour is nondeterministic, absolutely identical template applied sometime success sometime fail, it seems version does not matter.

TazzyMan commented 4 years ago

To be clear, it's actually not the PNP template that's causing this. Just about any ExecuteQuery() or ExecuteQueryRetry() can fail with a 500 server error at the moment, so it's clearly Microsoft's doing. Retrying will fix it. However ExecuteQueryRetry() doesn't account for the 500 server error. It does so for a 429 (Too busy) and 503 (Unavailable), but not for 500. The easy fix would be to add 500 to the list for ExecuteQueryRetry(). Apparently 500 is also a retryable error.

Moutah commented 4 years ago

Here is a breakdown of what is happening on my side:

main_script.ps1

Connect-PnPOnline -Url "site/url"
$site = Get-PnPWeb
$templates = list, of, template, paths

foreach ($templates as $template) {
    Apply-PnPProvisioningTemplate -Path $template -Web $site
}

template.xml

<?xml version="1.0"?>
<pnp:Provisioning xmlns:pnp="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema">
  <pnp:Preferences Generator="OfficeDevPnP.Core, Version=2.16.1706.0, Culture=neutral" />
  <pnp:Templates>
    <pnp:ProvisioningTemplate Version="1">
      <pnp:Files>

        <pnp:Directory Src=".\path\to\my\source" Folder="Relative-Url/To/My/Destination" Overwrite="true" Level="Published" Recursive="true" ExcludedExtensions="*.xml,*.ps1" />

      </pnp:Files>
    </pnp:ProvisioningTemplate>
  </pnp:Templates>
</pnp:Provisioning>

This uses

And we end up with the following output

2019-11-19T10:32:22.9656283Z powershell.exe Error: 0 : 2019-11-19 10:32:22.9566 [OfficeDevPnP.Core] [0] [Error] ExecuteQuery threw following exception: System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
2019-11-19T10:32:22.9657074Z    at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
2019-11-19T10:32:22.9657213Z    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
2019-11-19T10:32:22.9658831Z --- End of stack trace from previous location where exception was thrown ---
2019-11-19T10:32:22.9659010Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2019-11-19T10:32:22.9659111Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2019-11-19T10:32:22.9659219Z    at Microsoft.SharePoint.Client.SPWebRequestExecutor.<ExecuteAsync>d__0.MoveNext()
2019-11-19T10:32:22.9659282Z --- End of stack trace from previous location where exception was thrown ---
2019-11-19T10:32:22.9659343Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2019-11-19T10:32:22.9659598Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2019-11-19T10:32:22.9659770Z    at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryToServerAsync>d__6.MoveNext()
2019-11-19T10:32:22.9660078Z --- End of stack trace from previous location where exception was thrown ---
2019-11-19T10:32:22.9660233Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2019-11-19T10:32:22.9660324Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2019-11-19T10:32:22.9660398Z    at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryAsync>d__0.MoveNext()
2019-11-19T10:32:22.9660461Z --- End of stack trace from previous location where exception was thrown ---
2019-11-19T10:32:22.9660544Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2019-11-19T10:32:22.9660672Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2019-11-19T10:32:22.9661064Z    at Microsoft.SharePoint.Client.ClientRuntimeContext.<ExecuteQueryAsync>d__0.MoveNext()
2019-11-19T10:32:22.9661161Z --- End of stack trace from previous location where exception was thrown ---
2019-11-19T10:32:22.9661253Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2019-11-19T10:32:22.9661348Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2019-11-19T10:32:22.9661921Z    at Microsoft.SharePoint.Client.ClientContext.<ExecuteQueryAsync>d__4.MoveNext()
2019-11-19T10:32:22.9662084Z --- End of stack trace from previous location where exception was thrown ---
2019-11-19T10:32:22.9662220Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2019-11-19T10:32:22.9662279Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2019-11-19T10:32:22.9662396Z    at Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__7.MoveNext(). 0ms 
2019-11-19T10:32:22.9663548Z powershell.exe Information: 0 : 2019-11-19 10:32:22.9566   [Provisioning]  [9] [Debug] Code execution scope ended  14123ms e0667854-6c95-43cf-85fa-6730b585a849
2019-11-19T10:32:23.0846150Z Apply-PnPProvisioningTemplate : The remote server returned an error: (500) 
2019-11-19T10:32:23.0846518Z Internal Server Error.
2019-11-19T10:32:23.0846908Z At \Path\To\main_script.ps1:00 char:00
2019-11-19T10:32:23.0847811Z + ...             Apply-PnPProvisioningTemplate -Path $templatefilepath.Rep ...
2019-11-19T10:32:23.0848583Z +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2019-11-19T10:32:23.0849157Z     + CategoryInfo          : WriteError: (:) [Apply-PnPProvisioningTemplate], 
2019-11-19T10:32:23.0849966Z     WebException
2019-11-19T10:32:23.0850195Z     + FullyQualifiedErrorId : EXCEPTION,SharePointPnP.PowerShell.Commands.Prov 
2019-11-19T10:32:23.0850606Z    isioning.Site.ApplyProvisioningTemplate
2019-11-19T10:32:23.0851246Z  
2019-11-19T10:32:23.1879977Z ##[error]PowerShell exited with code '1'.

There is no specific template that succeed or fail.

SchauDK commented 4 years ago

To be clear, it's actually not the PNP template that's causing this. Just about any ExecuteQuery() or ExecuteQueryRetry() can fail with a 500 server error at the moment, so it's clearly Microsoft's doing. Retrying will fix it. However ExecuteQueryRetry() doesn't account for the 500 server error. It does so for a 429 (Too busy) and 503 (Unavailable), but not for 500. The easy fix would be to add 500 to the list for ExecuteQueryRetry(). Apparently 500 is also a retryable error.

@TazzyMan Exactly, we added check for "Internal Server Error" to ExecuteQueryRetry() two months ago. You'll sometimes still hit the default RetryCount limit, but that can also happen when throttled. In fact this is probably also caused by throttling.

netti01 commented 4 years ago

Raised a premier support ticket for SPO. I will inform you about the results!

jansenbe commented 4 years ago

Would advise anyone that can log a premier ticket to have their customer log a ticket if they're impacted. Having more similar tickets with repro information and correlation id's will speed up the support process

tarjeil commented 4 years ago

I have raised a ticket as well

tarjeil commented 4 years ago

Template and stack trace pasted below PnP PowerShell Version 3.15.1911.0 - SharePoint Online

<?xml version="1.0"?>
<pnp:Provisioning xmlns:pnp="http://schemas.dev.office.com/PnP/2018/05/ProvisioningSchema">
  <pnp:Preferences Generator="OfficeDevPnP.Core, Version=3.15.1911.0, Culture=neutral, PublicKeyToken=null"></pnp:Preferences>
  <pnp:Templates ID="CONTAINER-MODULE">
    <pnp:ProvisioningTemplate ID="MODULE" Version="1">
      <pnp:SiteFields xmlns:pnp="http://schemas.dev.office.com/PnP/2018/05/ProvisioningSchema">  
        <Field Type="Text" DisplayName="F1" Required="FALSE" EnforceUniqueValues="FALSE" Indexed="FALSE" MaxLength="255" Group="CustomField" ID="{4cebd48a-a677-4e9f-a307-1d44fdd168ca}" SourceID="{6522ecd9-f1db-4e2d-a786-2882aa246f8a}" StaticName="F1" Name="F1">
        </Field>
      </pnp:SiteFields>
      <pnp:ContentTypes xmlns:pnp="http://schemas.dev.office.com/PnP/2018/05/ProvisioningSchema">
        <pnp:ContentType ID="0x01009B09915171B0784A8A246564F64A7998" Name="CT1" Description="" Group="CustomCT" >
          <pnp:FieldRefs>
            <pnp:FieldRef ID="4cebd48a-a677-4e9f-a307-1d44fdd168ca" Name="F1" />
          </pnp:FieldRefs>
        </pnp:ContentType>
      </pnp:ContentTypes>
    </pnp:ProvisioningTemplate>
  </pnp:Templates>
</pnp:Provisioning>
Message          : The remote server returned an error: (500) Internal Server Error.
Stacktrace       :    at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
                      at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction,
                   Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.SPWebRequestExecutor.<ExecuteAsync>d__0.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryToServerAsync>d__6.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryAsync>d__0.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.ClientRuntimeContext.<ExecuteQueryAsync>d__0.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.ClientContext.<ExecuteQueryAsync>d__4.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__7.MoveNext(
                   )
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__7.MoveNext(
                   )
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.ClientObjectExtensions.<EnsurePropertiesImplementation>d__8`1.Move
                   Next()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.ClientObjectExtensions.EnsureProperties[T](T clientObject, Express
                   ion`1[] propertySelector)
                      at Microsoft.SharePoint.Client.FeatureExtensions.<IsFeatureActiveInternal>d__12.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.SharePoint.Client.FeatureExtensions.IsFeatureActive(Web web, Guid featureID)
                      at Microsoft.SharePoint.Client.SiteExtensions.IsCommunicationSite(Site site)
                      at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.ObjectSiteFooterSettings.WillProvision
                   (Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation applyingInformation
                   )
                      at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)
                      at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.SiteToTemplateConversion.ApplyRemoteTe
                   mplate(Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation provisioningI
                   nfo, Boolean calledFromHierarchy, TokenParser tokenParser)
                      at SharePointPnP.PowerShell.Commands.Provisioning.Site.ApplyProvisioningTemplate.ExecuteCmdlet()
                      at SharePointPnP.PowerShell.Commands.PnPCmdlet.ProcessRecord()
ScriptLineNumber : 1
netti01 commented 4 years ago

Just had a teams meeting with Microsoft support (EMEA). They are aware of that there is a possible overall issue affecting different tenants. I send them logfiles of diferent tenants that we administrate. They will also collect the feedback of other actual tickets and raise/escalate them directly to the development team. Please ensure that your logs include Correlation or SP-RequestID's.

VesaJuvonen commented 4 years ago

Moving this to sp-dev-docs issue list as this is not PnP specific and good to get it tracked on that side.

msft-github-bot commented 4 years ago

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

ghost commented 4 years ago

Could this also be giving problems with get-pnpprovisioningtemplate? Got the same internal server errors, with also it says the home page could not be found, related? Script: `pushd $saveDir -ErrorAction SilentlyContinue

verbinding maken naar de template site

Write-Host "Connecting to: $siteURL" -ForegroundColor Yellow Connect-PnPOnline -Url $siteURL -Credentials $cred Write-Host "Connected!" -ForegroundColor Green

$web = Get-PnPWeb $sourceSite = $web.ServerRelativeUrl

Get all pages in the site pages library

$pages = Get-PnPListItem -List $library

Save current homepage

$currentHomePage = Get-PnPHomePage $pagesList = New-Object System.Collections.Generic.List[System.Object] $pageNumber = 1

foreach ($page in $pages) { if ($page.FileSystemObjectType -eq "File") { $pagePath = $page.FieldValues["FileRef"] $pageFile = $pagePath -replace $sourceSite, "" $pageTemplate = $pageFile -replace "/SitePages/", "" -replace ".aspx", ".xml" $pagesList.Add($($pageTemplate -replace "./TemplateWithPages", "")) 

    #set current page as home page
    Set-PnPHomePage -RootFolderRelativeUrl ($pagePath -replace ($sourceSite + "/"), "")

    Write-Host ("Saving page #" + $pageNumber + " - " + $pageTemplate)

    if ($pageNumber -eq 1) {
        Get-PnPProvisioningTemplate -Out $($saveDir + "\" + $pageTemplate) -PersistBrandingFiles
    }
    else {
        Get-PnPProvisioningTemplate -Out $($saveDir + $pageTemplate) -Handlers PageContents
    }

    $pageNumber++
}

}

$pagesList.ToArray()

apply default default homepage

Set-PnPHomePage -RootFolderRelativeUrl $currentHomePage

copy base template

Copy-item -path ($saveDir + $pagesList[0]) -destination ($saveDir + "\pnpprov-Commissies.xml")

open main xml

$mainFile = [xml][io.File]::ReadAllText($($saveDir + $("\pnpprov-Commissies.xml"))) $clientSidePages = $mainFile.Provisioning.Templates.ProvisioningTemplate.ClientSidePages

remove pages to avoid duplicates

$clientSidePages.RemoveChild($mainFile.Provisioning.Templates.ProvisioningTemplate.ClientSidePages.ClientSidePage)

foreach ( $page in $pagesList ) {

open page template xml

$xmlContents = [xml][io.File]::ReadAllText($saveDir + "\" + $page)
foreach ($node in $xmlContents.Provisioning.Templates.ProvisioningTemplate.ClientSidePages.ClientSidePage) {

    #copy nodes from page xml
    $importNode = $clientSidePages.OwnerDocument.ImportNode($node, $true);
    $clientSidePages.AppendChild($importNode) | Out-Null

    #save final tempate
    $mainFile.Save($($saveDir + $("\pnpprov-Commissies.xml")))

}

}

popd`

VesaJuvonen commented 4 years ago

It's not PnP specific issue in first place, but yes - you can get the 500 exception in any out-of-the-box API call currently regardless if that's done through native APIs or PnP abstraction. This is being closely investigated currently.

gabrielnovoa commented 4 years ago

As @netti01 said, we identified a potential issue where several HTTP 500 errors are returned in calls made from SharePoint Online Client Side Object Model (_vti_bin/client.svc/ProcessQuery) solutions. If you have samples of failed responses (with SPRequestID) or Fiddler Traces, please open a support ticket on Office 365 Admin portal and refer this issue for tracking.

JBestebreurtje commented 4 years ago

Any news?

netti01 commented 4 years ago

This is the actual Service degradation incident appearing in Microsoft Admin Center. We assume that it is only visible on tenants which raised the support request.

image

koltyakov commented 4 years ago

The description is not completely accurate. This is not only about admin or custom scripts, but any API calls including OOTB requests.

VesaJuvonen commented 4 years ago

This is still being actively worked on. This is not admin script specific issue. It seems to be specific to CSOM API usage though, which also PowerShell scripts use behind the scenes. That would explain the wrong conclusion around admin scripts in the above messaging... Any custom line-of-business application using CSOM APIs impacted.

If you are experiencing this, please open also support cases through the tenant admin support channels at the tenant admin UIs, so that the scale of impact is understood as that impacts on the prioritization of the investigations. Thanks everyone who have been already been active around this issue.

We will keep you up to date on the progress as there are any news.

tarjeil commented 4 years ago

In our tenant it is classified as an "Advisory" rather than an incident. Asked our Premier Support TAM to increase the severity.. image

VesaJuvonen commented 4 years ago

This issue is still being actively worked on. No ETA at this point.

Few things to notice:

We do apologize for the inconvenience and are actively working on resolving this. Thank you for your input around the issue.

farid-ics commented 4 years ago

Hello i have same isse with Apply-PnPProvisioningTemplate " Apply-PnPProvisioningTemplate : Le serveur distant a retourné une erreur : (500) Erreur interne du serveur. Au caractère Ligne:3 : 1