Random issues using PnP Powershell Cmdlets in Azure runbooks when running concurrently

antoniobriones commented 5 years ago

Reporting an Issue or Missing Feature

Issue

Expected behavior

PnP Cmdlets should work when running concurrently from Azure runbooks

Actual behavior

They randomly fail, but only when the same runbook runs more than one time in paralell. There are never two runbooks at the same time executing (any) command against the same site, they are always different.

Examples of the errors

Apply-PnPProvisioningTemplate: Object reference not set to an instance of an object.

Disconnect-PnPOnline : Object reference not set to an instance of an object. At line:51 char:5 + Disconnect-PnPOnline + ~~~~ + CategoryInfo : NotSpecified: (:) [Disconnect-PnPOnline], NullReferenceException + FullyQualifiedErrorId : System.NullReferenceException,SharePointPnP.PowerShell.Commands.Base.DisconnectSPOnline

Get-PnPList : The data is not available. The query may not have been executed. At C:\Temp\4qpzjsz2.ihm\FeatureOutput.ps1:37 char:14 + $listExist = Get-PnPList -Identity Lists/$listName + ~~~~~~~~~ + CategoryInfo : WriteError: (:) [Get-PnPList], ClientRequestException + FullyQualifiedErrorId : EXCEPTION,SharePointPnP.PowerShell.Commands.Lists.GetList

Add-PnPFieldToContentType : No connection, please connect first with Connect-PnPOnline At C:\Temp\x0dupgbv.o3e\SiteProvisioner_CustomAction.ps1:33 char:5 + Add-PnPFieldToContentType -Field "siteColumn1Text" -ContentType $ ... + ~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Add-PnPFieldToContentType], InvalidOperationException + FullyQualifiedErrorId : System.InvalidOperationException,SharePointPnP.PowerShell.Commands.ContentTypes.AddFieldToContentType

Set-PnPTenantSite : Object reference not set to an instance of an object. At C:\Temp\ocziidtj.c0l\SiteProvisioner_CustomAction.ps1:16 char:1 + Set-PnPTenantSite -Url $url -NoScriptSite:$false + ~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Set-PnPTenantSite], NullReferenceException + FullyQualifiedErrorId : System.NullReferenceException,SharePointPnP.PowerShell.Commands.SetTenantSite

Get-PnPListItem : Object reference not set to an instance of an object. At C:\Temp\psw4qfpz.2dp\FeatureOutput.ps1:39 char:17 + ... $listItems= Get-PnPListItem -List $listName -Query "<Whe ... + ~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Get-PnPListItem], NullReferenceException + FullyQualifiedErrorId : System.NullReferenceException,SharePointPnP.PowerShell.Commands.Lists.GetListItem

Add-PnPSiteCollectionAppCatalog : Object reference not set to an instance of an object. At line:34 char:5 + Add-PnPSiteCollectionAppCatalog -Site $url + ~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Add-PnPSiteCollectionAppCatalog], NullReferenceException + FullyQualifiedErrorId : System.NullReferenceException,SharePointPnP.PowerShell.Commands.Admin.AddSiteCollectionAppCatalog + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException

Steps to reproduce behavior

Create a runbook that connects to a site using Connect-PnP-Online, and then runs any of the commands that error out.
Create a webhook for it and get the url
Run it 10 times within the same second

Which version of the PnP-PowerShell Cmdlets are you using?

[ ] PnP PowerShell for SharePoint 2013
[ ] PnP PowerShell for SharePoint 2016
[X] PnP PowerShell for SharePoint Online

What is the version of the Cmdlet module you are running?

3.12.1908.1

How did you install the PnP-PowerShell Cmdlets?

[ ] MSI Installed downloaded from GitHub
[ ] Installed through the PowerShell Gallery with Install-Module
[X] Other means: installed module from the modules galery from within the automation account

ghost commented 5 years ago

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

garrytrinder commented 5 years ago

This isn’t an issue with the PnP PowerShell module.

When running many runbooks at the same time the chances of them running in the same sandbox is much higher and scripts can affect each other.

See first paragraph of the below article

https://docs.microsoft.com/en-us/azure/automation/automation-runbook-execution

“When you start a runbook in Azure Automation, a job is created. A job is a single execution instance of a runbook. An Azure Automation worker is assigned to run each job. While workers are shared by many Azure accounts, jobs from different Automation accounts are isolated from one another. You don't have control over which worker services the request for your job. A single runbook can have many jobs running at one time. The execution environment for jobs from the same Automation Account may be reused. The more jobs you run at the same time, the more often they can be dispatched to the same sandbox. Jobs running in the same sandbox process can affect each other, one example is running the Disconnect-AzureRMAccount cmdlet. Running this cmdlet would disconnect each runbook job in the shared sandbox process.”

I would reconsider your design to batch your processing into a single Runbook execution, rather than executing the same Runbook many times.

Runbooks are designed for long running operations not short frequent ones.

antoniobriones commented 5 years ago

Thanks @garrytrinder for your response. When I read that paragraph I also thought it could be due to the runbook execution concurrency, but then I found I could substantially reduce the amount of failures just by replacing the PnP cmdlets by the underlying CSOM. For ex, I replaced Get-PnPListItem by get-list() and building a CAML query and I didn't get errors anymore. So, I just connect with PnP to get the context and wherever I can, I use CSOM and the code runs without issues, but I still get them where I'm running PnP Cmdlets.

If the issue is not on the PnP side but is on the runbooks concurrency, do you think that replacing them by functions would have the same behaviour?

Thanks!

garrytrinder commented 5 years ago

Are you able to share the code that you are executing in parallel?

antoniobriones commented 5 years ago

@garrytrinder, here you go:

https://github.com/antoniobriones/samples/blob/master/SiteProvisioner.ps1 is a runbook that gets triggered after a sharepoint site is created (through a site script0
that repository contains other runbooks, all of them fail under concurrent scenarios

Thanks!

garrytrinder commented 4 years ago

@antoniobriones

I've taken a look at your script, try using the -ReturnConnection switch on Connect-PnPOnline and pass this to your cmdlets directly, example below

$connection = Connect-PnPOnline -AppId $appId -AppSecret $appSecret -Url $url -ReturnConnection
$web = Get-PnPWeb -Connection $connection
Disconnect-PnPOnline -Connection $connection

DaniCorretja commented 4 years ago

@antoniobriones

I've taken a look at your script, try using the -ReturnConnection switch on Connect-PnPOnline and pass this to your cmdlets directly, example below
$connection = Connect-PnPOnline -AppId $appId -AppSecret $appSecret -Url $url -ReturnConnection
$web = Get-PnPWeb -Connection $connection
Disconnect-PnPOnline -Connection $connection

That's definitively the solution you are after. I had similar issues in the past with Runbooks running in parallel executing PnP commands and producing unexpected results and that's exactly how I resolved the issue.

bryanhart73 commented 4 years ago

I was seeing the same issue. -ReturnConnection and -Connection params as described above resolved my issues.

pnp / PnP-PowerShell