search

pnp / PnP-PowerShell

SharePoint PnP PowerShell CmdLets
https://pnp.github.io/powershell
Other
985 stars 665 forks source link

Invalid field name "TaxCatchAll" on Apply-PnPProvisioningTemplate #1180

Closed vipulkelkar closed 6 years ago

vipulkelkar commented 6 years ago

Reporting an Issue or Missing Feature

Issue

Expected behavior

The PnP template gets applied to a Team site on running the Apply-PnPProvisioningTemplate command

Actual behavior

Exception.

Exception Message: Invalid field name. {f3b0adf9-c1a2-4b02-920d-943fba4b3611} This is an intermittent issue and does not occur on every execution.

Steps to reproduce behavior

Exception Message: Invalid field name. {f3b0adf9-c1a2-4b02-920d-943fba4b3611}

The tenant of the site from which the template is generated and to the site where it is applied is the same.

The template application is done through an Azure function. The provisioning fails intermittently for some executions and cannot be reproduced always !!

Which version of the PnP-PowerShell Cmdlets are you using?

What is the version of the Cmdlet module you are running?

2.19.1710.2

How did you install the PnP-PowerShell Cmdlets?

vipulkelkar commented 6 years ago

Closing the issue. It turned out that after creating a site collection using New-PnPSite command, a delay of about 30-35 seconds before I ran the Apply-PnPProvisioning template wasn't sufficient. Sometimes it takes longer for the fields etc to get provisioned in the site. I had to delay the template application step by a couple of minutes and it is working well now !

AaronWoods commented 5 years ago

Hi @vipulkelkar, you are not alone here. I am experiencing the exact same issue, I had delays after template provisioning failed where I would re-add message to Queue. With my experience it would continue to fail a few times before it would succeed eventually. I will add the delay just after site creation of 2 minutes. Is this around the same duration of a delay you used?

vipulkelkar commented 5 years ago

@AaronWoods Yes, I added approximately the same delay..my provisioning process uses microsoft flow so I added the delay in the flow itself between the site provisioning and template application.

lennertcc commented 5 years ago

Update 2: I was able to resolve this problem without use of a delay, by adding the Taxonomy feature to the PnP template. This ensures the field to be available before PnP starts processing the list views.

`

`
jeffseiffert commented 5 years ago

I have tried all the suggestions and still get this error when applying to modern sites.

dharm-4 commented 5 years ago

@lennertcc After adding Taxonomy feacture to PnP template it is still not working. do you have any idea about workaround solution for this problem? we are facing problem with Appli-PnPProvisioningTemplate and it is not finding "TaxCatchAll" & "TaxCatchAllLabel" fieldref in target sharepoint site.

Looking forward for solution to this problem.

lennertcc commented 5 years ago

@dharm-4 sorry I have no other solution. You could try to introduce a delay of about 5 minutes. Other than that, I have no idea.

You could open a new issue to request addition support.

nateforsyth commented 4 years ago

@lennertcc After adding Taxonomy feacture to PnP template it is still not working. do you have any idea about workaround solution for this problem? we are facing problem with Appli-PnPProvisioningTemplate and it is not finding "TaxCatchAll" & "TaxCatchAllLabel" fieldref in target sharepoint site.

Looking forward for solution to this problem.

@vipulkelkar I know I am late to the party, but from what we've seen during local debugging through CSOM in Azure Functions in the last day or so, the processing of the PNP template appears to be hierarchical in nature, meaning it'll process in the order that the XML elements are entered into the template.

We noticed that exporting of templates places the <pnp:Features> section near the bottom of the template, seemingly after all Content Types and Fields have already been provisioned.

We've only noticed the OPs described issue when there is significant latency on the SPO back end.

We're in the process of testing the addition of the Taxonomy feature to the <pnp:Features> section, but found it was still failing, until we moved it near the top of the template.

This is circumstantial at best at the moment though, because the latency on the back end appears to have gone - the problems we were encountering were yesterday.

This may help others, though I'd assume by now that you already have a solution after almost 2 years (please share it if you do).

Here's the order in which we are adding the sections now (we use token replacement to build custom templates):

<?xml version="1.0"?>
<pnp:Provisioning xmlns:pnp="http://schemas.dev.office.com/PnP/2018/07/ProvisioningSchema">
  <pnp:Preferences Generator="OfficeDevPnP.Core, Version=3.4.1812.0, Culture=neutral, PublicKeyToken=5e633289e95c321a" />
  <pnp:Templates ID="CONTAINER-TEMPLATE-095C6DB0876A48A1A163A8620F016148">
    <pnp:ProvisioningTemplate ID="TEMPLATE-095C6DB0876A48A1A163A8620F016148" Version="1" BaseSiteTemplate="GROUP#0" Scope="RootSite">
      <pnp:WebSettings RequestAccessEmail="" NoCrawl="false" WelcomePage="SitePages/Home.aspx" SiteLogo="{site}/_api/GroupService/GetGroupImage?id='ea36f9b1-e966-4490-8c7c-b3f6de92b68d'&amp;hash=636801576130677177" AlternateCSS="" MasterPageUrl="{masterpagecatalog}/seattle.master" CustomMasterPageUrl="{masterpagecatalog}/seattle.master" CommentsOnSitePagesDisabled="false" QuickLaunchEnabled="false" />
      <pnp:RegionalSettings AdjustHijriDays="0" AlternateCalendarType="None" CalendarType="Gregorian" Collation="25" FirstDayOfWeek="Sunday" FirstWeekOfYear="0" LocaleId="5129" ShowWeeks="false" Time24="false" TimeZone="17" WorkDayEndHour="5:00PM" WorkDays="62" WorkDayStartHour="8:00AM" />
      <pnp:PropertyBagEntries>
        <!--<pnp:PropertyBagEntry Key="_PnP_ProvisioningTemplateComposedLookInfo" Value="{themecatalog}/15/theme.spcolor&quot;,&quot;FontFile&quot;:&quot;&quot;,&quot;BackgroundFile&quot;:&quot;&quot;,&quot;Version&quot;:0}" Overwrite="false" />-->
        <pnp:PropertyBagEntry Key="ThemePrimary" Value="" Overwrite="false" Indexed="true" />
        <pnp:PropertyBagEntry Key="SiteCollectionGroupIdd94abdce-9723-4ff5-9c1f-fdfc03e109c7" Value="95b44d86-a55c-4e75-a9ae-7b39c250d490" Overwrite="false" />
      </pnp:PropertyBagEntries>
      <!--<pnp:Security></pnp:Security>-->
      [0]
      <!--<pnp:Navigation></pnp:Navigation>-->
      [1]
      <!--<pnp:Features></pnp:Features>-->
      [5]
      <!--<pnp:SiteFields></pnp:SiteFields>-->
      [2]
      <!--<pnp:ContentTypes></pnp:ContentTypes>-->
      [3]
      <!--<pnp:Lists></pnp:Lists>-->
      [4]
      <!--<pnp:CustomActions></pnp:CustomActions>-->
      [6]
      <!--<pnp:Files>
        <pnp:File Src="..\..\_Templates\BAU Team Site\Solar.spcolor" Folder="{themecatalog}/15" Overwrite="true" Level="Draft" />
      </pnp:Files>
      <pnp:ComposedLook Name="Custom by PnP PowerShell" ColorFile="{themecatalog}/15/theme.spcolor" FontFile="" BackgroundFile="" Version="0" />-->
    </pnp:ProvisioningTemplate>
  </pnp:Templates>
</pnp:Provisioning>

Please feel free to close this again.

Zerg00s commented 4 years ago

When we create a new site using New-PnPSite or even manually (does not matter), we still need to wait for TaxCatchAllField (f3b0adf9-c1a2-4b02-920d-943fba4b3611) field to be provisioned. This field does not exist until after 20-60 seconds.

So when we run Apply-PnPProvisioningTemplate immediately after New-PnPSite we hit a problem.

Workaround

Our current workaround is to wait for this field to be available using a while + Start-Sleep approach.

# Let OOTB taxonomy TaxCatchAllField column to be provisioned. This column is a depencency we cannot skip
$TaxCatchAllField = $null
while($null -eq $TaxCatchAllField){
    $TaxCatchAllField =  Get-PnPField -Identity "f3b0adf9-c1a2-4b02-920d-943fba4b3611" -ErrorAction SilentlyContinue
    Write-Host $TaxCatchAllField
    Log-Info "Waiting for TaxCatchAllField column to be provisioned..." 
    Start-Sleep -Seconds 5
}

 Apply-PnPProvisioningTemplate -Path ./PnP_Core.xml
vipulkelkar commented 4 years ago

@nateforsyth no actually. We don't have a proper solution. We have been using delay like @Zerg00s has mentioned before we start applying the template, however sometimes it even fails with a total of 5 mins of delay. I guess its the TaxonomyFieldAdded feature that provisions this field ?? Probably enabling this feature as soon as the site is created might provision the field. Might test this out later.

SiteCollectionFeatures

vipulkelkar commented 4 years ago

I tested this by creating a site through the "New-PnPSite" command and enabling the TaxonomyFieldAdded feature immediately as the site url is returned. This does not make the TaxCatchAll field available. There must be a scheduled task that runs frequently to provision this field just as on a new site the features get enabled one after the other. @VesaJuvonen What would be the correct channel to get inputs on this ? Is delaying the template application only solution ? The delay is sometimes 10 mins or even more.

jhholm commented 4 years ago

Can reproduce just today. This makes provisioning new sites with managed metadata a bit tricky as I've seen delays up to 10 minutes until the field "arrives". So far I've noticed this in a way that the site in question does not get all the default site scoped features when the site has been created.

jhholm commented 4 years ago

So far we've done a workaround and provision the content types via Content Type HUB, then provisioning the content types during provisioning time via PnP schema without TaxCatchAll and TaxCatchAllLabel fields included. When the content type is provisioned later on via content type hub, the fields seem to appear on lists.

vipulkelkar commented 4 years ago

@jhholm yup, we were testing this same approach of removing TaxCatchAll and TaxCatchAllLabel fields from the XML template and applying the template. I get the error "Value does not fall within the expected range." at a step that provisions a library which contains a Taxonomy field. On applying the template again it works, probably because the TaxCatchAll field is provisioned by then. Tested this on about 4-5 sites, same result.

jhholm commented 4 years ago

Yeah, unfortunately I get the same error now. So this is still broken.

nateforsyth commented 4 years ago

This is 100% due to server performance, meaning that if it's degraded you sometimes need to wait for 10-15 minutes for the site to complete the provisioning process before applying the template.

It's unfortunate, but this is how it works and likely won't change (there's an absolute truck load of things going on when you create a site, you sometimes just have to wait, the taxcatchall fields will turn up eventually).

lennertcc commented 4 years ago

This is 100% due to server performance, meaning that if it's degraded you sometimes need to wait for 10-15 minutes for the site to complete the provisioning process before applying the template.

It's unfortunate, but this is how it works and likely won't change (there's an absolute truck load of things going on when you create a site, you sometimes just have to wait, the taxcatchall fields will turn up eventually).

That's all understandable and fine. What I expect as a solution for this, is that the function call for enabling a feature will not return until that feature is fully enabled. So that the next feature, or list, or view or whatever, can depend on the order of execution.

thunderstorm654 commented 4 years ago

This is causing me real problems at the moment. I had built in retry/delay up to waiting a total of 30 minutes but today that seems it is not enough. TaxCatchAll is still not available on a site provisioned today after 90 minutes. Could really do with a way to force this field to be available or some other workaround. It's a problem when needing to provision MMD fields/content types with MMD fields - you can't just comment out TaxCatchAll as will get the other errors noted by @vipulkelkar yesterday.

Grateful for any other solutions other than the delay which is still proving not to be sufficient for us as it is very inconsistent such as the further 'degraded performance' experienced today.

jonathanhotono commented 4 years ago

It's even worse for me.. I've waited for 2 days and the TaxCatchAll field is not even provisioned at all on my tenants. I tested against 3 tenants and 2 tenants got long delays..

thunderstorm654 commented 4 years ago

The sites I provisioned last week still don't have TaxCatchAll either..! I am also seeing this in multiple tenants. Seems they will never get it at this point as it's been 6 days. I've found this problem is consistently happening on communication sites, but Office 365 group enabled team sites seem to be OK (for now, sometimes it's a bit slow but they do turn up). Do you see any difference between site types @jonathanhotono?

@vipulkelkar any chance you could reopen this issue?

Without a reliable solution to this it's looking like we cannot use managed metadata in automated provisioning solutions at all right now which is problematic when many of our client requirements suit using it. I know it's an 'old' feature and improvements/changes may come to this area, but this was working with delays until very recently so it seems something has broken entirely in the past week or so.

jonathanhotono commented 4 years ago

@thunderstorm654 I haven't really tested this on Team site but my colleague did got the same issue. This is now happening across more than 4 tenants.

We managed to resolve the issue (not sure if it's actually the correct way as discussed above), by adding Taxonomy feature to address invalid field TaxCatchAll

<pnp:Feature ID="73ef14b1-13a9-416b-a9b5-ececa2b0604c" Description="Taxonomy feature" />

Then in the Content Type PnP, I removed TaxCatchAll fields. Adding the feature isn't exactly making sure TaxCatchAll field is provisioned.. I am just hoping that by enabling the taxonomy feature the TaxCatchAll field can be eventually created and applied into the content type automatically.

thunderstorm654 commented 4 years ago

@jonathanhotono I also found this does 'get round it' (by removing the TaxCatchAll fields and enabling the above Taxonomy feature), so we can provision the fields and complete the process - but I think it has knock on effects related to SharePoint search. I'm not 100% on the following but from what I can gather, TaxCatchAll is needed to make sure you can use MMD fields in search e.g. refiners, and I believe crawled properties use the values from TaxCatchAll rather than directly on the MMD field. There's also some event receivers activated in the background with the Taxonomy feature and I guess these may also be missing.

Whilst we aren't currently using the MMD fields in refiners/search customisations, it makes me a little nervous that the TaxCatchAll fields are no longer turning up after nearly a week, since I'd rather not leave production sites with missing pieces... I noticed some other OOTB features are also missing from the provisioned sites. It seems to point to there being something going fundamentally wrong with the OOTB provisioning of the site before we start applying any customisations.

I know things take time to activate, but features are still missing days later and in the case of MMD fields, manually enabling the Taxonomy feature later is not adding the columns.

jonathanhotono commented 4 years ago

@thunderstorm654 You're right. I just tested on fresh environment that just recently provisioned without the TaxCatchAll field. A major side effect that I noticed is the Managed Property is NOT created automatically and the crawled property is missing ows_taxIdXXX. Please do not use my method above.

@VesaJuvonen sorry, we might need to reopen this as TaxCatchAll field now seems to stop/heavily delayed from being provisioned on most tenants.

jonathanhotono commented 4 years ago

@thunderstorm654 I followed this article and it seems to works fine now. All managed property is provisioned. https://sprunner.net/2019/05/20/office-365-taxonomy-fields-on-newly-created-site/

Basically as usual, enable the feature or add via xml Enable-PnPFeature -Identity “73ef14b1-13a9-416b-a9b5-ececa2b0604c” -Scope Site

Update the provisioning xml SiteFields to add the TaxCatchAllField manually, I have made a little modification from the article above to automatically use existing site TaxonomyHiddenList id and site id using existing pnp parameter.

<?xml version="1.0"?>
<pnp:Provisioning
  xmlns:pnp="http://schemas.dev.office.com/PnP/2018/01/ProvisioningSchema">
  <pnp:Preferences Generator="OfficeDevPnP.Core, Version=2.20.1711.0, Culture=neutral, PublicKeyToken=3751622786b357c2" />
  <pnp:Templates ID="TaxonomyPrerequisite-TEMPLATES">
    <pnp:ProvisioningTemplate ID="ValoPrerequisite-v-1.0" Version="1" Scope="RootSite">
      <pnp:SiteFields>
         <Field Type="LookupMulti" DisplayName="Taxonomy Catch All Column" StaticName="TaxCatchAll" Name="TaxCatchAll" ID="{f3b0adf9-c1a2-4b02-920d-943fba4b3611}" ShowInViewForms="FALSE" List="{{listid:TaxonomyHiddenList}}" WebId="{siteid}" Required="FALSE" CanToggleHidden="TRUE" ShowField="CatchAllData" SourceID="{484c8c59-755d-4516-b8d2-1621b38262b4}" Mult="TRUE" Sortable="FALSE" AllowDeletion="TRUE" Sealed="TRUE" Hidden="TRUE" />
        <Field Type="LookupMulti" DisplayName="Taxonomy Catch All Column1" StaticName="TaxCatchAllLabel" Name="TaxCatchAllLabel" ID="{8f6b6dd8-9357-4019-8172-966fcd502ed2}" ShowInViewForms="FALSE" List="{{listid:TaxonomyHiddenList}}" WebId="{siteid}" Required="FALSE" Hidden="TRUE" CanToggleHidden="TRUE" ShowField="CatchAllDataLabel" FieldRef="{F3B0ADF9-C1A2-4b02-920D-943FBA4B3611}" SourceID="{484c8c59-755d-4516-b8d2-1621b38262b4}" ReadOnly="TRUE" Mult="TRUE" Sortable="FALSE" AllowDeletion="TRUE" Sealed="TRUE" />
      </pnp:SiteFields>
    </pnp:ProvisioningTemplate>
  </pnp:Templates>
</pnp:Provisioning>
sunithakm commented 4 years ago

I am still getting this issue with communication site. Apply-PnPProvisioningTemplate : Invalid field name. {f3b0adf9-c1a2-4b02-920d-943fba4b3611}

thunderstorm654 commented 4 years ago

Thanks so much @jonathanhotono that is working for me too! Hooray! I still think it's worth @VesaJuvonen perhaps getting someone to take a look at comm site creation though, as some of the other OOTB features aren't enabling after many days, there could be some other functionality not getting added to these sites at the moment (applies whether you create the site via PnP cmdlets or just via the SP Admin Center/SP home).

thunderstorm654 commented 4 years ago

As another example of OOTB features not being enabled, I found that the default SharePoint groups are (sometimes/always?) not being provisioned either on comm sites. Easy fix, just enable the DefaultSPGroupsAsync feature using Enable-PnPFeature- the ID is e233eb34-e720-4ff9-9f53-a5aabc706d12. But seems to me something funky is going on.

tdwhite0 commented 4 years ago

I had this issue too, but I also had to add two more fields to @jonathanhotono solution. `

`
thunderstorm654 commented 4 years ago

Thanks @tdwhite0! Do you see any issues with the 'Result Types' page on your sites even with these modifications? I just get an error. Cheers!

tdwhite0 commented 4 years ago

@thunderstorm654 are you talking about the page at _layouts/15/manageresulttypes.aspx?level=sitecol ? I don't see any issue with mine.

thunderstorm654 commented 4 years ago

@tdwhite0 Yes - thanks for confirming. Most likely unrelated then :)

guillaume-kizilian commented 4 years ago

On Modern Communication Site, we are still having this issue despite trying 4 times (3 minutes between each attempt). I think this issue should stay open as we need more investigation.

Cheers.

BraunJonathan commented 4 years ago

I created a new issue (#2269) and put a link to this one in it... The problem is now blocking for usin production as we provide an on-demand site creation system which uses pnp templating with content-types...

baskb commented 4 years ago

I experienced the same error while applying template. Using tips above, i changed the order of executing commands.

write-host "  - Creating site"
New-PnpSite -Title "$projectNumber - $projectName" -Alias $projectNumber -Lcid 1043 -Owners $requesterEmail -Type TeamSite -HubSiteId 51f1ea29-072c-46e2-b453-9ab98a09c1ac -IsPublic

write-host "  - Adding fields
Apply-PnPProvisioningTemplate -Path $pathProjectSiteTemplate -Handlers Fields

write-host "  - Enabling taxonomy feature"
Enable-PnPFeature -Identity 73ef14b1-13a9-416b-a9b5-ececa2b0604c -Scope Site

# wait until the taxonomy field is available
$docSoort = $null
while($null -eq $docSoort){
    $docSoort =  Get-PnPField -Identity "Documentsoort" -ErrorAction SilentlyContinue
    write-host "Waiting..." 
    Start-Sleep -Seconds 5
}

write-host "  - Apply template to new site"
Apply-PnPProvisioningTemplate -Path $pathProjectSiteTemplate -ClearNavigation -ExcludeHandlers Fields

In this order it works OK.

holylander commented 4 years ago

Update 2: I was able to resolve this problem without use of a delay, by adding the Taxonomy feature to the PnP template. This ensures the field to be available before PnP starts processing the list views.

<pnp:Features> <pnp:SiteFeatures> <pnp:Feature ID="73ef14b1-13a9-416b-a9b5-ececa2b0604c" Description="Taxonomy feature" /> </pnp:SiteFeatures> </pnp:Features>

This is what worked for me. Many thanks!