search

SitecoreUnicorn / Unicorn

A Sitecore utility designed to simplify deployment of Sitecore items across environments automatically
MIT License
269 stars 117 forks source link

Not syncing all fields on first run #283

Closed wene37 closed 5 years ago

wene37 commented 6 years ago

Do you want to request a feature or report a bug? Bug

What is the current behavior? If I run a sync and Unicorn adds a new item from the file system to the Sitecore database, this item is created but not all fields are with correct values (e.g. sortorder, icon). If I run Unicorn sync again, the fields are updated to the correct value. So it always takes two runs to have an item correctly synced from file system to the database. The problem as well is, if you use DeployOnce configuration, the items are created wrong on first run and on second run they are not updated, because they are already existing.

Following the log (I tried with and without Dilithium. But it doesn't change anything.):

First run

Project.Portal is being synced with Dilithium SQL + Serialized enabled.
[A] master:/sitecore/content/Portal/Inventory/Website/search (c4ba007b-b100-419d-8069-3afcdf823b43)
[A] master:/sitecore/content/Portal/Inventory/Website/search/Content Elements (c853c728-a79b-4884-af62-534f2776567a)
[A] master:/sitecore/content/Portal/Inventory/Website/search/Content Elements/Container (d5ac0f1e-fe71-4511-a1bc-18cf514d13c5)
[A] master:/sitecore/content/Portal/Inventory/Website/search/Content Elements/Hero (ba362854-c1dc-4009-9b07-be40f55ac71b)
Project.Portal sync complete: 831 items evaluated, 4 items modified (4 added, 0 updated, 0 recycled) in 1306ms (~1.57ms/item).

Second run

Project.Portal is being synced with Dilithium SQL + Serialized enabled.
[U] master:/sitecore/content/Portal/Inventory/Website/search (c4ba007b-b100-419d-8069-3afcdf823b43)
> Field Metadata Robots - Serialized '{881E7AB0-5ECB-472C-924C-5EF5954C0160}', Source '{9267ECE8-053E-4AE9-8DCF-CC6A0C307B8A}'
* [U] __Icon
* [U] Hide from menu
* [U] Metadata Robots
* [U] __Sortorder
* [U] __Renderings
Project.Portal sync complete: 828 items evaluated, 1 item modified (0 added, 1 updated, 0 recycled) in 6158ms (~7.44ms/item).

If the current behavior is a bug, please provide the steps to reproduce. See above

What is the expected behavior? An item should be created correctly on first sync.

Please mention your Sitecore version and Unicorn version. Sitecore: Sitecore.NET 9.0.1 (rev. 171219) Unicorn: 4.0.3

cassidydotdk commented 6 years ago

Does this happen exclusively with DeployOnce configuration? Because I just went through the steps here, and could not reproduce the behaviour.

image

cassidydotdk commented 6 years ago

Just tested with NewItemsOnlyEvaluator. Same result.

image

cassidydotdk commented 6 years ago

Can you share your config, please?

wene37 commented 6 years ago

No, it's happening in all situations. The only difference is, that with "normal" sync after a second sync everything is ok. But with NewItemsOnly there won't be any update, as the items are already existing

Here I send you my configs: serialization.zip

cassidydotdk commented 6 years ago

I am still unable to narrow this down. A couple of questions:

1) In the 2 logfiles above - there is nothing going on anywhere between those two calls? 2) Is is exclusively happening when you use the NewItemsOnlyEvaluator?

My best theory at this time would be, that some templates got updated in the first run after initial serialization happened, but there is nothing to indicate this in the logs you've shared.

wene37 commented 6 years ago

No, there's nothing going on between the two sync calls. And no, it's not only happening on NewItemsOnlyEvaulator. The example above is with the normal sync evaluator. But with the normal one, I just can run sync again and then everything is fine. But with NewItemsOnlyEvaluator this doesn't work, as the item then already exists. I will try to invest some more time in the next weeks. Maybe I can figure out more.

JaspervdM80 commented 6 years ago

Hi, we are experiencing the same issue as described above. Could it be a racing condition? In the initial sync the branch values are filled in the (some) newly created items. In the second run the serialized values are used.

stewpocalypse commented 5 years ago

I'm coming across the same issue. I can replicate it consistently.

  1. Using SXA create a new Tenant
  2. Serialize the created Tenant items using Unicorn 4.0.4 (tested against 4.0.0 as well) with the below configuration: `

    `

  3. Store serialized files somewhere else as a backup
  4. Remove /sitecore/Content/Tenant item using default Sitecore Delete (not the Tenant removal script)
  5. As the removal of the item will cause the yml to be removed, re add the backed up files to serialized location
  6. Do a sync to add the Tenant back into Sitecore
  7. Examine the Tenant and notice that various settings are empty tenant missing data
  8. Do another sync
  9. Examine the Tenant and notice that the settings are back tenant data returned

I've tried to simplify my problem down to something easily replicated but the problem is more wide spread. An easy way that I've seen to identify the problem is with no items on a fresh Sitecore installation do a sync then right after do a reseralization. Some files that were just synced now show up as changes have occurred.

Tenant.zip This is what the above config serializes for me. I'm really at a loss for what the issue might be.

stewpocalypse commented 5 years ago

Using Powershell in Sitecore I was able to pass my Tenant.yml contents into Import-RainbowItem and the item appeared as expected, with all the fields populated.

`$yaml = @'

ID: "0bdb6e11-af3a-453d-8d29-f89d835b36de" Parent: "0de95ae4-41ab-4d01-9eb0-67441b7c2450" Template: "7f99e7af-c67a-4dd4-8ba7-977c6b33c4b0" Path: /sitecore/content/Tenant DB: master BranchID: "9a4aad67-383b-4504-93b9-d3502be2b59d" SharedFields:

$yaml | Import-RainbowItem`

stewpocalypse commented 5 years ago

I continued some more digging into the code and am finding out that in Rainbow's DefaultDeserializer.Deserialize() that PasteSharedFields() and PasteVersions() cause the problem of missing fields on first sync. Watching the database while running the method I see that the Shared Fields are present after PasterSharedFields() is ran but after PasteVersions() runs the Shared Fields Values are removed from the database. The weird part is not all of the Shared Fields Values are removed, just some of them. I've also tried swapping PasteSharedFields() and PasteVersions() to see if the order makes a difference and in that case PasteSharedFields() will wipe the versions from the database.

stewpocalypse commented 5 years ago

Found more details pertaining to this issue.

When an item is added, if the item is based on a Branch and the item has Shared Fields, then those Shared Fields which differ from the Branch are lost.

I'm not sure what the solution or fix would be, but from everything I'm seeing at this point this is a very specific problem. A manual fix is either to do a double sync (hard to do with NewItemOnlyEvaluator) or remove the BranchID setting from the yml file.

caviyacht commented 5 years ago

Building on what stewpocalypse is talking about. The main issue is being caused by the removal of the __Originator field from the field filters setting (ref: https://github.com/SitecoreUnicorn/Unicorn/commit/f7d073b255891ec487e9a092fb2fa172ea77b9ac). This appears to be removed due to an issue with transparent sync, which we are not using.

This was not causing problems in a previous solution we had as it was not relying on shared fields, but in the new SXA world, it is heavily using branches with shared fields.

Adding the __Originator back to the field filters has completely solved the problem when syncing.

A fix for this could be that when using transparent sync that the __Originator field is removed from the field filters, but is there by default like it used to be.

cassidydotdk commented 5 years ago

We've found the root cause of this, and currently have an open ticket with Sitecore support.

Not to deflect; but the problem doesn't lie with Unicorn/Rainbow - the above scenarios can be replicated using standard Serializer/Deserializer tech. We have a number of issues all rooted in the same bug; will try and collect them under one ticket.

caviyacht commented 5 years ago

@cassidydotdk , I don't know if it was mentioned or not (can't seem to find it) but we ran the same yaml using the pure Rainbow PowerShell command in Sitecore and it worked like a charm.

cassidydotdk commented 5 years ago

The bug is definitely related to Branches. This situation happens on Shared fields on items created from a Branch template. It can be reproduced on vanilla instances of 8.2, 9.0 and 9.1 by following these steps:

1) Create a template; "Test Template". 2) On that template, create 1 field "Shared Field" and make it "Shared" 3) Create a Branch Template based on "Test Template". 4) On the Branch Template Item ($name) - set the field value of "Shared Field" to $name 5) Create a Common Folder under /sitecore/content named "Test Folder" 6) Create an item under "Test Folder" based on the created Branch Template. Name it "Test Item" 7) At this point, you should have /sitecore/content/test folder/test item - and the value of "Shared Field" would be set to "Test Item" 8) Now change the value of "Shared Field" on "Test Item" to "A new value". Save that. 9) Go to "Test Folder" and select "Serialize Tree" from the Developer Ribbon 10) Verify that "Test Item" was serialized correctly in your serialization folder 11) Now delete "Test Item" in the Content Editor 12) Navigate to "Test Folder" 13) Select "Revert Tree" from the Developer ribbon 14) Check the value of "Shared Field" on the now restored item. It will be "Test Item" and NOT the expected "A new value".

cassidydotdk commented 5 years ago

By commenting out the code in Rainbow that sets the correct BranchId (created from in the Content Editor interface) on newly created items, everything works as it should. But if BranchId is set, we get this mess.

And this is whether it's from Rainbow, native API, Sitecore Packager.

caviyacht commented 5 years ago

@cassidydotdk , I see you updated it to ignore the BranchId logic. Just curious, what type of problems will this cause if that is ignored during deserialization? Will this cause issues on other developers machines when attempting to keep them in sync?

Just an FYI, I know I mentioned it already, but we have been running fine for a month now with our fix of ignoring the mentioned __Originator field. Any reason that ignoring this field during a non-transparent sync (seems it was not excluded by default because of transparent sync) was not considered?

Remember I mentioned that using the $yaml | Import-RainbowItem command worked. Looking around it appears that in the YamlCommandBase the default fieldFilter is still excluding the __Originator field YamlCommandBase line 17 but the main Unicorn config is not: f7d073b. This would explain why it worked for me to use the PowerShell command vs Unicorn directly.

Could it be that this __Originator field could be causing the issue as well vs the ChangeBranch logic?

cassidydotdk commented 5 years ago

__Originator influenced how Transparent Sync worked/works. But it's been working fine even without that field excluded for a while now.

BranchId is not a field in the traditional sense. It's a property on the base Item. And even without ignoring __Originator I have been able to reproduce the problems outlined above. Only avoiding touching the BranchId under Item Creation have I been able to produce a reliable and expected result. Sitecore Support will eventually come back with some commentary as well I imagine.

This will have no influence and cause no problems that I am aware of, for developers or otherwise. The Content Editor interface might stop showing "Created From" - but Kam and I and people I've spoken to have not been able to identify any detrimental effect to this. Since it is not a field, it will not even trigger an item to be updated - BranchId is not a part of the field comparison logic.

It might work under PowerShell because it these CommandLets don't run under the EventDisabler. Nevertheless, I have been able to reproduce and deproduce the issue with the change here.

The change is optional. Feel free to toggle the default config setting if it suits you better.

JaspervdM80 commented 5 years ago

We've found the root cause of this, and currently have an open ticket with Sitecore support.

Not to deflect; but the problem doesn't lie with Unicorn/Rainbow - the above scenarios can be replicated using standard Serializer/Deserializer tech. We have a number of issues all rooted in the same bug; will try and collect them under one ticket.

@cassidydotdk, thanks for finding the root cause. Could you share the Sitecore support ticket number(s), so we can get an official patch as soon as possible?

cassidydotdk commented 5 years ago

Ticket id is 521933.

For a lot of complicated yet probably non-sensical reasons, the ticket may not move through their internal systems unless it is escalated by multiple sources. I suggest you ask your local and regional sales representatives for an escalation of the above.

For now, the released workaround works. As long as you're using Unicorn, and not built in tools.

cassidydotdk commented 5 years ago

There is now a public reference number for this bug. 215244.

Sitecore Support has no hotfix for 8.x, but claim to have hotfixes available for 9.01 and above.

SaintSkeeta commented 5 years ago

This is something that I reported to Sitecore back in May 2018. 215244 was one part of the fix for serialization, but only partially fixed the serialization issue. If you can also quote the public reference number 13657 as well in your requests, hopefully you will also get the complete fix for this issue, and it can be fixed in the full platform soon.

cassidydotdk commented 5 years ago

@SaintSkeeta They've known about the issue internally since 2015 so I'll probably opt to not hold my breath ;-)

Anyway - for all practical purposes I can think of, ignoring this BranchId property on an item should have no side effects. I do have a theory on how to get completely around this problem but it involves a lot more work than I feel it's worth right now - there are other things to focus on.