"Publish to..." wants to delete items that shouldn't be deleted

[Nuklon]

Describe the bug I'm having trouble synchronizing two instances from time to time. Today, it's happening again.

It basically wants to delete half the nodes from the destination server (pull from server to local instance has the same problem). I normally just pull twice so the nodes get first deleted and then recreated, but now I have tried to copy everything to destination server manually, including the database. It's still detecting the same nodes to be deleted.

It doesn't happen all the time, but quite frequently. What can cause this? I've compared some of the nodes it wants to delete (to trash) and the Umbraco id's match, and so does the history.

I'm using uSync v8.9.2 and Umbraco 8.12.0 (the previous versions have the same problem)

[Nuklon]

I just found out that it only seems to happen from a non-root node. If I push a root node (all settings default so descendants are also checked), it's working and detecting no changes. Any descendant of a root node has this problem (including descendants of the descendants, etc).

[KevinJump]

Hi

based on my notes below: can you add (or update) the following setting in the uSync8.config file on the server (it is needed on the side that is running the actual report)

 <CacheFolderKeys>False</CacheFolderKeys>

This will disable internal key caching - which has the potential to be the issue. so working through it would be good to check.

---

Working out loud (as much for my own notes as anything)

• When the Delete missing items check box is checked, that causes the export to create name_clean.config files.
• these name_clean.config files are what trigger a usync process to look at deleing items.
• they contain the guid of the item to which you want childrend deleted. and an action 'clean'

(this code is open source as its part of the uSync8 repo)

When uSync encounters a clean file - it hits the CleanFolder() method which:

• reads all the xml files in the folder and gets the key values from them
• gets all the children from the parent item out of umbraco
• returns a list of values that are in the umbraco list but not the list from the disk.

Possible issues (again thinking outloud here) :

• We cache the list of items in the folder based on the folder name . GetFolderKeys(folder)
  • The cache is based on the hash of the folder path, and its cleared at the start and end of a report/import - but maybe its not caching properly or being cleared properly ?
  • the cache is the whole folder name,. and that changes for each pull - so its unlike to be this.
  • if there are no files listed we don't do this (we report that the folder is empty)

The caching of the keys can be turned off. there is a CacheFolderKeys setting you can put in the uSync8.config file that will stop it from using the cache to store the folder keys, if its a caching issue then turning off the cacheFolderKeys should fix it, because everything will run against the folder each time.

• We then get the item of the item which is a clean (this is called GetCleanParent - but actually we are getting the item from the clean file, because that is the parent of the folder we want to clean).
• once we have the parent we get all its children from the Umbraco services.
• DeleteMissingItems then takes the parent content node and the keys.
  • it then loops through the items where their key is not in the list of keys we have from the folder.
  • these are then deleted or reported as deleted depending on the mode.

[Nuklon]

Thanks for the information Kevin. I have added <CacheFolderKeys>False</CacheFolderKeys> in Backoffice section in uSync.config (I assume it must be added there) both to server and locally, but it still wants to delete ~20 nodes. What I find really odd is that it does work from a root node, but not from any descendant. I know there's a difference in the number of related nodes depending on the node, but the root node should always have more.

[KevinJump]

yeah 😕 and from a technical point of view it shouldn't actually matter because internally uSync publisher puts all the files in a flat structure, so they are loaded for all nodes .

in order to say - there is stuff to delete it must not be getting the files from disk that match those of the nodes :(

I am trying to think of a good way to debug this, it might involve some copy of folders at the right point in the process - if i get a process down i will hopefully be able to give you a few steps you could go through so we could get some sample info.

[KevinJump]

Hi,

Could you do the following:

assuming you are pushing from source to target

1. Start the push process from your site (source)
2. When the site reports back - do not close any of the dalogs.
3. on the server (target) there should be a folder in App_Data/Temp/uSync/Receive - with a new timestamp.
4. zip that folder up.

on the server (target) back office : 1 . go to usync dashboard, and the exporter tab 2 create a sync pack with the content node you are pushing to from the other server.

3. include children, dependencies and ancestors.

4. download that sync pack.

if you could then send these two files to us kevin@jumoo.co.uk - we can take a look see if anything looks odd.

(what this process does is first capture the actual files sent between the two sites, and then also capture what the server thinks it has installed in the sync pack).

[Nuklon]

There's some sensitive data in it (intranet), so I'd rather not send pack files. Anything else I can do to give this a try?

On home page this is also shown (is this checked differently?):

("Live" is my destination server).

I did do your steps and compared them myself, quite a number of files are only in the receive folder and not in the pack. In the Content folder, most of these are ..._clean.config. Others are also normal guid.config files. They do seem to be read correctly (also <Trashed>false</Trashed>).

What I have also tried: create a sync pack from the same node locally and compared that with the "Live" environment. Those packs are completely identical (all files binary compared). I have also tried to import the sync pack on the destination server and it's detecting no changes. So it looks like the pack sent through "Publish to..." is created differently than the sync pack?

[KevinJump]

: ( the sync packs and publish packs use the same code the only diffrence (and it is what is causing the issue) are the _clean files. these are the files that kickoff the delete checking. so as above these get a list of children and see if they are in the folder.

I think we might have to look at adding some more logging of something to a version of uSync and seeing what we can fathom from that, because for some reason the list of nodes and the list of files are not matching on that check.

[Nuklon]

I think we might have to look at adding some more logging of something to a version of uSync and seeing what we can fathom from that, because for some reason the list of nodes and the list of files are not matching on that check.

Sounds good to me, I can give that a try.

Isn't there some difference then with how these _clean files are generated compared to how the comparison is done? As importing a sync pack isn't detecting all these nodes to be deleted.

[KevinJump]

yep, just adding some now.

just to confirm a couple of things: can you check if the 'Use flat' setting is set on either the source or the target server. Just checking here, i can't reproduce with this setting, but wondering if a combination causes it.

or in the usync8.config file

<FlatFolders>True</FlatFolders>

Kevin

[KevinJump]

does look like it is likely the flat files setting - if you can set this to true. and it will fix it (until we patch).

[KevinJump]

Will be fixed by : https://github.com/KevinJump/uSync8/issues/211 but also #99 will mitigate this in uSync.Publisher.

[Nuklon]

I have (and had) Flat structure enabled on local machine and also on destination machine (they're copies).

[KevinJump]

Can you update uSync/uSync.ContentEdition to v8.8.5 ? this might fix it, as it has the updates to the clean stuff?

there is also an update in the next publisher, but we are just finalizing that. - would be good to know if the 8.8.5 stuff has an effect

[Nuklon]

I have updated uSync (on both ends) but it still wants to delete 22 out of 45 items.

I'll try out the publisher version once it's out.

[KevinJump]

Hi,

can you try the new publisher version (just released) - it forces some of the settings for publishing (so ignores any config) for example we don't care about file names in publisher, and this might also help.

[Nuklon]

Thanks, I'll give it a try, hopefully tomorrow, otherwise next week.

[Nuklon]

I have updated to the new uSync version and unfortunately, it still wants to delete the items.

[Nuklon]

Unchecking Advanced options > Include linked pages also works (besides unchecking Delete missing items) I just discovered. Which is odd as the node in question has zero linked pages. Or does it also apply to ancestors?

[KevinJump]

Ahh, that might be it, linked pages is quite all encompasing - mostly i don't think you would really want that set , but it makes sense now.

I think we will need to patch that so that linked pages don't try to do the delete missing items thing.

[Nuklon]

Alright, I can give that a try then. Although I don't really understand how it's possible that a root node with linked pages enabled works correctly, but a child node not (especially because there are no linked pages, it's just a single document type not used elsewhere).

[KevinJump]

probably (and a bit of a guess) when you sync the root, all pages are included (because they are children of the root) so linked pages doesn't actually add anything.

when you sync a child node, linked pages (its all the pages that are linked to in pickers, or directly in content) are found across the site and included too, some of these pages will be in folders that you are not syncing,

the linked page is also carrying the 'delete missing' flag, so it lays a file telling uSync to check, but because only the linked pages are included it thinks you should delete all the pages that are not linked in the other folder.

so we need to remove the 'delete missing' flag when we include linked pages.

[Nuklon]

Makes sense, thanks :)

[KevinJump]

Fix to not removed linked items siblings is in latest version (but i don't think you need to turn linked back on, its probably best to keep it off in most cases).