Note Duplication

[kazdonkai]

Hi, Thanx for the useful plugin. I often get duplication of the same file. According to the conditions, the notes are moved to another folder, but soon after, they are restored to the original folder. So after that, the destination folder is commented that the file has already exists in the destination folder. I am syncing with iCloud, and I imagine that this is why Obsidian is resurrecting the files in the sync destination. Or it could be that I have the same rules on my Mac and iPad. But I don't know for sure. Thank you in advance for your consideration.

[farux]

What if the vault is temporarily excluded from syncing? If the file is locked or real-time synced from outside Obsidian, unfortunately the plugin cannot solve the problem.

[kazdonkai]

I will try it out over time.

[kazdonkai]

I think the file doubling is probably due to the Auto Move plugin conflicting with the save timing when the file is renamed to move the file. So, would it be possible to set the delay time when the file is renamed in the settings? By changing the delay time, it may be possible to avoid conflicts in different environments.

[farux]

No, that is not possible. Because the api command does the renaming and moving all at once. Also, if ANM handles a large number of files (edits by other plugins, obsidian updates, vault location changes, etc.), it is undesirable to add a delay for each rename event. This problem seems to be caused not by ANM, but by iCloud's compatibility with Obsidian.

https://www.reddit.com/r/ObsidianMD/comments/rtz69e/using_icloud_for_sync_creates_duplicate_files_eg/

https://www.reddit.com/r/ObsidianMD/comments/ts0yp0/duplicate_notes_with_icloud/

https://www.reddit.com/r/ObsidianMD/comments/t7j7uv/obsidian_is_unusable_with_icloud_in_windows_but/

You may want to consider a different sync method.

[kazdonkai]

I think the duplication issue and the iCloud duplication issue are two different things when moving.

The iCloud issue is only an issue with the Windows version of the iCloud app and does not occur on Mac or iOS.

[farux]

To begin with, ANM is only moving files, not copying them. There is also a checking mechanism for files with the same name. If duplication occurs, it can only be due to external factors. I ask the same question, what happens if the vault is excluded from iCloud sync? Does the problem still occur?

[kazdonkai]

It is not easy to temporarily stop iCloud, so I ran this plugin with the network connection set to no network connection, and there were no errors.

This means that there is a relationship between iCloud and this plugin.

But then this would mean that all users of iCloud would not be able to use this plugin.

[farux]

ANM is just moving files in a very ordinary way. If it causes problems, it is an iCloud bug that affects other programs as well. I recommend reporting this to Apple.

[kazdonkai]

Is it really?

Moving files with Obsidian does not cause problems with iCloud syncing. (except on Windows).

So, the only possible cause is that the timing of saving the note when changes are made to the note and the timing of moving the note is too short for this plugin.

[farux]

There is no file system that does or does not duplicate a file depending on when the file is saved, nor should there be. It is the responsibility of the 'Sync' program to keep the files up-to-date. It should not copy files without permission.

[kazdonkai]

I'm afraid the issues don't quite mesh.

My understanding is that this plugin works as follows?

(1) The plugin detects that a file has been renamed or tagged and saved. (2) Then, it moves the file according to the conditions.

In this case, the file is moved, but the original file ends up remaining.

The reason why it would remain is because iCloud synchronizes and saves the original file to other Valts, so even if the original file is erased by this plugin, it would in turn synchronize to the original Valt from other Valts. This is what I mean.

So, if there is time between (1) and (2), there will be no conflict.

[farux]

The moved state should be synchronized. Why is the state of the file there or not there not synced? Other syncing programs such as Dropbox and OneDrive work that way. If a file is moved or deleted, its most recent operation should be synced to the other client, and should not restore the files without permission.

[kazdonkai]

It's time you understand.

The time between saving a file and moving it with this plugin is too short for iCloud sync to keep up.

I am not manipulating the files.

[farux]

Usually, if synchronization is not completed in time, a queue is created or a list of files is retrieved to keep it up-to-date. Even if a delay were added, it would not solve this problem. With the current iCloud specs, duplicate files, restoring old files, etc. can happen in other programs as well if the network is unstable.

[kazdonkai]

I have not experienced any duplication issues with Obsidian other than with this plugin, so anyway, I think the problem may be caused by the plugin, not iCloud.

How about adding a delay to try and see?

[farux]

It is not easy to temporarily stop iCloud, so I ran this plugin with the network connection set to no network connection, and there were no errors.

As you can see, iCloud is restoring a file that does not exist and is not ANM. There is no way to know if iCloud synchronization has finished. Even if we add a uniform delay of a few seconds, it does not guarantee that duplicates will not occur completely, since the speed of synchronization varies depending on file size, system busy state, or network conditions. This is a problem that should be addressed by the sync program in the first place.

[kazdonkai]

My hypothesis was that due to the lack of time difference between file changes and deletions in Valt A, iCloud would not be able to keep up. And iCloud wrote back the updated files stored in Valt B from Valt B to Valt A. To verify this, I experimented with the plugin's behavior manually instead of automatically.

(1) Save the file according to the rules. Then iCloud will sync A and B and save the changed file.

(2) Run the plugin manually. Then, as is usual with Obsidian, files should be moved and deleted in both Valts.

When I ran it, the results were as expected and no duplicate files were generated.

So this is a problem that should be solved on the plugin side.

[farux]

As a precondition, a program should not inadvertently know what other programs are running on the system. Doing so is a security risk. Who would expect a editor plugin to scan for programs on the system? The plugin cannot know that iCloud is being used. So it can't wait for the task of what it doesn't know to be done.

https://technology.siprep.org/icloud-sync-creating-duplicate-files-when-saving/ According to this article and the comments section, the iCloud file duplication issue has been ongoing since at least 2016 to the present. Search for iCloud and duplicates of files, photos, contacts, etc. and you will find many results. This is because iCloud does not have the features that other Sync programs have that are required of Sync programs. In other words, it's a bug in iCloud.

[kazdonkai]

This problem does not occur when running the plugin manually, so I will run it manually from now on.

As a known issue, I think it would be helpful for those who face the same problem if it is noted in the manual that if you use iCloud and run it automatically, you may encounter this issue and that there is a thread on this.

[farux]

Thanks for the suggestion. I have added it to the troubleshooting section.