zedtools commented 3 months ago

Abstract

I recently switched from syncing hidden files to using customisation sync.

When customisation sync updates a file, it seems to update the timestamp of the file to the time the local file was modified. This is different to the way LiveSync updates .md files in the vault.

On top of that, customisation sync seems to both notify and update files when the timestamp of a file has changed. This leads to excessive updates of config files.

The only way around this is to "Select All Shiny" and manually apply individual changes where the contents have changed. This is something which should really be automatic, as it is a tedious process to do this each time.

Expected behaviour

If a configuration file's contents has not changed, do not notify if another device has a more recently modified file.
Alternatively, when updating a configuration file, set the file's timestamp to the time it was modified on the other device.

Actually happened

If Device 1 make a change to a configuration file, Device 2 will notify a file has changed and allow the changed to be applied.
After Device 2 applies the change, it saves a file with a newer timestamp.
Device 1 detects the same configuration file with a newer timestamp, and notifies the configuration has changed.
Clicking "Select All Shiny" followed by "Apply All Selected" will update the timestamps to a later time again.

With two devices, this is kind of manageable, but I sync four devices (desktop, laptop, iPhone, iPad), so this issue is magnified because one small configuration changes requires me to go back and forth between the four devices multiple times until the changes have been applied.

Reproducing procedure

I have the following options enabled under customisation sync:
- Per-file-saved customisation sync
- Enable customisation sync
- Notify customised
Open the dialog and set all files to "Selective".
On Device 1, make a change to a plugin's configuration (e.g. change a setting in the Dataview plugin), and wait for it to sync.
Open Obsidian on Device 2, and wait for the notification "Some configuration has been arrived...".
Click "Select All Shiny". The changed file comes up, so apply the changes by clicking "Apply All Selected".
Go back to Device 1, and wait for the notification "Some configuration has been arrived...".
Click "Select All Shiny". The changed file comes up again, so click "Apply All Selected".
Go back to step 4 and repeat.

Report materials

Report from the LiveSync

Report from hatch

``` ---- Obsidian info ---- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.4.16 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36 ---- remote config ---- cors: credentials: "true" headers: accept, authorization, content-type, origin, referer max_age: "3600" methods: GET, PUT, POST, HEAD, DELETE origins: app://obsidian.md,capacitor://localhost,http://localhost chttpd: bind_address: any max_http_request_size: "4294967296" port: "5984" require_valid_user: "true" admins: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 vendor: name: The Apache Software Foundation feature_flags: partitioned||*: "true" chttpd_auth: authentication_redirect: /𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷/_utils/session.html hash_algorithms: sha256, sha require_valid_user: "true" indexers: couch_mrview: "true" prometheus: additional_port: "false" bind_address: 127.0.0.1 port: "17986" httpd: WWW-Authenticate: Basic realm="couchdb" bind_address: 127.0.0.1 enable_cors: "true" port: "5986" smoosh: state_dir: ./data couch_httpd_auth: authentication_db: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 secret: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 authentication_redirect: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 couchdb_engines: couch: couch_bt_engine couchdb: database_dir: ./data max_document_size: "50000000" single_node: "true" uuid: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 view_index_dir: ./data ---- Plug-in config --- version:0.23.18 remoteType: "" useCustomRequestHandler: false couchDB_URI: self-hosted(HTTPS) couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 liveSync: true syncOnSave: false syncOnStart: false savingDelay: 200 lessInformationInLog: false gcDelay: 0 versionUpFlash: "" minimumChunkSize: 20 longLineThreshold: 250 showVerboseLog: false suspendFileWatching: false trashInsteadDelete: true periodicReplication: false periodicReplicationInterval: 60 syncOnFileOpen: false encrypt: false passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 usePathObfuscation: false doNotDeleteFolder: false resolveConflictsByNewerFile: false batchSave: false batchSaveMinimumDelay: 5 batchSaveMaximumDelay: 60 deviceAndVaultName: macbook usePluginSettings: false showOwnPlugins: false showStatusOnEditor: true showStatusOnStatusbar: true showOnlyIconsOnEditor: false usePluginSync: true autoSweepPlugins: false autoSweepPluginsPeriodic: false notifyPluginOrSettingUpdated: true checkIntegrityOnSave: false batch_size: 25 batches_limit: 25 useHistory: true disableRequestURI: true skipOlderFilesOnSync: true checkConflictOnlyOnOpen: false showMergeDialogOnlyOnActive: false syncInternalFiles: false syncInternalFilesBeforeReplication: false syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/, \/workspace(-mobile)?.json syncInternalFilesInterval: 60 additionalSuffixOfDatabaseName: 32e6163397f8cc82 ignoreVersionCheck: false lastReadUpdates: 23 deleteMetadataOfDeletedFiles: false syncIgnoreRegEx: "" syncOnlyRegEx: "" customChunkSize: 50 readChunksOnline: false watchInternalFileChanges: true automaticallyDeleteMetadataOfDeletedFiles: 0 disableMarkdownAutoMerge: false writeDocumentsIfConflicted: false useDynamicIterationCount: false syncAfterMerge: false configPassphraseStore: "" encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 permitEmptyPassphrase: false useIndexedDBAdapter: true useTimeouts: false writeLogToTheFile: false doNotPaceReplication: false hashCacheMaxCount: 300 hashCacheMaxAmount: 50 concurrencyOfReadChunksOnline: 30 minimumIntervalOfReadChunksOnline: 25 hashAlg: xxhash64 suspendParseReplicationResult: false doNotSuspendOnFetching: false useIgnoreFiles: false ignoreFiles: .gitignore syncOnEditorSave: false pluginSyncExtendedSetting: {} syncMaxSizeInMB: 50 settingSyncFile: "" writeCredentialsForSettingSync: false notifyAllSettingSyncFile: false isConfigured: true settingVersion: 0 enableCompression: false accessKey: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 bucket: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷(0 letters) endpoint: Not configured or AWS region: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷(4 letters) secretKey: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 useEden: false maxChunksInEden: 10 maxTotalLengthInEden: 1024 maxAgeInEden: 10 disableCheckingConfigMismatch: false displayLanguage: "" enableChunkSplitterV2: false disableWorkerForGeneratingChunks: false processSmallFilesInUIThread: false notifyThresholdOfRemoteStorageSize: 800 usePluginSyncV2: true usePluginEtc: false configPassphrase: "" preset: "" syncMode: LIVESYNC dummy: 0 ```

Plug-in log

``` 28/07/2024, 13:32:50->Initialized, NOW TRACKING! 28/07/2024, 13:32:50->Scanning customizations... 28/07/2024, 13:32:50->Scanning customizing files. 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/app.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/appearance.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/bookmarks.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/community-plugins.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/core-plugins-migration.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/core-plugins.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/global-search.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/graph.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/hotkeys.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/types.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/workspace-mobile.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/workspace.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/dataview.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/homepage.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/metadata-menu.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-linter.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-livesync.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-mind-map.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-minimal-settings.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-outliner.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-plugin-update-tracker.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-tasks-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-zoom.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/omnisearch.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/quickadd.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/templater-obsidian.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/todoist-sync-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/actions-uri.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/buttons.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/dataview.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/homepage.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/metadata-menu.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-importer.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-linter.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-livesync.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-mind-map.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-outliner.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-plugin-update-tracker.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-style-settings.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-tasks-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-zoom.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/omnisearch.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/quickadd.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/templater-obsidian.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/todoist-sync-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/S - Checkboxes.css.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/metadata-menu.css.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/outliner.css.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/THEME/ITS Theme.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/THEME/Minimal.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->Chunks saved: doc: macbook/CONFIG/workspace.json%workspace.json ,chunks: 1 (new:0, recycled:1, cached:0) 28/07/2024, 13:32:51->STORAGE --> DB:ix:macbook/CONFIG/workspace.json%workspace.json: (config) Done 28/07/2024, 13:32:51->Scanning customizations : done 28/07/2024, 13:32:51->Cache initialized 300 / 50000000 28/07/2024, 13:32:51->Modifying callback of the save command 28/07/2024, 13:32:51->Additional safety scan.. 28/07/2024, 13:32:51->Checking storage sizes 28/07/2024, 13:32:51->Before LiveSync, start OneShot once... 28/07/2024, 13:32:51->OneShot Sync begin... (pullOnly) 28/07/2024, 13:32:51->Remote storage size: 198.37MB 28/07/2024, 13:32:52->Replication paused 28/07/2024, 13:32:52->Replication completed 28/07/2024, 13:32:52->LiveSync begin... 28/07/2024, 13:32:53->Replication activated 28/07/2024, 13:32:53->There are no conflicted files 28/07/2024, 13:32:53->Additional safety scan done 28/07/2024, 13:32:54->Replication paused 28/07/2024, 13:32:55->Replication closed 28/07/2024, 13:33:00->Before LiveSync, start OneShot once... 28/07/2024, 13:33:00->OneShot Sync begin... (pullOnly) 28/07/2024, 13:33:00->Replication paused 28/07/2024, 13:33:00->Replication completed 28/07/2024, 13:33:00->LiveSync begin... 28/07/2024, 13:33:00->Replication paused 28/07/2024, 13:33:00->Replication paused 28/07/2024, 13:33:02->STORAGE --> DB:ix:macbook/PLUGIN_DATA/dataview%data.json: (config) Done 28/07/2024, 13:33:02->Replication activated 28/07/2024, 13:33:02->Replication paused 28/07/2024, 13:33:06->Replication closed 28/07/2024, 13:33:14->Before LiveSync, start OneShot once... 28/07/2024, 13:33:14->OneShot Sync begin... (pullOnly) 28/07/2024, 13:33:15->Replication activated 28/07/2024, 13:33:15->Replication paused 28/07/2024, 13:33:15->Replication completed 28/07/2024, 13:33:15->LiveSync begin... 28/07/2024, 13:33:15->Replication activated 28/07/2024, 13:33:15->Replication paused 28/07/2024, 13:33:15->Replication paused ```

Screenshots

Here is an example showing files where the configuration has not changed, but the timestamps are different:

Other information, insights and intuition.

Launching Obsidian also seems to update the last modified timestamps for several files, such as app.json, appearance.json, etc.

This adds to the excessive update notifications on other devices. Since these timestamp changes are beyond the control of this plugin, perhaps ignoring timestamp changes and just looking at the file contents is the most robust solution.

vrtmrz commented 3 months ago

Thank you for your very detailed report! This report quite helps me!

As you mentioned, I missed the following two logics for skipping non-changed files:

Checking the actual content (As you mentioned), And keeping the pair of mtime for two changes.
If the storage file and stored one have different mtimes, check they are already paired.

And one more thing, the differences in time are also shown correctly.

This has been fixed at v0.23.20. Would you mind if I ask you to check the fixes, please?

zedtools commented 3 months ago

Thanks for the quick update.

Yes, the mtime looked a bit odd, but I did not pay much attention to that. Now the files are showing a much more reasonable change in mtime such as several days.

As for the syncing of updates, I played around with it a little, and it looks like after updating the plugin I also had to click Select All Shiny and then Apply All Selected at least once, after which I no longer get the excessive notifications.

It looks like the behaviour now is:

Select All Shiny shows all files where the mtime or contents have changed.
Apply All Selected only applies changes where the contents have changed.
After applying changes, Select All Shiny still shows files with a more recent from another device mtime.

zedtools commented 3 months ago

On a side note, from a usability perspective it would help a lot to only display files whose contents have changed. I have 21 plugins, and it takes a while to scroll down to see what files may or may not have changed.

The ideal behaviour for me would be:

When opening the dialog, by default show only files whose contents have changed.
If I want to see more details, I can click a button or a checkbox to show all files, including mtime changes.

This way, if I get a notification that new configuration is available, I can open the dialog and quickly see what the relevant changes were. The more plugins installed, the more useful this behaviour would be.

Is this something you would consider?

zedtools commented 3 months ago

I also have a question about the four options for each file:

Selective
Automatic
Ignore
Flagged Selective

The Automatic option does not seem to do anything. If I enable that for a file, I sometimes get a notification that a hidden file was updated, but the file contents are not changed. Any settings I changed do not propagate to a device with Automatic selected.

If I change it back to Selective, I can manually update it fine.

How is Automatic supposed to work?

zedtools commented 3 months ago

Something is still not correct with the differences in mtime.

I just updated the Tasks plugin on my Mac and iPhone, but not yet on my iPad. Customisation sync is showing the iPad as having a newer version, which is incorrect.

I used Textastic to check the file modification times and grabbed the following screenshots. Note that all dates are dd/mm/yyyy.

Screenshots

These screenshots show the contents of .obsidian/plugins/obsidian-tasks-plugin on each platform.

File modified time

Mac

iPhone

iPad

Note that these are almost one month older:

Customisation sync

Why are my Mac and iPhone showing that my iPad has a newer version of MAIN?

Mac

iPhone

0zd0 commented 1 month ago

maybe I configured something wrong (half of the settings are described unclearly), but when opening obsidian makes its settings seem new, but in fact the new settings from the database do not have time to arrive and the plugin considers them obsolete (as I understood from the logs). as a result, new changes are rolled back when opening another device, as happens with iconize

vrtmrz commented 1 month ago

Sorry for being late! @zedtools

How is Automatic supposed to work?

This means Leave it to the hidden file sync. We have to enable Hidden file sync. But it is certainly more complicated since it was first created. Indeed I think you should put one over the other, but there are pros and cons to both, and turning this to automatic when both are enabled can be very problematic.

And, some mtime problems have been fixed in recent versions. I am sorry to trouble you, but could you check if the same problem occurs again?

P.S. I found that handling mtime is necessary in some cases. So, the item can be selected even if there is only a difference in mtime.

zedtools commented 1 month ago

And, some mtime problems have been fixed in recent versions. I am sorry to trouble you, but could you check if the same problem occurs again?

I think I saw it again about a week ago, specifically:

I updated a plugin on Windows desktop.
On Windows, customisation sync's Select All Shiny button selected another device (I think it was iPad).
Any other device I updated did the same.
Once I updated all devices to the latest plugin version, Select All Shiny no longer showed changes.

I can certainly keep an eye out for it and capture some more screenshots and data from the CouchDB database when it happens again.

zedtools commented 1 month ago

I just had this happen again.

Since it seems to be a separate issue from this one, I have logged a separate issue for under #506.

vrtmrz / obsidian-livesync

Customisation Sync timestamp changes lead to excessive updates #475

Abstract

Expected behaviour

Actually happened

Reproducing procedure

Report materials

Report from the LiveSync

Plug-in log

Screenshots

Other information, insights and intuition.

Screenshots

File modified time

Mac

iPhone

iPad

Customisation sync

Mac

iPhone