search

vrtmrz / obsidian-livesync

MIT License
4.53k stars 146 forks source link

Customisation Sync timestamp changes lead to excessive updates #475

Open zedtools opened 1 month ago

zedtools commented 1 month ago

Abstract

I recently switched from syncing hidden files to using customisation sync.

When customisation sync updates a file, it seems to update the timestamp of the file to the time the local file was modified. This is different to the way LiveSync updates .md files in the vault.

On top of that, customisation sync seems to both notify and update files when the timestamp of a file has changed. This leads to excessive updates of config files.

The only way around this is to "Select All Shiny" and manually apply individual changes where the contents have changed. This is something which should really be automatic, as it is a tedious process to do this each time.

Expected behaviour

Actually happened

With two devices, this is kind of manageable, but I sync four devices (desktop, laptop, iPhone, iPad), so this issue is magnified because one small configuration changes requires me to go back and forth between the four devices multiple times until the changes have been applied.

Reproducing procedure

  1. I have the following options enabled under customisation sync:
    • Per-file-saved customisation sync
    • Enable customisation sync
    • Notify customised
  2. Open the dialog and set all files to "Selective".
  3. On Device 1, make a change to a plugin's configuration (e.g. change a setting in the Dataview plugin), and wait for it to sync.
  4. Open Obsidian on Device 2, and wait for the notification "Some configuration has been arrived...".
  5. Click "Select All Shiny". The changed file comes up, so apply the changes by clicking "Apply All Selected".
  6. Go back to Device 1, and wait for the notification "Some configuration has been arrived...".
  7. Click "Select All Shiny". The changed file comes up again, so click "Apply All Selected".
  8. Go back to step 4 and repeat.

Report materials

Report from the LiveSync

Report from hatch ``` ---- Obsidian info ---- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.4.16 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36 ---- remote config ---- cors: credentials: "true" headers: accept, authorization, content-type, origin, referer max_age: "3600" methods: GET, PUT, POST, HEAD, DELETE origins: app://obsidian.md,capacitor://localhost,http://localhost chttpd: bind_address: any max_http_request_size: "4294967296" port: "5984" require_valid_user: "true" admins: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 vendor: name: The Apache Software Foundation feature_flags: partitioned||*: "true" chttpd_auth: authentication_redirect: /𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷/_utils/session.html hash_algorithms: sha256, sha require_valid_user: "true" indexers: couch_mrview: "true" prometheus: additional_port: "false" bind_address: 127.0.0.1 port: "17986" httpd: WWW-Authenticate: Basic realm="couchdb" bind_address: 127.0.0.1 enable_cors: "true" port: "5986" smoosh: state_dir: ./data couch_httpd_auth: authentication_db: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 secret: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 authentication_redirect: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 couchdb_engines: couch: couch_bt_engine couchdb: database_dir: ./data max_document_size: "50000000" single_node: "true" uuid: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 view_index_dir: ./data ---- Plug-in config --- version:0.23.18 remoteType: "" useCustomRequestHandler: false couchDB_URI: self-hosted(HTTPS) couchDB_USER: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 liveSync: true syncOnSave: false syncOnStart: false savingDelay: 200 lessInformationInLog: false gcDelay: 0 versionUpFlash: "" minimumChunkSize: 20 longLineThreshold: 250 showVerboseLog: false suspendFileWatching: false trashInsteadDelete: true periodicReplication: false periodicReplicationInterval: 60 syncOnFileOpen: false encrypt: false passphrase: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 usePathObfuscation: false doNotDeleteFolder: false resolveConflictsByNewerFile: false batchSave: false batchSaveMinimumDelay: 5 batchSaveMaximumDelay: 60 deviceAndVaultName: macbook usePluginSettings: false showOwnPlugins: false showStatusOnEditor: true showStatusOnStatusbar: true showOnlyIconsOnEditor: false usePluginSync: true autoSweepPlugins: false autoSweepPluginsPeriodic: false notifyPluginOrSettingUpdated: true checkIntegrityOnSave: false batch_size: 25 batches_limit: 25 useHistory: true disableRequestURI: true skipOlderFilesOnSync: true checkConflictOnlyOnOpen: false showMergeDialogOnlyOnActive: false syncInternalFiles: false syncInternalFilesBeforeReplication: false syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/, \/workspace(-mobile)?.json syncInternalFilesInterval: 60 additionalSuffixOfDatabaseName: 32e6163397f8cc82 ignoreVersionCheck: false lastReadUpdates: 23 deleteMetadataOfDeletedFiles: false syncIgnoreRegEx: "" syncOnlyRegEx: "" customChunkSize: 50 readChunksOnline: false watchInternalFileChanges: true automaticallyDeleteMetadataOfDeletedFiles: 0 disableMarkdownAutoMerge: false writeDocumentsIfConflicted: false useDynamicIterationCount: false syncAfterMerge: false configPassphraseStore: "" encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 permitEmptyPassphrase: false useIndexedDBAdapter: true useTimeouts: false writeLogToTheFile: false doNotPaceReplication: false hashCacheMaxCount: 300 hashCacheMaxAmount: 50 concurrencyOfReadChunksOnline: 30 minimumIntervalOfReadChunksOnline: 25 hashAlg: xxhash64 suspendParseReplicationResult: false doNotSuspendOnFetching: false useIgnoreFiles: false ignoreFiles: .gitignore syncOnEditorSave: false pluginSyncExtendedSetting: {} syncMaxSizeInMB: 50 settingSyncFile: "" writeCredentialsForSettingSync: false notifyAllSettingSyncFile: false isConfigured: true settingVersion: 0 enableCompression: false accessKey: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 bucket: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷(0 letters) endpoint: Not configured or AWS region: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷(4 letters) secretKey: 𝑅𝐸𝐷𝐴𝐢𝑇𝐸𝐷 useEden: false maxChunksInEden: 10 maxTotalLengthInEden: 1024 maxAgeInEden: 10 disableCheckingConfigMismatch: false displayLanguage: "" enableChunkSplitterV2: false disableWorkerForGeneratingChunks: false processSmallFilesInUIThread: false notifyThresholdOfRemoteStorageSize: 800 usePluginSyncV2: true usePluginEtc: false configPassphrase: "" preset: "" syncMode: LIVESYNC dummy: 0 ```

Plug-in log

Plug-in log ``` 28/07/2024, 13:32:50->Initialized, NOW TRACKING! 28/07/2024, 13:32:50->Scanning customizations... 28/07/2024, 13:32:50->Scanning customizing files. 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/app.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/appearance.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/bookmarks.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/community-plugins.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/core-plugins-migration.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/core-plugins.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/global-search.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/graph.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/hotkeys.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/types.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/workspace-mobile.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/CONFIG/workspace.json.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/dataview.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/homepage.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/metadata-menu.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-linter.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-livesync.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-mind-map.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-minimal-settings.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:50->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-outliner.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-plugin-update-tracker.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-tasks-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/obsidian-zoom.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/omnisearch.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/quickadd.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/templater-obsidian.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_DATA/todoist-sync-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/actions-uri.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/buttons.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/dataview.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/homepage.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/metadata-menu.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-importer.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-linter.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-livesync.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-mind-map.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-outliner.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-plugin-update-tracker.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-style-settings.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-tasks-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/obsidian-zoom.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/omnisearch.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/quickadd.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/templater-obsidian.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/PLUGIN_MAIN/todoist-sync-plugin.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/S - Checkboxes.css.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/metadata-menu.css.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/SNIPPET/outliner.css.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/THEME/ITS Theme.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->STORAGE -x> DB:ix:macbook/THEME/Minimal.md: (config) already deleted (Not found on database) 28/07/2024, 13:32:51->Chunks saved: doc: macbook/CONFIG/workspace.json%workspace.json ,chunks: 1 (new:0, recycled:1, cached:0) 28/07/2024, 13:32:51->STORAGE --> DB:ix:macbook/CONFIG/workspace.json%workspace.json: (config) Done 28/07/2024, 13:32:51->Scanning customizations : done 28/07/2024, 13:32:51->Cache initialized 300 / 50000000 28/07/2024, 13:32:51->Modifying callback of the save command 28/07/2024, 13:32:51->Additional safety scan.. 28/07/2024, 13:32:51->Checking storage sizes 28/07/2024, 13:32:51->Before LiveSync, start OneShot once... 28/07/2024, 13:32:51->OneShot Sync begin... (pullOnly) 28/07/2024, 13:32:51->Remote storage size: 198.37MB 28/07/2024, 13:32:52->Replication paused 28/07/2024, 13:32:52->Replication completed 28/07/2024, 13:32:52->LiveSync begin... 28/07/2024, 13:32:53->Replication activated 28/07/2024, 13:32:53->There are no conflicted files 28/07/2024, 13:32:53->Additional safety scan done 28/07/2024, 13:32:54->Replication paused 28/07/2024, 13:32:55->Replication closed 28/07/2024, 13:33:00->Before LiveSync, start OneShot once... 28/07/2024, 13:33:00->OneShot Sync begin... (pullOnly) 28/07/2024, 13:33:00->Replication paused 28/07/2024, 13:33:00->Replication completed 28/07/2024, 13:33:00->LiveSync begin... 28/07/2024, 13:33:00->Replication paused 28/07/2024, 13:33:00->Replication paused 28/07/2024, 13:33:02->STORAGE --> DB:ix:macbook/PLUGIN_DATA/dataview%data.json: (config) Done 28/07/2024, 13:33:02->Replication activated 28/07/2024, 13:33:02->Replication paused 28/07/2024, 13:33:06->Replication closed 28/07/2024, 13:33:14->Before LiveSync, start OneShot once... 28/07/2024, 13:33:14->OneShot Sync begin... (pullOnly) 28/07/2024, 13:33:15->Replication activated 28/07/2024, 13:33:15->Replication paused 28/07/2024, 13:33:15->Replication completed 28/07/2024, 13:33:15->LiveSync begin... 28/07/2024, 13:33:15->Replication activated 28/07/2024, 13:33:15->Replication paused 28/07/2024, 13:33:15->Replication paused ```

Screenshots

Here is an example showing files where the configuration has not changed, but the timestamps are different:

image

Other information, insights and intuition.

Launching Obsidian also seems to update the last modified timestamps for several files, such as app.json, appearance.json, etc.

This adds to the excessive update notifications on other devices. Since these timestamp changes are beyond the control of this plugin, perhaps ignoring timestamp changes and just looking at the file contents is the most robust solution.

vrtmrz commented 1 month ago

Thank you for your very detailed report! This report quite helps me!

As you mentioned, I missed the following two logics for skipping non-changed files:

And one more thing, the differences in time are also shown correctly.

This has been fixed at v0.23.20. Would you mind if I ask you to check the fixes, please?

zedtools commented 1 month ago

Thanks for the quick update.

Yes, the mtime looked a bit odd, but I did not pay much attention to that. Now the files are showing a much more reasonable change in mtime such as several days.

image

As for the syncing of updates, I played around with it a little, and it looks like after updating the plugin I also had to click Select All Shiny and then Apply All Selected at least once, after which I no longer get the excessive notifications.

It looks like the behaviour now is:

  1. Select All Shiny shows all files where the mtime or contents have changed.
  2. Apply All Selected only applies changes where the contents have changed.
  3. After applying changes, Select All Shiny still shows files with a more recent from another device mtime.
zedtools commented 1 month ago

On a side note, from a usability perspective it would help a lot to only display files whose contents have changed. I have 21 plugins, and it takes a while to scroll down to see what files may or may not have changed.

The ideal behaviour for me would be:

  1. When opening the dialog, by default show only files whose contents have changed.
  2. If I want to see more details, I can click a button or a checkbox to show all files, including mtime changes.

This way, if I get a notification that new configuration is available, I can open the dialog and quickly see what the relevant changes were. The more plugins installed, the more useful this behaviour would be.

Is this something you would consider?

zedtools commented 1 month ago

I also have a question about the four options for each file:

The Automatic option does not seem to do anything. If I enable that for a file, I sometimes get a notification that a hidden file was updated, but the file contents are not changed. Any settings I changed do not propagate to a device with Automatic selected.

If I change it back to Selective, I can manually update it fine.

How is Automatic supposed to work?

zedtools commented 1 month ago

Something is still not correct with the differences in mtime.

I just updated the Tasks plugin on my Mac and iPhone, but not yet on my iPad. Customisation sync is showing the iPad as having a newer version, which is incorrect.

I used Textastic to check the file modification times and grabbed the following screenshots. Note that all dates are dd/mm/yyyy.

Screenshots

These screenshots show the contents of .obsidian/plugins/obsidian-tasks-plugin on each platform.

File modified time

Mac

image

iPhone

image

iPad

Note that these are almost one month older:

iPad-files

Customisation sync

Why are my Mac and iPhone showing that my iPad has a newer version of MAIN?

Mac

image

iPhone

image