Some hidden files fail to sync and may cause sync paused

ntlv1 commented 5 months ago

Google Translate was used for some of the content, please let me know if something is not clear!

Abstract

The synchronisation stuck and show Working read storage processes icon ⏳ when hidden file sync enabled.

Expected behaviour

Synchronization should not be paused.

Actually happened

After enabling hidden file synchronization, the ⏳ icon appears in the status bar, and synchronization paused. The number of files being processed is 2.
I found that adding appearance.json and core-plugins.json to the blacklist avoids the above problem.
The problem appeared after the plugin version was updated to 0.22.17. There were no problems previously.

Reproducing procedure

Deleted obsidian config in Windows AppData folder.
A folder with some markdown files. (.obsidian folder has been deleted).
Open folder as vault.
Install self-hosted livesync plugin.
Enable hidden file sync.
There is the problem.

Report materials

Report from the LiveSync

Report from hatch

``` ---- Obsidian info ---- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.5.12 Chrome/120.0.6099.283 Electron/28.2.3 Safari/537.36 ---- remote config ---- cors: credentials: "true" headers: accept, authorization, content-type, origin, referer max_age: "3600" methods: GET, PUT, POST, HEAD, DELETE origins: app://obsidian.md,capacitor://localhost,http://localhost chttpd: bind_address: any max_http_request_size: "4294967296" port: "5984" require_valid_user: "true" admins: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 vendor: name: The Apache Software Foundation feature_flags: partitioned||*: "true" chttpd_auth: authentication_redirect: /_utils/session.html hash_algorithms: sha256, sha require_valid_user: "true" indexers: couch_mrview: "true" prometheus: additional_port: "false" bind_address: 127.0.0.1 port: "17986" httpd: WWW-Authenticate: Basic realm="couchdb" bind_address: 127.0.0.1 enable_cors: "true" port: "5986" smoosh: state_dir: ./data couch_httpd_auth: authentication_db: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 secret: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 authentication_redirect: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 couchdb_engines: couch: couch_bt_engine couchdb: database_dir: ./data max_document_size: "50000000" single_node: "true" uuid: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 view_index_dir: ./data ---- Plug-in config --- version:0.22.17 couchDB_URI: self-hosted(HTTPS) couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 liveSync: true syncOnSave: false syncOnStart: false savingDelay: 200 lessInformationInLog: false gcDelay: 0 versionUpFlash: "" minimumChunkSize: 20 longLineThreshold: 250 showVerboseLog: true suspendFileWatching: false trashInsteadDelete: true periodicReplication: false periodicReplicationInterval: 60 syncOnFileOpen: false encrypt: false passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 usePathObfuscation: false doNotDeleteFolder: false resolveConflictsByNewerFile: false batchSave: false deviceAndVaultName: "" usePluginSettings: false showOwnPlugins: false showStatusOnEditor: true showStatusOnStatusbar: true showOnlyIconsOnEditor: false usePluginSync: false autoSweepPlugins: false autoSweepPluginsPeriodic: false notifyPluginOrSettingUpdated: false checkIntegrityOnSave: false batch_size: 25 batches_limit: 25 useHistory: true disableRequestURI: true skipOlderFilesOnSync: true checkConflictOnlyOnOpen: false showMergeDialogOnlyOnActive: false syncInternalFiles: true syncInternalFilesBeforeReplication: false syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/, app\.json, workspace, community-plugins, hotkeys syncInternalFilesInterval: 60 additionalSuffixOfDatabaseName: 0150fae42ffccb54 ignoreVersionCheck: false lastReadUpdates: 22 deleteMetadataOfDeletedFiles: false syncIgnoreRegEx: "" syncOnlyRegEx: "" customChunkSize: 50 readChunksOnline: true watchInternalFileChanges: true automaticallyDeleteMetadataOfDeletedFiles: 0 disableMarkdownAutoMerge: false writeDocumentsIfConflicted: false useDynamicIterationCount: false syncAfterMerge: false configPassphraseStore: "" encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷 permitEmptyPassphrase: false useIndexedDBAdapter: true useTimeouts: false writeLogToTheFile: false doNotPaceReplication: false hashCacheMaxCount: 300 hashCacheMaxAmount: 50 concurrencyOfReadChunksOnline: 30 minimumIntervalOfReadChunksOnline: 25 hashAlg: xxhash64 suspendParseReplicationResult: false doNotSuspendOnFetching: false useIgnoreFiles: false ignoreFiles: .gitignore syncOnEditorSave: false pluginSyncExtendedSetting: {} syncMaxSizeInMB: 50 settingSyncFile: "" writeCredentialsForSettingSync: false notifyAllSettingSyncFile: false isConfigured: true settingVersion: 0 ```

Obsidian debug info

Debug info

``` 024/4/16 11:11:18->Cache initialized 300 / 250000000000 2024/4/16 11:11:18->loading plugin 2024/4/16 11:11:18->Self-hosted LiveSync v0.22.17 0.22.17 2024/4/16 11:11:18->xxhash for plugin initialised 2024/4/16 11:11:18->Waiting for ready... 2024/4/16 11:11:18->Cache initialized 10 / 1000000000 2024/4/16 11:11:18->Cache initialized 300 / 50000000 2024/4/16 11:11:18->Newer xxhash has been initialised 2024/4/16 11:11:18->Opening Database... 2024/4/16 11:11:18->Database info 2024/4/16 11:11:18->{ "doc_count": 749, "update_seq": 751, "db_name": "Wiki-0150fae42ffccb54-livesync-v2-indexeddb", "auto_compaction": false, "adapter": "indexeddb" } 2024/4/16 11:11:18->Database is now ready. 2024/4/16 11:11:18->Log window opened 2024/4/16 11:11:18->Initialize and checking database files 2024/4/16 11:11:18->Checking deleted files 2024/4/16 11:11:18->Collecting local files on the storage 2024/4/16 11:11:18->Collecting local files on the DB 2024/4/16 11:11:18->Collecting local files on the DB: 25 2024/4/16 11:11:18->Collecting local files on the DB: 50 2024/4/16 11:11:18->Collecting local files on the DB: 75 2024/4/16 11:11:18->Collecting local files on the DB: 100 2024/4/16 11:11:19->Collecting local files on the DB: 125 2024/4/16 11:11:19->Collecting local files on the DB: 150 2024/4/16 11:11:19->Collecting local files on the DB: 175 2024/4/16 11:11:19->Collecting local files on the DB: 200 2024/4/16 11:11:19->Opening the key-value database 2024/4/16 11:11:19->Updating database by new files 2024/4/16 11:11:19->UPDATE DATABASE: Nothing to do 2024/4/16 11:11:19->UPDATE STORAGE: Nothing to do 2024/4/16 11:11:22->Initialized, NOW TRACKING! 2024/4/16 11:11:22->Synchronizing hidden files... 2024/4/16 11:11:22->Scanning hidden files. 2024/4/16 11:11:25->Hidden files scanned: 3 files had been modified 2024/4/16 11:11:25->Synchronizing hidden files done ```

Screenshots

PixPin_2024-04-16_11-14-24

ZiGmaX809 commented 5 months ago

My problem is the same as yours.

vrtmrz commented 4 months ago

Sorry for being late! And thank you so much for your complete information!

I initially thought this was a display problem. Because for some reason the process itself was complete.

However, by tracing each file and each operation, I found a bit serious problem (It is abled by your information; appearance.json had reproducibility).

The conclusion was that when the same file, the same operation, which should be serialised and processed, was queued for a short period, some queues placed in the middle had their processing results ignored.

This is usually not so fatal. However, in some situations, it could be so harmful. I would love to thank you all for this report!

This issue has been fixed at v0.23.7. Would you mind if I ask you to check the behaviour, please?

ntlv1 commented 4 months ago

Thanks for the reply. So happy, cuz I've really been waiting for a long time😊 Just now I tested the latest version of the plugin, everything works fine, thanks!

ntlv1 commented 4 months ago

Problem was solved, so I'll close this issue.

vrtmrz / obsidian-livesync