search

vrtmrz / obsidian-livesync

MIT License
3.96k stars 134 forks source link

Help me! Fly.io Volume Usage 100%. 1. Sync is not possible. 2. Perform cleaning is failed. #312

Closed dlphnkck closed 7 months ago

dlphnkck commented 8 months ago

I am upset, so I apologize if my writing is clumsy.

Abstract

  1. 2GB volume of Fly.io's CouchDB is almost 100% used and cannot be synchronized.
    • Local file size is about 200MB.
    • The number of notes is about 7,000.
  2. Perform cleaning is failed.
  3. Is there no other way but to run Rebuild everything? I've done this many times before, but it takes several hours and I don't really want to do it ๐Ÿ˜….

Expected behaviour

The local Obsidian file size is 200MB, so I hope it will be reduced to that size.

Actually happened

  1. Volume Usage on Fly.io has increased from 823MB to 1.93GiB in the last month, but since Fly.io only has records for the last 30 days, I do not know how fast it increased before that.
  2. I ran the Perfom cleanup on Fly.io's remote databases, but the process seems to have failed and the disk usage has not decreased.

Reproducing procedure

I tried Perform cleaning on Remote databases (and Local databases) from the Maintain tab, but it failed.

2023/10/27 hh:mm:ss->Cleaning remote database
2023/10/27 hh:mm:ss->Database active-size: 955 MB, external-size:896 MB, file-size: 1 GB
2023/10/27 hh:mm:ss->Collecting unreferenced chunks on ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท
2023/10/27 hh:mm:ss->Lock remote database to prevent data corruption
2023/10/27 hh:mm:ss->Failed to make design document for operating chunks
2023/10/27 hh:mm:ss->{
  "error": "error",
  "reason": "internal_server_error",
  "status": 500,
  "name": "error",
  "message": "internal_server_error",
  "stack": "Error\n    at generateErrorFromResponse (plugin:obsidian-livesync:4670:35)\n    at fetchJSON (plugin:obsidian-livesync:9623:13)\n    at async HttpPouch.o._put (plugin:obsidian-livesync:9916:16)",
  "docId": "_design/chunks"
}
2023/10/27 hh:mm:ss->Could not prepare design document for operating chunks
2023/10/27 hh:mm:ss->No unreferenced chunks! ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท
2023/10/27 hh:mm:ss->Compacting database...
2023/10/27 hh:mm:ss->Compacting database done
2023/10/27 hh:mm:ss->Processed database active-size: 955 MB, external-size:896 MB, file-size: 1 GB
2023/10/27 hh:mm:ss->Reduced sizes: active-size: -, external-size:-, file-size: -
2023/10/27 hh:mm:ss->Cleaning remote database up: Done
2023/10/27 hh:mm:ss->Cleaning local database
2023/10/27 hh:mm:ss->Collecting unreferenced chunks on ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท-livesync-v2

Report materials

If the information is not available, do not hesitate to report it as it is. You can also of course omit it if you think this is indeed unnecessary. If it is necessary, I will ask you.

Report from the LiveSync

remote config ``` ----remote config---- cors: credentials: "true" origins: app://obsidian.md,capacitor://localhost,http://localhost chttpd: bind_address: any max_http_request_size: "4294967296" port: "5984" require_valid_user: "true" admins: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท vendor: name: The Apache Software Foundation feature_flags: partitioned||*: "true" chttpd_auth: hash_algorithms: sha256, sha require_valid_user: "true" indexers: couch_mrview: "true" prometheus: additional_port: "false" bind_address: 127.0.0.1 port: "17986" httpd: WWW-Authenticate: Basic realm="couchdb" bind_address: 127.0.0.1 enable_cors: "true" port: "5986" smoosh: state_dir: ./data couch_httpd_auth: authentication_db: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท secret: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท authentication_redirect: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท couchdb_engines: couch: couch_bt_engine couchdb: database_dir: ./data max_document_size: "50000000" uuid: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท view_index_dir: ./data ```
Plug-in config ``` ---- Plug-in config --- version:0.20.6 couchDB_URI: self-hosted couchDB_USER: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท couchDB_PASSWORD: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท couchDB_DBNAME: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท liveSync: true syncOnSave: false syncOnStart: false savingDelay: 200 lessInformationInLog: false gcDelay: 0 versionUpFlash: "" minimumChunkSize: 20 longLineThreshold: 250 showVerboseLog: true suspendFileWatching: false trashInsteadDelete: false periodicReplication: false periodicReplicationInterval: 60 syncOnFileOpen: false encrypt: true passphrase: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท usePathObfuscation: false doNotDeleteFolder: false resolveConflictsByNewerFile: false batchSave: false deviceAndVaultName: "" usePluginSettings: false showOwnPlugins: false showStatusOnEditor: true usePluginSync: false autoSweepPlugins: false autoSweepPluginsPeriodic: false notifyPluginOrSettingUpdated: false checkIntegrityOnSave: false batch_size: 250 batches_limit: 40 useHistory: true disableRequestURI: true skipOlderFilesOnSync: true checkConflictOnlyOnOpen: true syncInternalFiles: true syncInternalFilesBeforeReplication: false syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/,\/workspace$ ,\/workspace.json$ syncInternalFilesInterval: 60 additionalSuffixOfDatabaseName: "" ignoreVersionCheck: false lastReadUpdates: 19 deleteMetadataOfDeletedFiles: false syncIgnoreRegEx: "" syncOnlyRegEx: "" customChunkSize: 0 readChunksOnline: true watchInternalFileChanges: true automaticallyDeleteMetadataOfDeletedFiles: 0 disableMarkdownAutoMerge: false writeDocumentsIfConflicted: false useDynamicIterationCount: false syncAfterMerge: false configPassphraseStore: "" encryptedPassphrase: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท encryptedCouchDBConnection: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท permitEmptyPassphrase: false useIndexedDBAdapter: false useTimeouts: false writeLogToTheFile: false doNotPaceReplication: false hashCacheMaxCount: 300 hashCacheMaxAmount: 50 concurrencyOfReadChunksOnline: 100 minimumIntervalOfReadChunksOnline: 333 hashAlg: xxhash64 suspendParseReplicationResult: false doNotSuspendOnFetching: false useIgnoreFiles: false ignoreFiles: .gitignore syncOnEditorSave: false pluginSyncExtendedSetting: {} useV1: false workingUseDynamicIterationCount: false ```

Obsidian debug info

Debug info ``` SYSTEM INFO: Obsidian version: v1.4.16 Installer version: v1.4.16 Operating system: Windows 10 Home 10.0.19045 Login status: not logged in Insider build toggle: off Live preview: off Legacy editor: off Base theme: dark Community theme: Minimal v7.2.3 Snippets enabled: 5 Restricted mode: off Plugins installed: 21 Plugins enabled: 18 1: Calendar v1.5.10 2: Commander v0.5.1 3: Excalidraw v1.9.26 4: Outliner v4.8.0 5: Plugin Update Tracker v1.5.2 6: ReadItLater v0.3.1 7: Self-hosted LiveSync v0.20.6 8: Templater v1.16.4 9: Various Complements v9.0.1 10: Vimrc Support v0.9.0 11: Search on Internet v0.5.0 12: Toggl Track v0.11.0 13: 2Hop Links Plus v0.37.0 14: Advanced URI v1.38.1 15: Version History Diff v2.1.9 16: Editing Toolbar v2.3.1 17: QuickAdd v1.4.0 18: Word Splitting for Simplified Chinese in Edit Mode and Vim Mode v1.10.0 ```

Plug-in log

Sorry, I didn't know where the "Document box icon" was.

Network log

Failed to load resource: the server responded with a status of 500 ()
POST https://๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท.fly.dev/๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท/_bulk_docs 500
fetch @ plugin:obsidian-livesync:22087
ourFetch @ plugin:obsidian-livesync:9600
fetchJSON @ plugin:obsidian-livesync:9616
HttpPouch.o._bulkDocs @ plugin:obsidian-livesync:9904
await in HttpPouch.o._bulkDocs (async)
eval @ plugin:obsidian-livesync:6861
eval @ plugin:obsidian-livesync:4926
eval @ plugin:obsidian-livesync:4890
eval @ plugin:obsidian-livesync:4883
bulkDocs @ plugin:obsidian-livesync:1931

Screenshots

Chart of Volume Usage in Fly.io. 2023-10-27 Fly io Volume Usage Chart

Other information, insights and intuition.

That is all. The questions are,

  1. Why has Fly.io's Volume Usage increased so dramatically in the last 30 days alone?
  2. How to reduce this kind of Volume Usage.

That is all.

I apologize for the length of this article. I was upset, so I know there is a lot of material missing . Please feel free to ask.

Thanks for reading.

dlphnkck commented 8 months ago

Sorry.

  1. Report from the LiveSync
  2. Obsidian debug info

I'll reformat it and repost them.

Report materials

Report from the LiveSync

remote config ``` ----remote config---- cors: credentials: "true" origins: app://obsidian.md,capacitor://localhost,http://localhost chttpd: bind_address: any max_http_request_size: "4294967296" port: "5984" require_valid_user: "true" admins: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท vendor: name: The Apache Software Foundation feature_flags: partitioned||*: "true" chttpd_auth: hash_algorithms: sha256, sha require_valid_user: "true" indexers: couch_mrview: "true" prometheus: additional_port: "false" bind_address: 127.0.0.1 port: "17986" httpd: WWW-Authenticate: Basic realm="couchdb" bind_address: 127.0.0.1 enable_cors: "true" port: "5986" smoosh: state_dir: ./data couch_httpd_auth: authentication_db: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท secret: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท authentication_redirect: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท couchdb_engines: couch: couch_bt_engine couchdb: database_dir: ./data max_document_size: "50000000" uuid: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท view_index_dir: ./data ```
Plug-in config ``` ---- Plug-in config --- version:0.20.6 couchDB_URI: self-hosted couchDB_USER: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท couchDB_PASSWORD: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท couchDB_DBNAME: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท liveSync: true syncOnSave: false syncOnStart: false savingDelay: 200 lessInformationInLog: false gcDelay: 0 versionUpFlash: "" minimumChunkSize: 20 longLineThreshold: 250 showVerboseLog: true suspendFileWatching: false trashInsteadDelete: false periodicReplication: false periodicReplicationInterval: 60 syncOnFileOpen: false encrypt: true passphrase: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท usePathObfuscation: false doNotDeleteFolder: false resolveConflictsByNewerFile: false batchSave: false deviceAndVaultName: "" usePluginSettings: false showOwnPlugins: false showStatusOnEditor: true usePluginSync: false autoSweepPlugins: false autoSweepPluginsPeriodic: false notifyPluginOrSettingUpdated: false checkIntegrityOnSave: false batch_size: 250 batches_limit: 40 useHistory: true disableRequestURI: true skipOlderFilesOnSync: true checkConflictOnlyOnOpen: true syncInternalFiles: true syncInternalFilesBeforeReplication: false syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/,\/workspace$ ,\/workspace.json$ syncInternalFilesInterval: 60 additionalSuffixOfDatabaseName: "" ignoreVersionCheck: false lastReadUpdates: 19 deleteMetadataOfDeletedFiles: false syncIgnoreRegEx: "" syncOnlyRegEx: "" customChunkSize: 0 readChunksOnline: true watchInternalFileChanges: true automaticallyDeleteMetadataOfDeletedFiles: 0 disableMarkdownAutoMerge: false writeDocumentsIfConflicted: false useDynamicIterationCount: false syncAfterMerge: false configPassphraseStore: "" encryptedPassphrase: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท encryptedCouchDBConnection: ๐‘…๐ธ๐ท๐ด๐ถ๐‘‡๐ธ๐ท permitEmptyPassphrase: false useIndexedDBAdapter: false useTimeouts: false writeLogToTheFile: false doNotPaceReplication: false hashCacheMaxCount: 300 hashCacheMaxAmount: 50 concurrencyOfReadChunksOnline: 100 minimumIntervalOfReadChunksOnline: 333 hashAlg: xxhash64 suspendParseReplicationResult: false doNotSuspendOnFetching: false useIgnoreFiles: false ignoreFiles: .gitignore syncOnEditorSave: false pluginSyncExtendedSetting: {} useV1: false workingUseDynamicIterationCount: false ```

Obsidian debug info

Debug info ``` SYSTEM INFO: Obsidian version: v1.4.16 Installer version: v1.4.16 Operating system: Windows 10 Home 10.0.19045 Login status: not logged in Insider build toggle: off Live preview: off Legacy editor: off Base theme: dark Community theme: Minimal v7.2.3 Snippets enabled: 5 Restricted mode: off Plugins installed: 21 Plugins enabled: 18 1: Calendar v1.5.10 2: Commander v0.5.1 3: Excalidraw v1.9.26 4: Outliner v4.8.0 5: Plugin Update Tracker v1.5.2 6: ReadItLater v0.3.1 7: Self-hosted LiveSync v0.20.6 8: Templater v1.16.4 9: Various Complements v9.0.1 10: Vimrc Support v0.9.0 11: Search on Internet v0.5.0 12: Toggl Track v0.11.0 13: 2Hop Links Plus v0.37.0 14: Advanced URI v1.38.1 15: Version History Diff v2.1.9 16: Editing Toolbar v2.3.1 17: QuickAdd v1.4.0 18: Word Splitting for Simplified Chinese in Edit Mode and Vim Mode v1.10.0 RECOMMENDATIONS: Custom theme and snippets: for cosmetic issues, please first try updating your theme and disabling your snippets. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community theme and snippets. Community plugins: for bugs, please first try updating all your plugins to latest. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community plugins. ```
vrtmrz commented 8 months ago

Sorry for being late! Looks like we do not have enough space even to clean it up. However, I think that we do not have to worry so much. We can rebuild databases. This will certainly free up more space in the database than a database clean-up. And, rebuilding the database should not affect existing files (assuming all devices are fully synchronised. However, if not, we should back the vault up in all devices). The only thing that we will lose during the rebuild is a history of file editing. We can rebuild the local database and the remote database in Rebuild everything in the Maintain database pane. Could you please try rebuilding once?

dlphnkck commented 8 months ago

Thanks for taking the time to reply.

"Rebuild everything" was done 9 months ago and it was documented to be done in 6 hours with 5 devices.

I think the number of notes has doubled since 9 months ago, but I'm going to run "Rebuild everything" now.

I hope you are not too sick as you seem to be getting a lot of requests. I hope everyone can do a better job of attaching the documents in advance so that the burden on Mr. vorotamoroz can be reduced. Thank you very much for your time and answer. I hope it will ease your burden as much as possible.

dlphnkck commented 8 months ago

Sorry, "Rebuild everything" did not work.

Abstract

The volume usage of CouchDB in Fly.io is now at 100%, so I tried to Rebuild everything 4 times on Windows and once from my iPad, but when I restarted the sync, the note changes were not reflected on the other devices.

Expected behaviour

  1. execute "Rebuild everything" on the master device -> "Fetch rebuilt DB" is requested on the other 4 devices
  2. change of note is reflected to other devices.
  3. volume usage of CouchDB in Fly.io is reduced from 100%.

Actually happend

  1. execute "Rebuild everything" on the master device -> "Fetch rebuilt DB" is requested on the other 4 devices. โ†’The result is that nothing is displayed on the other devices.
  2. changes in notes were not reflected between devices.
  3. volume usage of CouchDB in Fly.io was still 100%.

Reproducing procedure

  1. enable "Write logs into the file",

  2. go to Obsidian->Settings->Self-hosted LiveSync->Maintain databases tab->Click "Rebuild everything" button.

  3. in Windows, press "Rebuild everything" 4 times. 1-1. I also got the "Lock remote database" popup in the top right corner of the screen.

Diary:2023/11/8 HH:mm:ss->Lock remote database to prevent data corruption
1-2. the following popup would sometimes appear in the upper right corner of the screen immediately after starting Obsidian.

Obsidian is indexing your vault... This should only happen once. Some functionality may not be available until this is complete. (1/233)

Image from Gyazo

  1. 1 time to rebuild everything on the iPad
Log right after starting "Rebuild everything" on Windows ```log Diary:2023/11/8 HH:mm:ss->CRYPT LOGIC OK Diary:2023/11/8 HH:mm:ss->Hidden files and plugin synchronization have been temporarily disabled. Please enable them after the fetching, if you need them. Diary:2023/11/8 HH:mm:ss->All synchronization have been temporarily disabled. Please enable them after the fetching, if you need them. Diary:2023/11/8 HH:mm:ss->Replication closed Diary:2023/11/8 HH:mm:ss->Cache initialized 300 / 50000000 Diary:2023/11/8 HH:mm:ss->Hidden files and plugin synchronization have been temporarily disabled. Please enable them after the fetching, if you need them. Diary:2023/11/8 HH:mm:ss->Cache initialized 300 / 50000000 Diary:2023/11/8 HH:mm:ss->Database closed for reset Database. Diary:2023/11/8 HH:mm:ss->Opening Database... Diary:2023/11/8 HH:mm:ss->Database info Diary:2023/11/8 HH:mm:ss->{ "doc_count": 0, "update_seq": 0, "idb_attachment_format": "binary", "db_name": "Diary-livesync-v2", "auto_compaction": false, "adapter": "idb" } Diary:2023/11/8 HH:mm:ss->Database is now ready. Diary:2023/11/8 HH:mm:ss->Local Database Reset Diary:2023/11/8 HH:mm:ss->Database closed (by close) Diary:2023/11/8 HH:mm:ss->Waiting for ready... Diary:2023/11/8 HH:mm:ss->Cache initialized 10 / 1000000000 Diary:2023/11/8 HH:mm:ss->Cache initialized 300 / 50000000 Diary:2023/11/8 HH:mm:ss->Newer xxhash has been initialised Diary:2023/11/8 HH:mm:ss->Opening Database... Diary:2023/11/8 HH:mm:ss->Database info Diary:2023/11/8 HH:mm:ss->{ "doc_count": 1, "update_seq": 1, "idb_attachment_format": "binary", "db_name": "Diary-livesync-v2", "auto_compaction": false, "adapter": "idb" } Diary:2023/11/8 HH:mm:ss->Database is now ready. Diary:2023/11/8 HH:mm:ss->Initializing Diary:2023/11/8 HH:mm:ss->Initialize and checking database files Diary:2023/11/8 HH:mm:ss->Checking deleted files Diary:2023/11/8 HH:mm:ss->Collecting local files on the storage Diary:2023/11/8 HH:mm:ss->Collecting local files on the DB Diary:2023/11/8 HH:mm:ss->Opening the key-value database Diary:2023/11/8 HH:mm:ss->Database looks empty, save files as initial sync data Diary:2023/11/8 HH:mm:ss->Updating database by new files Diary:2023/11/8 HH:mm:ss->UPDATE DATABASE ```
"Rebuild everything" log on iPad ```log Diary:2023/11/8 HH:mm:ss->Database closed for reset Database. Diary:2023/11/8 HH:mm:ss->Opening Database... Diary:2023/11/8 HH:mm:ss->Database info Diary:2023/11/8 HH:mm:ss->{ "doc_count": 0, "update_seq": 0, "db_name": "Diary-livesync-v2-indexeddb", "auto_compaction": false, "adapter": "indexeddb" } Diary:2023/11/8 HH:mm:ss->Database is now ready. Diary:2023/11/8 HH:mm:ss->Local Database Reset Diary:2023/11/8 HH:mm:ss->Database closed (by close) Diary:2023/11/8 HH:mm:ss->Waiting for ready... Diary:2023/11/8 HH:mm:ss->Cache initialized 10 / 1000000000 Diary:2023/11/8 HH:mm:ss->Cache initialized 300 / 50000000 Diary:2023/11/8 HH:mm:ss->Newer xxhash has been initialised Diary:2023/11/8 HH:mm:ss->Opening Database... Diary:2023/11/8 HH:mm:ss->Database info Diary:2023/11/8 HH:mm:ss->{ "doc_count": 1, "update_seq": 1, "db_name": "Diary-livesync-v2-indexeddb", "auto_compaction": false, "adapter": "indexeddb" } Diary:2023/11/8 HH:mm:ss->Database is now ready. Diary:2023/11/8 HH:mm:ss->Initializing Diary:2023/11/8 HH:mm:ss->Initialize and checking database files Diary:2023/11/8 HH:mm:ss->Checking deleted files Diary:2023/11/8 HH:mm:ss->Collecting local files on the storage Diary:2023/11/8 HH:mm:ss->Collecting local files on the DB Diary:2023/11/8 HH:mm:ss->Opening the key-value database Diary:2023/11/8 HH:mm:ss->Database looks empty, save files as initial sync data Diary:2023/11/8 HH:mm:ss->Updating database by new files Diary:2023/11/8 HH:mm:ss->UPDATE DATABASE ```

Report materials

Report from the LiveSync

Since this is the same as last time, I will omit it.

Obsidian debug info

Since this is the same as last time, I will omit it.

Plug-in log

Since this is the same as last time, I will omit it.

Network log

Sorry. I forgot to pick it up and only saw it from the middle.

plugin:obsidian-livesync:22087     POST https://***.fly.dev/***/_bulk_docs 500
Log at the end of "Rebuild everything" ```log plugin:obsidian-livesync:22816 2023/11/8 HH:mm:ss->Lock remote database to prevent data corruption plugin:obsidian-livesync:22087 PUT https://*****.fly.dev/*****/_local/obsydian_livesync_milestone 500 fetch @ plugin:obsidian-livesync:22087 ourFetch @ plugin:obsidian-livesync:9600 fetchJSON @ plugin:obsidian-livesync:9616 HttpPouch.o._put @ plugin:obsidian-livesync:9916 await in HttpPouch.o._put (async) putDoc @ plugin:obsidian-livesync:6497 eval @ plugin:obsidian-livesync:6519 eval @ plugin:obsidian-livesync:4926 eval @ plugin:obsidian-livesync:4890 eval @ plugin:obsidian-livesync:4883 r.put @ plugin:obsidian-livesync:1991 await in r.put (async) doMethod @ plugin:obsidian-livesync:1850 eval @ plugin:obsidian-livesync:1864 markRemoteLocked @ plugin:obsidian-livesync:19975 await in markRemoteLocked (async) markRemoteLocked @ plugin:obsidian-livesync:23281 rebuildEverything @ plugin:obsidian-livesync:20992 ```

Screenshots

None.

Other information, insights and intuition.

None.

That is all. Sorry for the complication. Thank you for your time.

vrtmrz commented 8 months ago

Thank you for your try! 6 hours sounds a bit disconcerting, but I hope that it could be better in recent versions. (If not or it has gotten longer, please inform me. Actually, I suspect a bit so).

And, I appreciate your kindness! I am one of the happiest maintainers!

However, while writing the above in the draft, the situation seems to change again. Thanks to the detailed information, we had not missed the indication. CouchDB still refusing the request. Perhaps we have really little storage in the remote. Would you mind if I ask you to delete the remote database manually from the admin tool of CouchDB? We can access it in browser by https://example.com/_utils

dlphnkck commented 8 months ago

Thank you for your help with this issue. I was able to successfully rebuild CouchDB.

The volume usage of Fly.io was 572MB right after I finished building CouchDB, but after synchronizing 5 devices, the volume usage of Fly.io was finally Used 1.01 GiB. However, after running "Perform cleaning" from the Maintain databases tab, it temporarily increased from Used 1.01 GiB to Used 1.73 GiB, and then successfully decreased to Used 484 MB! Thank you very much for your help. Image from Gyazo

P.S.

Quote from the article [Obsidian] A closer look at Self-hosted LiveSync and data synchronization! (Dialogue 02) ๏ผˆSelf-hosted LiveSyncใจใƒ‡ใƒผใ‚ฟๅŒๆœŸใซ่ฟซใ‚‹๏ผ (ๅฏพ่ซ‡ไผ็”ป02)) 2022.09.20

It is recommended to periodically "Rebuild everything" to dump and rebuild the database contents. ๏ผˆๅฎšๆœŸ็š„ใซใ€ŒRebuild everythingใ€ใงใƒ‡ใƒผใ‚ฟใƒ™ใƒผใ‚นใฎๅ†…ๅฎนใ‚’ไธ€ๆ—ฆๆจใฆใฆๅ†ๆง‹็ฏ‰ใ™ใ‚‹ไบ‹ใ‚’ๆŽจๅฅจใ—ใฆใ„ใ‚‹๏ผ‰

It is written that I should manually "Rebuild everything" periodically? ๏ผˆใจๆ›ธใ‹ใ‚Œใฆใ„ใพใ™ใŒใ€ๅฎšๆœŸ็š„ใซๆ‰‹ๅ‹•ใง"Rebuild everything"ใ‚’่กŒใฃใŸๆ–นใŒใ‚ˆใ„ใฎใงใ—ใ‚‡ใ†ใ‹๏ผŸ๏ผ‰

I learned my lesson this time and tried to see if I could use a Webhook to periodically notify Used in Volume Usage of CouchDB running on Fly.io, and now Fly.io is down, so I've been trying to get to...๐Ÿ˜ญ

I am very sorry for the trouble you went through. It was very helpful. Thank you very much.

vrtmrz commented 7 months ago

I am so relieved to hear that! I have seen your report and noticed that cleaning the database collects unused chunks on the server-side query, so that it may use extra storage temporarily. This may indicated as the 1.73GB-spike. However, to begin with, decreasing free space along newer devices getting synchronised is a little strange. Possibly there are some bugs around the conflict resolution or something. I will investigate it after this!

It is written that I should manually "Rebuild everything" periodically?

Yes, How often we should? is the case depending on the files in our vault and the way of editing them, but it is true. Observing status - your idea - would be useful.

I learned my lesson this time and tried to see if I could use a Webhook to periodically notify Used in Volume Usage of CouchDB running on Fly.io, and now Fly.io is down, so I've been trying to get to...๐Ÿ˜ญ

From Self-hosted LiveSync, we can see the usage in DryRun of cleaning up. However, it may cause a spike. Possibly It is good to be able to check the server status only.

dlphnkck commented 7 months ago

Hello. Thank you for taking the time to reply. I understand the importance of periodically executing "Rebuild everything."

I can now comfortably use Obsidian on multiple devices and am once again very grateful for the availability of the LiveSync plugin. Thank you for your support.

I know you are busy, but please take care of yourself.

vrtmrz commented 7 months ago

Hi! Thank you very much for your kindness! I am very honoured that your obsidian life is so awesome again. However, please do not hesitate to ask me. I will answer without overwork. I am trying to pace it in recently ๐Ÿ•Š๏ธ.