search

nextcloud / desktop

💻 Desktop sync client for Nextcloud
https://nextcloud.com/install/#install-clients
GNU General Public License v2.0
2.97k stars 784 forks source link

Should have an option (configfile) to be more tolerant to errors #1642

Open tstyopin opened 4 years ago

tstyopin commented 4 years ago

Hello! I have an ARM boards with attached hard drives, which runs Nextcloud. Boards are rather slow, so nothing on them works with 100% reliability. Please add an option to ignore errors on file transmission (timeouts especially), so desktop client can download some other files, rather that begin sync from very beginning. Also, there is no option to limit simultaneous downloads (setting env variable seems don't work on Windows) in interface, can you add this to config file or interface directly?

Expected behaviour

If there is time-out on one file while other files exist in download queue desktop client should try to download next file from queue instead of failing whole sync process There should be an configurable option in interface or config file to limit simultaneous downloads (today i saw 43 threads, which literally hang my board by swap)

Actual behaviour

Sync process failing, when one thread failed, no other files are tried to downloading There is no configurable option to limit simultaneous downloads

Steps to reproduce

  1. Have tiny small ARM boards
  2. Have Nextcloud up and running on them (i have mysql and nginx+php-fpm on separate boards)
  3. Try to download 20k-40k files in one run in fast home network (on slow networks not failing so much)

Client configuration

2.6.1 Stable. Build 20191105

Operating system: Windows 10 OS language: Russian/English

Qt version used by client package (Linux only, see also Settings dialog):

Client package (From Nextcloud or distro) (Linux only):

Installation path of client: Default

Server configuration

Arm64 nginx/1.14.0 + php-fpm 7.2.24-0ubuntu0.18.04.1 Nextcloud version: 17.0.1

Storage backend (external storage): USB-attached hard-drives

Logs

  1. Client logfile: There is no particular in client logfile, when first thread timed out, others interrupted and mark as cancelled

  2. Web server error log:

  3. Server logfile: nextcloud log (data/nextcloud.log):

tstyopin commented 4 years ago

part of nextcloudcmd log showing cancelling 7 sync operations after one thread failed due to timeout: 11-26 07:43:03:663 [ warning nextcloud.sync.networkjob ]: Network job timeout QUrl("https://my.nextcloud.url/remote.php/dav/files/TS/Games/Flashpoint/Data/Images/Flash/Box - Front/9991cb6c-321d-bf3b-d681-ca434246af5a-01.png") 11-26 07:43:03:664 [ warning nextcloud.sync.networkjob.get ]: Timeout QUrl("https://my.nextcloud.url/remote.php/dav/files/TS/Games/Flashpoint/Data/Images/Flash/Box - Front/9991cb6c-321d-bf3b-d681-ca434246af5a-01.png") 11-26 07:43:03:665 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Connection timed out" QVariant(Invalid) 11-26 07:43:03:668 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/9991cb6c-321d-bf3b-d681-ca434246af5a-01.png" by OCC::PropagateDownloadFile(0x149333aeba0) with status 1 and error: "Connection Timeout" 11-26 07:43:03:669 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:670 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/9968c7f9-a281-4c78-9b75-afa706dcdb63-01.png" by OCC::PropagateDownloadFile(0x1493514e600) with status 3 and error: "Operation canceled" 11-26 07:43:03:671 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:672 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/998df1f0-0914-4e8b-9f13-b96a4cabb223-01.png" by OCC::PropagateDownloadFile(0x149333ae510) with status 3 and error: "Operation canceled" 11-26 07:43:03:673 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:674 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/9995726e-f757-6cc6-dcbe-b874f171b1b7-01.png" by OCC::PropagateDownloadFile(0x149333ae6f0) with status 3 and error: "Operation canceled" 11-26 07:43:03:675 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:675 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/999a219d-a22f-40a9-8986-665b87dd67d6-01.PNG" by OCC::PropagateDownloadFile(0x149333ae8d0) with status 3 and error: "Operation canceled" 11-26 07:43:03:676 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:677 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/Atlantis Squarepantis Bus Rush-01.png" by OCC::PropagateDownloadFile(0x149333ae240) with status 3 and error: "Operation canceled" 11-26 07:43:03:679 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:679 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/Awaken - Front Line-01.png" by OCC::PropagateDownloadFile(0x149333ae330) with status 3 and error: "Operation canceled" 11-26 07:43:03:680 [ warning nextcloud.sync.networkjob ]: QNetworkReply::OperationCanceledError "Operation canceled" QVariant(Invalid) 11-26 07:43:03:681 [ warning nextcloud.sync.propagator ]: Could not complete propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/Daily Sodoku-01.png" by OCC::PropagateDownloadFile(0x1494fbe0f70) with status 3 and error: "Operation canceled" 11-26 07:43:03:682 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/Dexters Laboratory Clone-A-Doodle-Doo-01.png" by OCC::PropagateDownloadFile(0x1491cbed9c0) 11-26 07:43:03:683 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/F.P. Squad-01.png" by OCC::PropagateDownloadFile(0x1491cbedab0) 11-26 07:43:03:683 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/One Will Survive 2-01.png" by OCC::PropagateDownloadFile(0x1491cbed330) 11-26 07:43:03:684 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/Shootout-01.png" by OCC::PropagateDownloadFile(0x1491cbedba0) 11-26 07:43:03:685 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/a0003fa2-c803-6614-9c71-e46c8c44be30-01.png" by OCC::PropagateDownloadFile(0x1491cbed420) 11-26 07:43:03:685 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/a0064316-d8df-b12e-78ed-473853893ca2-01.png" by OCC::PropagateDownloadFile(0x1491cbed510) 11-26 07:43:03:686 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_CONFLICT propagation of "Games/Flashpoint/Data/Images/Flash/Box - Front/a009483e-f8db-40c0-6ded-7a9c6b578c0a-01.PNG" by OCC::PropagateDownloadFile(0x1491cbedc90) 11-26 07:43:10:016 [ info nextcloud.sync.database ]: Closing DB "S:/Storage/Nextcloud/._sync_b589b6dbd9b8.db" 11-26 07:43:10:017 [ info nextcloud.sync.engine ]: CSync run took 14106746 ms 11-26 07:44:01:181 [ info nextcloud.sync.database ]: sqlite3 version "3.24.0" 11-26 07:44:01:181 [ info nextcloud.sync.database ]: sqlite3 journal_mode= "wal" 11-26 07:44:03:062 [ info nextcloud.sync.database ]: Closing DB "S:/Storage/Nextcloud/._sync_b589b6dbd9b8.db"

tstyopin commented 4 years ago

Option --max-sync-retries seems not work, as no retries are made (dont saw in logfile) Option --unsyncedfolders has no documentation or examples, so i cant sync only one folder