search

pydio / pydio-sync

Python version of the Pydio synchronization client
https://pydio.com
GNU General Public License v3.0
82 stars 32 forks source link

Unrecoverable filesystem sync when lots of files change (KeyError) #180

Open mcr-ksh opened 5 years ago

mcr-ksh commented 5 years ago

Latest PydioSync 2.2.0 produces unrecoverable error on pydio-sync and Pydio8.

System:

When moving lots of files (373 and 13GB) in my case to a folder the client will react the following and will never recover the filesystem:

image

To notice a difference between a functioning sync and this error: The line "Processing XX changes" is printed after "Reducing changes for-my-files" On a non-functioning sync the "Processing XX changes" is not present, leading to the "KeyError". I could only guess that this is somehow related to a race-condition of the Filesystem changes not recognised before the actual sync kicks off. Any clue?

`
2019-08-06 22:22:13 INFO123145479970816 Thread-1 pydio.job.continous_merger : 495 | run | Loading remote changes with sequence 141841 for job id-my-files 2019-08-06 22:22:14 INFO123145479970816 Thread-1 pydio.job.continous_merger : 543 | run | Loading local changes with sequence 11504 for job id-my-files 2019-08-06 22:22:14 INFO123145479970816 Thread-1 pydio.job.continousmerger : 572 | run | Reducing changes for-my-files_ 2019-08-06 22:22:47 ERROR 123145479970816 Thread-1 pydio.job.change_stores : 643 | stat_path | u'/Private/Pictures/Something/IMG_1832.jpg' Traceback (most recent call last): File "/Users/charles/Desktop/TeamCity/buildAgent/work/19045f4c63bc432e/build/pydio-agent/out01-PYZ.pyz/pydio.job.change_stores", line 641, in stat_path KeyError: u'/Private/Pictures/Something/IMG_1832.jpg' 2019-08-06 22:22:48 ERROR 123145479970816 Thread-1 pydio.job.change_stores : 643 | stat_path | u'/Private/Pictures/Something/IMG_1834.jpg'
Traceback (most recent call last): File "/Users/charles/Desktop/TeamCity/buildAgent/work/19045f4c63bc432e/build/pydio-agent/out01-PYZ.pyz/pydio.job.change_stores", line 641, in stat_path KeyError: u'/Private/Pictures/Something/IMG_1834.jpg' 2019-08-06 22:22:49 ERROR 123145479970816 Thread-1 pydio.job.change_stores : 643 | stat_path | u'/Private/Pictures/Something/IMG_1835.jpg'
Traceback (most recent call last): File "/Users/charles/Desktop/TeamCity/buildAgent/work/19045f4c63bc432e/build/pydio-agent/out01-PYZ.pyz/pydio.job.change_stores", line 641, in stat_path KeyError: u'/Private/Pictures/Something/IMG_1835.jpg' 2019-08-06 22:22:50 ERROR 123145479970816 Thread-1 pydio.job.change_stores : 643 | stat_path | u'/Private/Pictures/Something/IMG_1839.jpg'
Traceback (most recent call last): File "/Users/charles/Desktop/TeamCity/buildAgent/work/19045f4c63bc432e/build/pydio-agent/out01-PYZ.pyz/pydio.job.change_stores", line 641, in stat_path KeyError: u'/Private/Pictures/Something/IMG_1839.jpg' 2019-08-06 22:22:51 ERROR 123145479970816 Thread-1 pydio.job.change_stores : 643 | stat_path | u'/Private/Pictures/Something/IMG_1841.jpg'
Traceback (most recent call last): File "/Users/charles/Desktop/TeamCity/buildAgent/work/19045f4c63bc432e/build/pydio-agent/out01-PYZ.pyz/pydio.job.change_stores", line 641, in stat_path KeyError: u'/Private/Pictures/Something/IMG_1841.jpg'

mcr-ksh commented 5 years ago

I was just able to confirm that v1.2.8 does not have this Problem and detected the > 1000 changes and is uploading, while the 2.2.0 failed around 100 changes.