search

Jwink3101 / syncrclone

Python-based bi-direction sync tool for rclone
MIT License
150 stars 13 forks source link

Error when "-P/--progress" rclone flag is set #1

Closed CcydtN closed 3 years ago

CcydtN commented 3 years ago

When the "--progress" flag is set, it always shows "ERROR: JSONDecodeError". The same error shows when testing on an empty folder with a new config.

Environment:

Jwink3101 commented 3 years ago

Thanks for trying out syncrclone. This is not really an issue per se. since syncrclone wraps rclone, it is (a) setting output formats with its own flags and (b) capturing all input and output.

So flags that syncrclone set may conflict with ones you set to modify the output.

FWIW, while nowhere near as pretty as rclone's, syncrclone always shows a running log of transfers including rclone's progress on large files.

When I'm back at my computer, I'll add some notes to the documentation/config comments saying you can't set flags like this or --log-level, etc. There are some other flags too that'll cause issues like checksums since syncrclone does that. I can't enumerate all of them but I'll try to make it more clear that you should only bet setting flags like fast-list, transfers, etc.

I am planning to minorly revamping how progress is shown for the pre-sync stuff so it's easier to parse but that's just to be faster to read.

CcydtN commented 3 years ago

OK, I see. Thank you for answering that.

I think you can also mention the "--drive-skip-gdocs" flag in the document as Google Document in my remote (Google Drive) was deleted by syncrclone. I fix it by adding this flag. I think rclone ignores them when pulling from remote to local, but the checking in syncrclone assume the gdoc was deleted locally.

Jwink3101 commented 3 years ago

Interesting. I'll add that note too. I've never used it with gdrive so I haven't experienced it. I guess it's causing a filter to be applied asymmetrically. It's a downside to relying on separate rclone calls for listing.

Just to make certain, when you say it deleted them, it moved it to the backup directory, right?

My priorities are no data loss even if some edge case messes with the sync. So if that happened, I really want to dive down and fix it.

CcydtN commented 3 years ago

Just to make certain, when you say it deleted them, it moved it to the backup directory, right?

Yes, it moved into the backup folder. A copy with a time tag is also created cause I set "newer_tag" in conflict_mode.

Jwink3101 commented 3 years ago

Great! Like I said, my chief concern is that nothing is lost.

Let me know if the doc issue comes up or there isn't a way around it.

Also, I have no idea how many people are actively using this code but I do extensively and have been thinking of no longer calling it beta. If there's any suggestion, I'm very open to hearing them.

My biggest goal is to speed up the remote copies for backup and delete. I may thread them which is easy enough but means the user has to set threads in two places! I think I'm using the optimal set of flags to make it as fast as possible but not 100% sure.