cjnaz / rclonesync-V2

A Bidirectional Cloud Sync Utility using rclone
MIT License
355 stars 39 forks source link

Unexpected path edits for SFTP remotes prevents successful runs #44

Closed jayjlawrence closed 4 years ago

jayjlawrence commented 4 years ago

First I wish to start with a thanks to Chris et al for their work on this code. I am not a python programmer so I do not have a pull request.

I am using rclonesync to sync between a few local machines but ran into unexpected issues. I am already successful in using rclone to do unidirectional sync's like this:

rclone Projects remote:Projects -l

rclonesync.py called like this fails:

rclonesync.py --first-sync Projects remote:Projects --rclone-args -l

Complaining about an unknown directory. (not sure which - the logging does not tell us) Also not sure about what rclone command is being run I added some logging to tell me what's going on.

I discover that the rclone command being invoked is:

rclone -l Projects/ remote:/Projects/

What's with those slashes!?! This is very different than I would expect given I have invoked both programs similarly. Perhaps for other remotes require the leading and trailing slashes. IMHO the "pathparse" routine transforms my paths in unexpected and undesirable ways. My vote would be to defeat these transformations and use the exact same path syntax as rclone. Alternatively this needs to be clearly documented at the outset to advise users that there are differences and what to expect. Finally the current set up prevents me from using the "home" directory of the remote and I forced to use an absolute path.

Additionally I would recommend adding to the logging for rclone_cmd and rclone_lsl. I needed an additional logging statement to output the actual command being invoked.

Although a Ruby programmer I whacked in the following line after line 186 which reads logging.error(print_msg("ERROR", "rclone lsl failed. Specified path invalid? (Line {})".format(linenum)))

added statement:

     logging.error(print_msg("ERROR", "rclone command: {}".format(process_args)))

I'm sure I didn't count the number of spaces correctly so I may have this in the wrong scope. It is meant to be part of the rclone_lsl function and not between the definitions of rclone_lsl and rclone_cmd. :-}

cjnaz commented 4 years ago

Hello JayJ, Let's try to get to the heart of the actual problem. I see that you are passing the symbolic link switch to rclone. Please run a test between your machines without the --rclone-args -l to confirm this isn't part of the issue. From your issue post I don't have enough info to help. Please post a shortened/sanitized version of the output with -v --rc-verbose --rc-verbose that gives a clue what the misbehavior is.

In your invocation rclonesync.py --first-sync Projects remote:Projects --rclone-args -l, Projects must be a subdirectory of the cwd, since this is relative path. If Projects is elsewhere then specify a relative path from the cwd, or specify the full/absolute path.

If I'm not understanding what path you are trying to specify, please clarify.

Also, please see the TROUBLESHOOTING.md, which might have a few clues of use.

cjnaz commented 4 years ago

There is no way to reference a relative path on a remote (or am I missing something?).

An rclone STFP remote is new to me. I see that the default root is the user's home path, and I see that the / that rclonesync inserts after the remote colon will screw that up. Let me think about if that slash can be cleanly removed for all cases.

cjnaz commented 4 years ago

Please check out branch v2.10. I've removed the force leading slash on remote, and implemented the keyboard interrupt handler.

cjnaz commented 4 years ago

@jayjlawrence - any feedback?

jayjlawrence commented 4 years ago

(rats - had a response and forgot to commit it)

One point about adding the trailing slashes. I do not believe they are a material concern with rclone. What I can off is that with rsync the trailing slashes are very significant. Because of that experience I am personally apprehensive with the idea of amending the user's supplied paths. I mention this strictly as an informational note that may come in handy when a type of remote is very sensitive to slashes and where they are placed.

Thank you again and I hope my Ruby-centric viewpoint (and occasional uncalled-for comment) is seen as a friendly joke ... I find Python programmers as a group so serious I j! ;-)

cjnaz commented 4 years ago

Thanks Jay. I trust that this is now working well for your SFTP remotes? Be aware that rclonesync works not so well for active filesystems. See issue #24 . I see you're syncing a SQL db. Your results may vary.

regards, cjn

cjnaz commented 4 years ago

v2.10 published