mount: improve implementation of --daemon mode

[ivandeex]

References

This ticket:

• supersedes (?) #2618
• draws ideas from https://github.com/rclone/rclone/issues/2968#issuecomment-716286668
• and discussion at https://github.com/rclone/rclone/issues/2968#issuecomment-716679470 till https://github.com/rclone/rclone/issues/2968#issuecomment-728309447
• is related to #4764

Also related to a number of console password input tickets - see Password prompts in a comment below.

What problem are you are trying to solve?

Looking at the rclone initialization code from the --daemon point of view I can see stuff that should run in the child only and not in parent:

• rc-server - if enabled, it will bind to port in parent and cause "cannot bind" in daemonized child
• NewFsDir - it chains to backend-specific, unconstrained, generally fork-unsafe Fs.NewFs

This calls for refactoring, esp. initConfig can be split into a function common to all commands to be used in cobra.OnInitialize and into another function to contain child-only steps. Most commands would inherit it from rootCmd.PersistentPreRun. Daemonizing commands would disable it by PersistentPreRun: func(){return nil} and then call from child... just an idea

How do you think rclone should be changed to solve that?

We might refactor backgrounding to have rughly 3 stages :

• parent pre-init will run a hand-picked subset of fork-safe operations from current cmd.initConfig, before callsingos.StartProcess
• child setup will take place in the child performing other unsafe operations from initConfig e.g. start rc-server, calls fs.NewFs, set up vfs, start fuse mount and return its status to parent. This stage closely mimics current non-daemonizing flow. At this stage child has fully working stdin/stdout/stderr (either parent redirects child streams through iself or perhaps the child just sits attached to the parent's terminal session) and can log debug/error messages to stderr or request user input from stdin. Parent just pipes child's streams and waits for status or timeout.
• child detach - parent receives daemonization/mount status from child (or detects timeout), logs it and exits, while child:
  • finishes sending status to parent as a specially formatted line on stdout/stderr or via dedicated pipe,
  • replaces stdin/stdout/stderr by /dev/null,
  • stops output from stats and pacers,
  • closes extra files and pipes,
  • calls setsid(),
  • probably does setpgrp (detaches from process group) to ensure it will be later reaped by pid 1 when it exits or aborts,
  • reopens log file descriptor and syslog connection if needed (because if any file descriptors remain from parent, the child later ends up as a zombie),
  • probably detaches itself from original cgroup (if it helps with systemd),
  • proceeds with business as usual!

As a consequence, we have to:

• triage initConfig contents (it's rather small) between stages 1 and 2;
• think of switching child logger between file and redirected stdout(err);
• rethink config key;
• deactivate systemd log format in child;
• parent should issue warnings in advance if child's log file or stdin helper isn't configured.
• refactor all ReadLine, ReadPassword, fmt.Scanf and alikes in the core including fs/config and in backends (where they usually read 2fa secrets) into a common wrapper that can:
  • read terminal when available (and optionally hide user input if it's secret), or
  • read stdin (prompt on stdout) piped from parent at stage 2, or
  • use universally configured helper executables(files) when detached, or
  • log errors when not configured;

Another idea is to keep the StartDaemon where it was already butuse a sort of sync mechanism between the parent process and the child process (pipe?).

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ivandeex]

A recent --daemon patch https://github.com/rclone/rclone/commit/8b8a943dd82bd16ea749e92645ba0941c82c456a introduced the fs.IsDaemon function which is true in the child. Now we need fs.IsParent to detect early whether daemonizing was requested.

Another approach is to let some facilities activate early in the parent but then shutdown them right before we spawn a child. Just another idea :)

[ivandeex]

Password prompts in Backgound process

Interactive input from user on stdin/stdout is used by rclone core to get config key and by many backends to update such stuff as 2-factor authentication or storage pins.

The child process lacks stdin so it could request passwords piped from parent's stdin then return status code to parent, but the child might need secrets after parent exits, so..

Generally daemonization calls for a common rclone API and configuration options to allow for obtaining secrets from a wider range of sources:

• parent's stdin via pipe (while parent runs),
• helper executables,
• external files,
• user provided pipe/FIFO,
• in the parent - from good old stdin, plus: guarded by a common lock.

The API patch is expected to be intrusive - it will affect code in a lot of backends.

References:

• 524

• 3386

• 2346

• 5027

• other console input tickets

[CyberPoison]

I'm having some issue whenre runing the --daemon flag as i see in the Patch the is a new flag --daemon-wait but my question is how can i mount a google drive with rclone mount such as i described here #6173 i have some issues when running daemon flag there is any work arround on this ?

My rclone version is rclone: Version "v1.58.1" And does i need to upgrade or downgrade to some version to get it working or just i need to run some specfic new flag ?

[CyberPoison]

For those who have the same issue this is how i deployed my rclone as daemon.

echo "Starting Mounts ..." && sleep 5 && bash -c 'setsid sudo rclone mount --allow-other --dir-cache-time 24h --vfs-read-chunk-size 124M --vfs-read-chunk-size-limit 4G --buffer-size 512M --vfs-cache-mode writes Cloud\ Storage\ Server\ 1\ Uncrypt:/ /mnt/Cloud\ Storage/Cloud\ Storage\ Server\ 1/ </dev/null &>/dev/null & jobs -p %1' && echo "Done..." && sleep 5

[psb777]

It all must be a lot simpler than some would have it, fixing this problem. When I run mount from the command line and run it in the background "&" using a "--log-file=x" option rclone mount works fine.

There are well documented standard techniques for running any program as a daemon, and there exist Unix/Linux utilities to do this. There is no need to have a --daemon flag at all. One such tool is this

https://linux.die.net/man/1/daemonize

It's always pleasant to fix a bug by removing code, and here is such an opportunity. Let's lose the --daemon flag. It's an o/s specific problem and the o/s should handle it, or a simple o/s specific utility will. Next we'll be embedding output pagination in rclone like they do in systemd, rather than using less/more/pg.

[ncw]

Maybe we should document using daemonize as an alternative to the --daemon flag? That might be a good first step.

Fancy sending a PR to that effect @psb777 ?

[psb777]

Well, maybe, except that the use of daemonize to turn any process into a daemon is the same for any process, and the doc on how so to do is man daemonize. I note there exist other similar utilities like such as the predictably named daemon. And my technique, of using a logfile and putting rclone into the background using the /bin/[zkcr]sh command line suffix & just works. So, whereas I have installed daemonize reading the manual is just too much work I'm not even using it.

[nathanshearer]

I have run into a problem where rclone mount is processing the vfs and queuing up changes. While this is happening, something times out and the process is killed:

DEBUG : Daemon timed out. Terminating daemon pid

As a temporary workaround, I am running rclone in the foreground and backgrounding that process:

RCLONE_ARGS=(
    --log-file="$LOGFILE"
    --cache-dir /mnt/local01/rclonecache
    mount
    googledrive:/
    --bwlimit 8M
    --drive-impersonate "$ACCOUNT"
    --drive-use-trash=false
    #--fast-list # --fast-list does nothing on a mount
    --vfs-cache-mode full
    --vfs-cache-poll-interval 10m
    --vfs-cache-max-size 8G
    --vfs-cache-max-age 10h
    --vfs-write-back 1h
    --allow-other
    --umask 0
    #-vv
    /mnt/googledrive
)
rclone "${RCLONE_ARGS[@]}" 2>&1 >> "$LOGFILE" &

I have an unusual setup where I keep items in the cache for a long period. I have sparse block devices stacked on top of my rclone mount (it works great!), but the block devices and arbitrary filesystem background tasks can cause a lot of IO, so I have a large 1TB local cache to prevent thrashing the google server. The read+write to/from google is minimal and serialized.

[Masterxilo]

I have the same issue here, mounting a Google Drive.

Using --daemon gives

<7>DEBUG : Daemon timed out. Terminating daemon pid 79084
Fatal error: mount not ready

while it works just fine without --daemon... the feature is broken; rclone v1.63.1

[ncw]

Try adjusting --daemon-wait - if you use --daemon-wait 0 then I think rclone won't wait for the mount to come up at all so it will come up in the background.

[Masterxilo]

thanks @ncw, adding both "--daemon" and --daemon-wait 0 restored the behavior I observed with older rclone versions, the mount comes up just fine now and the program runs in the background