crond can't see drive contents

[Chovin]

I've not been able to get this running with crond Sticking some debug stuff in rclone.sh, shows that it can see that gdrive is a remote but it can't see its contents

...
  fi
  echo "Executing => $sync_command" >> /test
  eval "$sync_command" || send

  echo "rclone listremotes ===" >> /test
  rclone listremotes >> /test
  echo "===" >> /test
  echo "rclone ls ===" >> /test
  rclone ls gdrive: >> /test
  echo "===" >> /test

) 200>/run/rclone.lock
...

from the console it seems to work fine though

bash-4.4# rclone listremotes
2019/03/08 08:32:45 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
gdrive:
bash-4.4# rclone ls gdrive:
2019/03/08 08:33:00 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
  1560010 Getting started
bash-4.4# cat test
Executing => rclone copy --drive-shared-with-me /data GDRIVE:/Backups/testing
rclone listremotes ===
gdrive:
===
rclone ls ===
===

[Chovin]

Not sure if this is related to #12

[tspicer]

How are you running cron? Using ENV or a mounted cron conf file?

[Chovin]

I’m using a cron conf file. You can see the RCLONE_SYNC_COMMAND i’m using in the cat above

[tspicer]

I just tried it and I can run with crond. If you are using your own cron file, remember that cron often will not have access to the system ENV variables. You need to make sure your cron file contains the desired variables. The simplest approach might looks something like this;

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CACHE_PREFIX={{CACHE_PREFIX}}
AWS_ACCESS_KEY_ID={{AWS_ACCESS_KEY_ID}}
AWS_SECRET_ACCESS_KEY={{AWS_SECRET_ACCESS_KEY}}
AWS_S3BUCKET={{AWS_S3BUCKET}}
*/45 * * * * /usr/bin/env foo -c /usr/bin/foo 2>&1

There are a few references around that describe cron and env variables.

[Chovin]

This is my crontab.conf

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
*/1 * * * * /usr/bin/env bash -c "/rclone.sh run"

and this is my docker-compose.yml

version: '3'
services:
  rclone:
    image: openbridge/ob_bulkstash
    environment:
      - RCLONE_CONFIG_GDRIVE_TYPE=drive
      - RCLONE_CONFIG_GDRIVE_SCOPE=drive
      - RCLONE_CONFIG_GDRIVE_SERVICE_ACCOUNT_FILE=auth.json
      - RCLONE_CRONFILE=/cron/crontab.conf
      - RCLONE_SYNC_COMMAND=rclone copy --drive-shared-with-me /data GDRIVE:/Backups/$DESTINATION_PATH
    volumes:
      - $DATA_PATH:/data:ro
      - $AUTH_JSON:/auth.json:ro
      - $CRONTAB_CONF_FILE:/cron/crontab.conf
    command: tail -f /dev/null 

what env variables would I need? Once cron runs rclone.sh, it seems to have all it needs (RCLONE_SYNCE_COMMAND). It gets the command fine. Unless I'm missing something, the behavior I'm seeing is that with cron running rclone, it does not have access to google drive contents.

[tspicer]

It does not have access to the contents of gdrive because cron does not have access to the environment variables. This is why it works when you use the command line. However, cron is not running the same usr environment. Cron essentially is running its own shell.

If you are running rclone.sh, there should be a "/cron/rclone.env" file. Can you share the contents of that?

[tspicer]

Also, one other note. I'm not sure about this path:

RCLONE_CONFIG_GDRIVE_SERVICE_ACCOUNT_FILE=auth.json

I think it should be this based on your config above

RCLONE_CONFIG_GDRIVE_SERVICE_ACCOUNT_FILE=/auth.json

[Chovin]

ah that was the issue. it was running cron from /root so couldn't see the relative path. Thank you!

as a note, I believe cron isn't having any troubles with environment variables */1 * * * * curl -fsS --retry 3 ${RCLONE_HEALTHCHECK_URL}/start && /usr/bin/env bash -c "/rclone.sh run" && curl -fsS --retry 3 ${RCLONE_HEALTHCHECK_URL} runs fine without adding RCLONE_HEALTHCHECK_URL={{RCLONE_HEALTHCHECK_URL}}

is it perhaps getting those from rclone.sh sourcing the rclone.env?

[tspicer]

Cron should not have issues per se, since there is a direct import in rclone.sh

That is why this is here:

# Create the environment file for crond
printenv | sed 's/^\([a-zA-Z0-9_]*\)=\(.*\)$/export \1="\2"/g' | grep -E "^export RCLONE" > /cron/rclone.env

source /cron/rclone.env

However, hard to know what folks are doing so I need to point out that you can't make assumptions about cron and env variables. Stackoverflow is littered with questions about this topic.

glad you got it working.