Waitng on mounts...

[Ruta25]

Please assist with this :) I've tried reboot, reinstallin rclone, nothing works

Giving your system a nice little spring cleaning...

Updating Gooby

Cloning into '/opt/.Gooby'... remote: Enumerating objects: 206, done. remote: Counting objects: 100% (206/206), done. remote: Compressing objects: 100% (206/206), done. remote: Total 10830 (delta 115), reused 0 (delta 0), pack-reused 10624 Receiving objects: 100% (10830/10830), 1.90 MiB | 1.31 MiB/s, done. Resolving deltas: 100% (7316/7316), done.

Shutting everything down

Removing network docker_default WARNING: Network docker_default not found.

Updating Rclone if possible

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4426 100 4426 0 0 4488 0 --:--:-- --:--:-- --:--:-- 4484 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15 100 15 0 0 12 0 0:00:01 0:00:01 --:--:-- 12

The latest version of rclone rclone v1.48.0 is already installed.

Cleaning mount leftovers

mountpoint: /mnt/google: No such file or directory

Calling on Upgrade Service

Your system has already been upgraded... prodeeding

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15 100 15 0 0 28 0 --:--:-- --:--:-- --:--:-- 28 done Waiting on mounts to be created... Waiting on mounts to be created...

[TechPerplexed]

Well that's strange... it just suddenly started happening to you? Let's see where your process halts :)

Can you please try these commands one by one:

sudo systemctl daemon-reload
sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
sudo systemctl start rclonefs

And then see if your media shows in /mnt/rclone?

[Ruta25]

perplexed@PlexPod:/root$ sudo systemctl start rclonefs Job for rclonefs.service failed because the control process exited with error code. See "systemctl status rclonefs.service" and "journalctl -xe" for details. perplexed@PlexPod:/root$ sudo systemctl status rclonefs.service ● rclonefs.service - Mount Google Drive (Rclone) Loaded: loaded (/etc/systemd/system/rclonefs.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2019-07-30 08:19:25 PDT; 24s ago Process: 14186 ExecStartPre=/bin/mkdir -p ${RCLONEMOUNT} (code=exited, status=1/FAILURE) Main PID: 16700 (code=exited, status=2)

Jul 30 08:19:25 PlexPod systemd[1]: Failed to start Mount Google Drive (Rclone). Jul 30 08:19:25 PlexPod systemd[1]: rclonefs.service: Service hold-off time over, scheduling restart. Jul 30 08:19:25 PlexPod systemd[1]: rclonefs.service: Scheduled restart job, restart counter is at 5. Jul 30 08:19:25 PlexPod systemd[1]: Stopped Mount Google Drive (Rclone). Jul 30 08:19:25 PlexPod systemd[1]: rclonefs.service: Start request repeated too quickly. Jul 30 08:19:25 PlexPod systemd[1]: rclonefs.service: Failed with result 'exit-code'. Jul 30 08:19:25 PlexPod systemd[1]: Failed to start Mount Google Drive (Rclone).

[TechPerplexed]

Also, can you try this:

sudo chown -R $USER:$USER /mnt

And then run the above 4 lines again please?

[Ruta25]

It gives me this error unfortunately:

chown: cannot access '/mnt/rclone': Transport endpoint is not connected

[TechPerplexed]

Ok, run this first:

sudo systemctl daemon-reload
sudo systemctl stop mergerfs
sudo systemctl stop rclonefs

Then verify that your /mnt doesn't contain a folder /mnt/rclone nor /mnt/google (but it SHOULD have /mnt/local and /mnt/uploads)

[Ruta25]

Still no luck

perplexed@PlexPod:/root$ sudo systemctl daemon-reload perplexed@PlexPod:/root$ sudo systemctl stop mergerfs perplexed@PlexPod:/root$ sudo systemctl stop rclonefs perplexed@PlexPod:/root$ sudo chown -R $USER:$USER /mnt chown: cannot access '/mnt/rclone': Transport endpoint is not connected perplexed@PlexPod:/root$

In the /mtn folder only google local and uploads are visible. /mnt/rclone is not there

[TechPerplexed]

Try this:

/bin/fusermount -uz /mnt/rclone and /bin/fusermount -uz /mnt/google

Does that allow you to perform the 3 lines and then the sudo chown command?

[Ruta25]

Yes thank you! How do I get it back up working again?

perplexed@PlexPod:/root$ /bin/fusermount -uz /mnt/rclone perplexed@PlexPod:/root$ /bin/fusermount -uz /mnt/google /bin/fusermount: entry for /mnt/google not found in /etc/mtab perplexed@PlexPod:/root$ sudo systemctl daemon-reload perplexed@PlexPod:/root$ sudo systemctl stop mergerfs perplexed@PlexPod:/root$ sudo systemctl stop rclonefs perplexed@PlexPod:/root$ sudo chown -R $USER:$USER /mnt perplexed@PlexPod:/root$

[TechPerplexed]

Great, now let's see if rclone is less stubborn now :)

sudo systemctl start rclonefs

[Ruta25]

Ran the command, no errors given but when manually triggering upload:

Are you sure you want to trigger upload (y/N)?

--->

This will manually trigger your uploads. This process runs every 15 minutes regardless!

Starting sync at Tue Jul 30 08:46:58 PDT 2019 find: Failed to restore initial working directory: /root: Permission denied find: Failed to restore initial working directory: /root: Permission denied 2019/07/30 08:46:58 Failed to rc: connection failed: Post http://localhost:5572/core/stats: dial tcp [::1]:5572: connect: connection refused Queuing /XxXXXXX/XXXXXXXXXX.mkv of size 689M mkdir: cannot create directory ‘/mnt/rclone’: Transport endpoint is not connected 2019/07/30 08:46:58 Failed to rc: connection failed: Post http://localhost:5572/operations/movefile: dial tcp [::1]:5572: connect: connection refused Finished at Tue Jul 30 08:46:58 PDT 2019

---

trigger upload completed

---

All done! Press Enter to return to the menu

[TechPerplexed]

Hah, but we're nowhere near ready yet...

First, one of the errors you're seeing is because you are still in the root folder. You can avoid it by typing cd ~

The rest of the errors you see because you don't even have MergerFS up yet.

So let's continue the troubleshooting first ;)

Do you see your media in /mnt/rclone now? If so, let's bring MergerFS up:

sudo systemctl start mergerfs

Then verify that your media is ALSO visible at /mnt/google (plus the Downloads folder)

[Ruta25]

Thank you for being so patient with me! New to this as you can probably see

No, unable to see /mnt/rclone

When trying to start rclone getting this error again: perplexed@PlexPod:~$ sudo systemctl start rclonefs Job for rclonefs.service failed because the control process exited with error code. See "systemctl status rclonefs.service" and "journalctl -xe" for details.

[TechPerplexed]

Haha no worries, but yeah we need to sort one thing at the time for sure :) You're still getting errors when you run the 4 original commands?

sudo systemctl daemon-reload
sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
sudo systemctl start rclonefs

[Ruta25]

Unfortunately yes

perplexed@PlexPod:~ sudo systemctl daemon-reload perplexed@PlexPod:~ sudo systemctl stop mergerfs perplexed@PlexPod:~ sudo systemctl stop rclonefs perplexed@PlexPod:~ sudo systemctl start rclonefs Job for rclonefs.service failed because the control process exited with error code. See "systemctl status rclonefs.service" and "journalctl -xe" for details. perplexed@PlexPod:~ systemctl status rclonefs.service ● rclonefs.service - Mount Google Drive (Rclone) Loaded: loaded (/etc/systemd/system/rclonefs.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2019-07-30 09:04:42 PDT; 26s ago Process: 22609 ExecStartPre=/bin/mkdir -p ${RCLONEMOUNT} (code=exited, status=1/FAILURE) Main PID: 20121 (code=exited, status=2) perplexed@PlexPod:~$

[TechPerplexed]

Can you share the content of your .env file? nano /var/local/Gooby/Docker/.env

[TechPerplexed]

Thanks, I have deleted it for privacy reasons :) Let me look at it and get back to you... but so far that looks all right.

One last thing I can think of, has it ever worked for you before? Could it be that the vendor has not enabled FUSE on your system?

[Ruta25]

Thank you! I have used the same system and reinstalled. Ever since it has given me trouble.

Thank you for the time!

[TechPerplexed]

No worries... I'm just puzzled what it could be.

Can you run these lines once more:

/bin/fusermount -uz /mnt/rclone and /bin/fusermount -uz /mnt/google

and then uninstall and reinstall Rclone from the menu just to make sure?

[Ruta25]

Did all the above, then triggered manual upload which gave no errors. Then when viewing the activity in the menu:

[TechPerplexed]

What system are you currently using?

Also, can you verify that your mounts are up and showing your files (both /mnt/rclone and /mnt/google)?

[Ruta25]

OS version : Ubuntu 18.04.2 LTS Timezone : America/Los_Angeles Domain : xxxxx Email Address : xxxxxx IP Address : xxxxxx Media location : /Media Download location : /Media/Downloads Gooby version : 2.2.0

When retrying the trigger upload:

Then checking the filesystem, rclone missing:

[bdschuster]

Also, what is rclone config showing? is there 2 listed?

[Ruta25]

Yes Gdrive and local

[TechPerplexed]

If you type rclone lsd GDrive: do you see your folders on Google?

[Ruta25]

If you type rclone lsd GDrive: do you see your folders on Google?

Yes I do :)

[TechPerplexed]

Well, I'm getting stumped. So when you run the entire sequence again:

sudo systemctl daemon-reload
sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
/bin/fusermount -uz /mnt/google
/bin/fusermount -uz /mnt/rclone
sudo chown -R $USER:$USER /mnt
sudo systemctl start rclonefs

Then what error are you getting exactly? Still the same error as before? And still no mount at /mnt/rclone?

[bdschuster]

@Ruta25 , any update? I would like to try something, but want to see what errors you get from the above @TechPerplexed requested.

[Ruta25]

Typed in the above as is: , then rclone folder shows up. Start mergerfs, then rclone disappears and google folder shows up.

Then triggered manual upload, all went well. Go into rclone activity and get this error again:

Then, I go back and try to trigger upload again but then this error shows up:

So it seems as if everything goes well, up until the first file starts uploading and then rclone dismounts/malfunctions.

[bdschuster]

Ok, one thing before I start going crazy and make you delete a bunch of stuff...lol...can you log out of your session, so completely drop the SSH connection, then re-connect as perplexed (not root then switch to perplexed, just login as perplexed), just run sudo nano and then enter, does it prompt you for a password?

[TechPerplexed]

Also, what do you mean by "then rclone folder shows up. Start mergerfs, then rclone disappears and google folder shows up."?

Please forget about the upload for a while, until we sort the underlying problem that is not your issue just yet ^_^

[Ruta25]

Thanks for all the assistance. I’ll run everything when I’m back from work tonight :)

[Ruta25]

Ok, one thing before I start going crazy and make you delete a bunch of stuff...lol...can you log out of your session, so completely drop the SSH connection, then re-connect as perplexed (not root then switch to perplexed, just login as perplexed), just run sudo nano and then enter, does it prompt you for a password?

This screen came up without asking for password

[Ruta25]

Well, I'm getting stumped. So when you run the entire sequence again:

sudo systemctl daemon-reload
sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
/bin/fusermount -uz /mnt/google
/bin/fusermount -uz /mnt/rclone
sudo chown -R $USER:$USER /mnt
sudo systemctl start rclonefs

Then what error are you getting exactly? Still the same error as before? And still no mount at /mnt/rclone?

After running this sequence again, the rclone, uploads and local folders show up in /mnt and I can see my files in the rclone folder and the pending uploads in the uploads folder.

Then when I run sudo systemctl start mergerfs, I see all 4 folders that is supposed to be there. Then when you wait a few minutes I can only see the folders google, uploads and local folders in /mnt (no error messages are being generated and rclone folder is not in /mnt)

[TechPerplexed]

So, you're saying that when you create all the mounts using the above commands + the mergerfs one, everything comes up correctly (as you said, your media is in /mnt/rclone and your media + uploads is in /mnt/google).

Then, after a few minutes, without any further action from you, /mnt/rclone disappears and /mnt/google only shows the uploads folder? Did I summarize that correctly? How many minutes roughly do your mounts stay stable?

[Ruta25]

So, you're saying that when you create all the mounts using the above commands + the mergerfs one, everything comes up correctly (as you said, your media is in /mnt/rclone and your media + uploads is in /mnt/google).

Then, after a few minutes, without any further action from you, /mnt/rclone disappears and /mnt/google only shows the uploads folder? Did I summarize that correctly? How many minutes roughly do your mounts stay stable?

Yes exactly! Approx 2 mins? I'm redoing everything now to see if I can recreate the problem.

[Ruta25]

Just re-ran all commands, all 4 folders came up, and then approx two minutes in the rclone folder disappears.

[TechPerplexed]

Ok, can you please try this... Just to help us pinpoint what could be wrong.

Run all the above commands once again, but this time do NOT fire up MergerFS. So basically stop at starting rclonefs. Does your /mnt/rclone still drop after a few minutes?

[Ruta25]

Ok, can you please try this... Just to help us pinpoint what could be wrong.

Run all the above commands once again, but this time do NOT fire up MergerFS. So basically stop at starting rclonefs. Does your /mnt/rclone still drop after a few minutes?

Ran these commands: (didn't start mergerfs) - and rclone appeared and could see files inside the folder.

Then two minutes later:

[bdschuster]

@Ruta25 Ok, so now we have narrowed it down to rclone. Can you please run sudo cat /var/log/syslog | grep rclone and post the output. also cat ~/logs/rclone.log and post that output as well please.

[Ruta25]

syslog's and rclone entries: Terminal Saved Output.txt

[bdschuster]

@Ruta25 , ok, can I see cat /var/local/Gooby/Docker/.env and cat /etc/systemd/system/rclonefs.service sorry for all the workaround, just trying to get this sorted out as this is not happening on any test boxes for me.

[Ruta25]

Thank you for all the help. Sorry for the delay but was not at home

Docker env.txt rclonefs_service.txt

[bdschuster]

well, everything looks good there. Is this a first time setup on this server?

Also, can I get the output of sudo lshw and df -h

[Ruta25]

No it ran flawlessly for more than a year, then it became slow so I decided to reinstall the OS (which was probably not the smartest thing to do) lshw.txt df-h.txt

[bdschuster]

@Ruta25 , is this a Physical server? Or are you using a VPS?

[Ruta25]

@Ruta25 , is this a Physical server? Or are you using a VPS?

It's a VPS. Would that cause the problem since it worked previously without a problem?

[bdschuster]

This is meant to work with VSP’s, just trying to rule out any hardware failure, like RAM since it is giving you and OOM error.

Ok, so at this point, would it be a big deal to completely wipe the server and start over fresh? Also, do you have the option with your VSP to install Debian instead of Ubuntu? We have been seeing a lot of issues with Ubuntu lately, I actually made the switch to Debian and never looked back, runs smooth.

[Ruta25]

That would not be a problem! I'll start doing that right now :) What version of Debian should I install?

[bdschuster]

I’m using 9, no issues here. Go with that.

[Ruta25]

I am not familiar with Debian at all When running this: sudo wget http://bit.ly/GetGooby -O /tmp/install.sh && sudo bash /tmp/install.sh

I get this: -bash: sudo: command not found