Closed Dulanic closed 7 years ago
When did you run bash <( curl -Ls https://github.com/madslundt/cloud-media-scripts/raw/master/INSTALL )
?
Try run bash setup.sh
So I ran the curl part first and I tried to load it to /cloud to store it there. That has not worked so far, I know it failed due to permission so I did sudo su and ran it and that works, but then the mount fails.
This may take a while because Plexdrive needs to cache your files setup.sh: line 120: /cloud/scripts/mount.remote: Permission denied Mount is up and running
I tried to run the mount manually and it failed still /w root saying permission denied.
Tried again without using the /cloud after and not I am getting permission denied again?
2017/10/10 16:16:25 mount helper error: fusermount: failed to chdir to mountpoint: Permission denied Plexdrive has successfully been updated
Tried /w sudo and still and error...
This may take a while because Plexdrive needs to cache your files
setup.sh: line 120: /cloud/cloud-media-scripts/scripts/mount.remote: Permission denied
Mount is up and running
I'm actually receiving the same error as well as a another.
This may take a while because Plexdrive needs to cache your files
/opt/cloud-media-scripts/scripts/mount.remote: 42: [: 7: unexpected operator
[ 2017-10-10@15:55:21 ] Mounting Google Drive mountpoint: /opt/cloud-media-scripts/.cloud-encrypt
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
^C
root@PleXXX:/opt/cloud-media-scripts# ls
config cron INSTALL LICENSE plexdrive rclone README.md scripts setup.sh uml_diagram.png
root@PleXXX:/opt/cloud-media-scripts#
OK found some of the problems... #1 it downloads some files to the wrong spots if you dont run it in the right spot. So If you run the curl from /opt then run the command it outputs, its wrong. It needs to be run from /opt/cloud not /opt.
It says to run this while your in opt. bash /opt/cloud/setup.sh
OK found some of the problems... #1 it downloads some files to the wrong spots if you dont run it in the right spot. So If you run the curl from /opt then run the command it outputs, its wrong. It needs to be run from /opt/cloud not /opt.
It says to run this while your in opt. bash /opt/cloud/setup.sh
Are you not using /opt/cloud-media-scripts/ for your folder? Try renaming your folder to that.
mv /opt/cloud/ /opt/cloud-media-scripts/
Then try running bash /opt/cloud-media-scripts/setup.sh
I still think the mount.remote script may have some issues, but I haven't had much time to scour it since I'm at work right now.
I use it /w cloud because the script allows that. I'll explain more in a few... Edit config file [/opt/cloud-media-scripts/config] and run setup.sh [bash /opt/cloud-media-scripts/setup.sh]
By default this will place cloud-media-scripts in the directory ./cloud-media-scripts. An extra argument can be added to change this
bash <( curl -Ls https://github.com/madslundt/cloud-media-scripts/raw/master/INSTALL ) [PATH]
Edit: OK here is my clarification. If you are in /opt and run the curl command it loads the data. At the end it says...
Edit config file [/opt/cloud-media-scripts/config] and run setup.sh [bash /opt/cloud-media-scripts/setup.sh]
It says this and returns you to /opt. So if you then just run bash /opt/cloud-media-scripts/setup.sh it loads SOME of the files into /opt instead of /opt/cloud-media-scripts/. So a catch or clarification needs to be done so you do NOT run it from /opt causing rclone & plexdrive to get pulled to /opt and then fail when it goes to cp them later in the script.
This doesn't address the other issues we are seeing. But quickly addresses calling the script from a dir other than from within the cloud-media-scripts folder.
Oh I see what you are saying. He needs to have his script changed then to contain $1 for the path and then have that referenced in the script so it loads that parameter in. If that is what you are referring to.
So it looks like it is continually looping this while loop:
while [ -z "$(ls -A ${cloud_encrypt_dir})" ]
do
echo "Waiting for mount ${cloud_encrypt_dir} ..."
sleep 30
done
That looks is normal, it's waiting for the dir to mount. The problem is it isn't mounting. For me I just see "/opt/cloud/setup.sh: line 121: /opt/cloud/scripts/mount.remote: Permission denied " I was able to get around that chmod +x all the scripts. But the rclone config is not sticking, I am 1000% sure I have the right password but it won't mount
2017/10/10 19:26:37 Failed to create file system for "local-crypt:": failed to decrypt password: base64 decode failed: illegal base64 data at input byte 0 Waiting for mount /opt/cloud/.cloud-decrypt ...
Added another push request because I found out the permission problem, it needs chmod +x because the files by default aren't executable. Sure the files themselves instead could be fixed.
Yea I had already made them executable, I wasn't getting an error like that either. Mine just kept looping, more of seemed like something wasn't responding. I thought maybe it was plexdrive, im not sure. Had a bit to drink, going to continue troubleshooting tomorrow morning.
Hey guys.
Thank you for your patience and specially thank you for your debugging skills. I'll look into the setup.sh when running from a different folder and try to enforce the problems on my own machine. I'll try to fix it today or tomorrow.
Again thanks a lot
I'm not home at the moment so it's not easy for me to test. Would you mind trying again with the easy install? I think I've fixed the problem with permissions
chmod: cannot access '/opt/cloud/scripts/*': No such file or directory
Might be the order in the script run? Because the chmod works fine when I run it manually. Check my original pull request, it works there.
OK so with manually running the chmod when it asks me to run rclone config, I got around that point. Now it stalled on .cloud-decrypt.
[ 2017-10-11@07:32:31 ] Mounting Google Drive mountpoint: /opt/cloud/.cloud-encrypt
[ 2017-10-11@07:32:36 ] Mounting decrypted Google Drive: /opt/cloud/.cloud-decrypt
My rclone.conf is:
[gd]
type = drive
client_id = <CLIENT_ID>
client_secret = <CLIENT_SECRET>
token = {"access_token":"ya29.GlvhBHNAunhVbv5KNwRzALfiWPykXp3mF6yodq6I-51oX44G8$
[gd-crypt]
type = crypt
remote = gd:
filename_encryption = standard
password = *redacted*
password2 =
[local-crypt]
type = crypt
remote = /opt/cloud/.cloud-encrypt
filename_encryption = standard
password = *redacted*
password2 =
Edit: Let remote.mont run longer and it just runs forever...
Waiting for mount /opt/cloud/.cloud-decrypt ... Waiting for mount /opt/cloud/.cloud-decrypt ... Waiting for mount /opt/cloud/.cloud-decrypt ... Waiting for mount /opt/cloud/.cloud-decrypt ... Waiting for mount /opt/cloud/.cloud-decrypt ...
Okay so far so good. I'll look into it as soon as I get home and have some time for it.
I'll also make sure it doesn't run forever but at least gives an error after X tries.
So I got home quick and fixed the error with permissions.
I'll look into the waiting for mount
So seems just won't mount cloud-encrypt now. I've left it for long periods of time and no progress.
This may take a while because Plexdrive needs to cache your files
[ 2017-10-11@11:13:43 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud/.cloud-encrypt You are not the owner of /var/run/screen/S-root. [ 2017-10-11@11:13:43 ] Mounted successfully with Plexdrive Waiting for mount /opt/cloud/.cloud-encrypt ... Waiting for mount /opt/cloud/.cloud-encrypt ... Waiting for mount /opt/cloud/.cloud-encrypt ... Waiting for mount /opt/cloud/.cloud-encrypt ... Waiting for mount /opt/cloud/.cloud-encrypt ... Waiting for mount /opt/cloud/.cloud-encrypt ... Waiting for mount /opt/cloud/.cloud-encrypt ...
I see I forgot to change fuse.conf. If you pull the newest down it should work now.
Any reason you aren't using Plexdrive 5? Seems to remove the need for MongoDB since it uses an internal DB in itself. Causes more RAM usage tho, but that's not a problem for me.
Seems to still be waiting for mount.
This may take a while because Plexdrive needs to cache your files
###### Start Mount ######
[ 2017-10-11@15:31:56 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud-media-scripts/.cloud-encrypt
[ 2017-10-11@15:31:56 ] Mounted successfully with Plexdrive
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
Waiting for mount /opt/cloud-media-scripts/.cloud-encrypt ...
I have just a test folder on my Drive and nothing else, so I don't think it would take a long time for it to "Cache" my files.
Same for me, keeps waiting for cloud-encrypt
OK think I got my problem. I was running it in sudo su mode. I chowned /opt/cloud and it ran fine.
###### Start Mount ######
[ 2017-10-11@17:22:44 ] Mounting Google Drive with Plexdrive in mountpoint: /cloud-encrypt
[ 2017-10-11@17:22:44 ] Mounted successfully with Plexdrive
[ 2017-10-11@17:22:49 ] Mounting decrypted Google Drive with Rclone in mountpoint: /cloud-decrypt
[ 2017-10-11@17:22:49 ] Mounted successfully with Rclone
[ 2017-10-11@17:22:49 ] Final library mountpoint: /local-media already mounted.
###### Mounted successfully ######
I'm running it as root, but I think the issue on mine is this god damn MongoDB. I cannot get the authentication working with it. I've created an admin account and it still is giving me errors...
2017-10-11T16:33:08.043-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8db29BQ19WRjAyWm8" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.043-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dQWFGUlRWanlQWUE" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.043-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dWm55S1MyUGdYX00" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.043-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dSWhIdUZMRU5CV00" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.044-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dN3hmZjZLOGUyUEU" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.044-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8deFNqVWFpMGl0TE0" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.044-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dM0JzeVZERlRhdGs" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.044-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dOURJLS1zRmlYR0E" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.044-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { delete: "api_objects", deletes: [ { q: { _id: "0B7irD2yyQw8dRXBQVk95LVpad0E" }, limit: 1 } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.045-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { update: "api_objects", updates: [ { q: { _id: "0B7irD2yyQw8dQzFwZEhLSENZR28" }, u: { _id: "0B7irD2yyQw8dQzFwZEhLSENZR28", name: "test-decrypt", isdir: true, size: 0, lastmodified: new Date(1507679894657), downloadurl: "https://www.googleapis.com/drive/v3/files/0B7irD2yyQw8dQzFwZEhLSENZR28?alt=media", parents: [ "0B7irD2yyQw8dRXBQVk95LVpad0E" ], cantrash: true }, upsert: true } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.045-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { update: "api_objects", updates: [ { q: { _id: "0B7irD2yyQw8dYzIwemFkWHFmSFE" }, u: { _id: "0B7irD2yyQw8dYzIwemFkWHFmSFE", name: "test-encrypt", isdir: true, size: 0, lastmodified: new Date(1507679885023), downloadurl: "https://www.googleapis.com/drive/v3/files/0B7irD2yyQw8dYzIwemFkWHFmSFE?alt=media", parents: [ "0B7irD2yyQw8dOURJLS1zRmlYR0E" ], cantrash: true }, upsert: true } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.045-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { update: "api_objects", updates: [ { q: { _id: "0B7irD2yyQw8dMVNzc0dKNzhXc2c" }, u: { _id: "0B7irD2yyQw8dMVNzc0dKNzhXc2c", name: "test", isdir: true, size: 0, lastmodified: new Date(1507742199732), downloadurl: "https://www.googleapis.com/drive/v3/files/0B7irD2yyQw8dMVNzc0dKNzhXc2c?alt=media", parents: [ "0B7irD2yyQw8dSEhaalV0NGZwajg" ], cantrash: true }, upsert: true } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.045-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { update: "api_objects", updates: [ { q: { _id: "0B7irD2yyQw8dSEhaalV0NGZwajg" }, u: { _id: "0B7irD2yyQw8dSEhaalV0NGZwajg", name: "PleXXX", isdir: true, size: 0, lastmodified: new Date(1507657493131), downloadurl: "https://www.googleapis.com/drive/v3/files/0B7irD2yyQw8dSEhaalV0NGZwajg?alt=media", parents: [ "0ALirD2yyQw8dUk9PVA" ], cantrash: true }, upsert: true } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:08.046-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { update: "page_token", updates: [ { q: { _id: "t" }, u: { _id: "t", token: "4543" }, upsert: true } ], writeConcern: { getLastError: 1 }, ordered: true }
2017-10-11T16:33:36.327-0600 I ACCESS [conn1] Unauthorized: not authorized on plexdrive to execute command { find: "api_objects", filter: { parents: "0ALirD2yyQw8dUk9PVA" }, skip: 0 }
Something else is wrong, it's not unencrypting any of my data, so Im going to have to play around and find out whats wrong. I see no data in cloud-encrypt so that is prob the root of the problem.....
Ya I just checked my processes, there is no running rclone or plexdrive. Going to keep digging.
OK Im done for today lol I;ll work on it again tomorrow. I keep getting waiting again.
Alright, so I restored to a backup on my VM, and reinstalled mongodb. I now have moved past the waiting for mounting point to hanging on:
This may take a while because Plexdrive needs to cache your files
###### Start Mount ######
[ 2017-10-11@18:47:51 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud-media-scripts/.cloud-encrypt
[ 2017-10-11@18:47:51 ] Mounted successfully with Plexdrive
[ 2017-10-11@18:47:56 ] Mounting decrypted Google Drive with Rclone in mountpoint: /opt/cloud-media-scripts/.cloud-decrypt
I'm done for today as well. Gonna spend time with the wife.
I'm wondering if switching to Plexdrive 5 in your scripts would have any affect on potential problem areas. I think just having plexdrive have it's own internal DB would eliminate a lot of potential issues. Tomorrow I'm going to look into changing the script for PD5.
I've been attempting to swap it to plexdrive 5 /w a fork but was having issues so I manually tried it. I kept getting screen need 777 errors which is odd. I ended up just running it without screen and using & at the end instead. But then it wouldnt run the rclone properly then. Swapping to 5.0 is pretty simple just need to change a few commands and get rid of mongo options.
Why not just run as a service without screen? Just create a plexdrive.service under /etc/systemd/system/plexdrive.service.
[Unit]
Description=Plexdrive Application
[Service]
User=plexdrive (im lazy and use root)
Group=plexdrive (im lazy and use root)
ExecStart=/opt/cloud-media-scripts/plexdrive/plexdrive-linux-amd64 -then-whatever-parameters-you-need
[Install]
WantedBy=multi-user.target
Something like that, im kinda out of it right now... I would definitely double check all that lol
I don't run with Plexdrive 5 because it uses memory cache. My cache is currently on a 4 TB hard drive and Plexdrive 5 can't handle that.
I recommend installing mongodb in a docker container. I'm thinking about integrating an installer for mongodb in the scripts, but first I need to make sure the scripts are working.
I'll look into your errors again later today. Maybe rewrite it to not use screen. Thank you for your patience.
OBS. Remember to change google_drive_media_directory
if you have your media files located in a sub folder in google drive. This will point rclone to the sub folder and starts decrypt from there
OK so I made progress.... I didn't see google_drive_media_directory as it's kind of buried. Now I figured I should remove all the folders and just let the scripts take care of it.
Works great... but then I get to local-media. It shows 1 out of 3 folders only, which is Books. Movies and Shows don't show.
*redacted*@*redacted*:/local-media/Media$ ls -l
total 4
drwxr-xr-x 2 root root 4096 Oct 12 07:54 Books
However, if I run this manually, you can see I have more:
*redacted*@*redacted*:/$ sudo rclone mount --allow-other decrypt: /test &
[2] 8309
*redacted*@*redacted*:/$ cd /test/Media
*redacted*@*redacted*:/test/Media$ ls -l
total 0
drwxr-xr-x 1 root root 0 Oct 7 19:29 Books
drwxr-xr-x 1 root root 0 Oct 9 18:24 calibre
drwxr-xr-x 1 root root 0 Oct 7 19:29 Movies
drwxr-xr-x 1 root root 0 Oct 7 19:29 Shows
OK I got it, it's because mount never finishes. I thought the cron job ran, and it did, but it never finishes because the log is empty. Again tried manually and same stuff... I'm going crazy! :) It threw me off since there was the Books dir in local-media, yet I had just deleted local-media so how did it appear if it didnt mount?
`redacted@redacted:/opt/cloud/scripts$ sudo bash mount.remote
[ 2017-10-12@08:18:53 ] Mounting Google Drive with Plexdrive in mountpoint: /cloud-encrypt [ 2017-10-12@08:18:53 ] Mounted successfully with Plexdrive [ 2017-10-12@08:18:58 ] Mounting decrypted Google Drive with Rclone in mountpoint: /cloud-decrypt ` and then it stalls there.
Here is my config files, if anything stands out... rclone.conf.txt config.txt
Hi again.
I fixed the problem with Rclone hanging. I'm wondering why my Rclone doesn't hang on my server with the previous command. However I've tested it with the new command and it seems to fix it.
I've approved your Pull Request. Good job finding the error!
I'm working on expanding setup.sh to ask if mongodb should be installed as well.
Glad to figure it out, I was going crazy! I'm working on some other improvements that I'll keep submitting too. Mostly around logging. Glad to submit what I find because I plan on using this long term and the work now makes it so much easier later.
On Thu, Oct 12, 2017 at 12:52 PM, Mads Lundt notifications@github.com wrote:
Hi again.
I fixed the problem with Rclone hanging. I'm wondering why my Rclone doesn't hang on my server with the previous command. However I've tested it with the new command and it seems to fix it.
I've approved your Pull Request. Good job finding the error!
I'm working on expanding setup.sh to ask if mongodb should be installed as well.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/madslundt/cloud-media-scripts/issues/3#issuecomment-336215352, or mute the thread https://github.com/notifications/unsubscribe-auth/ACz8bvkE21y0GVaALfjCDEs_7MGCNdFtks5srlHVgaJpZM4P0YAF .
OK now I have a new problem :( So it properly mounts /cloud-decrypt and /cloud-encrypt but /local-media is back to missing some folders. I went into webmin and I see it's back to mouting it to /opt/cloud/media even though the first run loaded it to /local-media/Media.
/opt/cloud/media | FUSE.UNIONFS | unionfs | 99 % | Yes | No
OK next issue :) Can something be added to allow others write access that arent root? My docker containers run under my UID (1000) and they get denied access when writing now. Is this unionfs?
There is deff something up /w the script/dir permissions. The cron job won't work for me, either as my userid or as root. I have to boot up then run sudo bash mount.remote. So it works, until I need to reboot.
*redacted*@*redacted*:/local-media/Media/Shows$ echo 1234 >> test
-bash: test: Permission denied
Wonder if this is going to work either.... I just realized after I boot up I can't see my /local-media/Media/Shows etc... until I restart plex. Which tells me this will have the same problem /w seeing mounts that reconnect...
I have a question about this entire thing that maybe one of you can answer. I'm not entirely sure what each directory is doing?
# DIRECTORIES
###############################################################################
media_dir="/opt/cloud-media-scripts"
# Cloud directories
cloud_encrypt_dir="${media_dir}/.cloud-encrypt" <--Seems to be the folder that contains GD Encrypted Mount
cloud_decrypt_dir="${media_dir}/.cloud-decrypt" <--Seems to be the decrypted files from GD Mount
# Local directory
local_decrypt_dir="${media_dir}/.local-decrypt/media" <--Seems to be where media goes that needs to be encrypted and uploaded to GD
plexdrive_temp_dir="${media_dir}/.local-decrypt/plexdrive" <--No idea maybe caching for Plexdrive?
# Media directory (FINAL)
local_media_dir="${media_dir}/media" <--I assume a Drone like folder for Sonarr/Radarr that then copies to .local-decrypt? or its the folder that is watched for stuff to be encrypted and uploaded.
I would suggest maybe better descriptions on these configuration parameters, or atleast have a descriptions of each variables use within the config file or in a read me. Just a thought. :)
They are in the config file :grinning: They're kind of self explatory. Cloud encrypted is the cloud data still encrypted, cloud decrypted is the cloud encrypted sheet after its been decrypted by rclone. Local media is cloud decrypted (read only) and local decrypted (rw) unioned together using unionfs. There is a picture even.
Oh temp dir is exactly that it's a small cache used by plexdrive.
I will rethink the readme file on how to describe the different directories. Dulanic is completely right on what the different directories are doing.
plexdrive_temp_dir
is plexdrive's cache. This is controlled by the Plexdrive parameter --clear-chunk-max-size
or, if this is removed, it is controlled by --clear-chunk-age
(default is --clear-chunk-age=24h
).
I'll introduce some extra documentations on Rclone, Plexdrive and UnionFS parameters.
Hopefully I have this done this week, but I can't promise anything :)
Hello,
I am attempting to use this version to see if it works around the docker issue I opened. However,m when I run setup.sh it fails /w this:
No Rclone config exists setup.sh: 69: setup.sh: Bad substitution
I am running ubuntu 16.04.