search

l3uddz / plex_autoscan

Script to assist sonarr/radarr with plex imports. Will only scan the folder that has been imported, instead of the whole library section.
GNU General Public License v3.0
396 stars 70 forks source link

Added Rclone Crypt Support for both Drive/Teamdrive #92

Closed daghaian closed 5 years ago

clg27 commented 5 years ago

I think a few things need to be done before this is merged.

  1. @daghaian's commits have the issue of showing every single line in each file as being changed, which makes it very hard to follow what has changed in the commit. I think it's from different line endings but I don't know how to fix it.
  2. Update the readme to explain the new settings, it took me a little bit of trial and error to figure them out and the correct format.
  3. Update the sample config.json file.
  4. I didn't look at it, but make sure that config.py generates the correct config.json.

I also had to edit rclone.py to allow for the fact that my google drive root is not encrypted and my crypt is nested one level down, i.e. "My Drive/crypt/" is my encrypted folder.

My rclone.py implementation looks like this:

import os
import logging

logger = logging.getLogger("RCLONE")

class RcloneDecrypter:
    def __init__(self,binary,crypt_mappings,config):
        self._binary = binary
        if self._binary == "" or not os.path.isfile(binary):
            self._binary = os.path.normpath(subprocess.check_output(["which", "rclone"]).decode().rstrip('\n'))
            logger.info("Rclone binary path located as {}".format(binary))

        self._config = config
        self._crypt_mappings = crypt_mappings

    def decrypt_path(self, path):
        for root_dir in self._crypt_mappings:
            if root_dir in path:
                logger.info("Root directory identified as {}".format(root_dir))
                file_path = path.lstrip(root_dir)
                logger.info("Root directory {} has mapping defined in config as remote {}. Attempting to decrypt".format(root_dir,self._crypt_mappings[root_dir]))
                logger.info("Raw query is {}".format(" ".join([self._binary,"cryptdecode",self._crypt_mappings[root_dir],file_path])))
                try:
                         decoded = subprocess.check_output([self._binary,"--config",self._config,"cryptdecode",self._crypt_mappings[root_dir],file_path],stderr=subprocess.STDOUT).decode().rstrip('\n')
                except subprocess.CalledProcessError as e:
                         logger.error("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
                         return None

                decoded = decoded.split(' ',1)[1].lstrip()

                if "failed" in decoded.lower():
                         logger.error("Failed to decode path {}".format(file_path))
                else:
                         logger.info("Deccoded path of {} is {}".format(file_path,decoded))
                         return [os.path.join(root_dir,decoded)]

        return None
desimaniac commented 5 years ago
  1. Fixed the line endings using dos2unix.

Develop has now been merged to Master and Readme is now up to date with both.

2-4. need to be done for this PR.

daghaian commented 5 years ago

3-4 are both done already. I will take a look at your suggested change to support your use case

m1lkman commented 5 years ago

@daghaian my crypt is also in a subfolder. Let me know how I can help with testing. Thank you.

m1lkman commented 5 years ago

@daghaian I pulled your latest commit and here's the debug output from when I see a new file come in if it helps you at all. Thanks.

 2019-05-14 16:13:44,807 -    DEBUG - requests_oauthlib.oauth2_session [140365638137600]: Invoking 0 protected resource request hooks.
 2019-05-14 16:13:44,807 -    DEBUG - requests_oauthlib.oauth2_session [140365638137600]: Adding token {u'access_token': u'ya29.GlwJB_11rlX7hYW6-gndxco6XAv9VG1EHphQFDdXW_MmzjUJ2-Ui7ANTt9kaFWazL24GAXQGrA8oS2hA8rItLsExNPfsvp8EiiRBbfmfrtL_nf7emdo0B4fpaF1qMA', u'expires_in': 3600, u'expires_at': 1557877477.499579, u'token_type': u'Bearer', u'scope': [u'https://www.googleapis.com/auth/drive'], u'refresh_token': u'1/usbSzEw9otHyHIPefMBNAaoQu1qh2f6XMnJlGqFU7haCqblviOakVGBLEGX2eqnc'} to request.
 2019-05-14 16:13:44,807 -    DEBUG - requests_oauthlib.oauth2_session [140365638137600]: Requesting url https://www.googleapis.com/drive/v3/changes using method GET.
 2019-05-14 16:13:44,807 -    DEBUG - requests_oauthlib.oauth2_session [140365638137600]: Supplying headers {u'Authorization': u'Bearer ya29.GlwJB_11rlX7hYW6-gndxco6XAv9VG1EHphQFDdXW_MmzjUJ2-Ui7ANTt9kaFWazL24GAXQGrA8oS2hA8rItLsExNPfsvp8EiiRBbfmfrtL_nf7emdo0B4fpaF1qMA'} and data None
 2019-05-14 16:13:44,808 -    DEBUG - requests_oauthlib.oauth2_session [140365638137600]: Passing through key word arguments {'allow_redirects': True, 'params': {'pageSize': 1000, 'includeTeamDriveItems': False, 'fields': 'changes(file(md5Checksum,mimeType,modifiedTime,name,parents,teamDriveId,trashed),fileId,removed,teamDrive(id,name),teamDriveId),newStartPageToken,nextPageToken', 'supportsTeamDrives': False, 'pageToken': u'4750549', 'includeRemoved': True}, 'timeout': 30}.
 2019-05-14 16:13:44,982 -    DEBUG -    GOOGLE [140365638137600]: Request URL: https://www.googleapis.com/drive/v3/changes?pageSize=1000&includeTeamDriveItems=False&fields=changes%28file%28md5Checksum%2CmimeType%2CmodifiedTime%2Cname%2Cparents%2CteamDriveId%2Ctrashed%29%2CfileId%2Cremoved%2CteamDrive%28id%2Cname%29%2CteamDriveId%29%2CnewStartPageToken%2CnextPageToken&supportsTeamDrives=False&pageToken=4750549&includeRemoved=True
 2019-05-14 16:13:44,982 -    DEBUG -    GOOGLE [140365638137600]: Request ARG: {'params': {'pageSize': 1000, 'includeTeamDriveItems': False, 'fields': 'changes(file(md5Checksum,mimeType,modifiedTime,name,parents,teamDriveId,trashed),fileId,removed,teamDrive(id,name),teamDriveId),newStartPageToken,nextPageToken', 'supportsTeamDrives': False, 'pageToken': u'4750549', 'includeRemoved': True}}
 2019-05-14 16:13:44,983 -    DEBUG -    GOOGLE [140365638137600]: Response Status: 200 OK
 2019-05-14 16:13:44,983 -    DEBUG -    GOOGLE [140365638137600]: Response Content:
{
 "newStartPageToken": "4750552",
 "changes": [
  {
   "removed": false,
   "fileId": "1R4uPlbgmSFnaEcSlxbzywldqoldSKrAJ",
   "file": {
    "name": "384opi89i4he694f2cb2eoicnreggplpisvciudlas6v8hin1ckpik3h6audtu7qsp4a6gtloq77mgo22ss9io7erkon7qk37tk61fg4rh2ftslrrhuhf5es210hhqu607k1tn860gjuvi7prr54s38chgjtpqmsnde3r95306e6j52pivlajsl71pghlgoo8d7cc0apak41e",
    "mimeType": "application/octet-stream",
    "trashed": false,
    "parents": [
     "1kcfKJaDFIH8DU7PvFJY0S6v-m2nPyLOk"
    ],
    "modifiedTime": "2019-05-14T11:23:19.000Z",
    "md5Checksum": "d64248e5ac27d607f60c0289291ee55e"
   }
  },
  {
   "removed": false,
   "fileId": "17iLp7O_8vULAl3ai8VH216kBfHhSmtFO",
   "file": {
    "name": "pcm329ij4gskfsnj5h838rki39qtvcs06emu95ebf1d1n6mr4kpbtd7mg18mmtj0n03r6fn5cla42",
    "mimeType": "application/octet-stream",
    "trashed": false,
    "parents": [
     "1Tq7Oy75e5no16vt4Ws7wbNYVOMyKDNZc"
    ],
    "modifiedTime": "2019-05-14T23:12:46.679Z",
    "md5Checksum": "9adbddf9af036e92228e9d3a836663b6"
   }
  }
 ]
}

 2019-05-14 16:13:44,985 -    DEBUG -    GOOGLE [140365638137600]: Updated page_token: 4750552
 2019-05-14 16:13:44,986 -     INFO -    GOOGLE [140365638137600]: Processing 2 changes
 2019-05-14 16:13:44,987 -     INFO -    GOOGLE [140365638137600]: Added '1R4uPlbgmSFnaEcSlxbzywldqoldSKrAJ' to cache: 384opi89i4he694f2cb2eoicnreggplpisvciudlas6v8hin1ckpik3h6audtu7qsp4a6gtloq77mgo22ss9io7erkon7qk37tk61fg4rh2ftslrrhuhf5es210hhqu607k1tn860gjuvi7prr54s38chgjtpqmsnde3r95306e6j52pivlajsl71pghlgoo8d7cc0apak41e
 2019-05-14 16:13:44,993 -     INFO -    RCLONE [140365638137600]: Root directory identified as My Drive
 2019-05-14 16:13:44,994 -    DEBUG -    GOOGLE [140365638137600]: Ignoring u'My Drive/encrypt/l2jqdshaa72ski0r33s5neqsp8/gjnj9alu5h50m7jb2r81863tm88bf7q6bsarc58h3fc046m6bfbb8fimnl9r5ljv8gepmk42j3df0/uf0j7jt9shqokf957k1dn77dmk/384opi89i4he694f2cb2eoicnreggplpisvciudlas6v8hin1ckpik3h6audtu7qsp4a6gtloq77mgo22ss9io7erkon7qk37tk61fg4rh2ftslrrhuhf5es210hhqu607k1tn860gjuvi7prr54s38chgjtpqmsnde3r95306e6j52pivlajsl71pghlgoo8d7cc0apak41e' because it was not an allowed extension
 2019-05-14 16:13:44,995 -     INFO -    GOOGLE [140365638137600]: Added '17iLp7O_8vULAl3ai8VH216kBfHhSmtFO' to cache: pcm329ij4gskfsnj5h838rki39qtvcs06emu95ebf1d1n6mr4kpbtd7mg18mmtj0n03r6fn5cla42
 2019-05-14 16:13:44,999 -     INFO -    RCLONE [140365638137600]: Root directory identified as My Drive
 2019-05-14 16:13:44,999 -    DEBUG -    GOOGLE [140365638137600]: Ignoring u'My Drive/encrypt/61t4aqgpi4i63c5cum5017mtc0/pcm329ij4gskfsnj5h838rki39qtvcs06emu95ebf1d1n6mr4kpbtd7mg18mmtj0n03r6fn5cla42' because it was not an allowed extension
 2019-05-14 16:13:45,000 -    DEBUG -    GOOGLE [140365638137600]: Added: {}
 2019-05-14 16:13:45,000 -    DEBUG -    GOOGLE [140365638137600]: Unwanted: [u'My Drive/encrypt/l2jqdshaa72ski0r33s5neqsp8/gjnj9alu5h50m7jb2r81863tm88bf7q6bsarc58h3fc046m6bfbb8fimnl9r5ljv8gepmk42j3df0/uf0j7jt9shqokf957k1dn77dmk/384opi89i4he694f2cb2eoicnreggplpisvciudlas6v8hin1ckpik3h6audtu7qsp4a6gtloq77mgo22ss9io7erkon7qk37tk61fg4rh2ftslrrhuhf5es210hhqu607k1tn860gjuvi7prr54s38chgjtpqmsnde3r95306e6j52pivlajsl71pghlgoo8d7cc0apak41e', u'My Drive/encrypt/61t4aqgpi4i63c5cum5017mtc0/pcm329ij4gskfsnj5h838rki39qtvcs06emu95ebf1d1n6mr4kpbtd7mg18mmtj0n03r6fn5cla42']
 2019-05-14 16:13:45,001 -    DEBUG -    GOOGLE [140365638137600]: Ignored: {}
 2019-05-14 16:13:45,001 -     INFO -    GOOGLE [140365638137600]: 0 added / 0 removed / 2 unwanted / 0 ignored
 2019-05-14 16:13:45,001 -    DEBUG -    GOOGLE [140365638137600]: Finished retrieving changes from all loaded drives
m1lkman commented 5 years ago

@daghaian I think I have this mostly worked out. I'm no python programmer but check out https://github.com/m1lkman/plex_autoscan/tree/develop. The only issue I'm having is the value it's returning back to drive.py isn't the right mime type. See my error below, any advice?

2019-05-21 20:25:34,938 -     INFO -    RCLONE [139662966126336]: Deccoded path of l2jqdshaa72ski0r33s5neqsp8/jqjt88ptlmhsnbedlleuj9m16sjn1juq009e5agaoec8b2bluegg/u6coh09ul6f4ou0c85uve2cras/st0dkfekrme6hvqnk6mae6u8oj8cakv4jtbikjilo6nilnejohe6c6undfs65cnefnpscvc4mbh4b46oh9la6idqgjk8mhqv9j7ifoum24rpqegnjgrfgvqf45tlhv6gmclhr83ut6od2ej02k7uoqt0lk is Television/Avatar- The Last Airbender/Season 1/Avatar- The Last Airbender - S01E01 - The Boy in the Iceberg Bluray-1080p x264 AC3 [EN].mkv
2019-05-21 20:25:34,938 -    DEBUG -    GOOGLE [139662966126336]: decrypting u'My Drive/encrypt/l2jqdshaa72ski0r33s5neqsp8/jqjt88ptlmhsnbedlleuj9m16sjn1juq009e5agaoec8b2bluegg/u6coh09ul6f4ou0c85uve2cras/st0dkfekrme6hvqnk6mae6u8oj8cakv4jtbikjilo6nilnejohe6c6undfs65cnefnpscvc4mbh4b46oh9la6idqgjk8mhqv9j7ifoum24rpqegnjgrfgvqf45tlhv6gmclhr83ut6od2ej02k7uoqt0lk'
2019-05-21 20:25:34,938 -    DEBUG -    GOOGLE [139662966126336]: decrypted u'My Drive/encrypt/Television/Avatar- The Last Airbender/Season 1/Avatar- The Last Airbender - S01E01 - The Boy in the Iceberg Bluray-1080p x264 AC3 [EN].mkv'
 2019-05-21 20:25:34,938 -    DEBUG -    GOOGLE [139662966126336]: Ignoring [u'My Drive/encrypt/Television/Avatar- The Last Airbender/Season 1/Avatar- The Last Airbender - S01E01 - The Boy in the Iceberg Bluray-1080p x264 AC3 [EN].mkv'] because it was not an allowed mime: application/octet-stream
2019-05-21 20:25:34,939 -    DEBUG -    GOOGLE [139662966126336]: Added: {}
2019-05-21 20:25:34,939 -    DEBUG -    GOOGLE [139662966126336]: Unwanted: [u'My Drive/encrypt/Television/Avatar- The Last Airbender/Season 1/Avatar- The Last Airbender - S01E01 - The Boy in the Iceberg Bluray-1080p x264 AC3 [EN].mkv']
2019-05-21 20:25:34,939 -    DEBUG -    GOOGLE [139662966126336]: Ignored: {}
2019-05-21 20:25:34,939 -     INFO -    GOOGLE [139662966126336]: 0 added / 0 removed / 1 unwanted / 0 ignored
clg27 commented 5 years ago

@m1lkman sorry for not replying about your last comment, got busy with work. Most likely you'll need to disable mimetype checking. It's the "Google"->"MIME_TYPES" setting. I think the default is "true", just set it to "false" and things should work for you.

m1lkman commented 5 years ago

Thanks @clg27, that seems to work. Easy fix. I'll keep it running tonight but it looks good. Should work for root and subfolder crypts as well as My Drive and Team Drives I think but would need to do some testing to validate.

m1lkman commented 5 years ago

I ran into something different today while importing some MP3s. It was able to decode most of them but then ran into an error on one file UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 390: ordinal not in range(128). The rclone cryptdecode command runs fine manually. The file name in question has a ’ in it. Any ideas @clg27 or @daghaian?

see log output below

 2019-05-22 13:37:04,108 -     INFO -    RCLONE [140281936529152]: Root directory identified as My Drive/encrypt/
 2019-05-22 13:37:04,108 -     INFO -    RCLONE [140281936529152]: Root directory My Drive/encrypt/ has mapping defined in config as remote gcrypt:. Attempting to decrypt
 2019-05-22 13:37:04,108 -     INFO -    RCLONE [140281936529152]: Raw query is /usr/bin/rclone cryptdecode gcrypt: g2cjo4qilkguaklfv403qpr1rs/bjagt7vmlc95qohsqb90qiknjk/u5jskbqi7v8fhr5c5d6dmuat8cs3d92voulpfu5h7jt7gbdi7cfic2bsk4ucqdh1g5ubcdpufbfp4/k94guq4ukdsf5qff1esjc586jl47uq29fgc4ugre2ma8j6hp8f8jl3vgi48lcg19895urb211ubru1p4uhudck8ngibug1rdf5v1cq8
 2019-05-22 13:37:04,154 -     INFO -    RCLONE [140281936529152]: Decoded path of g2cjo4qilkguaklfv403qpr1rs/bjagt7vmlc95qohsqb90qiknjk/u5jskbqi7v8fhr5c5d6dmuat8cs3d92voulpfu5h7jt7gbdi7cfic2bsk4ucqdh1g5ubcdpufbfp4/k94guq4ukdsf5qff1esjc586jl47uq29fgc4ugre2ma8j6hp8f8jl3vgi48lcg19895urb211ubru1p4uhudck8ngibug1rdf5v1cq8 is Music/Logic/Confessions of a Dangerous Mind (2019)/Logic - Confessions of a Dangerous Mind - 12 - Limitless.mp3
 2019-05-22 13:37:04,155 -     INFO -    GOOGLE [140281936529152]: Added '1itfXf-ytcOleKw79O0fAm2Fh5n8eu_ur' to cache: p7jm4tdcgvm8s1sbs6538eonsksrjavvgoiea5t15223ecthr8cscv71thec81027mge5kkimq98fu2u77jv1hdq34dbq6th9tp36u7s5jbevifel3s1a0rumine20rt8r90u37fl117m90k8ehqd01cq4
 2019-05-22 13:37:04,157 -     INFO -    RCLONE [140281936529152]: File Path identified as g2cjo4qilkguaklfv403qpr1rs/bjagt7vmlc95qohsqb90qiknjk/u5jskbqi7v8fhr5c5d6dmuat8cs3d92voulpfu5h7jt7gbdi7cfic2bsk4ucqdh1g5ubcdpufbfp4/p7jm4tdcgvm8s1sbs6538eonsksrjavvgoiea5t15223ecthr8cscv71thec81027mge5kkimq98fu2u77jv1hdq34dbq6th9tp36u7s5jbevifel3s1a0rumine20rt8r90u37fl117m90k8ehqd01cq4
 2019-05-22 13:37:04,157 -     INFO -    RCLONE [140281936529152]: Root directory identified as My Drive/encrypt/
 2019-05-22 13:37:04,157 -     INFO -    RCLONE [140281936529152]: Root directory My Drive/encrypt/ has mapping defined in config as remote gcrypt:. Attempting to decrypt
 2019-05-22 13:37:04,157 -     INFO -    RCLONE [140281936529152]: Raw query is /usr/bin/rclone cryptdecode gcrypt: g2cjo4qilkguaklfv403qpr1rs/bjagt7vmlc95qohsqb90qiknjk/u5jskbqi7v8fhr5c5d6dmuat8cs3d92voulpfu5h7jt7gbdi7cfic2bsk4ucqdh1g5ubcdpufbfp4/p7jm4tdcgvm8s1sbs6538eonsksrjavvgoiea5t15223ecthr8cscv71thec81027mge5kkimq98fu2u77jv1hdq34dbq6th9tp36u7s5jbevifel3s1a0rumine20rt8r90u37fl117m90k8ehqd01cq4
 2019-05-22 13:37:04,203 -    ERROR -    GOOGLE [140281936529152]: Exception sending request to https://www.googleapis.com/drive/v3/changes with kwargs={'params': {'pageSize': 1000, 'includeTeamDriveItems': False, 'fields': 'changes(file(md5Checksum,mimeType,modifiedTime,name,parents,teamDriveId,trashed),fileId,removed,teamDrive(id,name),teamDriveId),newStartPageToken,nextPageToken', 'supportsTeamDrives': False, 'pageToken': u'4754988', 'includeRemoved': True}}: 
Traceback (most recent call last):
  File "/opt/plex_autoscan/google/drive.py", line 179, in query
    callbacks['data_callback'](resp.json())
  File "/opt/plex_autoscan/google/drive.py", line 585, in _process_changes
    decrypted = self.decrypter.decrypt_path(item_paths[0])
  File "/opt/plex_autoscan/rclone.py", line 28, in decrypt_path
    decoded = subprocess.check_output([self._binary,"--config",self._config,"cryptdecode",mapped_remote,file_path],stderr=subprocess.STDOUT).decode().rstrip('\n')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 390: ordinal not in range(128)
m1lkman commented 5 years ago

I think I resolved it. Now it correctly processes file names with non-ascii characters. Please review and let me know if there are any issues with what I did here https://github.com/daghaian/plex_autoscan/pull/1/commits/30ada844c491881462ddcfb875993b2abe03f0f9.

clg27 commented 5 years ago

I don't see any issues with that. Good fix!