l3uddz / plex_dupefinder

Find and delete duplicate files in Plex
GNU General Public License v3.0
303 stars 54 forks source link

Not Skipping Files with pattern in the Path of SKIP_LIST #52

Open dabigc opened 3 years ago

dabigc commented 3 years ago

Describe the bug Even though the SKIP_LIST variable is set, files with the pattern specified are being considered when finding duplicates and the 1080p/720p copy is then deleted when running in automatic mode (which, base on the docs, is the only mode that SKIP_LIST is considered).

To Reproduce Steps to reproduce the behavior:

  1. Run plex_dupefinder once to generate the initial config file.
  2. Edit the config file to have the settings from the Additional Context section below.
  3. Run plex_dupefinder with the configuration set to automatic mode.
  4. Kill the script after the first file or two to avoid trashing valid copies of files.

Expected behavior That the SKIP_LIST be considered when the script is run in automatic mode and paths containing the pattern(s) from this list be excluded from the results that are evaluated for deletion.

Screenshots N/A

Logs

$ plex_dupefinder --loglevel=DEBUG

       _                 _                   __ _           _
 _ __ | | _____  __   __| |_   _ _ __   ___ / _(_)_ __   __| | ___ _ __
| '_ \| |/ _ \ \/ /  / _` | | | | '_ \ / _ \ |_| | '_ \ / _` |/ _ \ '__|
| |_) | |  __/>  <  | (_| | |_| | |_) |  __/  _| | | | | (_| |  __/ |
| .__/|_|\___/_/\_\  \__,_|\__,_| .__/ \___|_| |_|_| |_|\__,_|\___|_|
|_|                             |_|

#########################################################################
# Author:   l3uddz                                                      #
# URL:      https://github.com/l3uddz/plex_dupefinder                   #
# --                                                                    #
#         Part of the Cloudbox project: https://cloudbox.works          #
#########################################################################
#                   GNU General Public License v3.0                     #
#########################################################################

Initialized
Finding dupes...
Found 1059 dupes for section 'Movies'
Found 5982 dupes for section 'TV Shows'

Determining best media item to keep for '3 from Hell' ...
        Keeping  : 365248 - ['/mnt/media/4K/4K-Movies/3 from Hell (2019)/3 from Hell 2019 Remux-2160p.mkv']
        Removing : 497434 - ['/mnt/media/Movies/3 from Hell (2019)/3 from Hell 2019 Remux-1080p.mkv']
                Deleted media item: 497434

Determining best media item to keep for '6 Underground' ...
        Keeping  : 365160 - ['/mnt/media/4K/4K-Movies/Six Underground (0)/6 Underground 2019 WEBRip-2160p.mkv']
        Removing : 365382 - ['/mnt/media/Movies/Six Underground (0)/6 Underground 2019 WEBDL-1080p.mkv']
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/plex_dupefinder", line 465, in <module>
    delete_item(part_info['show_key'], media_id)
  File "/usr/local/bin/plex_dupefinder", line 203, in delete_item
    if requests.delete(delete_url, headers={'X-Plex-Token': cfg.PLEX_TOKEN}).status_code == 200:
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 152, in delete
    return request('delete', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1373, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 311, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 272, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt

System Information

Additional context Full config.json with sensitive bits redacted.

{
  "AUDIO_CODEC_SCORES": {
    "Unknown": 0,
    "aac": 1000,
    "ac3": 1000,
    "dca": 2000,
    "dca-ma": 4000,
    "eac3": 1250,
    "flac": 2500,
    "mp2": 500,
    "mp3": 1000,
    "pcm": 2500,
    "truehd": 4500,
    "wmapro": 200
  },
  "AUTO_DELETE": true,
  "FIND_DUPLICATE_FILEPATHS_ONLY": false,
  "FILENAME_SCORES": {
    "*.avi": -1000,
    "*.ts": -1000,
    "*.vob": -5000,
    "*1080p*BluRay*": 15000,
    "*720p*BluRay*": 10000,
    "*HDTV*": -1000,
    "*PROPER*": 1500,
    "*REPACK*": 1500,
    "*Remux*": 20000,
    "*WEB*CasStudio*": 5000,
    "*WEB*KINGS*": 5000,
    "*WEB*NTB*": 5000,
    "*WEB*QOQ*": 5000,
    "*WEB*SiGMA*": 5000,
    "*WEB*TBS*": -1000,
    "*WEB*TROLLHD*": 2500,
    "*WEB*VISUM*": 5000,
    "*dvd*": -1000
  },
  "PLEX_LIBRARIES": [
    "Movies",
    "TV Shows"
  ],
  "PLEX_SERVER": "http://127.0.0.1:32400",
  "PLEX_TOKEN": "<REDACTED>",
  "SCORE_FILESIZE": true,
  "SKIP_LIST": ["/4K/"],
  "VIDEO_CODEC_SCORES": {
    "Unknown": 0,
    "h264": 10000,
    "h265": 5000,
    "hevc": 5000,
    "mpeg1video": 250,
    "mpeg2video": 250,
    "mpeg4": 500,
    "msmpeg4": 100,
    "msmpeg4v2": 100,
    "msmpeg4v3": 100,
    "vc1": 3000,
    "vp9": 1000,
    "wmv2": 250,
    "wmv3": 250
  },
  "VIDEO_RESOLUTION_SCORES": {
    "1080": 10000,
    "480": 3000,
    "4k": 20000,
    "720": 5000,
    "Unknown": 0,
    "sd": 1000
  }
}