Introduction

Plex DupeFinder is a python script that finds duplicate versions of media (TV episodes and movies) in your Plex Library and tells Plex to remove the lowest rated files/versions (based on user-specified scoring) to leave behind a single file/version.

Duplicates can be either in bulk (automatic) or on-by-one (interactively).

Demo

Click to enlarge.

Requirements

Python 3.6+.
Required Python modules (see below).

Installation

Note: Steps below are for Debian-based distros (other operating systems will require tweaking to the steps).

Install Python 3 and PIP
```
sudo apt install python3 python3-pip
```

Clone the Plex DupeFinder repo.

sudo git clone https://github.com/l3uddz/plex_dupefinder /opt/plex_dupefinder

Find your user & group.
```
id
```
Fix permissions of the Plex DupeFinder folder (replace user/group with yours).
```
sudo chown -R user:group /opt/plex_dupefinder
```
Go into the Plex DupeFinder folder.
```
cd /opt/plex_dupefinder
```

Install the required python modules.

sudo python3 -m pip install -r requirements.txt

Create a shortcut for Plex DupeFinder.

sudo ln -s /opt/plex_dupefinder/plex_dupefinder.py /usr/local/bin/plex_dupefinder

Generate a config.json file.
```
plex_dupefinder
```

Fill in Plex URL and credentials at the prompt to generated a Plex Access Token (optional).

Dumping default config to: /opt/plex_dupefinder/config.json
Plex Server URL: http://localhost:32400
Plex Username: your_plex_username
Plex Password: your_plex_password
Auto Delete duplicates? [y/n]: n
Please edit the default configuration before running again!

Configure the config.json file.
```
nano config.json
```

Configuration

Sample

{
  "AUDIO_CODEC_SCORES": {
    "Unknown": 0,
    "aac": 1000,
    "ac3": 1000,
    "dca": 2000,
    "dca-ma": 4000,
    "eac3": 1250,
    "flac": 2500,
    "mp2": 500,
    "mp3": 1000,
    "pcm": 2500,
    "truehd": 4500,
    "wmapro": 200
  },
  "AUTO_DELETE": false,
  "FIND_DUPLICATE_FILEPATHS_ONLY": false,
  "FILENAME_SCORES": {
    "*.avi": -1000,
    "*.ts": -1000,
    "*.vob": -5000,
    "*1080p*BluRay*": 15000,
    "*720p*BluRay*": 10000,
    "*HDTV*": -1000,
    "*PROPER*": 1500,
    "*REPACK*": 1500,
    "*Remux*": 20000,
    "*WEB*CasStudio*": 5000,
    "*WEB*KINGS*": 5000,
    "*WEB*NTB*": 5000,
    "*WEB*QOQ*": 5000,
    "*WEB*SiGMA*": 5000,
    "*WEB*TBS*": -1000,
    "*WEB*TROLLHD*": 2500,
    "*WEB*VISUM*": 5000,
    "*dvd*": -1000
  },
  "PLEX_LIBRARIES": [
    "Movies",
    "TV"
  ],
  "PLEX_SERVER": "https://plex.your-server.com",
  "PLEX_TOKEN": "",
  "SCORE_FILESIZE": true,
  "SKIP_LIST": [],
  "VIDEO_CODEC_SCORES": {
    "Unknown": 0,
    "h264": 10000,
    "h265": 5000,
    "hevc": 5000,
    "mpeg1video": 250,
    "mpeg2video": 250,
    "mpeg4": 500,
    "msmpeg4": 100,
    "msmpeg4v2": 100,
    "msmpeg4v3": 100,
    "vc1": 3000,
    "vp9": 1000,
    "wmv2": 250,
    "wmv3": 250
  },
  "VIDEO_RESOLUTION_SCORES": {
    "1080": 10000,
    "480": 3000,
    "4k": 20000,
    "720": 5000,
    "Unknown": 0,
    "sd": 1000
  }
}

Foreword

The scoring is based on: non-configurable and configurable parameters.

Non-configurable parameters are: bitrate, duration, height, width, and audio channel.
Configurable parameters are: audio codec scores, video codec scores, video resolution scores, filename scores, and file sizes (can only be toggled on or off).
Note: bitrate, duration, height, width, audio channel, audio and video codecs, video resolutions (e.g. SD, 480p, 720p, 1080p, 4K, etc), and file sizes are all taken from the metadata Plex retrieves during media analysis.

Details

Audio Codec Scores

You can set AUDIO_CODEC_SCORES to your preference.
The default settings should be sufficient for most.

Auto Delete

Under AUTO_DELETE, set your desired option.
- "AUTO_DELETE": true, - Plex DupeFinder will run in automatic mode.
- "AUTO_DELETE": false, - Plex DupeFinder will run in interactive mode. (Default)
- Options:
  - Skip (i.e. keep both): 0
  - Choose the best one (and delete the rest): b
  - Select the item to keep (and delete the rest): # (i.e. 1, 2, 3, etc).

Find Duplicate File Paths Only

Finds duplicates that only share the same file path.
```
"FIND_DUPLICATE_FILEPATHS_ONLY": false,
```
This option has a very limited use case, i.e. in instances where Plex may have glitched and created multiple duplicates of the same media item.
If using this setting, we recommend using UnionFS-Fuse that can generate whiteout files (*_HIDDEN~) to prevent the deletion of the actual file on the system. The _HIDDEN~ files can then be removed afterwards or even during the dupe cleanup (e.g. watch -n 5 rm -rf /mnt/local/.unionfs-fuse/*).
The default settings should be sufficient for most.

Filename Scores

You can set FILENAME_SCORES to your preference.
The default settings should be sufficient for most.

Plex Libraries

Go to Plex and get all the names of your Plex Libraries you want to find duplicates in.
- Example Library:

Under PLEX_LIBRARIES, list your Plex Libraries exactly as they are named in your Plex.

Format:

"PLEX_LIBRARIES": [
"LIBRARY_NAME_1",
"LIBRARY_NAME_2"
],

"PLEX_LIBRARIES": ["LIBRARY_NAME_1", "LIBRARY_NAME_2"],

Example:
```
"PLEX_LIBRARIES": [
"Movies",
"TV"
],
```

Plex Server URL

Your Plex server's URL.
This can be any format (e.g. http://localhost:32400, https://plex.domain.ltd).

Plex Token

Obtain a Plex Access Token:
- Fill in the Plex URL and Plex login credentials, at the prompt, on first run. This only occurs when there is no config.json present.
  
  or
- Visit https://support.plex.tv/hc/en-us/articles/204059436-Finding-an-authentication-token-X-Plex-Token
Add the Plex Access Token to "PLEX_TOKEN" so that it now appears as "PLEX_TOKEN": "abcd1234",.
- Note: Make sure it is within the quotes (") and there is a comma (,) after it.

Filesize Scores

"SCORE_FILESIZE": true will add more points to the overall score based on the actual file size.
The default settings should be sufficient for most.
Note: In some situations (e.g. a bad encode resulting in a large size), this may be something you want to turn it off (i.e. false).

Skip List

In Auto Delete mode, any file paths matching the patterns (i.e folders), listed in SKIP_LIST, will be ignored.
Example:
```
"SKIP_LIST": ["/Movies4K/"]
```
The default settings should be sufficient for most.

Video Codec Scores

You can set VIDEO_CODEC_SCORES to your preference.
The default settings should be sufficient for most.

Video Resolution Scoring

You can set VIDEO_RESOLUTION_SCORES to your preference.
The default settings should be sufficient for most.

Plex

You will need to make sure that Allow media deletion is enabled in Plex.

In Plex, click the Settings icon -> Server -> Library.
Set the following:
- Allow media deletion: enabled
Click SAVE CHANGES.

Usage

Simply run the script/command:

plex_dupefinder

Donate

If you find this project helpful, feel free to make a small donation to the developer:

Monzo: Credit Cards, Apple Pay, Google Pay
Beerpay: Credit Cards
Paypal: l3uddz@gmail.com
BTC: 3CiHME1HZQsNNcDL6BArG7PbZLa8zUUgjL

l3uddz / plex_dupefinder

readme