search

terrelsa13 / MUMC

Multi-User Media Cleaner aka MUMC (pronounced Mew-Mick) will go through movies, tv episodes, audio tracks, and audiobooks in your Emby/Jellyfin libraries deleting media items you no longer want.
GNU General Public License v3.0
99 stars 6 forks source link

Playtime is not considered when deleting. #107

Closed Retr0astic closed 5 months ago

Retr0astic commented 5 months ago

Hi, I’m using the latest version of Jellyfin and MUMC, when deleting episode files, the script deletes half watched episodes.

Retr0astic commented 5 months ago

config

terrelsa13 commented 5 months ago

@Retr0astic I am not able to reproduce this on my end. I will ask you a few questions to help me understand what might be going on. If my questions do not point me in the right direction; I may need your help reproducing this issue on your setup with DEBUG=4 turned on.

For the questions below; make a mental note of one of the "half watched" deleted episodes.

Questions

  1. What is the total run-time on the "half watched" episode?
  2. What is the watched time on the "half watched" episode for user user_name: Retr_0?
  3. What is the watched time on the "half watched" episode for user user_name: Vicky?
  4. In Jellyfin go to http://192.168.1.203:8096/web/#/dashboard/playback/resume
    1. What is the value for Maximum resume percentage?

How MUMC determines if a media_item is played

There are 3 pieces of information showing played/unplayed status for a media_item. For an individual user, the script requires at least 2 of 3 are met to consider the media_item played.

Each of these are checked. For each that is True; itemIsPlayedControlCount is incremented by 1.

  1. LastPlayedDate exists and does not have a value of 'Unplayed'.
  2. PlayCount exists and is greater than zero.

  3. Played state exists and is True.

    # get played status of media item
def get_isItemPlayed(item):

#there are multiple pieces of data denoting if a media item is played/unplayed

#require at least this many are True before deciding the media item is played
itemIsPlayedControlFloor=2
itemIsPlayedControlCount=0
itemIsPlayed=False

if ('UserData' in item):
    if ('LastPlayedDate' in item['UserData']) and (not (item['UserData']['LastPlayedDate'] == 'Unplayed')):
        itemIsPlayedControlCount += 1
    if ('PlayCount' in item['UserData']) and (item['UserData']['PlayCount'] > 0):
        itemIsPlayedControlCount += 1
    if ('Played' in item['UserData']) and (item['UserData']['Played']):
        itemIsPlayedControlCount += 1

if (itemIsPlayedControlCount >= itemIsPlayedControlFloor):
    itemIsPlayed=True

return(itemIsPlayed)
Retr0astic commented 5 months ago

Questions

  1. What is the total run-time on the "half watched" episode?

Runtime: 00:59:31

  1. What is the watched time on the "half watched" episode for user user_name: Retr_0?

Watched: 24 mins 34 seconds

  1. What is the watched time on the "half watched" episode for user user_name: Vicky?

Entire Runtime

  1. In Jellyfin go to http://192.168.1.203:8096/web/#/dashboard/playback/resume

Maximum Resume Percentage : 90

terrelsa13 commented 5 months ago

Been giving this some thought. Trying to figure out what is going on here. Is there any chance user_name: Retr_0 had previously watched the episode, causing PlayedCount to be >0 and the LastPlayedDate to be set with the value the episode was last watched?

Retr0astic commented 5 months ago

The user Retr_0 is watching the episode for the first time.

There seems to be an issue with my script now, I cant seem to access jellyfin with a new config:

\

Time Stamp Start: 20240527114001 MUMC Version: 5.7.3 MUMC Config Version: 5.7.3 Jellyfin Version: 10.9.2 Python Version: 3.12.3 OS Info: Linux-6.8.10-300.fc40.x86_64-x86_64-with-glibc2.39

Start... Cleaning media for server at: http://192.168.1.203:8096

HTTPError: Unable to get information from server during processing of: get_single_user_info_for_95599ea488484c4b90cc0c6cb5db8791 Object: <bound method Request.header_items of <urllib.request.Request object at 0x7f6907509730>> URL: http://192.168.1.203:8096/Users/95599ea488484c4b90cc0c6cb5db8791 Method: Header: {'Authorization': 'MediaBrowser Client="mumc.py", Device="Multi-User Media Cleaner", DeviceId="MUMC", Version="5.7.3", Token="558e4c9784cd403cb6671148ae9539bf"', 'Content-type': 'application/json'} Data: None

HTTPError: 401 - Unauthorized

Check password and/or remove any GUI API keys for MUMC

Upon creating an API key for MUMC manually in jellyfin, and replacing the one in the config, I'm able to solve this issue, but a new one arises:

Time Stamp Start: 20240527114111 MUMC Version: 5.7.3 MUMC Config Version: 5.7.3 Jellyfin Version: 10.9.2 Python Version: 3.12.3 OS Info: Linux-6.8.10-300.fc40.x86_64-x86_64-with-glibc2.39

Start... Cleaning media for server at: http://192.168.1.203:8096

Get List Of Media For: Retr_0 - 95599ea488484c4b90cc0c6cb5db8791

EPISODE POST PROCESSING STARTED...

PROCESSING BLACKLISTED EPISODES...

MOVIE POST PROCESSING STARTED...

PROCESSING BLACKLISTED MOVIES...

PROCESSING WHITELISTED MOVIES...

PROCESSING BLACKTAGGED MOVIES...

PROCESSING WHITETAGGED MOVIES...

PROCESSING FAVORITED MOVIES...

MOVIE POST PROCESSING COMPLETE.

HTTPError: Unable to get information from server during processing of: playedPatternCleanup_for_c72552c81d0987a80595265f4b698223 Object: <bound method Request.header_items of <urllib.request.Request object at 0x7f88db53e4e0>> URL: http://192.168.1.203:8096/Users/bf6edc698eda45ca9a6ab7d9e2e06fae/Items/c72552c81d0987a80595265f4b698223?enableImages=False&enableUserData=True&Fields=ParentId,Genres,Tags,RecursiveItemCount,ChildCount,Type Method: Header: {'Authorization': 'MediaBrowser Client="mumc.py", Device="Multi-User Media Cleaner", DeviceId="MUMC", Version="5.7.3", Token="b3bc41924fae487a9812238941a74c5e"', 'Content-type': 'application/json'} Data: None

HTTPError: 404 - Not Found

Check password and/or remove any GUI API keys for MUMC Traceback (most recent call last): File "/home/sree/scripts/MUMC/mumc.py", line 110, in MUMC() File "/home/sree/scripts/MUMC/mumc.py", line 84, in MUMC deleteItems=sortDeleteLists(deleteItems_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sree/scripts/MUMC/mumc_modules/mumc_sort.py", line 25, in sortDeleteLists deleteItems_episode=deleteItems_dict['episode']


  File "<string>", line 2, in __getitem__
  File "/usr/lib64/python3.12/multiprocessing/managers.py", line 836, in _callmethod
    raise convert_to_error(kind, result)
KeyError: 'episode'
Retr0astic commented 5 months ago

Update on the previous issue, upon further reading of the log and comparing the item names, it seems the script is looking for playback value for item c72552c81d0987a80595265f4b698223 on user bf6edc698eda45ca9a6ab7d9e2e06fae, which is user: Theater, even though I've set up the config to not look for the above item since it is a TV show episode under the library path "/data/shows" .

I marked the item as unplayed, and it just checks another item if it's played under the user, even though the user is not supposed to access the library.

terrelsa13 commented 5 months ago

Ok. A lot is happening. Let's see if we can figure this out.

First let's make sure there are no API keys in the GUI containing MUMC in the name. Emby/Jellyfin does not allow some actions to be performed with GUI created API keys. And Emby/Jellyfin does not like when an existing GUI API key is already using the MUMC name. The API keys created via the "authenticate by name" API service is what MUMC uses so it has permission to perform all actions.

Next rename the current mumc_config.yaml to mumc_config.yaml.old

Run MUMC to create a new mumc_config.yaml file.

Once the new config is created, edit it to run the desired way.

Then set DEBUG: 4 in the config.

Rerun MUMC so the error happens again. (It will run more slowly because it is creating a debug file.)

Let MUMC fail due to the error.

Open /home/sree/scripts/MUMC/.

Find the file named mumc_DEBUG.log.

Attach both the mumc_config.yaml and the mumc_DEBUG.log files to your reply. This will help me get a better idea of what is happening.

Retr0astic commented 5 months ago

Deleted the old API keys and created a new config

mumc_DEBUG.log

`--- version: 5.7.3 basic_settings: filter_statements: movie: played: condition_days: 7 count_equality: '>=' count: 1 created: condition_days: -1 count_equality: '>=' count: 1 behavioral_control: true episode: played: condition_days: 1 count_equality: '>=' count: 1 created: condition_days: -1 count_equality: '>=' count: 1 behavioral_control: true audio: played: condition_days: -1 count_equality: '>=' count: 1 created: condition_days: -1 count_equality: '>=' count: 1 behavioral_control: true audiobook: played: condition_days: -1 count_equality: '>=' count: 1 created: condition_days: -1 count_equality: '>=' count: 1 behavioral_control: true advanced_settings: filter_statements: movie: query_filter: favorited: true whitetagged: false blacktagged: true whitelisted: false blacklisted: true episode: query_filter: favorited: true whitetagged: false blacktagged: true whitelisted: false blacklisted: true audio: query_filter: favorited: true whitetagged: false blacktagged: true whitelisted: false blacklisted: true audiobook: query_filter: favorited: true whitetagged: false blacktagged: true whitelisted: false blacklisted: true behavioral_statements: movie: favorited: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false extra: genre: 0 library_genre: 0 whitetagged: action: keep user_conditional: all played_conditional: ignore action_control: 0 dynamic_behavior: false tags: [] blacktagged: action: delete user_conditional: all played_conditional: any_played action_control: 0 dynamic_behavior: false tags: [] whitelisted: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false blacklisted: action: delete user_conditional: any played_conditional: any_played action_control: 3 dynamic_behavior: false episode: favorited: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false extra: genre: 0 season_genre: 0 series_genre: 0 library_genre: 0 studio_network: 0 studio_network_genre: 0 whitetagged: action: keep user_conditional: all played_conditional: ignore action_control: 0 dynamic_behavior: false tags: [] blacktagged: action: delete user_conditional: all played_conditional: any_played action_control: 0 dynamic_behavior: false tags: [] whitelisted: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false blacklisted: action: delete user_conditional: all played_conditional: all_played action_control: 3 dynamic_behavior: false audio: favorited: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false extra: genre: 0 album_genre: 0 library_genre: 0 track_artist: 0 album_artist: 0 whitetagged: action: keep user_conditional: all played_conditional: ignore action_control: 0 dynamic_behavior: false tags: [] blacktagged: action: delete user_conditional: all played_conditional: any_played action_control: 0 dynamic_behavior: false tags: [] whitelisted: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false blacklisted: action: delete user_conditional: any played_conditional: any_played action_control: 3 dynamic_behavior: false audiobook: favorited: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false extra: genre: 0 audiobook_genre: 0 library_genre: 0 track_author: 0 author: 0 library_author: 0 whitetagged: action: keep user_conditional: all played_conditional: ignore action_control: 0 dynamic_behavior: false tags: [] blacktagged: action: delete user_conditional: all played_conditional: any_played action_control: 0 dynamic_behavior: false tags: [] whitelisted: action: keep user_conditional: any played_conditional: ignore action_control: 3 dynamic_behavior: false blacklisted: action: delete user_conditional: any played_conditional: any_played action_control: 3 dynamic_behavior: false whitetags: [] blacktags: [] delete_empty_folders: episode: season: false series: false episode_control: minimum_episodes: 0 minimum_played_episodes: 0 minimum_episodes_behavior: Max Played Min Unplayed trakt_fix: set_missing_last_played_date: movie: true episode: true audio: true audiobook: true console_controls: headers: script: show: true formatting: font: color: '' style: '' background: color: '' user: show: true formatting: font: color: '' style: '' background: color: '' summary: show: true formatting: font: color: '' style: '' background: color: '' footers: script: show: true formatting: font: color: '' style: '' background: color: '' warnings: script: show: true formatting: font: color: '' style: '' background: color: '' movie: delete: show: true formatting: font: color: '' style: '' background: color: '' keep: show: true formatting: font: color: '' style: '' background: color: '' post_processing: show: true formatting: font: color: '' style: '' background: color: '' summary: show: true formatting: font: color: '' style: '' background: color: '' episode: delete: show: true formatting: font: color: '' style: '' background: color: '' keep: show: true formatting: font: color: '' style: '' background: color: '' post_processing: show: true formatting: font: color: '' style: '' background: color: '' summary: show: true formatting: font: color: '' style: '' background: color: '' audio: delete: show: true formatting: font: color: '' style: '' background: color: '' keep: show: true formatting: font: color: '' style: '' background: color: '' post_processing: show: true formatting: font: color: '' style: '' background: color: '' summary: show: true formatting: font: color: '' style: '' background: color: '' audiobook: delete: show: true formatting: font: color: '' style: '' background: color: '' keep: show: true formatting: font: color: '' style: '' background: color: '' post_processing: show: true formatting: font: color: '' style: '' background: color: '' summary: show: true formatting: font: color: '' style: '' background: color: '' UPDATE_CONFIG: false REMOVE_FILES: false admin_settings: server: brand: jellyfin url: http://192.168.1.203:8096 auth_key: 0eec431de4ad4622ac677892a329c32d admin_id: 95599ea488484c4b90cc0c6cb5db8791 behavior: list: blacklist matching: byId users: monitor_disabled: true users:

  • user_id: 95599ea488484c4b90cc0c6cb5db8791 user_name: Retr_0 whitelist:
  • lib_id: f137a2dd21bbc1b99aa5c0f6bf02a805 collection_type: movies path: /data/Movies network_path: null lib_enabled: true
  • lib_id: 7e64e319657a9516ec78490da03edccb collection_type: music path: /data/music network_path: null lib_enabled: true
  • lib_id: 7b2528c2eda373a3c97dbd246d4a413f collection_type: tvshows path: /data/tamil/Tamil_Series network_path: null lib_enabled: true
  • lib_id: 31c1f0d55826daaf8e3859b3e890f576 collection_type: movies path: /data/tamil/Movies network_path: null lib_enabled: true
  • lib_id: 9d7ad6afe9afa2dab1a2f6e00ad28fa6 collection_type: boxsets path: /config/data/collections network_path: null lib_enabled: true blacklist:
  • lib_id: a656b907eb3a73532e40e44b968d0225 collection_type: tvshows path: /data/shows network_path: null lib_enabled: true
  • lib_id: 0c41907140d802bb58430fed7e2cd79e collection_type: tvshows path: /data/anime network_path: null lib_enabled: true userPosition: 0
  • user_id: bf6edc698eda45ca9a6ab7d9e2e06fae user_name: Theater whitelist: [] blacklist:
  • lib_id: f137a2dd21bbc1b99aa5c0f6bf02a805 collection_type: movies path: /data/Movies network_path: null lib_enabled: true
  • lib_id: 31c1f0d55826daaf8e3859b3e890f576 collection_type: movies path: /data/tamil/Movies network_path: null lib_enabled: true
  • lib_id: 7b2528c2eda373a3c97dbd246d4a413f collection_type: tvshows path: /data/tamil/Tamil_Series network_path: null lib_enabled: true userPosition: 1
  • user_id: 163c3ce3ff344945b14d8eb28f78b4e2 user_name: Vicky whitelist:
  • lib_id: f137a2dd21bbc1b99aa5c0f6bf02a805 collection_type: movies path: /data/Movies network_path: null lib_enabled: true
  • lib_id: 7e64e319657a9516ec78490da03edccb collection_type: music path: /data/music network_path: null lib_enabled: true
  • lib_id: 0c41907140d802bb58430fed7e2cd79e collection_type: tvshows path: /data/anime network_path: null lib_enabled: true
  • lib_id: 7b2528c2eda373a3c97dbd246d4a413f collection_type: tvshows path: /data/tamil/Tamil_Series network_path: null lib_enabled: true
  • lib_id: 31c1f0d55826daaf8e3859b3e890f576 collection_type: movies path: /data/tamil/Movies network_path: null lib_enabled: true
  • lib_id: 9d7ad6afe9afa2dab1a2f6e00ad28fa6 collection_type: boxsets path: /config/data/collections network_path: null lib_enabled: true blacklist:
  • lib_id: a656b907eb3a73532e40e44b968d0225 collection_type: tvshows path: /data/shows network_path: null lib_enabled: true userPosition: 2 api_controls: attempts: 4 item_limit: 25 cache: size: 32 fallback_behavior: LRU minimum_age: 200 DEBUG: 4`
terrelsa13 commented 5 months ago
  1. Open up a browser on a computer that can access Jellyfin.

  2. Copy/Paste the link below (all one line) into the search bar of the browser:

    http://192.168.1.203:8096/Users/bf6edc698eda45ca9a6ab7d9e2e06fae/Items/2ce1e69359c039150d8556c0b279f99c?enableImages=False&enableUserData=True&Fields=ParentId,Genres,Tags,RecursiveItemCount,ChildCount,Type&api_key=0eec431de4ad4622ac677892a329c32d
  3. Press enter
  4. There should be JSON data returned from the server that looks similar to one of the attached images below.
  5. Attach a screenshot of what is shown in the browser.

#

Chrome - select the Pretty-print checkbox

Screenshot from 2024-05-27 16-39-41

#

Firefox

Screenshot from 2024-05-27 16-38-30

Retr0astic commented 5 months ago

image

terrelsa13 commented 5 months ago

  1. Please copy/paste the following link into a browser that can access Jellyfin. This will request information for user_name: theater.

    http://192.168.1.203:8096/users/bf6edc698eda45ca9a6ab7d9e2e06fae?api_key=0eec431de4ad4622ac677892a329c32d
    1. Select the COPY button at the top left.
    2. Next, create a file named episode_info.json.
    3. Then, paste the json data into the episode_info.json file.
    4. Finally, attach episode_info.json to the reply.

  2. Using the file explorer or the command line; navigate to /data/shows/Peripheral, The (2022) [tvdb-349187]/Season 01/.

    1. Does the file below exist in this folder? 
      The Peripheral (2022) - S01E01 - Pilot [WEBDL-2160p][HDR10Plus][EAC3 5.1][IT+EN][h265]-MeM.mkv

  3. Log into Jellyfin as user_name: theater.

    1. Go to the library containing the tv show The Peripheral.
    2. Play Season 01, Episode 01, named Pilot.
    3. Is user_name: theater able to access this file?
    4. Is user_name: theater able to play this file?
Retr0astic commented 5 months ago

  1. Please copy/paste the following link into a browser that can access Jellyfin. This will request information for user_name: theater.

       http://192.168.1.203:8096/users/bf6edc698eda45ca9a6ab7d9e2e06fae?api_key=0eec431de4ad4622ac677892a329c32d
    1. Select the COPY button at the top left.
    2. Next, create a file named episode_info.json.
    3. Then, paste the json data into the episode_info.json file.
    4. Finally, attach episode_info.json to the reply.

episode_info.json

2. Using the file explorer or the command line; navigate to `/data/shows/Peripheral, The (2022) [tvdb-349187]/Season 01/`.

   1. Does the file below exist in this folder?

   ```
   The Peripheral (2022) - S01E01 - Pilot [WEBDL-2160p][HDR10Plus][EAC3 5.1][IT+EN][h265]-MeM.mkv
   ```

Yes

3. Log into Jellyfin as `user_name: theater`.

   1. Go to the library containing the tv show `The Peripheral`.
   2. Play `Season 01`, `Episode 01`, named `Pilot`.
   3. Is `user_name: theater` able to access this file?
   4. Is `user_name: theater` able to play this file?

No, because theater is not given access to this file, theater isnt a "monitored" user for this path as well.

terrelsa13 commented 5 months ago

@Retr0astic Didn't forget about you.

Give the latest MUMC_dev branch (or the latest beta release v5.8.2-beta) a try.

Let me know if this fixes the issues.

Retr0astic commented 5 months ago

@Retr0astic Didn't forget about you.

Give the latest MUMC_dev branch (or the latest beta release v5.8.0-beta) a try.

Let me know if this fixes the issues.

Just installed the latest dev branch, the config does not have the advanced settings section, so it lacked the fine grain control I had before, after copying over my original config, I couldn't reproduce the issue. I'll keep this updated over this week after further testing.

terrelsa13 commented 5 months ago

Just installed the latest dev branch, the config does not have the advanced settings section, so it lacked the fine grain control I had before,

Correct. There was a request to simplify the config for non-tech savvy users. The complaint was, the config was getting intimidating because all of the config variables are shoved into the default config MUMC generates after the first run.

To satisfy my non-tech users I have simplified the default config MUMC generates. To satisfy my tech users all of the additional config variables are available on the wiki.

Starting at MUMC v5.8.0-beta the config no longer needs to have EVERY config variable. Only the minimum described above. Any additional config variables can be added individually. Anything not in the config, will have default values

couldn't reproduce the issue. I'll keep this updated over this week after further testing.

Good to hear. Let me know if you find something else. Or feel free to close the issue once testing is completed.

I will push the MUMC_dev changes to the MUMCv5 branch once this issue is closed.

Retr0astic commented 5 months ago

Update on this issue, I couldn't test for a few days as I lost my drive due to corruption, but I've set everything backup.

I've noticed that the script will delete any media that is half watched if the episode is half watched before script creation.

Suggestion for config files, ask the user if they want "Minimum" or "Full" during script creation and update. Optionally allow them to add full config just for specific media types.

terrelsa13 commented 5 months ago

This is interesting. It appears JF and Emby have been busy making changes to how this works.

Previously PlayCount was not incremented until the episode PlayedPercentage was >= the value set in the GUI. And LastPlayedDate was set at the same time as PlayCount.

Now it appears as soon as the episode starts playing the PlayCount is incremented and a new LastPlayedDate is set.

However Played remains false until the PlayedPercentage is >= the value set in the GUI.

UserData:
   PlayedPercentage: 66.5082812781887
   PlaybackPositionTicks: 4718789160
   PlayCount: 1
   IsFavorite: false
   LastPlayedDate: 2024-06-03T20:30:12.0000000Z
   Played: false

Yeah. You are correct. Looks like I need to figure out how to get the PlayedPercentage value set in the GUI. Then compare it to the PlayedPercentage value for each media_item before treating them as watched.

terrelsa13 commented 5 months ago

Actually, forget what I just said about grabbing the PlayedPercentage.

The reason MUMC was using the 2 of 3 approach was because there was inconsistency for when an item was considered played coming from Emby/JF. It appears this has been corrected. Instead only the UserData > Played will be considered for marking an item as watched.

If someone wants a media_item to be marked as watched using a different percentage than the default 90%, it should be changed in the GUI. Not in MUMC.

This has been fixed in the lastest v.5.8.9-beta release.

Retr0astic commented 5 months ago

Okay, I've installed the latest release, will test further, thanks.

Retr0astic commented 5 months ago

I've tested this, it works as intended now!!

This issue is resolved, thanks!