search

terrelsa13 / MUMC

Multi-User Media Cleaner aka MUMC (pronounced Mew-Mick) will go through movies, tv episodes, audio tracks, and audiobooks in your Emby/Jellyfin libraries deleting media items you no longer want.
GNU General Public License v3.0
93 stars 6 forks source link

Not deleting unplayed media past creation data criteria #109

Open Doug411 opened 3 months ago

Doug411 commented 3 months ago

Hi again! I have a folder that I am populating with podcasts. I'm using TubeArchivist to download them to a Podcast Library. Each individual podcast is a series folder on Jellyfin. Many of the podcasts are daily. I'd like to set a retention period for 14 days, and delete old content regardless of whether it has been watched or not (unless its favorited), I've tried various incarnations, but the only items it seems to put in my delete queue are items that were watched by the monitored user.

This is what I thought would work based on my understanding of the logic. (turn off played evaluation, set my created condition, and put the count <0). However, it only flagged the played items.

episode: played: condition_days: -1 count_equality: '>=' count: 1 created: condition_days: 14 count_equality: '>=' count: -1 behavioral_control: true

Also, what is the best way to potentially change the retention period at the podcast level? eg

Podcast (Jellyfin Series Library) -Podcast 1 (Sub Folder) 14 days -Podcast 2 (Sub Folder) 21 days -Podcast 3 (Sub Folder) 30 days

I also want to clear up my understanding of how the script works. If I use tags, do I still have to blacklist the library that I'm evaluating? In other words is it possible to NOT blacklist the podcast library, but blacktag items in a library that is whitelisted?

What happens if I tag the individual podcast folders, versus tagging the items? Tubeachivist doesn't have functionality to tag items upon download. I'm trying to figure out the best way to accomplish a folder based retain/purge logic. ideas?

I also want to understand more why you are allowing multiple monitor users. If I monitor 5 users played status.... Is the main use case to not delete until all 5 have watched and it meets the days criteria? Wow, that's complex if so.

Doug411 commented 2 months ago

Sure. I sent the results

terrelsa13 commented 2 months ago

I made you a "good news, bad news sandwich".

Top Slice Of Bread (good news)

I cannot find anything about the Megyn K. episodes that would prevent it from showing up.

Preferred Sandwich Contents (bad news)

Of course trying to look for clues to explain why the Megyn K. episodes do not show up, I instead discovered why episode_control is not working for you.

It appears Jellyfin is not aware of the episode numbers for Lex F., Tucker C., and Megyn K.. There should be two data items returned for each episode IndexNumber (episode number) and ParentIndexNumber (season number).

The shows mentioned have a lot of episode numbers shown as ??. This means Jellyfin did not return data for IndexNumber (episode number) and ParentIndexNumber (season number). MUMC has to fill in an unknown value, in this case ??, to allow it to proceed.

Unfortunately shows without legit episode numbers, will not work with episode_control.

Bottom Slice Of Bread (good news)

I have attached a zip file of MUMC that has additional/special debug messages to hopefully show why the Megyn K. episodes are not showing up.

  1. Download this MUMC_DEBUG.zip
  2. Move the MUMC_DEBUG.zip next to your current native MUMC folder
  3. Unzip MUMC_DEBUG.zip
  4. There should now be a MUMC_DEBUG folder
  5. Open the MUMC_DEBUG folder
  6. Open the mumc_config.yaml
  7. admin_settings > server > auth_key has been removed
    1. Go ahead and copy/paste the auth_key from the native MUMC folder's mumc_config.yaml
  8. Do not make any other changes to the config or script
  9. Run the script inside of the MUMC_DEBUG folder
  10. Gmail/Pastebin me the mumc_DEBUG.log
Doug411 commented 2 months ago

Just goes to show that all the good stuff in life is filled with carbs! :)

I'm not so concerned about episode control, I can live without it. It was more of a nice to have.

I'll give it a go.

Doug411 commented 2 months ago

remind me how the auth_key works again? is it simply the API key in jellyin? I ended up changing my api key this morning. Can i just paste that in, or is it some derivative that would require me running the mumc setup script again?

I had a five minute scare where my media dissapeared this morning. I could drill into folders but media wasnt playable or visible. I tried to stay calm and reboot docker containers, my SMB VM, etc. Eventually rebooting the whole PC and rescanning in jellyfin brought it back. As it was happening I kept telling myself the most likely problem was related to my USB drives passed through to a SMB VM. But after that litlle scare, I figured it was a message to not be lazy and actually change my api key. When the universe talks...I should probably listen.....:)

terrelsa13 commented 2 months ago

remind me how the auth_key works again? is it simply the API key in jellyin? I ended up changing my api key this morning. Can i just paste that in, or is it some derivative that would require me running the mumc setup script again?

Auth/API key is needed to send queries to the server.

The last I checked the GUI generated API keys will not work as Auth keys. BUT I saw a post late last year mentioning the JF was working on making the GUI generated API keys work the same as the Auth keys ones fetched via user authentication requests.

Long story short. Try with a GUI generated API key. If it does not work, you will have to rename the included current config. Trick MUMC into thinking it is the first time it's being run. Complete the configuration setup. Open the new configuration file. Copy the auth key for the new config. Paste it into the included config that was renamed earlier. Delete the new config. Rename the included config back to mumc_config.yaml.

I had a five minute scare where my media dissapeared this morning. I could drill into folders but media wasnt playable or visible. I tried to stay calm and reboot docker containers, my SMB VM, etc. Eventually rebooting the whole PC and rescanning in jellyfin brought it back. As it was happening I kept telling myself the most likely problem was related to my USB drives passed through to a SMB VM. But after that litlle scare, I figured it was a message to not be lazy and actually change my api key. When the universe talks...I should probably listen.....:)

100% do not blame you.

For remote access, I keep my server behind wireguard and use nginxPM (only available thru wireguard) to access it via HTTPS. That way in the very unlikely event wireguard somehow leaks server info, it's still encrypted.

Doug411 commented 1 month ago

Sorry for the delay. I was powerless for a while.... (literally...) Spending my first summer in Houston and that hurricane was brutal).

FYI... the API key worked. Sent the debug log to you in Gmail.

Doug411 commented 1 month ago

Ya- re: remote access. I had mine behind wireguard for a while too. Then I granted access to a few family members that are technically challenged, so I enabled access via SSL using a reverse proxy server as well. I have multi-factor authentication required for Jellyfin, as well as some seriously stringent password requirements and fail2ban monitoring logs..... But still, can never be too careful.

terrelsa13 commented 1 month ago

Np. Glad you made it through. Nothing like mixing large volumes of water with pretty much anything else to make the anything else worse. I've lived all over the country and have been in a few of mother nature's weather/environmental "tantrums". I will take anything we can predict is coming with a decent level or accuracy over the ones we guesstimate and/or the ones that surprise us.

Ok. What I found. I was hoping MUMC was somehow losing the episode number data. Unfortunately, the server is sending "??" as the episode number value. You can search the debug log recently sent for the text below to see what I mean.

        "IndexNumber": "??",
        "ParentIndexNumber": 2024,
        "SeriesName": "Lex Fridman",
        "SeriesId": "d6d87857b5e02eb535430dd2e6289b47",
        "IndexNumber": 74,
        "ParentIndexNumber": 2024,
        "SeriesName": "Lex Fridman",
        "SeriesId": "d6d87857b5e02eb535430dd2e6289b47",

From the episodes in the log it appears some shows (e.g. Bill M.) do a better job at having the episode number populated than others. The example above you can see Lex F. has one episode (# 74) populated, but the rest show "??".

I think I recall you mentioning there was another app/program being used to import the podcasts. May have to try reaching out to those developer(s) to see if there is anyway to populate the IndexNumber with the proper episode value; instead of "??".

Side Note: I see with a quick Google of "podcast episode numbering", the podcast creation community has some episode numbering inconsistencies to work thru on how episodes numbers will be shown. Taking a guess here, this may not be solvable until a standard way is agreed upon (e.g. like TV episodes). I do see Apple has made their position clear. But a quick read about it did not really show if any other major streaming media companies like Google, Amazon, Spotify, etc... have made a decision on what they prefer to see to help guide the masses and organize podcast episodes.

I am not a podcast creator and do not 100% know how this works. But I would expect when a podcast episode is uploaded there should be fields to enter the episode information (i.e. Title, Episode, Season, etc...). Maybe the podcast episode upload tools do not provide this option? Hence why we see some without episode numbers. I am working with a small sample size; it appears the episodes with S##E## in the title have both the season and episode numbers properly filled out.

Doug411 commented 1 month ago

That's really strange. A few iterations ago, prior to attempting to enable episode control, it was working for all my podcasts. At that time, all of the podcast series had items for deletion. I wonder if my metadata got updated inadvertently and it is unrelated to recent changes to the code?

Doug411 commented 1 month ago

Also, I noticed there were several items that seem to have no episode number that are being suggested for deletion. Why would it suggest to delete those, but not similar items from other podcast series?

eg.

[DELETED] Episode - Lex Fridman - s2024.e00??

terrelsa13 commented 1 month ago

That's really strange. A few iterations ago, prior to attempting to enable episode control, it was working for all my podcasts. At that time, all of the podcast series had items for deletion. I wonder if my metadata got updated inadvertently and it is unrelated to recent changes to the code?

The script does not modify any season/series metadata.

Also, I noticed there were several items that seem to have no episode number that are being suggested for deletion. Why would it suggest to delete those, but not similar items from other podcast series?

eg.

[DELETED] Episode - Lex Fridman - s2024.e00??

I am looking at the last log sent it and all x15 Lex F. episodes created >=10days ago were fake deleted. Same for Bill M. x11 episodes created >=10days ago were fake deleted.

Am I misunderstanding?

Doug411 commented 1 month ago

Nope. But I noticed that some of lex f had episode numbers and some didn't. Wondering why Megyn K has nothing suggested for deletion?

On Fri, Jul 26, 2024, 11:47 PM terrelsa13 @.***> wrote:

That's really strange. A few iterations ago, prior to attempting to enable episode control, it was working for all my podcasts. At that time, all of the podcast series had items for deletion. I wonder if my metadata got updated inadvertently and it is unrelated to recent changes to the code?

The script does not modify any season/series metadata.

Also, I noticed there were several items that seem to have no episode number that are being suggested for deletion. Why would it suggest to delete those, but not similar items from other podcast series?

eg.

[DELETED] Episode - Lex Fridman - s2024.e00??

I am looking at the last log sent it and all x15 Lex F. episodes created

=10days ago were fake deleted. Same for Bill M. x11 episodes created >=10days ago were fake deleted.

Am I misunderstanding?

— Reply to this email directly, view it on GitHub https://github.com/terrelsa13/MUMC/issues/109#issuecomment-2253759063, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHDODHSLIY3NISW5OQCQE3TZOMQ5LAVCNFSM6AAAAABIVXMBFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJTG42TSMBWGM . You are receiving this because you authored the thread.Message ID: @.***>

terrelsa13 commented 1 month ago

But I noticed that some of lex f had episode numbers and some didn't.

The ?? for episode numbers is happening outside of MUMC. My best guess is:

I think I recall you mentioning there was another app/program being used to import the podcasts. May have to try reaching out to those developer(s) to see if there is anyway to populate the IndexNumber with the proper episode value; instead of "??".

Wondering why Megyn K has nothing suggested for deletion?

Nothing was suggested for deletion because no Megyn K. episodes were queried (aka they did not make it past the filter_statements) for either of the users.

I had you disable querying for (un)played whitelisted episodes to help remove some of the noise. Now that we see whitetagging is working as expected. Make sure the Megyn K. series, seasons, and episodes are in one of these two whitelisted folders:

Once you have confirmed Megyn K. is in one of the two whitelisted folders shown above; make the change shown below.

How it looks now:

advanced_settings:
  filter_statements:
    episode:
      query_filter:
        whitelisted:
          played: false

How we want it to look:

advanced_settings:
  filter_statements:
    episode:
      query_filter:
        whitelisted:
          played: true

I am now able to make an assumption about the Megyn K. series, seasons, and/or episodes. They do NOT have a matching blacktag (i.e. Retain14d or auto-purge).

Explanation of why I am saying the Megyn K. series, seasons, and/or episodes are not blacktagged :

How it looks now:

advanced_settings: filter_statements: episode: query_filter: - What kind of queries should MUMC "ask" Emby/Jellyfin whitelisted: - For whitelist folders... favorited: false - do not specifically ask for favorited episodes whitetagged: false - do not specifically ask for whitetagged episodes blacktagged: true - do specifically ask for blacktagged episodes played: false - do not specifically ask for (un)played episodes

First query:

MUMC specifically asks Emby/Jellyfin to only answer with media_items that are in the whitelisted folders and have a blacktag matching either Retain14d or auto-purge.

Second query:

No second query.

How we want it to look:

advanced_settings: filter_statements: episode: query_filter: - What kind of queries should MUMC "ask" Emby/Jellyfin whitelisted: - For whitelist folders... favorited: false - do not specifically ask for favorited episodes whitetagged: false - do not specifically ask for whitetagged episodes blacktagged: true - do specifically ask for blacktagged episodes played: true - do specifically ask for (un)played episodes

First query:

Same as above... MUMC will specifically ask Emby/Jellyfin to only answer with media_items that are in the whitelisted folders and have a blacktag matching either Retain14d or auto-purge.

Second query:

  1. Now that played is true, the script will look at the basic_settings > filter_statements > episode > played & created > count_equality & count.

    1. For your specific config played is ignored because condition_days: -1.

    2. However, created is not ignored.

      1. The script uses count_equality & count to determine if it should query for "unplayed", "played", or "unplayed + played" media_items.
      2. In your case >= 0 overlaps both unplayed and played.

  2. MUMC will specifically ask Emby/Jellyfin to only answer with media_items that are in the whitelisted folders and are either "unplayed or played" (aka everything)

It is possible for a media_item to be all or none of the configured behaviors (e.g. favorited, whitetagged, blacktagged, and (un)played).

Because of this, MUMC will filter out any overlap. It does not want to process the same media_item more that once.

Next, post-processing is run using the behavioral_statements config values.

Finally media_items that have made it through both filter_statements and behavioral_statements are what is sent for deletion.

terrelsa13 commented 1 month ago

If the intention is for the Megyn K. series, seasons, episodes to be blacktagged. Then do not perform any of the steps I mentioned above. Instead go the the Megyn K. metadata and fix/correct the tag(s).

If that does not work; try moving the blacktags for one of the series we verified is working Bill M. and/or Lex F to match where they are for Megyn K. For example, move the Bill M. blacktag to the series level if the blacktag at the Megyn K series level is not working.

Now does this break Bill M. or does Bill M. still show correctly?

Doug411 commented 1 month ago

I didnt make the first set of changes (yet). I just removed my blacktags, and retagged them. When I add a blacktag to a series in Jellyfin, it automatically adds the blacktag to the season folders for that Series as well. I now have 9 podcasts blacktagged in the Youtube Library. Under my series library I only have 1 show blacktagged (Bill M). I played around with tagging and untagging some of the podcasts in the Youtube Library. I have noticed that it seems to flag for deletion episodes in the first Podcast alphabetically and the last Podcast alphabetically. It does not flag any episodes for deletion in between the first and last podcasts. I've untagged the last alphabetically, and it just shifts to the showing me the first and the (next) last Podcast alphabetically. It almost seems like there is a loop that isnt iterating through correctly. It's just grabbing the first and last series in the array.

I also think it is no longer flagging all the episodes within the first and last that match the filter criteria (just a subset).

As for the first set of changes, is that something I should try? What was that intended to fix? In answer to your question, my desire is to blacktag specific series in both those white tagged libraries (youtube, and series).

Also what issues do you think having episode eo??? will cause? I want to understand why you mentioned that as an issue. I know at one point everything was working as far as deletions (prior to me messing around with episode control). I don't think episode control is related, just as far as timing is concerned, if we scroll up in this thread. Maybe I should rollback to a prior version and see if it works?

terrelsa13 commented 1 month ago

Also what issues do you think having episode eo??? will cause? I want to understand why you mentioned that as an issue. I know at one point everything was working as far as deletions (prior to me messing around with episode control). I don't think episode control is related, just as far as timing is concerned, if we scroll up in this thread. Maybe I should rollback to a prior version and see if it works?

The ?? episode thing is only an issue when using episode_control.

As for the first set of changes, is that something I should try? What was that intended to fix? In answer to your question, my desire is to blacktag specific series in both those white tagged libraries (youtube, and series).

There is something else going on with your setup. Doing those changes is not going to help like I originally thought.

When I add a blacktag to a series in Jellyfin, it automatically adds the blacktag to the season folders for that Series as well.

Update: My jellyfin now has this behavior. I did not change anything. My best guess is I either was not looking in the right places or this behavior only happens on certain series/seasons.

When I blacktag a series the seasons are not automatically tagged by Jellyfin. It sounds like a cool feature though. Is there some Jellyfin setting you turned on to make it do this???

I didnt make the first set of changes (yet). I just removed my blacktags, and retagged them. [...] I now have 9 podcasts blacktagged in the Youtube Library. Under my series library I only have 1 show blacktagged (Bill M). I played around with tagging and untagging some of the podcasts in the Youtube Library. I have noticed that it seems to flag for deletion episodes in the first Podcast alphabetically and the last Podcast alphabetically. It does not flag any episodes for deletion in between the first and last podcasts. I've untagged the last alphabetically, and it just shifts to the showing me the first and the (next) last Podcast alphabetically. It almost seems like there is a loop that isnt iterating through correctly. It's just grabbing the first and last series in the array.

I also think it is no longer flagging all the episodes within the first and last that match the filter criteria (just a subset).

I just setup my JF server to mimic your setup (or at least how I think I understand your setup).

Are you running the latest version v5.8.29? I do not recall changing anything related to how the filter_statements work. But stranger things have happened.

terrelsa13 commented 1 month ago

Also attaching my mumc_config.yaml.txt for reference.

terrelsa13 commented 4 weeks ago

I re-ran my mimic version of your setup and I still see the same result I saw a few weeks ago.

Were you able to figure out if there was something different with your jellyfin install?

terrelsa13 commented 4 weeks ago

I have been working on your pre-defined tags idea. Re-reading your comment I realize you are asking for a way to have different retention periods for different media types. This is already I feature. From our past discussions I see you were using the global whitetag and global blacktag variables. There are also places for media_type specific tags.

Global Tag Location

advanced_settings:
  whitetags: [global,whitetags,go,here]
  blacktags: [global,blacktags,go,here]

Media Specific Tag Location

advanced_settings:
  behavioral_statements:
    movie:
      whitetagged:
        tags: [movie,specific,whitetags,go,here]
      blacktagged:
        tags: [movie,specific,blacktags,go,here]
    episode:
      whitetagged:
        tags: [series,season,episode,specific,whitetags,go,here]
      blacktagged:
        tags: [series,season,episode,specific,blacktags,go,here]