search

terrelsa13 / MUMC

Multi-User Media Cleaner aka MUMC (pronounced Mew-Mick) will go through movies, tv episodes, audio tracks, and audiobooks in your Emby/Jellyfin libraries deleting media items you no longer want.
GNU General Public License v3.0
92 stars 6 forks source link

v5 Errors #88

Closed terrelsa13 closed 10 months ago

terrelsa13 commented 11 months ago

Thank you for your detailed feedback. I've just reinstall all my set-up to V5 and I'm trying to set-up MUMC but every time I get an error:

  1. One of my libraries (Top Picks) does not have a "CollectionType". To solve the problem, I deleted this library, but I think I should add a check to verify that the variable is defined here: https://github.com/terrelsa13/MUMC/blob/93b282d6049f597a0fd16631c487df181f133c8f/mumc_modules/mumc_userlib_builder.py#L796 
    
Traceback (most recent call last):
File "/opt/MUMC_v5/mumc_modules/mumc_config_import.py", line 86, in importConfig
import mumc_config as cfg
ModuleNotFoundError: No module named 'mumc_config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/MUMC_v5/mumc.py", line 113, in main() File "/opt/MUMC_v5/mumc.py", line 31, in main cfg,init_dict=importConfig(init_dict,cmdopt_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/MUMC_v5/mumc_modules/mumc_config_import.py", line 102, in importConfig importHasException(init_dict,cmdopt_dict) File "/opt/MUMC_v5/mumc_modules/mumc_config_import.py", line 22, in importHasException build_configuration_file(init_dict) File "/opt/MUMC_v5/mumc_modules/mumc_config_builder.py", line 115, in build_configuration_file the_dict['admin_settings']['users']=get_users_and_libraries(the_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/MUMC_v5/mumc_modules/mumc_userlib_builder.py", line 796, in get_users_and_libraries if (not (folder['CollectionType'] == 'boxsets')): 


```

2. When I try to initialize the script to generate the configuration file, I have a problem with a "Version" key
```
Monitoring multiple users is possible.                                                                                                                                                                               
When multiple users are selected; the user with the oldest last played time will determine if media can be deleted.                                                                                                  
Select one user at a time.                                                                                                                                                                                           
Enter number of the next user to monitor; leave blank when finished:                                                                                                                                                 

----------------------------------------------------------------------------------------                                                                                                                             
----------------------------------------------------------------------------------------                                                                                                                             
Traceback (most recent call last):                                                                                                                                                                                   
  File "/opt/MUMC_v5/mumc_modules/mumc_config_import.py", line 86, in importConfig                                                                                                                                   
    import mumc_config as cfg                                                                                                                                                                                        
ModuleNotFoundError: No module named 'mumc_config'                                                                                                                                                                   

During handling of the above exception, another exception occurred:                                                                                                                                                  

Traceback (most recent call last):                                                                                                                                                                                   
  File "/opt/MUMC_v5/mumc.py", line 113, in <module>                                                                                                                                                                 
    main()                                                                                                                                                                                                           
  File "/opt/MUMC_v5/mumc.py", line 31, in main                                                                                                                                                                      
    cfg,init_dict=importConfig(init_dict,cmdopt_dict)                                                                                                                                                                
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                
  File "/opt/MUMC_v5/mumc_modules/mumc_config_import.py", line 102, in importConfig                                                                                                                                  
    importHasException(init_dict,cmdopt_dict)                                                                                                                                                                        
  File "/opt/MUMC_v5/mumc_modules/mumc_config_import.py", line 22, in importHasException                                                                                                                             
    build_configuration_file(init_dict)                                                                                                                                                                              
  File "/opt/MUMC_v5/mumc_modules/mumc_config_builder.py", line 156, in build_configuration_file                                                                                                                     
    yaml_configurationBuilder(the_dict)                                                                                                                                                                              
  File "/opt/MUMC_v5/mumc_modules/mumc_configuration_yaml.py", line 79, in yaml_configurationBuilder                                                                                                                 
    config_data={thekey:the_dict_copy[thekey] for thekey in the_dict_keys}                                                                                                                                           
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                           
  File "/opt/MUMC_v5/mumc_modules/mumc_configuration_yaml.py", line 79, in <dictcomp>                                                                                                                                
    config_data={thekey:the_dict_copy[thekey] for thekey in the_dict_keys}                                                                                                                                           
                        ~~~~~~~~~~~~~^^^^^^^^
KeyError: 'version'
```

I tried to figure out where it was coming from but couldn't. So I tried to remove the "version" key by replacing this line:
https://github.com/terrelsa13/MUMC/blob/93b282d6049f597a0fd16631c487df181f133c8f/mumc_modules/mumc_configuration_yaml.py#L78

with this:
```
    the_dict_keys=[,'basic_settings','advanced_settings','admin_settings','DEBUG']
```

But I think it's stuck somewhere. The configuration file is created, but every time I run the script again, it throws back the initial set-up to configure the server. I've checked that the parameter is set to False: `UPDATE_CONFIG: false`.

_Originally posted by @Floflobel in https://github.com/terrelsa13/MUMC/issues/83#issuecomment-1754962567_
terrelsa13 commented 11 months ago

@Floflobel Thanks for trying v5 out. I will take a look at these errors once I get home.

terrelsa13 commented 11 months ago

Quick suggestion. The script is supposed to convert the legacy config to the new yaml format, if it can find it. If it cannot find a legacy config or a yaml config it should assume a new install. Try copying the v4 config into the mumc_v5 folder. See if that helps.

Floflobel commented 11 months ago

Thanks for creating the new ticket, which will be much better. I'm actually going to try to use V4 config file, but I would have preferred to start from scratch. I'll wait for your feedback and keep looking on my side.

terrelsa13 commented 11 months ago

For the 2nd issue I see the problem. Not home yet, so cannot fix it. But if you are able, open mumc_config_builder.py. Go to line #32 press enter. Then type the_dict['version']=get_script_version()

Make sure you undo your change to the_dict_keys list.

Floflobel commented 11 months ago

Well done! It worked and I was able to generate the configuration. I came across another error:

Traceback (most recent call last):                                                                                                                                                                                   
  File "/opt/MUMC_v5/mumc.py", line 113, in <module>                                                                                                                                                                 
    main()                                                                                                                                                                                                           
  File "/opt/MUMC_v5/mumc.py", line 34, in main                                                                                                                                                                      
    cfgCheckYAML(cfg,init_dict)                                                                                                                                                                                      
  File "/opt/MUMC_v5/mumc_modules/mumc_yaml_check.py", line 3478, in cfgCheckYAML                                                                                                                                    
    raise RuntimeError('\n' + error_found_in_mumc_config_yaml)                                                                                                                                                       

RuntimeError:                                                                                                                                                                                                        
ValueError: admin_settings > users > user_id: 5417e14c6a3b4394866cbefe7bfaa025 > whitelist > network_path: None is not an expected value                                                                             
ValueError: admin_settings > users > user_id: 5417e14c6a3b4394866cbefe7bfaa025 > whitelist > network_path: None is not an expected value                                                                             
ValueError: admin_settings > users > user_id: 5417e14c6a3b4394866cbefe7bfaa025 > whitelist > network_path: None is not an expected value                                                                             
ValueError: admin_settings > users > user_id: 5417e14c6a3b4394866cbefe7bfaa025 > whitelist > network_path: None is not an expected value                                                                             
ValueError: admin_settings > users > user_id: 5417e14c6a3b4394866cbefe7bfaa025 > whitelist > network_path: None is not an expected value                                                                             
ValueError: admin_settings > users > user_id: 5417e14c6a3b4394866cbefe7bfaa025 > whitelist > network_path: None is not an expected value

Config example:

  users:
  - user_id: 5417e14c6a3b4394866cbefe7bfaa025
    user_name: User1
    whitelist:
    - lib_id: db4c1708cbb5dd1676284a40f2950aba
      collection_type: movies
      path: /media/storage3/films
      network_path: null
      lib_enabled: true

I temporarily replaced it with this to correct the problem: https://github.com/terrelsa13/MUMC/blob/5ed1217166de8bf258e787187bcd899e9bc970d8/mumc_modules/mumc_yaml_check.py#L119

check_item is None
Floflobel commented 11 months ago

I think there's another problem with whitelisted/blacklisted libraries. I've chosen libraries to be whitelisted or blacklisted, but I can never see it in the script output:

:[KEEPING] - Movie - [REC]⁴ : Apocalypse - Filmax - Unplayed - Play Count: 0 - Created 1046 days ago - Favorite: False - Whitetag: True - Blacktag: False - Whitelisted: False - Blacklisted: False - MovieID: 425196

It looks like they've gone into whitetag, even though I didn't activate this option at all during initialization. I can provide my config and logs if needed.

Edit: I have the impression that others are nowhere to be found while the library is well referenced.

:[KEEPING] - Movie - Annihilation - DNA Films - Played 1619 days ago - Play Count: 3 - Created 1018 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: False - MovieID: 1188033
terrelsa13 commented 11 months ago

Found it. Fixed it. Script was checking if black/whitetags where true with if (tag): but the tag variable is an empty string. Changed to if (tag == ''): and now it does not think everything is tagged. Should be a v5.0.2 on the beta branch. You can give that a try; but I think there might be something going on with the behavioral statements. Something I think should not be deleted appears to be getting deleted. I will look into that more tomorrow.

terrelsa13 commented 11 months ago

Ignore v5.0.2 and use v5.0.3 instead. Was able to fix the bug in the behavioral statements. On my side I am able to set the following and it appears to work. Let me know if it behaves the same way for you.

basic_settings:
  filter_statements:
    movie:
      played:
        condition_days: -1
        count_equality: '>='
        count: 1
      created:
        condition_days: 300
        count_equality: '>='
        count: 1
        behavioral_control: true
advanced_settings:
  behavioral_statements:
    movie:
    .
    .
    .
      blacklisted:
        action: delete
        user_conditional: any
        played_conditional: any_created
        action_control: 3

The above settings will delete movies in blacklisted libraries created 300 or more days ago:

terrelsa13 commented 11 months ago

Apologies, I made some assumptions in my last post.

I will specify the easiest path; set action_control: 0 to disable any bahavioral statements for favorited, whitetagged, blacktagged, and whitelisted.

advanced_settings:
  behavioral_statements:
    movie:
      favorited:
        action_control: 0
      whitetagged:
        action_control: 0
      blacktagged:
        action_control: 0
      whitelisted:
        action_control: 0
Floflobel commented 11 months ago

Okay, I've applied all your recommendations and it seems to want to remove the good media from what I can see. There's a little problem on the display when it processes the media where it says it's keeping it but in the end it deletes it:

:[KEEPING] - Movie - Snowpiercer : le Transperceneige - CJ Entertainment - Played 951 days ago - Play Count: 3 - Created 1110 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 1189029
:[KEEPING] - Movie - The Craft : Les nouvelles sorcières - Blumhouse Productions - Played 282 days ago - Play Count: 1 - Created 1073 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 374862
:[KEEPING] - Movie - Zookeeper - Happy Madison Productions - Played 213 days ago - Play Count: 1 - Created 1093 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 336027
MOVIE POST PROCESSING STARTED...
[DELETED]     Movie - Snowpiercer : le Transperceneige - 1189029
[DELETED]     Movie - The Craft : Les nouvelles sorcières - 374862
[DELETED]     Movie - Zookeeper - 336027

Thank you for your hard work !

Floflobel commented 11 months ago

I've just tried changing these parameters:

basic_settings:
  filter_statements:
    movie:
      played:
        condition_days: -1
        count_equality: '>='
        count: 1
      created:
        condition_days: 300
        count_equality: '=='
        count: 0
        behavioral_control: true

or

basic_settings:
  filter_statements:
    movie:
      played:
        condition_days: -1
        count_equality: '>='
        count: 1
      created:
        condition_days: 300
        count_equality: '<'
        count: 1
        behavioral_control: true

Logically, it should want to remove media older than 300 days with 0 plays on the counter. It doesn't seem to want to, because it's not deleting anything.

:[KEEPING] - Movie - Volte/Face - Krane Entertainment - Unplayed - Play Count: 0 - Created 1245 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 1176381
:[KEEPING] - Movie - Voyage à Tokyo - Shochiku Co., Ltd. - Unplayed - Play Count: 0 - Created 1107 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 318831
:[KEEPING] - Movie - Voyage au bout de l’enfer - Universal Pictures - Unplayed - Play Count: 0 - Created 993 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 618735
MOVIE POST PROCESSING STARTED...

Perhaps I've misunderstood something.

EDIT:

It seems that if I make this configuration everything goes well:

basic_settings:
  filter_statements:
    movie:
      played:
        condition_days: -1
        count_equality: '>='
        count: 1
      created:
        condition_days: 300
        count_equality: '=='
        count: 1
        behavioral_control: true
:[KEEPING] - Movie - Snowpiercer : le Transperceneige - CJ Entertainment - Played 951 days ago - Play Count: 3 - Created 1110 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 1189029
:[KEEPING] - Movie - Tenet - Warner Bros. Pictures - Played 30 days ago - Play Count: 1 - Created 36 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 1237391
:*[DELETE] - Movie - The Craft : Les nouvelles sorcières - Blumhouse Productions - Played 282 days ago - Play Count: 1 - Created 1073 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 374862
:*[DELETE] - Movie - Zookeeper - Happy Madison Productions - Played 213 days ago - Play Count: 1 - Created 1093 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 336027
MOVIE POST PROCESSING STARTED...
[DELETED]     Movie - The Craft : Les nouvelles sorcières - 374862
[DELETED]     Movie - Zookeeper - 336027
terrelsa13 commented 11 months ago

Ahh... I see where this is still getting hung up.

In def get_playedCreatedDays_playedCreatedCounts(the_dict,item,var_dict): the script was only running the necessary checks if the media item was played. Checks have now been added for unplayed media items. Give it another shot. v5.0.8 should be in the beta branch now.

Use these settings:

basic_settings:
  filter_statements:
    movie:
      played:
        condition_days: -1
        count_equality: '>='
        count: 1
      created:
        condition_days: 300
        count_equality: '=='
        count: 0
        behavioral_control: true
advanced_settings:
  behavioral_statements:
    movie:
      favorited:
        action: keep
        user_conditional: any
        played_conditional: ignore
        action_control: 0
        extra:
          genre: 0
          library_genre: 0
      whitetagged:
        action: keep
        user_conditional: all
        played_conditional: ignore
        action_control: 0
        tags: []
      blacktagged:
        action: delete
        user_conditional: all
        played_conditional: any_any
        action_control: 0
        tags: []
      whitelisted:
        action: keep
        user_conditional: all
        played_conditional: any_any
        action_control: 0
      blacklisted:
        action: delete
        user_conditional: any
        played_conditional: any_created
        action_control: 3
Floflobel commented 11 months ago

Wonderful, everything works fine with just one user. I tried with two users but it doesn't seem to do what I want with my configuration: I would like that if one of my users has seen the media, it is not deleted. Currently: DELETE + KEEPING = DELETE Desired: DELETE + KEEPING = KEEPING

With several users, this should go in the same direction: DELETE + DELETE + DELETE + DELETE + ... + KEEPING = KEEPING

I've tried to change the configuration in this direction but it doesn't work:

User1 - 5417e14c6a3b4394866cbefe7bfaa025
:*[DELETE] - Movie - Free Guy - Berlanti Productions - Unplayed - Play Count: 0 - Created 727 days ago - Favorite: False - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 1098811
User2 - 4fd64ef2422e40e0ba6e095eb9f74ecd
:[KEEPING] - Movie - Free Guy - Berlanti Productions - Played 589 days ago - Play Count: 4 - Created 727 days ago - Favorite: True - Whitetag: False - Blacktag: False - Whitelisted: False - Blacklisted: True - MovieID: 1098811
MOVIE POST PROCESSING STARTED...
[DELETED]     Movie - Free Guy - 1098811

config:

version: 5.0.7
basic_settings:
  filter_statements:
    movie:
      played:
        condition_days: -1
        count_equality: '>='
        count: 1
      created:
        condition_days: 600
        count_equality: ==
        count: 0
        behavioral_control: true
...
advanced_settings:
  behavioral_statements:
...
      blacklisted:
        action: delete
        user_conditional: all
        played_conditional: all_created
        action_control: 3

EDIT: Same with the 5.0.10

terrelsa13 commented 11 months ago

My fault.

  1. Movie is added to the delete list during the data fetch for user1.
  2. Movie is not add added to the delete list during the data fetch for user2.

During post processing:

Change action_control: 3 to action_control: 5.

During post processing:

Floflobel commented 11 months ago

It works perfectly! Thanks for the explanation. The doc is indeed missing, but I think you're waiting to finish v5 before creating it, or maybe I haven't seen it.

I think I've reached my ideal setup, thank you very much for your time and this wonderful project.

Where can I send you a small donation?

terrelsa13 commented 11 months ago

Awesome! Glad we got it working the way you want.

Oh yes, the documentation is indeed missing. I'm working on learning how to create a wiki to make it more intuitive to explain/read how the different options make the script work. Hopefully I can get that done sometime soon.

There is a donate button at the bottom of the README. Thank you for showing your support!

terrelsa13 commented 10 months ago

@Floflobel Sounds like we are good. I am going to close this issue. If you need help or run into a bug go ahead and open a new issue.