search

ilarramendi / BetterCovers

BetterCovers is a fully customizable script to automaticaly generate covers and backdrops with embeded ratings, mediainfo, language, certifications, age ratings, source, production companies, etc!
GNU General Public License v3.0
40 stars 2 forks source link

Improving hit ratio for movies #4

Closed samcro1967 closed 2 years ago

samcro1967 commented 2 years ago

I have 1,500+ movies. 400 of them are not matched. Any way I can improve the hit ratio thru BetterCovers config? Movies are stored in separate folders. Folder and files are named "MovieName (Year).ext".

Sample of container logs of a run Link

ilarramendi commented 2 years ago

Helo, sorry for the late response! Can are you still having the problem? i cant access the logs

samcro1967 commented 2 years ago

Yes, here is a snippet of the logs.

Link

ilarramendi commented 2 years ago

Let me take a look at that, but on the meantime if its a posibility with radarr you can rename to the prefered naming scheme: {Movie Title} ({Release Year}) [imdbid={ImdbId}] [tmdbid={TmdbId}].ext This dosnt require title matching and just grabs the id

samcro1967 commented 2 years ago

I've got some other apps in the workflows dependent on the naming convention. Not sure they will all support that format and would be quite a bit of work to try to retrofit everything.

A Google search of any movie name along with the year will always return the IMDB link as the first in my experience. Maybe that could be an secondary search method if the method you are using now fails. Or use that method first if the filename is not in your preferred format. I'm sure that method will be much higher than 66% for non preferred file name formats.

ilarramendi commented 2 years ago

Hit ratio should have improved, please pull the latest version or update the docker container. Previous metadata, config and templates are obsolete, so i would suggest deleting the BetterCovers directory and downloading again.

When you have time please try it and tell me how it went, feedback is always apreciated. Also if you are still having trouble with a specific movie try running with parameters: -o -w 1 --log-level 10 --no-colors --json only for that specific movie ( /movies/movieFolderName) and send me the log and metadata.json for further analysis

samcro1967 commented 2 years ago

The new container is not creating a default /config/config/config.json and I do not see one in the repo. Am I missing something or is there a place to get the latest template?

ilarramendi commented 2 years ago

I think the last update i did to the container had something in cache, im updating it again. Unrelated with the container problem i also added the config,json file that was missing to github, thanks for noticing.

samcro1967 commented 2 years ago

Here is what I am seeing against just one movie that is not matching. Permissions on the file are 777 and the container is running as root. It seems like it did not process the file for some reason. Guessing I am not doing something quite right for it to process just one movie?

docker run

docker run -i --rm -v "/mnt/nas/xbmc/Movies/Almost Famous (2000)":/media -v /home/user/Documents/Docker/better-covers:/config -e parameters="-o -w 1 --log-level 10 --no-colors --json" -e TZ=America/Chicago ilarramendi/better-covers

Logs

Succesfully updated IMDB Ratings Dataset
Succesfully updated IMDB Episodes Dataset
[05/21/2022 08:42:48] --> Starting BetterCovers for directory: /media/*
[05/21/2022 08:42:48] --> Not updating  library, API key not set.
[05/21/2022 08:42:48] --> Done, total time was: 0:00:00 and generated: 0 images.

metadata.json

{
       "items": [],
       "version": 6
}
ilarramendi commented 2 years ago

Mounting a single movie in the media volume is not supported by the container yet, i should fix that. For now for testing a single file you can try: docker run -i --rm -v /mnt/nas/xbmc/Movies/:/media -v /home/user/Documents/Docker/better-covers:/config -e parameters="-o -w 1 --log-level 10 --no-colors --json" -e TZ=America/Chicago -e fileMask="Almost Famous (2000)" ilarramendi/better-covers
Cant realy test if this syntax works right now but im guessing it should.

samcro1967 commented 2 years ago

That produced the following error:

[8] Failed to execute script BetterCovers
Traceback (most recent call last):
  File "BetterCovers.py", line 210, in <module>
  File "json/__init__.py", line 296, in load
  File "json/__init__.py", line 348, in loads
  File "json/decoder.py", line 337, in decode
  File "json/decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ilarramendi commented 2 years ago

And just running normaly? docker run -i --rm -v /mnt/nas/xbmc/Movies/:/media -v /home/user/Documents/Docker/better-covers:/config -e TZ=America/Chicago ilarramendi/better-covers

samcro1967 commented 2 years ago

That was with

docker run -i --rm -v /mnt/nas/xbmc/Movies/:/media -v /home/user/Documents/Docker/better-covers:/config -e parameters="-o -w 1 --log-level 10 --no-colors --json" -e TZ=America/Chicago -e fileMask="Almost Famous (2000)" ilarramendi/better-covers
ilarramendi commented 2 years ago

You are using the wrong container lol. The correct name is ilarramendi/bettercovers i just edited your command and didnt notice, that was the old container name. Run: docker run -i --rm -v /mnt/nas/xbmc/Movies/:/media -v /home/user/Documents/Docker/better-covers:/config -e TZ=America/Chicago ilarramendi/bettercovers and it should work, probably (?

samcro1967 commented 2 years ago

Oh boy, my bad. I did not realize it had been renamed. Below is what I got. It did not match the correct movie or download artwork, but it looks like the artwork was not available for downland.

Logs

Succesfully updated IMDB Ratings Dataset
Succesfully updated IMDB Episodes Dataset
[05/21/2022 13:49:00] --> Starting BetterCovers for directory: /media/Almost Famous (2000)
[05/21/2022 13:49:00] --> Processing: Almost Famous
[05/21/2022 13:49:11] --> Finished getting TMDB metadata for: My Almost Famous Family in: 0:00:10
[05/21/2022 13:49:11] --> Found IMDB rating in dataset for: My Almost Famous Family: 8.4, in: 0:00:00
[05/21/2022 13:49:16] --> Finished getting OMDB metadata for: My Almost Famous Family in: 0:00:05
[05/21/2022 13:49:16] --> No results found on RottenTomatoes for: My Almost Famous Family
[05/21/2022 13:49:16] --> No result found in TVTime for: My Almost Famous Family
[05/21/2022 13:49:17] --> Finished getting Trakt ratings for: My Almost Famous Family in: 0:00:06
[05/21/2022 13:49:19] --> Error getting MetaCritic ratings for: My Almost Famous Family
[05/21/2022 13:49:19] --> Missing cover image for: My Almost Famous Family
[05/21/2022 13:49:19] --> Missing backdrop image for: My Almost Famous Family
[05/21/2022 13:49:19] --> Finished getting metadata for: My Almost Famous Family, and generated 0 tasks in: 0:00:19
[05/21/2022 13:49:19] --> Not updating  library, API key not set.
[05/21/2022 13:49:19] --> Done, total time was: 0:00:19 and generated: 0 images.

metedata.json

{
       "items": [
              {
                     "ageRating": "NR",
                     "certifications": [],
                     "folder": "/media/Almost Famous (2000)",
                     "ids": {
                            "IMDB": "tt1464668",
                            "TMDB": "18834"
                     },
                     "images": {
                            "backdrops": [],
                            "covers": [],
                            "logos": []
                     },
                     "media_info": {},
                     "path": "/media/Almost Famous (2000)",
                     "production_companies": [],
                     "ratings": {
                            "IMDB": {
                                   "icon": "IMDB",
                                   "value": "8.4"
                            },
                            "Trakt": {
                                   "icon": "Trakt",
                                   "value": "7.6"
                            }
                     },
                     "release_date": "2022-05-21 13:49:00.914541",
                     "seasons": [],
                     "title": "My Almost Famous Family",
                     "trailers": [],
                     "type": "tv",
                     "updates": {
                            "IMDB": "2022-05-21 13:49:11.384406",
                            "LB": "1970-01-01 00:00:01",
                            "MC": "1970-01-01 00:00:01",
                            "OMDB": "2022-05-21 13:49:16.466720",
                            "RT": "1970-01-01 00:00:01",
                            "TMDB": "2022-05-21 13:49:11.313986",
                            "TVTime": "1970-01-01 00:00:01",
                            "Trakt": "2022-05-21 13:49:17.097283",
                            "mediaInfo": "1970-01-01 00:00:01"
                     },
                     "urls": {},
                     "year": 2000
              }
       ],
       "version": 6
}
ilarramendi commented 2 years ago

It worked but it didnt match the correct movie, ill take a look at that and maby revert the changea that i did since maby the low hit rate was in a older version

ilarramendi commented 2 years ago

If you run it again with all movies can you tell me if the hitrate improved and if you see any missmatch?

samcro1967 commented 2 years ago

It is running now. Will post results later when it is finished. Still seeing this error message periodically:

Exception in thread Thread-155:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/BetterCovers/src/Movie.py", line 460, in updateMetadata
    _getTMDB() # Get general information first
  File "/BetterCovers/src/Movie.py", line 194, in _getTMDB
    if result['name'].lower() == self.title.lower():
KeyError: 'name'
samcro1967 commented 2 years ago

It is done running and I have tried to summarize the results. It is difficult to determine from the logs when a mismatch occurs. Can you add a line to the logs to make it easier to identify? It would then be easier to identify.

[TAG  ][Date Time] --> Matched: "FileName" to "Match"
[Info  ][05/21/2022 13:49:00] --> Matched: "Almost Famous (2000)" to "My Almost Famous Family"

Log file

Container logs

ilarramendi commented 2 years ago

Not updating library, API key not set. This line runs at the end of the script so these logs are for multiple runs? ill fix the name error for next version, as well as the match log!

samcro1967 commented 2 years ago

Here is a grouping of them together in one run. They are a few seconds apart

[Error  ][05/21/2022 14:04:07] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:11] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:14] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:18] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:21] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:26] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:05] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:08] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:11] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:15] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:19] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:23] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:29] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:39] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:55] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:06:23] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:07:18] --> Not updating  library, API key not set.
samcro1967 commented 2 years ago

I might also suggest consolidating some error messages to reduce the log output. Perhaps something like this:

[Error ][05/21/2022 13:49:00] --> Failed getting metadata for MetaCritic, OMDB, RT, TMDB, Trakt, and TVTime" [Info ][05/21/2022 13:49:00] --> Finished getting metadata for IMDB and TMDB"

The same could be done for No need to update, but it does not seem to generate as many messages.

ilarramendi commented 2 years ago

Here is a grouping of them together in one run. They are a few seconds apart

[Error  ][05/21/2022 14:04:07] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:11] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:14] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:18] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:21] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:04:26] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:05] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:08] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:11] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:15] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:19] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:23] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:29] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:39] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:05:55] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:06:23] --> Not updating  library, API key not set.
[Error  ][05/21/2022 14:07:18] --> Not updating  library, API key not set.

Thats not good, do you see the same number of times the line: Starting BetterCovers for directory: ...?

I might also suggest consolidating some error messages to reduce the log output. Perhaps something like this:

[Error ][05/21/2022 13:49:00] --> Failed getting metadata for MetaCritic, OMDB, RT, TMDB, Trakt, and TVTime" [Info ][05/21/2022 13:49:00] --> Finished getting metadata for IMDB and TMDB"

The same could be done for No need to update, but it does not seem to generate as many messages.

Also this is a good idea, the reason its like that its because each metadata provider is in a different thread but i can make a print when all end!

samcro1967 commented 2 years ago

I see Starting BetterCovers for directory 255 times. I ran it a few times on one movie and then once on the entire library. So probably started it 5 or 6 times.

ilarramendi commented 2 years ago

Logs are reduced by a lot on the new version, i think i was uploading my logs to the container, it should only start one time now, Please again delete all BetterCovers related files and run it again! Also i couldnt replicate the 'name' error, if you are still having it it the latest version please let me know.

PD: probably has a couple of bugs since a lot of things changed since last version

samcro1967 commented 2 years ago

The container crashes on startup with this error message:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/BetterCovers/src/Movie.py", line 463, in updateMetadata
    _getTMDB()
  File "/BetterCovers/src/Movie.py", line 200, in _getTMDB
    if result['name'].lower() == self.title.lower():
KeyError: 'name'

Here are the steps I used:

git clone https://github.com/ilarramendi/BetterCovers.git .
cd bettercovers && docker build -t bettercovers .
docker rm -f bettercovers
rm -Rf *
create config.json file in the bettercovers folder using the github example.  Replace tmdbApi with mine.
docker-compose up -d
ilarramendi commented 2 years ago

Can you try pulling the container from dockerhub? the name is ilarramendi/bettercovers You just need to delete the folder you pass as /config first. Also can you see any log message before that line? Maby try increasing log level.

If this dosnt work you cant try adding the line: print(result) in line 199 of Movie.py and building the container again to see what the server is actualy returning, since i cant replicate this problem with my movie library.

ilarramendi commented 2 years ago

PD: TMDB api key dosnt need to be changed since it has an unlimited api hit limit / hits per second, if you want you can add an OMDB api key to get more results

samcro1967 commented 2 years ago

Got the same results with the Docker Hub image. Adding print(result) though has generated some logs now before it crashes. I double checked the folders it marked as empty and they are not. The file perms are 777 and owned by 1000:1000. The container is running as root.

https://pastebin.com/ukjaP4ad

ilarramendi commented 2 years ago

i dont see the print statement you added in those logs, it should print a json object somewhere in the code (it wont be stored in BetterCovers.log tho)

Also can you send me the complete file name of a movie that says its empty?

PD: can you see the file /config/templates/cover.html?

samcro1967 commented 2 years ago

This is in the container logs right as it starts, but does not show up in BetterCovers.log.

Exception in thread Thread-3:                                                                                                                                                                                    Traceback (most recent call last):                                                                                                                                                                                 File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner                                                                                                                                        self.run()                                                                                                                                                                                                     File "/usr/local/lib/python3.9/threading.py", line 917, in run                                                                                                                                                     self._target(*self._args, **self._kwargs)                                                                                                                                                                      File "/BetterCovers/src/Movie.py", line 463, in updateMetadata                                                                                                                                                     _getTMDB()                                                                                                                                                                                                     File "/BetterCovers/src/Movie.py", line 200, in _getTMDB                                                                                                                                                           if result['name'].lower() == self.title.lower():                                                                                                                                                             KeyError: 'name'                                                                                                                                                                                                 Exception in thread Thread-41:                                                                                                                                                                                   Traceback (most recent call last):                                                                                                                                                                                 File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner                                                                                                                                        self.run()                                                                                                                                                                                                     File "/usr/local/lib/python3.9/threading.py", line 917, in run                                                                                                                                                     self._target(*self._args, **self._kwargs)                                                                                                                                                                      File "/BetterCovers/src/Movie.py", line 463, in updateMetadata                                                                                                                                                     _getTMDB()                                                                                                                                                                                                     File "/BetterCovers/src/Movie.py", line 200, in _getTMDB
    if result['name'].lower() == self.title.lower():
KeyError: 'name'
Exception in thread Thread-47:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner                                                                                                                                        self.run()
  File "/usr/local/lib/python3.9/threading.py", line 917, in run                                                                                                                                                     self._target(*self._args, **self._kwargs)
  File "/BetterCovers/src/Movie.py", line 463, in updateMetadata                                                                                                                                                     _getTMDB()
  File "/BetterCovers/src/Movie.py", line 200, in _getTMDB
    if result['name'].lower() == self.title.lower():
KeyError: 'name'
Exception in thread Thread-58:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner                                                                                                                                        self.run()
  File "/usr/local/lib/python3.9/threading.py", line 917, in run                                                                                                                                                     self._target(*self._args, **self._kwargs)
  File "/BetterCovers/src/Movie.py", line 463, in updateMetadata                                                                                                                                                     _getTMDB()
  File "/BetterCovers/src/Movie.py", line 200, in _getTMDB
    if result['name'].lower() == self.title.lower():
KeyError: 'name'
No need to update IMDB Ratings Dataset
No need to update IMDB Episodes Dataset
[05/25/2022 15:56:35][Info   ] --> Starting BetterCovers for directory: /media/*
samcro1967 commented 2 years ago

print (results) is not working. I have tired both tabs and spaces to make it looks like this:

198                           for result in res['results']:
199                                 print (results)
200                                 # TODO error here
201                                 if result['name'].lower() == self.title.lower():

Container logs

TabError: inconsistent use of tabs and spaces in indentation
Traceback (most recent call last):
  File "/BetterCovers/src/BetterCovers.py", line 15, in <module>
    from Movie import Movie
  File "/BetterCovers/src/Movie.py", line 199
    print (results)
TabError: inconsistent use of tabs and spaces in indentation
samcro1967 commented 2 years ago

1917 (2019).m4v is an example filename of one is says is an empty folder.

ilarramendi commented 2 years ago

1917 (2019).m4v is an example filename of one is says is an empty folder.

m4v type was not supported, i added it to the media extensions list! If you have any media file extension that is not in ['mkv', 'mp4', 'avi', 'm2ts', 'm4v'] please let me know so i can add it, those where the common ones i could remember.

print (results) is not working. I have tired both tabs and spaces to make it looks like this:

198                           for result in res['results']:
199                                 print (results)
200                                 # TODO error here
201                                 if result['name'].lower() == self.title.lower():

Container logs

TabError: inconsistent use of tabs and spaces in indentation
Traceback (most recent call last):
  File "/BetterCovers/src/BetterCovers.py", line 15, in <module>
    from Movie import Movie
  File "/BetterCovers/src/Movie.py", line 199
    print (results)
TabError: inconsistent use of tabs and spaces in indentation

Also i renamed my library to remove the [tmdbid=xxx] from names and found the issue, it will be fixed for the next version too!

Besides from this can you check if the folder /config has the same content as: https://github.com/ilarramendi/BetterCovers/tree/main/config (including subfolders) Any missing file should be generated at the start of the container.

samcro1967 commented 2 years ago

Everything matches file for file and folder for folder. Most of mine are mp4. I have a few that are m4v. I should not have any others that I know of.

LMK when you have updated master and I can test again. Glad you were able to find the issue. Was beginning to think I was crazy.

ilarramendi commented 2 years ago

Its up (github or dockerhub), also in your logs i saw a lot of lines that said Error opening: /config/templates/cover.html, lets see if that problem continues!

samcro1967 commented 2 years ago

It is running now. Looking much better. Keyerror is gone from the container logs. Not seeing Error opening: /config/templates/cover.html anymore in BetterCovers.log either. Logs also look much cleaner now. Still seeing some that are not matched. Going to stop it, and add the OMDB key and will report some stats once it is done.

samcro1967 commented 2 years ago

It is also picking up m4v files as well...

samcro1967 commented 2 years ago

Here is a summary of the results.

samcro1967 commented 2 years ago

I spent about 15 mins this morning looking through and I would say we are at almost 100% of covers being created and getting the correct match. Ones that were not matched were naming issues. They were either misspelled or named differently than the official movie title. Ones that had been previously mismatched are now matched correctly. Appreciate your amazing work on this.