Ignoring: 'Movie Title' with IMDB ID: tt123456 supplies no valid rating := 'null'

[zotoro]

I've got a few items that are showing the error above even though IMDB certainly has ratings available. Here's an example:

Ignoring: 'Good Boys' with IMDB ID: tt7343762 supplies no valid rating := 'null'

When I go to IMDB with the listed IMDB ID, it shows that Good Boys has a rating of 6.7/10 with 62,836 votes. What is causing these items to be ignored and how do I fix it?

[mynttt]

This error is caused by the item not being present in the exported ratings that IMDB offers. To confirm this you can open up the rating_set.tsv that UpdateTool downloads and search for tt7343762 (it yields no result on my dataset).

Normally this happens with newer entries so I'm suprised that a movie from 2019 is not included in their dataset. As a quickfix you can either write the current rating in the rating_set.tsv yourself (be aware that it is redownloaded every 2 days) so it can get updated once at least.

Since this is an issue that I've recived multiple tickets for I'll add a way to lookup the IMDB rating via screen scraping over their website if the ID is not included in their exported data set with the next update.

[mynttt]

1.5.5 has that screen scraping capability now.

[zotoro]

Is there something I need to do to enable screen scraping? I'm running the GUI version and after updating I'm still getting the same error for the same movie among others.

[mynttt]

It is enabled automatically in version 1.5.5. In the log it would appear as:

[INFO ] - 2021-05-06 20:30:10 @ ImdbScraper.scrapeFallback: Scraped rating 6.7 for id tt7343762. Cached for 7 day(s)

[zotoro]

That does not show in the log. Here is the log from a run I just did using the Windows GUI. I do not see any lines referencing scraping.

Started UpdateTool:

[INFO ] - 2021-05-06 16:27:27 @ Main.loadImplementation: Loaded implementation << imdb-docker << updatetool.imdb.ImdbDockerImplementation
[INFO ] - 2021-05-06 16:27:27 @ Main.main: Running version: 1.5.4
[INFO ] - 2021-05-06 16:27:27 @ Main.main: Args: {imdb-docker={schedule=12}}
[INFO ] - 2021-05-06 16:27:27 @ Main.main: << INIT: imdb-docker @ {schedule=12} >>
[INFO ] - 2021-05-06 16:27:27 @ ImdbDockerImplementation.bootstrap: No TMDB API key detected. Will not process TMDB backed Movie and TV Series libraries and TMDB orphans.
[INFO ] - 2021-05-06 16:27:27 @ ImdbDockerImplementation.bootstrap: No TVDB API authorization string detected. Will process TVDB backed TV Series libraries.
[INFO ] - 2021-05-06 16:27:27 @ ImdbDockerImplementation.bootstrap: Starting IMDB Watchdog
[INFO ] - 2021-05-06 16:27:27 @ ImdbDockerImplementation.bootstrap: Plex data dir: C:\Users\Plex Server\AppData\Local\Plex Media Server
[INFO ] - 2021-05-06 16:27:27 @ ImdbDockerImplementation.bootstrap: Capabilities: [VERBOSE_XML_ERROR_LOG]
[INFO ] - 2021-05-06 16:27:27 @ Main.main: << INIT SUCCESS >>
[INFO ] - 2021-05-06 16:27:27 @ Scheduler.go: Scheduler is loading tasks... Blocking until completely set-up and ready to go.
[INFO ] - 2021-05-06 16:27:27 @ Scheduler.go: Scheduling tasks...
[INFO ] - 2021-05-06 16:27:27 @ Scheduler.lambda$go$0: Scheduled imdb-docker task to run @ every 12 hour(s).
[INFO ] - 2021-05-06 16:27:27 @ Scheduler.lambda$go$1: Queued task imdb-docker for immediate execution.
[INFO ] - 2021-05-06 16:27:27 @ Scheduler.go: Running supplied tasks immediately NOW!
[INFO ] - 2021-05-06 16:27:27 @ TaskWrapper.run: ================================================
[INFO ] - 2021-05-06 16:27:27 @ TaskWrapper.run: Starting task: imdb-docker | Execution count: 0
[INFO ] - 2021-05-06 16:27:27 @ TaskWrapper.run: ================================================
[INFO ] - 2021-05-06 16:27:28 @ ImdbDockerImplementation$ImdbBatchJob.lambda$run$1: New TV Show Agent Library with ID: 2 will not be processed by UpdateTool. Please register with environment variable UNLOCK_FOR_NEW_TV_AGENT to unlock proceessing capabilities.
[WARN ] - 2021-05-06 16:27:28 @ ImdbDatabaseSupport.updateNewAgentMetadataMapping: No external metadata provider id associated with this guid plex://movie/5f1d13f4c406e9003f3dabfb (Archenemy). This item will not be processed any further.
[INFO ] - 2021-05-06 16:27:28 @ ImdbDockerImplementation$ImdbBatchJob.run: Library IDs on ignore list: []
[INFO ] - 2021-05-06 16:27:28 @ ImdbRatingDatasetFactory.requestSet: IMDB Dataset has the timestamp: 1620181951752 and violates the update every 86400000 ms constraint. Refreshing dataset...
[INFO ] - 2021-05-06 16:27:28 @ ImdbRatingDatasetFactory.downloadData: Downloading IMDB rating set from: https://datasets.imdbws.com/title.ratings.tsv.gz
[INFO ] - 2021-05-06 16:27:32 @ ImdbRatingDatasetFactory.downloadData: Download succeeded @ ./__tmp_rating.gz
[INFO ] - 2021-05-06 16:27:32 @ ImdbRatingDatasetFactory.extractData: Extracting dataset...
[INFO ] - 2021-05-06 16:27:33 @ ImdbRatingDatasetFactory.extractData: Extraction completed.
[INFO ] - 2021-05-06 16:27:33 @ ImdbRatingDatasetFactory.readData: Reading data...
[INFO ] - 2021-05-06 16:27:34 @ ImdbRatingDatasetFactory.readData: 1148911 lines read.
[WARN ] - 2021-05-06 16:27:34 @ NewPlexAgentToImdbResolvement.<init>: No TMDB fallback set. Will not resolve new plex agent items if they only have a TMDB id associated.
[WARN ] - 2021-05-06 16:27:34 @ NewPlexAgentToImdbResolvement.<init>: No TVDB fallback set. Will not resolve new plex agent items if they only have a TVDB id associated.
[INFO ] - 2021-05-06 16:27:34 @ ImdbDockerImplementation$ImdbBatchJob.run: [MOVIE] Movies has 1050 item(s)
[INFO ] - 2021-05-06 16:27:34 @ ImdbDockerImplementation$ImdbBatchJob.run: Processing [MOVIE] Movies with UUID 613ef216-2a2e-4f6e-a0c9-908f65dab562 at stage: CREATED
[INFO ] - 2021-05-06 16:27:34 @ ImdbPipeline.analyseDatabase: Resolving IMDB identifiers for items. Only warnings and errors will show up...
[INFO ] - 2021-05-06 16:27:34 @ ImdbPipeline.analyseDatabase: Items that show up here will not be processed by further stages of the pipeline.
[ERROR] - 2021-05-06 16:27:34 @ NewPlexAgentToImdbResolvement.resolve: No external id associated with guid plex://movie/5f1d13f4c406e9003f3dabfb (Archenemy).
[INFO ] - 2021-05-06 16:27:34 @ ImdbPipeline.analyseDatabase: Filtered 1 invalid item(s).
[WARN ] - 2021-05-06 16:27:35 @ ImdbTransformer.needsUpdate: Ignoring: 'Father' with IMDB ID: tt8542198 supplies no valid rating := 'null'
[WARN ] - 2021-05-06 16:27:35 @ ImdbTransformer.needsUpdate: Ignoring: 'Good Boys' with IMDB ID: tt7343762 supplies no valid rating := 'null'
[WARN ] - 2021-05-06 16:27:35 @ ImdbTransformer.needsUpdate: Ignoring: 'African Kung Fu Nazis' with IMDB ID: tt13399862 supplies no valid rating := 'null'
[INFO ] - 2021-05-06 16:27:35 @ ImdbPipeline.transformMetadata: 1049 item(s) need no update.
[INFO ] - 2021-05-06 16:27:35 @ ImdbPipeline.transformMetadata: Transforming 0 item(s)
[INFO ] - 2021-05-06 16:27:35 @ ImdbPipeline.transformMetadata: Transformed entries for 0 items(s).
[INFO ] - 2021-05-06 16:27:35 @ ImdbPipeline.updateDatabase: Nothing to update. Skipping...
[INFO ] - 2021-05-06 16:27:35 @ ImdbPipeline.updateXML: Updating XML fallback files for 0 item(s).
[INFO ] - 2021-05-06 16:27:35 @ ImdbPipeline.updateXML: Completed updating of XML fallback files.
[INFO ] - 2021-05-06 16:27:35 @ ImdbDockerImplementation$ImdbBatchJob.run: Job returned PASS : Job finished correctly
[INFO ] - 2021-05-06 16:27:35 @ ImdbDockerImplementation$ImdbBatchJob.run: Job finished successfully for [MOVIE] Movies with UUID 613ef216-2a2e-4f6e-a0c9-908f65dab562
[INFO ] - 2021-05-06 16:27:35 @ ImdbDockerImplementation$ImdbBatchJob.run: Completed batch successfully.
[INFO ] - 2021-05-06 16:27:35 @ TaskWrapper.run: ================================================
[INFO ] - 2021-05-06 16:27:35 @ TaskWrapper.run: Suggesting JVM to run the GC as soon as possible (Request might be ignored!)
[INFO ] - 2021-05-06 16:27:35 @ TaskWrapper.run: Completed imdb-docker in 7.7674061s. - Invoking next task or going to sleep. It is safe to suspend execution if no other task is being invoked immediately.

[mynttt]

Running version: 1.5.4 is the problem. Restarting the GUI will cause the tool to download the recent 1.5.5 version that added this feature.

[zotoro]

It is updated to v1.5.5. That's the latest log from v1.5.5 Windows GUI (see pic link)

Windows GUI v1.5.5

Log from v1.5.5

[mynttt]

Does the log at the beginning (not in your SS) say 1.5.5? There might be a bug with the automated update that keeps the jar on 1.5.4.

[zotoro]

Ah-ha, that is the issue. Before running, it shows version 1.5.5, but once I start running the log shows "Main.main: Running version: 1.5.4" right at the beginning.

Seems like there is a bug with the automated update after all.

[mynttt]

I won't fix the bug in the updater because it's used so rarely if it happens again just delete the JAR again and force it to update automatically.

[zotoro]

Ok, well with the release of 1.5.6 the gui now runs 1.5.5 so scraping is working. Seems like it consistently runs the previous version despite saying it updated to the latest one. Even after re-downloading the gui jar file, it still runs the previous version. Now, with the release of 1.5.6b, it's running 1.5.6 See image

Just wanted to make sure you're aware, in case you decide this is worth investigating and fixing.

[mynttt]

Thanks for your investigation, that seems to be the curlpit! The downloader used the old version read from the local FS instead of the newly recieved version (but then wrote the newer version as the current version while downloading version n-1 again) which perfectly explains this behavior. I've just pushed a fix for this with g1.0.3. Let me know if there are any issues with it!