Support new Plex Movie agent

[docmeth02]

Should be a pretty easy fix. The new agent is called tv.plex.agents.movie instead of com.plexapp.agents.imdb for the legacy scanner. But there should probably be some kind of detection what kind of agent is used for the library. I can give it a shot and submit a pull request if you want. Or maybe you prefer to implement it yourself.

For anyone looking to fix this NOW. Change in plexutils.py:

if library_type == 'movie':
            params['type'] = 'movie'
            params['agent'] = 'tv.plex.agents.movie'
            params['scanner'] = 'Plex Movie Scanner'

Be aware that this change will make legacy movie libraries stop working.

[adamgot]

@docmeth02 Please feel free to submit a PR if you want. Detecting “default” library agent might be useful, not sure, keep in mind that agent can differ from item to item, so they’ll all have to be considered.

[Migz93]

@docmeth02 I've tried your changes but it doesn't work. From what i understand the only bit that changed was changing the agent to tv.plex.agents.movie?

I'm going to assume because currently the new plex agent doesn't expose any GUIDs for external services, i.e that a movie is xx on IMDB or yy on TMDB that when the script checks what movies are in the Plex library it can't see any GUIDs it understands so thinks there are no movies. Plex have advised that they'll be making these GUIDs available again in the XML, once those are available this (and many other 3rd party scripts) should be able to work with the new scanner. Unsure if it'll need some changes to the script itself though. https://forums.plex.tv/t/native-plex-agents-allow-access-to-external-provider-ids-for-media-eg-imdb-tmdb-tvdb/619090/130

[Mnk3y]

External ids are now available in version 1.20.1.3213. And yes, I think changes are still needed because it still does not work with the new server version.

Edit: Or you have to refresh the metadata again after the update so that the ID is entered by the agent.

Edit 2: Okay before refresh there is this guid="com.plexapp.agents.imdb://tt0054215". After refresh it looks like this guid="plex://movie/5d7768264de0ee001fcc87e0" and there are new entries for example <Guid id="imdb://tt0054215"/> or <Guid id="tmdb://539"/>.

Edit 3: Apparently the python plex api cannot parse these new entries at the moment.

[ghost]

any fix yet for new plex agent ?

[Mnk3y]

No, we have to wait for an update of the Plex API, because it cannot parse the new entries yet. https://github.com/pkkid/python-plexapi/issues/534

[morganzero]

Does it work for anyone yet?

[PearsonFlyer]

Plexapi has been updated. Any chance we can take advantage of this yet? I miss my trending movies library :(

[morganzero]

I 2nd this @PearsonFlyer

[adamgot]

This is not yet in plexapi, waiting for this: https://github.com/pkkid/python-plexapi/issues/557

[xfouloux]

Should be a pretty easy fix. The new agent is called tv.plex.agents.movie instead of com.plexapp.agents.imdb for the legacy scanner. But there should probably be some kind of detection what kind of agent is used for the library. I can give it a shot and submit a pull request if you want. Or maybe you prefer to implement it yourself.

For anyone looking to fix this NOW. Change in plexutils.py:

if library_type == 'movie':
            params['type'] = 'movie'
            params['agent'] = 'tv.plex.agents.movie'
            params['scanner'] = 'Plex Movie Scanner'

Be aware that this change will make legacy movie libraries stop working.

Best way would be to get the currently used agent for the library, then use it

Thanks

[xfouloux]

Also tautulli states that : The external metadata provider IDs have been added back in PMS 1.20.1.

So if using new metadata agent, well upgrade whole library for refresh all match with new agent, then use the proposed fix by @docmeth02 i guess, upgrading lots of movies metadata right now so it will take time :(

if you still use old agent well everything should still work =)

[PearsonFlyer]

I made the change suggested above, and it still does not work. Also, someone said that only newly-scanned movies would have the new scraper format, and that is incorrect. Once you change, your whole movie library changes to the new scraper format.

[morganzero]

Just to give some nuance in the matter.

This is what I did;

1. I changed the agent to Plex Movies on my Movie library, but not my dynamic library. Executed script.

Result: script could not recognise newly added movies with metadata from new Plex Movie Agent.

2. I deleted all the symlink and metadata from my dynamic library to start from scratch. I changed both my Movies library and the dynamic library to the new Agent. Executed script.

Result: script could not recognise newly added movies with metadata from new Plex Movie Agent. Nothing was added to to dynamic library.

3. I switched both libraries back to Plex Movie (Legacy) agent. Executed script.

Result: The script works as intended and I'm back at square one.

Dynamic library = libraries created by python-plexlibrary.

[xfouloux]

Yeah after hours of updating all my movie library metadata, well it's still not working, we need to check on how tautulli gets the guid from tmdb and imdb with the new scrapper and adapt :/

[josh-gaby]

I've submitted a pull request (#96) that works around this issue, at least until the plexapi has been updated to support the tmdb and imdb ids and then it could be removed.

[samwiseg0]

After speaking with @blacktwin. Searching with the GUID might not be possible. So https://github.com/pkkid/python-plexapi/issues/557 is not going to be a solution.

It seems like the best way would be to pull the entire library once and create a reversed dictionary lookup. This pull request https://github.com/pkkid/python-plexapi/pull/562 would need to be merged to make the reversed dictionary lookup work.

[josh-gaby]

@samwiseg0 even with that pull request merged it would still require an api call for every single individual movie in the users libraries.

What are peoples thoughts on directly accessing the plex database for this? I guess the biggest limitation would be that you could only run this on the server that Plex is running on, not sure how much of a limitation that really is though?

It's a simple sqlite connection and a very simple query like SELECT t.tag, mdi.guid FROM metadata_items mdi JOIN taggings tg ON tg.metadata_item_id = mdi.id JOIN tags t ON t.id = tg.tag_id AND t.tag_type = 314 WHERE mdi.metadata_type = 1 GROUP BY mdi.id, t.tag returns the following data set:

plex://movie/5d9f3c6fe4fc29001eb6f051 imdb://tt3321300 plex://movie/5d9f3c6fe4fc29001eb6f051 tmdb://292040 plex://movie/5d9f3c6fe4fc29001eb6f051 tvdb://4167 plex://movie/5d776bad7a53e9001e72cef7 imdb://tt3703908 plex://movie/5d776bad7a53e9001e72cef7 tmdb://344041 plex://movie/5d776bad7a53e9001e72cef7 tvdb://5926 plex://movie/5d776b3d96b655001fe09882 imdb://tt3733774 plex://movie/5d776b3d96b655001fe09882 tmdb://301608 plex://movie/5d776b3d96b655001fe09882 tvdb://7515

I only have a little over 3000 movies in my database and the above query only takes 359ms to return, I don't imagine there would be any way to make that many http requests in a similar timeframe.

Edit: and if I drop the group by clause it returns in 41ms, group by wouldn't be needed since we would create a hash from the results anyway

Edit: I have an initial direct db access version working here, it uses a new module I'm working on to map IMDB/TMDB/TVDB ids to the current plex guid python-plexmovieagentmapper

[xfouloux]

Well i find this horrible, with more than 9k movies, let me tell you it will be horrible How about forking or modifying the project to use radarr as source (as an option ? because not everyone use radarr)

[samwiseg0]

What are peoples thoughts on directly accessing the Plex database for this?

I am very opposed to anything that looks at the database. SQLite only allows one connection to a database so connecting to it while Plex is online is out of the question. Some other projects have done it but it's a terrible idea and frankly I would never use it if the project went that route.

[samwiseg0]

Well i find this horrible, with more than 9k movies, let me tell you it will be horrible How about forking or modifying the project to use radarr as source (as an option ? because not everyone use radarr)

Not everyone would have radarr. We should figure out another way to pull this data from the plex API.

[josh-gaby]

What are peoples thoughts on directly accessing the Plex database for this?

I am very opposed to anything that looks at the database. SQLite only allows one connection to a database so connecting to it while Plex is online is out of the question. Some other projects have done it but it's a terrible idea and frankly I would never use it if the project went that route.

What if a temporary copy of the database was taken to do the lookups on?

I've got it working without copying it and haven't had issues with my testing so far, but maybe taking a temporary copy of the database at initialisation would prevent any issues with Plex accessing it. My library is 3000+ movies and 26000+ tv episodes and the database is only ~300mb, copying that to a temp file wouldn't be to bad.

In saying that, SQLite does allow for multiple connections and since we are only ever reading from the database it shouldn't be a problem?

[samwiseg0]

In saying that, SQLite does allow for multiple connections and since we are only ever reading from the database it shouldn't be a problem?

This is my issue. "But only one process can be making changes to the database at any moment in time, however." What if Plex is writing to the database at the time you are trying to read it? It would be in a locked state. Again, multiple things talking to a database without knowing about each other is a bad idea.

I've got it working without copying it and haven't had issues with my testing so far, but maybe taking a temporary copy of the database at initialization would prevent any issues with Plex accessing it.

I think taking a copy would be ideal. But we should be pulling it just like the admin would from Plex web. https://plex.server.com:32400/diagnostics/databases?X-Plex-Token=XXXXXXXXXXXXXXXX. Not sure if this is built into plexapi.

[adamgot]

I would also very much like to avoid directly accessing the database. If that’s the only way it would be possible I think plexapi should do that and not this project, mainly so that we don’t have to keep up with database changes...

Hopefully Plex might make this searchable in a future update?

[josh-gaby]

@adamgot Guid search has been removed from plexapi and blacktwin said that it is not likely to be resolved in that project https://github.com/pkkid/python-plexapi/issues/557#issuecomment-707778230

[samwiseg0]

We should be able to do something like this. Again, we need https://github.com/pkkid/python-plexapi/pull/562 merged.

guid_lookup = {guid: movie for movie in library.section('Movies').all() for guid in movie.guids}

guid = 'imdb://tt8695030'
movie = guid_lookup.get(guid)

[adamgot]

@samwiseg0 this seems like the best way forward... will have to experiment with the performance. What I’m thinking is to save a cache of the entire library with all guids as lookup keys, and keep that cache updated by checking for recently added movies since last time.

[josh-gaby]

We should be able to do something like this. Again, we need https://github.com/pkkid/python-plexapi/pull/562 merged.

guid_lookup = {guid: movie for movie in library.section('Movies').all() for guid in movie.guids}

guid = 'imdb://tt8695030'

movie = guid_lookup.get(guid)

Agreed, that would be ideal but unless Plex change where/when they return the extra guids,that just won't be possible.

The problem is that library.section('Movies').all() will return the results of the library/sections/{section_id}/all Plex URL which just doesn't have those extra guids available so no pull request will make that work unless Plex changes the API again

[morganzero]

Could the "solution" above affect huge libraries in a bad way?

[adamgot]

TV isn’t affected at all, but whatever solution will be adapted I will make sure is working fine with very large libraries as well.

[morganzero]

Thanks, that's good to know! Keep up the good work!

[samwiseg0]

TV isn’t affected at all, but whatever solution will be adapted I will make sure is working fine with very large libraries as well.

It will be in the future. Rumor has it that they are working on their own for TV shows. I don't see this as an issue since we only will be pulling the master list once.

If we track changes from recently added then this is a non issue. The first run will take longer but frankly this is not a time sensitive application so I don't see why it would matter if it took 10 seconds or 10 minuets to complete. As long as we are not drowning the Plex server in API calls I don't see the problem.

[josh-gaby]

Does the recently added endpoint return items that have been modified or only newly added stuff? the cache wont be able to track someone fixing mismatched movies or changing the agent for an entire library if its is only returning new stuff

[samwiseg0]

It pulls what has been recently added. If someone fix matches something that is not in recently added then no, it wont be included.

We could put a flag to track changes only or do the entire lib when pulling. Even with 20000 items pulling the whole thing should not be too bad. I think we would have to pull it in 9999 increments anyway. So that is 3 calls.

[JonnyWong16]

Agreed, that would be ideal but unless Plex change where/when they return the extra guids,that just won't be possible.

The problem is that library.section('Movies').all() will return the results of the library/sections/{section_id}/all Plex URL which just doesn't have those extra guids available so no pull request will make that work unless Plex changes the API again

No, you're wrong. You're confusing the Plex API /library/sections/<sectionID>/all with the plexapi method library.LibrarySection.all().

>>> from plexapi.server import PlexServer
>>> plex = PlexServer("http://localhost:32400", token="xxxxxxxxxx")
>>> library = plex.library.section("Movies (preview)")
>>> guid_lookup = {guid.id: movie for movie in library.all() for guid in movie.guids}
>>> guid_lookup['imdb://tt0470752']
<Movie:245:Ex-Machina>

For the record, I'm the one that told @samwiseg0 to use a reversed dictionary lookup.

If you want to cache, then update the cache based on the updatedAt timestamp. That should include recently added items and items have had metadata refreshed.

library.section("Movies (preview)").all(sort="updatedAt:desc")

Or only retrieve the items updatedAt newer than your last cache timestamp.

library.section("Movies (preview)").all(sort="updatedAt:desc&updatedAt>=1602623286")

[josh-gaby]

@JonnyWong16 You're right I was completely mixing them up, what is the performance like for your "Movies (preview) " library and how many items are in it? I've just done a test with a small library of only 269 movies and it took ~33 seconds to load, that's around 8 items loaded per second for me (not a local connection though, separate server on the same network).

Definitely think a cache is going to be important if this method is used.

[JonnyWong16]

It's a test library on my development server so it only has less than 10 movies.

You're not doing a time critical task. So it doesn't matter if it takes longer the first time.

[josh-gaby]

@morganzero how big is your database file with that many entries? I'm curious if that many entries would make it large enough to be problematic for the copying of the database method because I have that working and it's super quick

[morganzero]

@josh-gaby my com.plexapp.plugins.library.db is 1.10GB

[xfouloux]

Indeed it would seem that reversed dictionary cache and UpdateAt methods are the way to go, downloading the DB all the time and doing queries in it is fine but not that great. Even if the 1st time take hours, then the other task would just take a few minutes at most to add new movies from the recently added items. Also it allows you to not have to maintain a SQL query somewhere, and wonder at each update of plex if the db has been changed and breaks the query.

[ghost]

https://github.com/mza921/Plex-Auto-Collections this got fix with new plex agent , are you planning to update ?

[JonathanLew1s]

Does Tautulli API contain enough information to be able to replace using the Plex API to determine if an item exists in the library?

[PearsonFlyer]

Just following up on this. I appears that PlexApi is not going to handle this. Does this mean PlexLibrary is dead for anything movie-related? I'm getting an error now on my trending movies run. Just wondering if I need to just delete my Trending Movies library from Plex and forget about it.

AttributeError: 'MovieSection' object has no attribute 'ALLOWED_FILTERS'

[morganzero]

@PearsonFlyer It's working like a charm for me as I'm still using the Legacy Agent. Not switching to new one unless this gets fixed or someone creates another way to enjoy both plexlibraries and the new Plex Agent.

[PearsonFlyer]

@morganzero Of course it's working for legacy. This entire issue is related to the new movie agent. It doesn't apply to people using legacy.

[xfouloux]

well the version in PR done by @josh-gaby works fine for me using the DB i've done the explanation steps for docker and such in the PR if you wanna try it. seems like the way to go for now unfortunately, because using tautulli, radarr or else is not something viable (some people don't use those tools)

[danz1995]

Hey @xfouloux and @josh-gaby, how can I find the new Plex Agent guid and relate it to an IMDb id? I am still looking for the IMDb id inside the XML file using: {baseurl}:{port}/libraries/sections/{key}/all/?X-Plex-Token={token}.

If I should be looking for this information inside another place, can you please let me know and give me some guidance? Thanks!

[josh-gaby]

@danz1995 which direction are you trying to go? are you wanting to use a Plex Agent guid to look up the IMDb id or are you trying to go the other direction?

[danz1995]

@josh-gaby I am able to obtain Plex agent guid through XML, so, ideally, use that to look up for the IMDb id

[josh-gaby]

@danz1995 the results from the "/library/sections/{key}/all" query will contain a key for each item like "/library/metadata/{db_id}", if you make a request to that URL you will be able to find the imdb/tmdb/tvdb ids in the data returned.

You will need to make a request for each media item you are trying to find the IMDb id for.

This will only work if the library being checked has had its metadata refreshed using Plex Media Server version 1.20.1.3213 or newer.