Casvt / Plex-scripts

Plex, the arr's and tautulli scripts coming from user requests
GNU General Public License v3.0
344 stars 30 forks source link

Plex_sync + Plex_exporter_importer: Support for intro markers #44

Closed Ashes2022 closed 2 years ago

Ashes2022 commented 2 years ago

Only one feature should be requested per issue. Please make multiple issues if you have multiple feature requests

In what file should the feature be added? Plex_sync.py

What is the feature request? In addition to syncing watch history, sync intro markers across plexes. Requesting that a flag be introduced to isolate as well. E.g. only sync intro markers or only sync watch history in the event you don't want both. Thanks in advance for your consideration!

Casvt commented 2 years ago

Request accepted. Doesn't mean it's per se possible. Just that I accept the challenge and will do some research and testing.

edrock200 commented 2 years ago

Ohh nice one. I would love to see this too :)

andrewmcd7 commented 2 years ago

I third that! This would be awesome!

UnknownWitcher commented 2 years ago

Request accepted. Doesn't mean it's per se possible. Just that I accept the challenge and will do some research and testing.

This may help

https://github.com/animosity22/homescripts/blob/master/scripts/plex-missing-intro

Edit: being able to also trigger plex to scan those files for missing intro's would also be cool. not sure if you'd want me to create another post for that request.

Casvt commented 2 years ago

I'm going to make this a feature of the plex_exporter_importer.py script instead of the plex_sync.py script. The reason for this is that, in order for this to work, we need to directly edit the database file. The idea behind plex_sync.py is that both servers are on at the same time and that we sync the two using the network (api requests). The script cannot edit both databases (read from source and apply in target) at the same time. That is for the simple reason that the script won't have access to both database files at the same time, because they are on different machines.

Now I know that you have a special setup with some sort of file sync where the machine has access to both database files at the same time, but that setup is extremely rare. So if I edit plex_sync.py to support your request, basically only you can use the feature and nobody else. That's not good.

So I'm going to make it a feature of plex_exporter_importer.py. This script is intended for exporting and importing plex metadata. Basically the non-live version of plex_sync.py, by saving everything into files (exporting) and reading from them when needed (importing). If I make this request a feature of this script, I can just export the data from the database file into a file alongside the media and when importing insert it into the accessible database.

This is way more user friendly and covers a lot more people's setup.

For you, this would mean the following: you run the script in export mode on the source server to export the intro markers into a file alongside the media. Then on the target server, you run the script in import mode which will read the just-created-file and apply it to it's database. This is exactly how all the other things work in plex_exporter_importer.py (posters, metadata). This does require though that the folder with the media is accessible and imported into plex on both servers.

It's either this or a feature for plex_sync.py that nobody but you can use. And I've decided that I'm going with the plex_exporter_importer.py route.

edrock200 commented 2 years ago

That sounds awesome bud, thank you! Does Plex need to be stopped or can it be running during import? Can intros be selected independently of all else (posters, meta, etc)? Thanks again!!

Casvt commented 2 years ago

Okay I've been trying stuff out and been digging through the raw database file. I haven't written any code yet but I just have some notes written and some sqlite commands with which I think I can make this work! So I'm going to say that this has a 90% chance of working!

That this request is going to work is good in it self. But there's more good news: I've found a way that we only need to access the target database. No need to access the source database. This is amazing news because that means that we can just run the script on the target server, grab the intro data via the api from the source server and edit the db file on the target server (on which the script is run) to apply the intro data there.

So this is my plan:

I'm going to implement this feature request in both plex_exporter_importer.py AND plex_sync.py.

And that is the important thing here: it will work, but we can only apply intro markers to the plex server that the script is running on. So plex_sync.py needs to be run on the target server in order for this feature to work. And plex_exporter_importer.py needs to be run on the target server when -t is set to import and -p to (the non-existent yet) intro_marker.

For you, this means that you will be able to run plex_sync.py on the target server and sync the intro markers from source (remote) to target (the machine itself) with no problems.

Reaction to @edrock200: When using both scripts, plex can stay on at all times. No need to shut it down. It even works "live". When you run the script and are looking at your library via the web-ui, you'll see the posters and titles changing, collections being made, etc. And using both scripts, you can indeed do actions individually (you select what to export/import/sync; posters, collections, metadata and soon intro markers). So if you want to only sync/import intro markers, on plex_sync.py you will be able to do -S intro_markers and on plex_exporter_importer.py -p intro_marker.

If there are anymore questions feel free to ask them ;)

edrock200 commented 2 years ago

Amazing!!! This is great! Thank you so much.

Casvt commented 2 years ago

Oh my god I think I've made it work! This is amazing. I've even surprised myself with the fact that I was able to make this work. The only problem is that it has only been integrated into plex_sync.py for now. Might add it to plex_exporter_importer tomorrow or later.

For now, you can download plex_sync.py here with the new feature integrated: https://github.com/Casvt/Plex-scripts/blob/Casvt/issue44/multiple_servers/plex_sync.py. This version should work, but I haven't tested it fully yet so I'm eager to hear how it goes on your side. Please let me know because if it works for you, then I know it works and I can push it to main. Just find an episode that doesn't have intro markers, run the script and then see if it does. It should even work live.

I'm eager to hear the result from your testing!

EDIT: Just to help you along, the following argument list should be sufficient to test this: -s Main -S intro_markers, assuming that the script is being run on the backup server. Keep in mind that this affects every episode in every show library.

edrock200 commented 2 years ago

So awesome!! And will do, thank you! I'll give it a shot later today.

edrock200 commented 2 years ago

One question, will this add or sync? E.g. if source Plex has intros for showA and backup Plex has intros for showB, and I sync will backup only have intros for showA or both A and B?

edrock200 commented 2 years ago

hmm. installed aiohttp, but now getting this error File "./introsynctest.py", line 40, in <module> from asyncio import gather, run ImportError: cannot import name 'run'

edrock200 commented 2 years ago

Got passed that error by upgrading python. Next error it didn't think I was running on target system, but my plex is in a docker container so I hardcoded the db path around line 323. It's running now. Just says "Intro Markers." So I'm guessing it's doing its thing. Can plex be running on target or should it be stopped? Thanks again!

edrock200 commented 2 years ago

ran for a good bit, then spit out this error:


Intro Markers
Traceback (most recent call last):
  File "./plex_sync.py", line 552, in <module>
    response = instance.start_sync()
  File "./plex_sync.py", line 157, in start_sync
    response = self._intro_markers()
  File "./plex_sync.py", line 350, in _intro_markers
    target_ratingkey = self.__find_on_target(guid=episode.get('Guid',[]), title=episode.get('title',''), type='episode').get('ratingKey','')
AttributeError: 'NoneType' object has no attribute 'get'```
Casvt commented 2 years ago

One question, will this add or sync? E.g. if source Plex has intros for showA and backup Plex has intros for showB, and I sync will backup only have intros for showA or both A and B?

Let's say source has ShowA with intro's and target not. Target has ShowB with intro's but source not. If you run the script on target, showA's intro markers from source will be applied on target showA and the markers on ShowB on target are kept. It only adds or edit's intro markers, not remove them.


Got passed that error by upgrading python.

Yeah someone else got that same problem too just a few hours ago haha. At least 3.7 is required I think. If your version was lower than 3.7 before, it was a good idea to update anyway.


Can plex be running on target or should it be stopped?

Both the plexes on source and target can be kept running. They both don't need to be shut down at any point. This all happens "live".


Just says "Intro Markers." So I'm guessing it's doing its thing.

Good that you're saying that because I indeed forgot to add print statements that tell the user where the script is. I've added those now. It will now look like this:

Intro Markers
    Show Lib 1
    Show Lib 2
    Show Lib 3

This way the user knows that the script is actually doing something and just about where it is.


ran for a good bit, then spit out this error

Fixed it in a new update. Please re-download the script to get the fixed version. Still here: https://github.com/Casvt/Plex-scripts/blob/Casvt/issue44/multiple_servers/plex_sync.py.


Thank you very much for all the feedback it really helps.

edrock200 commented 2 years ago

Thanks bud! Trying again. It's ok that I hardcoded the db in the code since I'm running Plex in docker correct?

Casvt commented 2 years ago

Yeah that's good. Maybe I should add a variable at the top of the script that people can use to override/hardcode the location of the db. The script is coded in such way that it can find the location itself, but with special setups like with docker, it falls apart and there should be an easy option to hardcode the location. I'll add that too tomorrow (00:57 here now so I'm going to sleep in an hour so I'll do it tomorrow).

edrock200 commented 2 years ago

OK it's running again. I had to fix the indentation on line 419 before it would run but otherwise it seems to be working so far :) I'll keep you posted,. Thanks again! Oh and re ashes, I know him, we are testing together. :)

Casvt commented 2 years ago

Uhm the indentation on line 419 is just fine?

edrock200 commented 2 years ago

It wouldnt run until I made "handled.series" line up to "self.target" above it. Kept giving me a line 419 indention error. I noticed just before handled series are spaces instead of a tab. Not sure if that matters?

Casvt commented 2 years ago

I updated the script and re-did the indentation of the line (and the ones above and below it) so it should really work now. You can use the link shared earlier to download the new version. The handled_series line should be one indentation less than self.target. Or you can just re-download the script.

edrock200 commented 2 years ago

I updated the script by pulling it down from your link. I don't get the indendation error now but the original error appears to have returned:


Intro Markers
        4K TV Shows
Traceback (most recent call last):
  File "./plex_sync.py", line 557, in <module>
    response = instance.start_sync()
  File "./plex_sync.py", line 158, in start_sync
    response = self._intro_markers()
  File "./plex_sync.py", line 339, in _intro_markers
    self.result_json.append(episode[['ratingKey']])
TypeError: unhashable type: 'list'
Casvt commented 2 years ago

Different error but just a simple typo. Fixed it already so again just use the link above to re-download the script. Sorry for the stupid little mistake.

edrock200 commented 2 years ago

haha please stop apologizing for helping everyone with an awesome tool! trying it now.

edit ok it's running again, and no errors on launch this time. :) I'll report back when its done!

edrock200 commented 2 years ago

OK so this time it finished running. I had one show as a test - Last Week Tonight With John Oliver. On source, skip intro appears, but on destination it doesn't appear to show. Do you know which db field to look at to confirm sync?

Casvt commented 2 years ago

In the web-ui, go to one of the episodes -> [three dots] -> "get information" (last option) -> "show XML" (left bottom). At the bottom of the info that pops up, there should be a "Marker" with type="intro" SmartSelect_20220618-133927_Samsung Internet

You can do this on the source to check if it's there and then on the target to check if it's there too after sync.

Edit: oh I already know why it doesn't work. The script currently can update intro markers but not add intro markers. I'll fix that when I have the time in the next few hours.

edrock200 commented 2 years ago

Yeah I see that on source but not on destination

edrock200 commented 2 years ago

What field are you using as the unique identifier between the two instances? The file path & name/hash?

Casvt commented 2 years ago

Well you fill in the info for both servers and give each server a name (standard name is 'Main' and 'Backup' which you can define with the variables inside the script) and then you say "source is Main". Then it will make connections to that server to grab info and then make connections to the other server and applies them.

But I'm not sure if that's what you mean?

edrock200 commented 2 years ago

I guess what unique identifier is it using to match showA/SeasonX/EpisodeY to ensure they are the same on main and backup before syncing the intro data

Casvt commented 2 years ago

Guid, title and type. They all need to match. It works well

edrock200 commented 2 years ago

In the web-ui, go to one of the episodes -> [three dots] -> "get information" (last option) -> "show XML" (left bottom). At the bottom of the info that pops up, there should be a "Marker" with type="intro" SmartSelect_20220618-133927_Samsung Internet

You can do this on the source to check if it's there and then on the target to check if it's there too after sync.

Edit: oh I already know why it doesn't work. The script currently can update intro markers but not add intro markers. I'll fix that when I have the time in the next few hours.

I somehow completely missed this post that you already found the issue lol. Great! Let me know when its updated and I'll give it another go. :)

Casvt commented 2 years ago

Hey there, I'm working on it and I think I've made it work. However, there is one thing that I need to make sure. I'm not going to explain what's going on but can you just do the following please (do exactly as stated)?

  1. Open a terminal on the source server
  2. Navigate to the folder with the databases (on linux, that is /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases)
  3. When you're in the folder inside your terminal, run the following command:
    python3

    This should open a python shell in the database folder

  4. Inside the python shell, execute the following commands one by one:
    import sqlite3
    cursor = sqlite3.connect('com.plexapp.plugins.library.db').cursor()
    cursor.execute("SELECT DISTINCT tag_id FROM taggings WHERE text = 'intro';")
    cursor.fetchall()

    Please share the output of the last command. Now I'm going to look into a magic glass ball and predict that the outcome will be [(34408,)]. I'm hoping very much that that will be the output, because otherwise I'm going to have a pretty big problem haha. Ask me questions if you get stuck ;)

edrock200 commented 2 years ago

>>> cursor = sqlite3.connect('com.plexapp.plugins.library.db').cursor()
>>> cursor.execute("SELECT DISTINCT tag_id FROM taggings WHERE text = 'intro';")
<sqlite3.Cursor object at 0x7f6b1b902f80>
>>> cursor.fetchall()
[(308935,)]
>>>
edrock200 commented 2 years ago

If that scraped all the intro data from source, it did so MUCH faster than via API calls. Took ~30 seconds vs hours. :)

Casvt commented 2 years ago

If that scraped all the intro data from source

It sort of did but not to the extend that we can use it. So we still have to do it via the api. And by the way, the reason it takes hours is not because of all the API calls, it's that it needs to make thousands of files. Creating a file is really slow compared to the rest. If you let the script complete once, and then run it again, it will run like three times as fast because it doesnt have to create the files anymore; only update the content of it.

And by the way, even though the output didn't match up, I already know how I'm going to fix it. Even though you have a different number, we both have only one number. So I just have to get that number and use that. Anyway, thanks. And I'll look into ways to optimise the script more. Did that old script take hours too? How long did that old script take?

edrock200 commented 2 years ago

Thank you! Old script said: Intro markers time: 25464.95s so just over 7h. But my library is abnormally larger than most. :)

Casvt commented 2 years ago

Oh I'm sorry. I thought I was talking to someone else. plex_sync.py doesn't make any files. So don't listen to the whole story about making files and stuff haha. That is for plex_exporter_importer.py. 7h is still very long though...

Fetching the contents of the taggings table inside the db took 30 seconds while for me it was practically instantly. So maybe that's what's being so slow. Everytime we add an intro marker on target, it takes super long because the table is already massive. Or it's the commit that's taking so long... Either way, I'm going to take a look at it.

Casvt commented 2 years ago

Oh and also: I've updated the script so it now supports both editing existing intro markers and also adding new ones. It should all work now. So please redownload the script and try again! And by the way, you don't have to let it run for the complete 7 hours. You can just look at the very first episodes in the library the script displays on the target server after a few minutes and then the first few episodes should be edited so you can just take a look then and confirm that it's working before letting it go the complete 7 hours.

edrock200 commented 2 years ago

Awesome! Thanks. Will give it a go!

edrock200 commented 2 years ago

It's running :) Question - for intro markers only, does the backup ip/port/token need configured if I'm hardcoding the db file path? It is configured now, but just curious.

Casvt commented 2 years ago

Yes it does. Though it's not many, there are a few requests made to the target server. So those credentials are required yes.

edrock200 commented 2 years ago

It worked! Took 6600 seconds but it worked! 😁 Thank you!

Casvt commented 2 years ago

Well 1h48m is less than 7h. But I'm still going to try to make it faster.

edrock200 commented 2 years ago

Indeed it is! Didn't mean to sound negative. My apologies if so. One question, at the risk of "looking a gift horse in the mouth", would it be difficult to dump the intro markers out to a single file, then import them as a file? Reason being, it could be a good way for the community to share intro markers as well as make it much faster for a scenario with multiple servers syncs.

Casvt commented 2 years ago

Didn't mean to sound negative. My apologies if so.

Oh don't worry. It's not you, it's me. I set a high standard to myself and my scripts and I just find 1h48m way to long. IMO that's not acceptable, so I want to make it faster for you, but also just for myself so I'm happy with it.

at the risk of "looking a gift horse in the mouth"

The whole idea of this project is that you can request stuff so don't worry.

would it be difficult to dump the intro markers out to a single file, then import them as a file?

Then you're looking at plex_exporter_importer.py. It doesn't have intro marker support yet but it will get it soon. That script saves to files when exporting and reads from files when importing. However, it currently saves metadata for each media item to a separate file along side the media. So it's not really useful now for your use case:

it could be a good way for the community to share intro markers as well as make it much faster for a scenario with multiple servers syncs.

However, see this comment on a different issue: https://github.com/Casvt/Plex-scripts/issues/21#issuecomment-1159781739. I'm going to overhaul the script so that it saves everything to one SQL database. That file, people can move, upload and download whatever they want. And then you pass it to the script when importing and everything in the database is applied. That would be perfect for your use case:

  1. See the comment linked above but shortly, it's going to be way faster.
  2. Everything, literally everything, is going to be stored in one DB file, which you can easily move around everywhere and share.
  3. You just pass the db file to the script in import mode and everything is applied that is chosen by the user.

So just wait for that and I'll notify you when it's done.

edrock200 commented 2 years ago

You are the man! Err..kid! 😁 Convince your folks to accept donations for your college fund! 👍

Casvt commented 2 years ago

I'm already looking into ways to recieve money (internationally) but I haven't found any way. Is saw PayPal.me but that requires that I'm 18 years or older. I haven't found any other way yet they would work.

And I've received an offer for an internship at Plex so no worries haha I have myself covered for uni.

Casvt commented 2 years ago

I've updated the script to have a variable at the top which allows you to set a custom location for the database folder. Saves you having to edit the script.

Casvt commented 2 years ago

Feature has been added to both scripts so request completed.

For the record: this is the pull request that added these features to main https://github.com/Casvt/Plex-scripts/pull/109

@edrock200 eventhough this issue will be closed, I'll comment here to let you know when the upgrade of plex_exporter_importer has been completed :)