Plex_exporter_importer:Music Libraries

[RBeatse]

Only one feature should be requested per issue. Please make multiple issues if you have multiple feature requests

In what file should the feature be added? Plex_exporter_importer

What is the feature request? Add support to gather all of the pertinent info from Music libraries (album and artist pictures and play info)

[Casvt]

Request completed. Music library support! This took a good 3 hours to make but hey a whole new library type supported is a good trade off for three hours. Please re-download the script to get the new feature. Enjoy!

[RBeatse]

It took me a while to figure out the new parms and what they meant and how to use them (and I am still not sure I am using them correctly) but I finally got it to run and it ran through the first 2+ artists and got this

    Home for Christmas - D1T4   - Breath of Heaven (Mary’s Song)
    Home for Christmas - D1T5   - O Come All Ye Faithful

Traceback (most recent call last): File "c:\Plex_util\plex_exporter_importer\plex_exporter_importer.py", line 715, in response = plex_exporter_importer( File "c:\Plex_util\plex_exporter_importer\plex_exporter_importer.py", line 615, in plex_exporter_importer print(f' {track["parentTitle"]} - D{track["parentIndex"]}T{track["index"]} - {track["title"]}') File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u2010' in position 35: character maps to

The arguments I am using are python3 plex_exporter_importer.py -t export --AllMusic -p metadata -p watched_status -p poster -p art 1> "U:\Plex_exporter_importer Logs\script_output.txt" 2>&1

The whole log is attached script_output.txt

[Casvt]

What is the title of "Home for Christmas" Disk 1 Track 6? There seems to be such a weird character in the title that the script crashes. It's not even the fault of my script but instead of python. There is some character in the title that python doesn't know what to do with it.

[RBeatse]

Here is the listing. Nothing sticks out

[Casvt]

First of all download the newer version. Then go into the file and go to line 611:

print(f'        {track["parentTitle"]} - D{track["parentIndex"]}T{track["index"]}   - {track["title"]}')

Change the line so that it looks like this:

print(f'        {track["parentTitle"].encode("utf-8")} - D{track["parentIndex"]}T{track["index"]}   - {track["title"].encode("utf-8")}')

In theory it should fix it. However, it's still strange. The character that's bugging out python, \u2010, is in fact just the code for a hyphen: -. And no where in the filename of track 6 is there a hyphen present. So why does python struggle with a hyphen, and where does it come from? I just don't have an answer and it has probably something to do with your file and not my script. Maybe rename the file by first giving it the name x.mp3 and then rename it back to 06 Grown Up Christmas List.mp3 typing it over and not copy pasting. Maybe that fixes it? I don't know. But for now, you can edit the file yourself and change as described above and it might help? Let's just see.

[RBeatse]

I updated the code, ran it, same issue. I retyped the name of the song, ran it, same issue I changed the code as you showed, ran it, and it got through it but then died again on a different record. Here is the log and the screenshot of the file. This is weird script_output.txt

[Casvt]

Oh but that error is fixable. The problems with the files I don't know what to do with. I'll come back to that later. I'm at a party so I'll fix it tomorrow hahah sorry

[RBeatse]

Don't ever be sorry! I'm getting what I paid for!!!

[Casvt]

Script has been updated to fix the error. You can download the new version and if you do the fix above, it should run fine. Still don't know what to do with the file naming

[RBeatse]

It keeps getting further but still dying :( (I did update the code line 611 as instructed)

    b'In the Swing of Christmas' - D1T12    - b'Christmas Is Just Around the Corner'

Traceback (most recent call last): File "c:\Plex_util\plex_exporter_importer\plex_exporter_importer.py", line 715, in response = plex_exporter_importer( File "c:\Plex_util\plex_exporter_importer\plex_exporter_importer.py", line 576, in plex_exporter_importer print(f' {artist["title"]}') File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u2010' in position 8: character maps to

Is it possible to look for the error and just skip that record if it sees it?

[Casvt]

Is it possible to look for the error and just skip that record if it sees it?

Yeah good idea. I'll just add a try-except statement which tries to print out the title and otherwise just prints "D{n}T{n}".

[Casvt]

Maybe try this?

https://stackoverflow.com/questions/32382686/unicodeencodeerror-charmap-codec-cant-encode-character-u2010-character-m

[Casvt]

By the way, you've told me a few times that the script takes a long time to complete. Probably hours right? Well, I've already optimised the snot out of the current script so no speed gains to be made in it anymore. Unless I re-do the complete infrastructure of the script...

My suspicions for why the script is so "slow":

1. We create thousands and thousands of files. Creating files is very slow compared to the rest of the script.
2. We have to json encode and decode all the metadata every time when we put it in and read from the files.
3. Because of all the files we target a different part of the storage drive every time which has an impact on performance.

Then there are also some problems with the script:

1. When the filename of the media changes, the metadata and poster files don't match anymore and won't be picked up when importing. (issue #100 )
2. The metadata can't be stored anywhere else than alongside the media file currently. (issue #95 )
3. There is no logical place to store the file for general info like playlists. (issue #23 and #65 )
4. The script has to complete before posters are saved, so you can't interrupt the script or you'll loose the posters.

I could fix these all in one go and make it faster too. What if...

---

Instead of making thousands of files, we create one sqlite3 database in the same folder as the script and that's it; no more files than that one db file. Then we put everything in there. No need to json encode the metadata before we put it in a file because we use a sql database instead of a json file. We write only once at the end when we commit the database, so disk usage is low. We can handle multiple media entries at the same time and just create a "queue" to write to the database. All the posters are also not put in a file but their raw bytes are just stored in the database.

When the user interrupts the script, we just do a quick commit to the database before we exit and all progress is saved for how far we were. We can note down the last time the media was updated and when exporting again, and the time hasn't changed, we just skip the media as nothing has changed, skipping 1-5 requests per media skipped.

After the script has completed (or it was interrupted, so stopped) the user can move the database file (which will be GB's big but won't take more space than all those thousands of files together) where ever he/she/they wants, as it's just one (big) file. And no worries about file names not matching after changing the media file, because there are no filenames because there aren't any files anymore.

I could go on and on but the red line is that I'm going to give this script a massive face lift, make it so much faster that it's almost scary and fix 4 issues at the same time with it.

[RBeatse]

This sounds good. However, the only thing I would suggest is, like the Plex token, could there be a variable that gives the location of where to save the SQLite db? For example, my C: drive is an SDD that is small and so adding a possibly large db file may be an issue. If I could designate a location, the. That would let me place it where the space isn't an issue.

Other than that, I love the idea. Especially because 99% of items dont change so not redoing those every time is a good idea. It would also make the collection extraction process easier.

[Casvt]

However, the only thing I would suggest is, like the Plex token, could there be a variable that gives the location of where to save the SQLite db?

Oh sure. The default location will be alongside the script but I'll just add an argument where you can define an alternative folder to put the db in. Users could, for example, set the location to a folder that is on a really fast ssd, so that the script goes faster (though disk usage will be way lower so you won't notice it that much I expect).

Especially because 99% of items dont change

When the script goes through a library, it first makes one request to get all the media that is in that library. In that output, there is, for every media entry, a key called "last_updated" or something like that, that is a timestamp which describes the last time something has changed (I don't know if poster changes also count; I'll find that out). Currently, we also make a request for every media item, as the library output doesn't show all metadata that we need so we need to individually request all media too. But what we can do is write down the value of "last_updated" in the database. Then, the next time the script exports (because you have it running weekly for example), it will look in the already existing database, only make the library request, and if the "last_updated" value is the same in the db and in the media entry, we just skip it as nothing has changed. This saves one request for requesting the individual media and one/two request(s) for the posters which saves a lot of time. If you have built the database, the script could take just a few minutes to cover your complete server as it will just skip 99% of all requests.

It's going to be pretty amazing. I'm already waiting for those time results: "Yeah I tried it and it was done 7 minutes later instead of 7 hours" hahaha.

[RBeatse]

Sounds great! Good luck!

[Casvt]

Upgrade has been completed. You can now download the new version. It's all in one database file now. So after you've completed one run with the new script (created a database), you can delete all json files and jpg posters from your plex library as those aren't needed anymore. They're all in the database now! And keeping up-to-date (exporting but already giving a database to update instead of making a new one; handy for running the script weekly) happens in seconds! Enjoy and ask me any questions when needed.

[RBeatse]

OK, here is what happened when I ran it. The script I am running is (to test Movies)

python3 plex_exporter_importer.py -t export --AllMovie -p metadata -p watched_status -p poster -p art -p intro_marker 1> "U:\Plex_exporter_importer Logs\script_output.txt" 2>&1

The output is attached (it took about a minute or two to fail). TBH, it could be me screwing things up with the process tags. I am not following real well and if I did --All, do I then have all of the process tags even if some only make sense for some of the library types? script_output.txt

PS The db file is 88 kb at this point

[Casvt]

TBH, it could be me screwing things up with the process tags. I am not following real well and if I did --All, do I then have all of the process tags even if some only make sense for some of the library types?

The arguments that you gave to the script are perfect. They're correct and I agree with the choices you made. You're exporting: Plex metadata, all posters and arts except episode posters and episode arts, the watched status of the media for every user and the intro marker for all episodes (the timestamps for when an intro in an episode starts and ends; normally this would require analysing the media which could be recourse intensive so this is a good solution if you want to carry over those markers). And you're targeting all movie libraries.

Btw, you're currently exporting intro markers for episodes. That's totally fine. However, when you want to import them back, you'll be required to run the script with root and on the same computer on which the targeted Plex instance is running, as we need to directly edit the Plex database file to apply the intro markers. Just letting you know. But for exporting, no problem and you don't have to do anything special.

--All targets all libraries. It does not apply all process tags. If you were to use --All and, for example, --AllMovie, you'd get an error saying conflicting target selection because you'd target all libraries but also only movie libraries which is conflicting.

Don't worry about applying process tags that aren't available for that media type. This script is smart enough to skip over that when needed. So doing -t export -p episode_poster --AllMovie would just do nothing.

Script has been updated to fix the error.

[RBeatse]

Updated the script and got the following error. DB is still at 88 kb

BTW, what does the parm "verbose" do?

script_output.txt

[Casvt]

Do you have the same movie in two different libraries (e.g. movie1 in 1080p in the normal library and in 4k in the special library). That would explain the error. I forgot to handle that but no big problems.

The db is at 88kb because everything is saved (= when the size increases) when 1) the script ends, 2) a library is finished and 3) when the user interrupts (CTRL + C). Getting an other error interrupts the script in such way that it doesn't commit to the database. I should change that to just any interruption so that when there's an error in the script, your progress doesn't get lost.

BTW, what does the parm "verbose" do?

Without it, only the library names are printed out while the script is doing it's thing. When you add -v to the argument list, it will also print out all the media while it's handling it. See:

#no verbose
Lib1
Lib2

#verbose on
Lib1
    Movie1
    Movie2
Lib 2
    Series1
        S1E1 - Title
        S1E2 - Title2
    Series2
etc.

Don't enable verbose printing unless you really need to. The script is so fast that in verbose mode, it sends out print commands faster than the terminal can print them out. So what you get is that the script is already completed but the terminal is still printing out everything it received. In my testing, in verbose mode, the script was done after 16 seconds but the terminal stopped printing after 1m20s. So if you enable verbose printing, you're just waiting 1m4s in my case for your terminal to catch up, even though the script is already done. The definition of wasting time, so I removed printing out each media item unless asked for (by adding -v).

[RBeatse]

Thanks for the Info on "verbose".

Yes, I have the same movie in more than one library. For example, I have an "All Movies" library that has...wait for it...all movies! But I also have a library just for the 800+ Disney/Pixar movies. The Disney movies are included in the All Movies as well but it is easier to point kids to the Disney library.

Thanks.

[Casvt]

Ah thanks yeah that explains the error. I'll fix that. And that the script will commit on any error instead of just user interrupt.

[Casvt]

Script has been updated. Both things are fixed now.

[RBeatse]

Latest message...do I need to run it with verbose to give you more to go on?

script_output.txt

[Casvt]

Delete the database file at c:\Plex_util\plex_exporter_importer\plex_exporter_importer.db and try again. Verbose mode isn't needed thank you

[RBeatse]

AllMovies ran to completion!

Exporting to c:\Plex_util\plex_exporter_importer\plex_exporter_importer.db

All Movies Christmas Disney & Pixar Disney Christmas Disney+ Movies Hallmark: Hall of Fame Movies Hallmark: Mysteries & Movies LDS Related & Religious Movies Marvel & Junelle's Favorites Exercise Time: 390.648s

I am now going to run --All and I'll report back.

[RBeatse]

BTW, if I remove something from a library, will the script remove it from the DB?

[Casvt]

Hmm no it doesn't, do you think it should? I think it should be an option

[RBeatse]

If I happen to move a bunch of things around, you wouldn't want all of their old info hanging around, so I would think it should be an accurate representation.

Another question, just not clear to me from the instructions... If I use the --Location parm, would I put the full path including file name or just to the folder? For example --Location "x:\Plex_util\plex_exporter_importer\plex_exporter_importer.db" or --Location "x:\Plex_util\plex_exporter_importer\"

And, can I name the file anything I want? ("x:\Plex_util\plex_exporter_importer\plex backup1.db")

[RBeatse]

It was doing really well (got through movies very fast, of course and then did the tv shows) and then died again in the Music files

Exporting to c:\Plex_util\plex_exporter_importer\plex_exporter_importer.db Database file already exists, so updating Exporting complete plex library

All Movies Christmas Disney & Pixar Disney Christmas Disney+ Movies Hallmark: Hall of Fame Movies Hallmark: Mysteries & Movies LDS Related & Religious Movies Marvel & Junelle's Favorites Beatse Family Christmas TV Disney + TV Shows Disney Christmas TV Disney TV Embroidery LDS Related & Religious TV Shows Lind Family Produced Videos Lind Home Videos Reed TV Shows Christmas Music Traceback (most recent call last): File "c:\Plex_util\plex_exporter_importer\plex_exporter_importer.py", line 838, in response = plex_exporter_importer( File "c:\Plex_util\plex_exporter_importer\plex_exporter_importer.py", line 714, in plex_exporter_importer album_info = album_info.json()['MediaContainer']['Metadata'] KeyError: 'Metadata'

[RBeatse]

Also, DB size did not change, I think because it didn't fail gracefully

[Casvt]

Twice wrong button lol

1. When exporting, if you pass a folder, it will put the db file in that folder
2. I'll edit the script to allow passing a file when exporting and that will be the db file that everything will be put in then.
3. When importing, you pass the file path and that database will be read when importing.

The error is fixable I'll do it now even

[Casvt]

Script has been updated for the error catching and error that you got

[RBeatse]

Tried again. I deleted the DB, I wanted to start clean. Failed in a more cryptic way script_output.txt

DB is now 5.5 GB

[Casvt]

Script is updated. Seeing that the progress was saved, you should just jump right to where you were

[RBeatse]

updated the script, ran it again, got the following error message. It looks like it was using the previous status but failed in an area it did not fail before. Also, nothing was added to that library since it ran yesterday. script_output.txt

[Casvt]

Script has been updated

[Casvt]

BTW, I updated the script again and now you get a nice summary of what will happen when you start the script. You'll understand what I mean when you try it! It helps with people that aren't sure if their argument list will do what they want it to do. Just run the new script and look at the outputted text at the top :). I also edited the epilog (text at bottom of help page -h) to make it more clear what values you can give for the location argument and what effect it will have.

[RBeatse]

Well that was anti-climactic! LOL I updated the code, ran it and it immediately failed. script_output.txt

I see you added variables for plex user and plex group that are defaulted. Should I be changing something with those?

[Casvt]

File permissions in python work differently on windows apprrently. I'll take a look for how I can fix it.

Leave the Plex user and group to the default value.

[Casvt]

Script has been updated it now should work for you again. Keep in mind that importing chapter thumbnails is only supported on Linux (exporting is supported on all OSes though) so you are not able to use the chapter thumbnail feature unfortunately :(

[RBeatse]

Thanks. Luckily for me, I don't need the chapter thumbnails. I need to make sure that the parks I am using do not include those. My PC died over the weekend so I will test it in the next day or so when I get it all up again.

[Casvt]

All the things you're exporting are fully supported on all OSes. Chapter thumbnail importing is the only one not supported on win.

And I'm sorry for your computer. One of the HDDs in my server died yesterday (was setup in a mirror so no worries about data) and I felt terrible. That's even while all my data is still saved on the other drive of the mirror and I'm getting a free new one because it broke due to bad shipping. Can't imagine losing a whole pc...

[RBeatse]

OK, I am back!! Not too bad, everything is fine now. I just ran the newest code and we didn't get very far (line 31). I think it is something in the Linux pieces that isn't translating well. script_output.txt

[Casvt]

Script has been updated

[RBeatse]

OK, tried it again and getting the same message.

Looks the Environmental Variables section needs the "Linux==" also

script_output.txt

[Casvt]

Damn it you're right. Bit of a dumb mistake. Update the script.

[RBeatse]

Well it got so much further!! Looks like it had a problem with a Photo library which I don't think you were even doing anything with! script_output.txt

[RBeatse]

Well, it happened again (I got an email that siad to update the script but I now see that the comment was deleted, so maybe I wasn't supposed to do it yet).

script_output.txt

I also was wondering if, at the end, since you can't see anything happening in the database, could that output say something like

XXX records were added YYY records were updated ZZZ records were deleted

Just a thought