search

jellyfin / jellyfin-kodi

Jellyfin Plugin for Kodi
https://jellyfin.org
GNU General Public License v3.0
820 stars 110 forks source link

Library Cleanup removes all Jellyfin entries from the Library #409

Open gersilex opened 3 years ago

gersilex commented 3 years ago

Describe the bug

A Kodi library cleanup removes all Jellyfin library elements.

Excerpt of other reports this issue might replace or complement:

To Reproduce

  1. Have a working Kodi library with Jellyfin set up
  2. Run Clean Library from the Kodi Settings

Expected behavior

The library should remain populated with the Jellyfin media entries together with local files and entries of other addons.

Actual behavior All Jellyfin entries are gone. Local files and entries of other addons remain populated.

Logs

N/A

Screenshots

N/A (see https://kodi.wiki/view/Settings/Media/Library#Video_Library for Library Cleanup Function)

System (please complete the following information):

Additional context

Jellyfin writes directly to the Kodi database. Direct access to the Kodi database is not allowed according to the official Kodi Add-on rules. Maybe this relates to the issue.

The issue has been reported as different issues but it all boils down to the library entries and is not related to any other addon. Thanks to @CastagnaIT from the Netflix Addon for finding this out.

CastagnaIT commented 3 years ago

I think you made some confusion with what I was trying to explain, but I can understand, it is not always easy to understand the complexities of kodi...

kodi video library supports various type of sources that can be: -sources as real directories that can cointains: STRM or all video files extensions like MKV and so on, these organized for tvshws or movies -sources of network type, like Samba, DLNA etc... -source of "plugin://" type, where the kodi library browse an addon to add all movies/tvshow to kodi library -and other types...

here jellyfin try to use "plugin://" type path and it injiect the paths directly to the kodi library but here something is not done right, and then all the paths added will be deleted at kodi library Cleaning

I believe that the source paths (that are hidden) that jellyfin inserts manually are not full browseable by Kodi's library, where it should be instead, and so the files are recognized as broken (only my opinion).

it wouldn't be possible to use STRM files?

so if you use the "plugin://" as source, you could avoid to use STRM (thing that netflix addon actually not support, but there is a feature requested here https://github.com/CastagnaIT/plugin.video.netflix/issues/877) but unfortunately the real use of "plugin" as source, leads to some compromises of working not always easy to do (like you browse the addon instead kodi library also when you are in Kodi library, can sound strange but it is really so, tested personally)

I think for Kodi 18 (and so relative 19) the only praticable things are:

For Kodi 19 I can officially publish the changes on Kodi 19 having a partial VideoLibrary.Clean based on a directory is also intended to speed up a lot the operation on a big database (of course if kodi dev accept this change)

gersilex commented 3 years ago

Oh sorry, I looked into the STRM files you generate and thought this is the way to go. I think I should rewrite the issue to just report the problem instead of proposing design changes I have no clue about :sweat_smile:

oddstr13 commented 3 years ago

I would love to completely rip out all the current database interaction code, but the addon API is too limited in this regard from what I've found so far.

We could go the STRM file way, and there are JSON-RPC calls to trigger updates of specific directories, but there's no way to actually set up a library as far as I can tell.

It may still be desirable to use STRM + nfo where we can, and only interact directly with the database when absolutely required, such as configuring the media libraries and the paths they point at.

I'm open to suggestions and documentation links to how this could be done – any info on how to replace the raw SQL we're doing right now, really.

https://kodi.wiki/view/Internet_video_and_audio_streams#The_.STRM_file_method: https://kodi.wiki/view/NFO_files#Movie_nfo https://kodi.wiki/view/JSON-RPC_API/v12#VideoLibrary.Scan

In any case, testability (unit tests) should be keept in mind when reworking these parts.

mcarlton00 commented 3 years ago

I'd looked into this before because I don't like modifying the database directly (albeit not involving strm files). I don't see any options for adding folders through the JSON RPC api. Probably equally frowned upon by the Kodi folks as modifying the database, but we could likely edit the $KODI_HOME/userdata/sources.xml file to add a directory of our choosing. It might even help with our handling of mixed content to drop them as "Files" instead of guessing Movie vs TV Show.

<sources>
    <programs>
        <default pathversion="1"></default>
    </programs>
    <video>
        <default pathversion="1"></default>
        <source>
            <name>Movies</name>
            <path pathversion="1">/home/matt/NAS/JF-Testing/Movies/</path>
            <allowsharing>true</allowsharing>
        </source>
    </video>
    <music>
        <default pathversion="1"></default>
    </music>
    <pictures>
        <default pathversion="1"></default>
    </pictures>
    <files>
        <default pathversion="1"></default>
    </files>
    <games>
        <default pathversion="1"></default>
    </games>
</sources>

And then populate that directory with strm files. If we do get that working, we should be able to trigger more targeted content updates judging by this thread: https://stackoverflow.com/questions/46765815/how-to-scan-only-a-particular-directory-in-kodi-via-json-rpc

Might have to watch out when using embedded devices though. We've already had reports of users on FireTV and similar devices running out of storage space trying to handle images, dropping a few thousand strm files on the disk won't help that

CastagnaIT commented 3 years ago

For STRM i can give some tips (i use it for netflix addon)

Might have to watch out when using embedded devices though. We've already had reports of users on FireTV and similar devices running out of storage space trying to handle images, dropping a few thousand strm files on the disk won't help that

an example: Agent of shield 6 seasons are 123 strm files, for a total of 10,4KB

so take in account about 10KB every 120 files

For the NFO files, are useful only if you want to use the "Local information only" screaper in the folder NFO files can take up a lot more space depending on how much information you want to contains

But you could avoid writing NFO files if you set a Kodi screaper (e.g. TMDB)

For the STRM management:

Problems/Side effects:

sidenote: there is also the MediaImport kodi project, I am not sure if it would help, i heard a rumor of the devs will be probably integrated in Kodi 19, but you will have to wait...

mcarlton00 commented 3 years ago

I think we'd have to do NFO files. Some people have spent a decent chunk of time getting their metadata in the server adjusted just right, and wouldn't appreciate getting different metadata in Kodi vs other clients. In addition to people that have custom shows/data for items. Possible alternative is writing our own scraper to query JF instead, but that sounds like another can of worms.

Sounds like there's plenty of other extra quirks involved in strm files that need to be thought through. I appreciate you sharing what you've found so far, definitely gives us a good heads up of things we'll surely run into, so big thanks for that.

Either way, it sounds like this path is going to slow down the sync speeds since we have to go through the api layer (and write extra files) instead of writing to a database. In my mind the ideal solution is to store everything we want in our own local database (jellyfin.db, netflix.db, etc) without touching Kodi's built in libraries. This would give us the speed benefits of database queries but stops conflicts with other addons/local libraries. I just haven't found a good way of doing that at this point in time and have it populate into Kodi's internal library. We can have write our own databases and have our addons query that, but then users have to customize their home views or go through the addon menu which is less than ideal, and something I'm currently looking for a solution for with with JellyCon and the Embuary skin.

We've been contacted about the MediaImport project a few times, but it's been several months since I've heard any updates on it. I haven't personally had much success with it, but my testing was a good while ago. In theory they're supporting Plex and Emby, and with our api 99% the same as Emby it should just work. However since our api is likely to be changing more over time I do have some concerns about it in the future. Especially if our api changes in the middle of a Kodi release cycle, I imagine that's not going to go over super well. User "CopperYang" in our matrix chat has been keeping tabs on it and was the one to bring it to our attention.

gersilex commented 3 years ago

I don't see any options for adding folders through the JSON RPC api

Would this be a thing we should open an issue for at the Kodi project? Editing an XML sounds doable with some classic CRUD operations. There seems to be a clear demand for this and it would even help the Netflix and other addons.

mcarlton00 commented 3 years ago

I've posted a message to their dev slack asking about adding directories or importing libraries, so we'll see if we get a response that way. But yes, it would definitely need to be an upstream thing, and with Kodi 19 coming it's probably a good time to broach that subject so it can get into a release.

I've been playing with the strm files and working through the beginnings of a plan to move to them instead of hitting the database directly, but there's still several questions and a lot more testing to do. Some in response to @CastagnaIT, some are just my own findings and ramblings.

When you play a STRM the video resume is broken (due to a bug of Kodi) so video start every time from beginning

From my testing tonight, this is super inconsistent. If I play a video halfway and stop, then try to play again, it will let me resume. However, if I play a video halfway and stop, then restart Kodi, when I try to play again it will start from the beginning. Seems like a fairly easy workaround (just catch the play action, query the server for userdata and ask the user if they want to jump to previous position), but very odd overall. I'm guessing this is the mentioned bug.

STRM not allow to include video info like length of video or format of audio/video

I'm not quite sure where this comes into play. On the Amber skin at least it shows up fine when accessing with a strm file, both in the skin and during playback. I wonder if JF is providing a different payload than Netflix that includes that info. More testing needed.

As for the new things I found or things that need more research:

The ramblings of a madman: How this could work

Now, the giant hurdle for this plan (outside of the things i mentioned in the previous section): Metadata.

We want to match what's in the server, so scanning against built in providers in Kodi won't work. The way I see it, we have 2 options: NFO files, or write our own metadata provider for JF.

There has to be holes here. It sounds far too straightforward. Not an insignificant amount of code, but straightforward. @oddstr13 tell me I'm wrong. Find the holes, before I start exploring this rabbit hole further.

CastagnaIT commented 3 years ago

I'm not quite sure where this comes into play. On the Amber skin at least it shows up fine when accessing with a strm file, both in the skin and during playback. I wonder if JF is providing a different payload than Netflix that includes that info. More testing needed.

you have to take into account only the media-flag of the skin directory, because when you play the video the media-flags can be updated/replaced automatically according to the played streams

External subs are right out. Again, should be relatively straightforward fix. On playback start, query the server for external subs and pull them separately.

not sure what you mean exactly but for Kodi19 there is also a new jsonRPC in progress to add on the fly the subs to the player: https://github.com/xbmc/xbmc/pull/18445 perhaps you could be interested

The sync would hit a separate jellyfin.db instead of the Kodi DBs. This basically just serves as an index of item_id to file_path so we don't have to try to parse a file tree constantly. Sqlite is way faster than that.

maybe you already knew it but i found out recently.......so pay attention to how many operations at the db do sqlite is fast, but on low-performance sdcards or mechanical harddisk is extremely slow

e.g. if you have to do 100 sqlite same similar "insert" op

This take a lot of time: open db > write "insert" > close .... open db > write "insert" > close ... and so on

This take less than 1/10 of time of single insert method: BEGIN TRANSACTION; + executemany + COMMIT;

so with my addon i have improved the situation from about 20secs to less than 1secs! to do same operations, but of course it depends on the type of implementation and what you have to do...just a suggestion

oddstr13 commented 3 years ago

So, one wild idea I just had is the possibility of using the http network source combined with a server-side addon in Jellyfin, presenting a browsable http tree with strm, nfo files and artwork.

The strm would still be pointing to plugin://plugin.video.jellyfin/, (probably, hopefullt) giving the addon control over things such as transcoding and playback.

This way, we could leave syncing and cleaning to the standard Kodi way of doing it, or trigger partial rescan via the JSON-RPC on initial connection or websocket updates.

Keep in mind that the file and directory naming must be proper for TV series & episodes to be picked up properly by Kodi.

This would avoid the problem of a on-filesystem directory structure bloating used storage space.

mcarlton00 commented 3 years ago

An http directory isn't a bad idea, but that will largely fall on you and the server team. My C# is .... shall we say "not good." First thought is that there might be access speed issues when scanning. Typically any http browsing I've done has had some delay, but that may be more a side effect of like nginx file indexing over the actual technology. The server should be faster in that regard. Of course, it also largely renders the addon ineffective unless a user needs transcoding. It would basically exist purely for playback reporting and transcoding.

gersilex commented 3 years ago

I looked over some HTTP server access logs I found on the kodi.tv forums and it looks like at least Kodi 18.3 does not use HTTP/2 will be somewhat inefficient scanning huge libraries. That's not meant to be negative. Not at all. This means if Kodi decides to go and use a HTTP/2 http client for the HTTP sources, the connections can be re-used and will get a lot faster. Probably somewhere between 10-100x times faster (with a large number of small files).

The HTTP index, that has to be implemented in Jellyfin can be generated as part of the Scan all libraries scheduled task or a new one. This is sometime referred to as a "Static API". It behaves like an API, but it doesn't hit an underlying database, as all the data is pre-rendered like a pre-rendered static website. If the webserver hands out these pre-rendered files, requests from Kodi will be ansered extremely fast, even with HTTP/1.1. Another way would be to have the webserver cache the results, but that brings all the pros and cons of caching with it as you never truly know what the client will receive. This can be combined with refreshing the data shortly after delivering the cached data, so the HTTP responses are always cached AND exactly one call behind the truth. I'm getting way to deep into this, sorry.

However. I really like idea idea and I think we have a lot off leverage to optimize if we see that the concept works out but performs badly.

oddstr13 commented 3 years ago

@mcarlton00 well, it doesn't really do that much more now either. Making sure the library is synchronized and up to date (could still trigger re-scans of affected files), making sure watched state is syncronized between clients, and transcoding if needed (and hopefully soon also syncplay).

@gersilex HTTP/1.1 can reuse connections, but I think what HTTP/2 brings is multiple concurrent requests over the same socket (several transport streams encapsulated in one connection or something).

The structure could be dynamically created, but the nfo cached (the server already has the capability of generating nfos) – This shouldn't be any slower than the current API requests.

I think @mcarlton00's idea of possibly also combining the nfo with a metadata provider that queries the Jellyfin server should be considered, altho this could complicate implementing multi-server functionality – tho, xbmc/xbmc#14972 would be desirable for this.

Long-Man commented 3 years ago

Hi All. It would seem to me that most of problems been tackled here are solved with MediaImport. My question, why not pool your efforts with Montellese to get MediaImport merged rather than pursuing alternate solutions. https://github.com/Montellese/xbmc/wiki/Media-Import. Please engage Montellese. He has encountered most if not all the problems you are trying to solve.

mcarlton00 commented 3 years ago

MediaImport both is and isn't a solution, and only partly helps with the big picture here.

It does help with:

It doesn't help with:

Unknowns:

I've had reports of some users already using it and having good results. I was never able to get it working personally, but the Emby importer works for us. https://github.com/Montellese/xbmc/wiki/Emby-or-Jellyfin-Media-Importer. At some point the APIs will diverge more and we'll have to write our own but for now users do have a path forward with that if they'd like.

It seems like MediaImport will be a good direction in the future. However, (my opinion), at this point we can't rely on it. Until it's been merged into Kodi proper, it's effectively a hard fork with an extra feature. And if we put a bunch of our effort into refactoring planning on MediaImport and then it doesn't get merged before Kodi 19 releases for whatever reason, we're kinda up shit creek. So my personal stance is that until it's merged and included in Team Kodi's official builds, it can't be the central piece of our official plan for the future. Not to mention that we're sure to have Kodi 18 users for years to come still, though we haven't fully fleshed out how to handle that once 19 releases with whatever other breaking changes get thrown our way between now and then.

Long-Man commented 3 years ago

It does support External Subtitles and Playback Reporting. Could you expand on transcoding and remote control sessions?

What problem are you having getting it to work. Maybe I can help. Please bear in mind that the current solution still need effort to get it to work. Why not engage @Montellese to find out what is required to get it into v19. There are four skilled programmers here. Why not put the effort there?

mcarlton00 commented 3 years ago

Transcoding is the process of re-encoding the files on the fly so that they play on the device better. Often caused by the file being a codec that's not supported by the client or bandwidth limits. Trying to pump a 50GB 4k bluray remux through a 3Mbps pipe, for example. And since the URLs and various parameters are often determined at the time you press play, I really can't think of any good way to handle that at the database layer. So it works great if your server is on your lan, but if it's remote it increases the chance of issues.

The remote control is something I don't anticipate mediaimport ever supporting, but it's something we support now. You're able to control Kodi (or other clients') playback through the Jellyfin web interface or mobile apps. It has nothing to do with getting the content into Kodi, which is why I don't see it being included in mediaimport. This is why I say it is and isn't a solution. It solves some things for us, but not others.

As for helping, not all programmers are equal. Kodi and MediaImport are both written in C++. We don't know C++. Me trying to help work on it would be roughly equivalent to trying to read Shakespeare written in Wingdings. Between not knowing the language and just not having enough free time to get into another project, options are kinda limited.

Long-Man commented 3 years ago

MediaImport itself does not support Remote control but it doesn't prevent it. I install the Jellyfin Kodi addon and control playback from the web client. I have been doing that for all of this year. I have done the same with Emby for Kodi and EmbyCon. But I have kept Jellyfin Kodi. I chose what tto play and add to the playqueue.

Technically I cannot see why it would not support transcoding. MediaImport stores a playback url as the path to the file. If it is currently supported in Jellyfin Kodi it should be possible.

Help can be defined in many ways. You understand the server, you know what is the experience that users desire. And finally you have maintained Jellyfin Kodi. I am sure that as a programmer you understand that testing and feedback from a person who understand can be like a boost of nitro to a programmer.

Please give it spin.

oddstr13 commented 3 years ago

Our addon currently stores the following URI in the files table in the video database; plugin://plugin.video.jellyfin/f137a2dd21bbc1b99aa5c0f6bf02a805/?filename=Big+Buck+Bunny+%282008%29+%28HDTV-2160p+-+x264+MP3%29.mp4&id=4d67615aa1fbd62293d1b6e08c82b809&dbid=36&mode=play.

And, as @mcarlton00 said, I'm not particularly good at C++ either – can piece together some Arduino code, and understand the basic syntax, but that's it.

We still have a fair number of users on Kodi 17 (I'd love to know why), so I don't see us dropping Kodi 18 support anytime soon. (even though I'd like to drop Python 2)

Long-Man commented 3 years ago

Please give it a spin. I promise It won't kill you. https://forum.kodi.tv/showthread.php?tid=224794&pid=2980146#pid2980146 Please give it a spin. The repo is included but it is disabled. Enable it and install the mediaimpoter addon.

Montellese commented 3 years ago

Hey guys

I can fully understand that you currently can't rely on my media import work because it's not part of Kodi mainline and I can't provide any realistic timeline because I'm very busy with RL. Even if media import would be part of Kodi mainline if you have to support a large number of Kodi v17 / v18 users it's not really an option.

Nonetheless I want to provide some input and feedback on what is possible with media import:

As @Long-Man has pointed out I'm always happy for new ideas, feature requests, bug reports and other feedback from testers. I'm also not a very good python programmer and don't know Kodi's add-on system that well. Therefore progress on the media import add-ons is slow. I'm currently working on providing a version of media import for Matrix Alpha 2.

CastagnaIT commented 3 years ago

Thanks for this explaination, in the future I will take a look at plugin.video.waipu.tv to understand better how it is implemented, i would like try to adopt this system

mcarlton00 commented 3 years ago

Thanks for the info @Montellese. I'm cautiously excited to see where it goes over time when other addons start implementing it. It sounds like that will significantly simplify things in the future, though given the amount of our users on legacy Kodi versions I'm not sure how soon we'll be able to take advantage of it. We could implement it but route code differently for older Kodi versions, but that sounds less than fun to maintain (though possibly still easier than our database code).

On topic for this issue as a whole, I'm feeling like the strm file route may be the answer for now, but more testing is certainly needed before we can forge full steam ahead.

My reasoning against the http directory route:

It's not a flawless solution, and @oddstr13 is correct that it does complicate how we can ever implement multi-server functionality. I do have the beginnings of an idea for that, but it's not going to get fully fleshed out until we can confirm that other things will work as expected.

I've also realized that our playback functionality shouldn't have to change much. Instead of the strm file containing a link directly to the server (https://{server_url}/Items/{item_id}/Download?api_key={access_token}), we could likely have the strm file point to the addon itself with something along the lines of plugin://plugin.video.jellyfin/{item_id}?mode=play. Still some questions overall, but I think it's worth investigating more at least.

mcarlton00 commented 3 years ago

As usual, there's another hangup. It doesn't seem possible to do this in a way that allows us to get away from modifying the database at all. We can add a bunch of strm files and include that in sources.xml, but the method to tell Kodi what metadata provider to use for a given directory is in the database.

sqlite> select * from path;
idPath|strPath|strContent|strScraper|strHash|scanRecursive|useFolderNames|strSettings|noUpdate|exclude|dateAdded|idParentPath
1|/home/matt/Movies/|movies|metadata.themoviedb.org|28d9889cec58db2c060b82b6be2b4429|2147483647|0|<settings version="2"><setting id="certprefix" default="true">Rated </setting><setting id="fanart">true</setting><setting id="imdbanyway" default="true">false</setting><setting id="keeporiginaltitle" default="true">false</setting><setting id="landscape" default="true">false</setting><setting id="language" default="true">en</setting><setting id="RatingS" default="true">TMDb</setting><setting id="tmdbcertcountry" default="true">us</setting><setting id="trailer">true</setting></settings>|0|0||

This seems nicer than writing all of the metadata to the database directly, but still not a particularly great solution

mcarlton00 commented 3 years ago

After lots of monkeying around in the database and experimenting, I can run a clean library and not purge my library contents. It's not going to get us away from the modifying the database though, and the longer I spend on this the more I think we may need to give up on that particular dream.

Changes required for us to survive a clean library operation:

  1. Modify ~/.kodi/userdata/sources.xml to include a local file path. 
    <source>
<name>Jellyfin Movies</name>
<path pathversion="1">/home/matt/.kodi/userdata/addon_data/plugin.video.jellyfin/libraries/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/</path>
<allowsharing>true</allowsharing>
</source>
  2. Update the strPath column in the path table in the database to point to this directory instead of the plugin directly. old: plugin://plugin.video.jellyfin/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/ new: /home/matt/.kodi/userdata/addon_data/plugin.video.jellyfin/libraries/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805
  3. Update the strFilename column in the files table to use a strm filename instead of the plugin path example old: plugin://plugin.video.jellyfin/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/?dbid=3&mode=play&id=dc3a0d09-9368-0538-46d7-0d8e50ac63a0&filename=V+for+Vendetta+%282005%29.mkv example new: dc3a0d09-9368-0538-46d7-0d8e50ac63a0.strm
  4. Create strm file mentioned above for each item in the library with a link to the old plugin:// path from the database example name: /home/matt/.kodi/userdata/addon_data/plugin.video.jellyfin/libraries/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/dc3a0d09-9368-0538-46d7-0d8e50ac63a0.strm example contents: plugin://plugin.video.jellyfin/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/?dbid=3&mode=play&id=dc3a0d09-9368-0538-46d7-0d8e50ac63a0&filename=V+for+Vendetta+%282005%29.mkv

Pros:

Cons:

Conclusion: This adventure is going to make me pull my hair out. But at this moment I think this is the most sane route forward.

mcarlton00 commented 3 years ago

While discussing this offline, oddstr brought up a good point that writing thousands and thousands of strm files to the disk is likely to wear out sd card based systems and nuke any longevity of those systems

Long-Man commented 3 years ago

TBH. Jellyfin Kodi is not meant for the use case in the topic. Jellyfin Kodi makes Kodi a client for the exclusive use with Jellyfin Server. In such a case _you should not be using Clean Library._

You should be using Perform local database reset or the entries under Manage Libraries on the settings section of the addon. If you want to use other Kodi Addons alongside your Jellyfin server items, MediaImport is what you should be using. Don't try to fit a square peg in a round hole.

Use the right tool for the job, you will keep your hair.

mcarlton00 commented 3 years ago

You're missing the point. We are not clicking clean library. Other things are. We are fully at the mercy of other addons, Kodi updates, or any number of other situations that could trigger the clean library function, like the netflix one mentioned above. That's not good. It's annoying. It causes needless re-syncs, and annoys users.

I'm going to say this as nice as I can, and for the last time. MediaImport is not the solution. Full stop. It may become the solution in the future. It certainly has the promise to do so. But right now, and for the foreseeable future, it is not. It's not been merged into mainline Kodi yet. Given that we're at beta releases of Matrix, the odds of it being merged in now are slim to none (and if it does get merged, I'll have to seriously question the Kodi devs' opinions on what alpha and beta are for if you merge a large feature after theym). Not usable in vanilla Kodi = not an option here. I'm not telling thousands of users "oh we don't support vanilla Kodi anymore, go use this soft fork." Not to mention we have the large majority of our users on older versions that MediaImport is never going to support. It's not an option. I mean absolutely no disrespect to Montellese when I say this, but it's effectively vaporware. They've done an immense amount of work and have impressive results right now, and I can only imagine it's going to get better over time. But not being in vanilla Kodi and not being able to support the large majority of our users means that it's effectively useless to me. End of story.

We have options on how to make this work now, for all of our existing users. It's just exploring through those and deciding what makes sense. It will not be the most elegant solution, because we're still tricking Kodi into doing things it was not designed to do. But we can make it work. And that's what all of my comments in this thread have been, exploring those options. Now I could be doing those in private and just announce a decision at the end and be done with it, but I feel like the users who are interested in this issue would like to see the thought process and reasoning about why things are the way they are and some of the issues for why it's taking so long to find a solution. If you're tired of watching my ramblings as I continue to explore how to break Kodi in new and creative ways, the unsubscribe button is right over there to the right under "Notifications". You won't hurt my feelings, promise.

MediaImport is not the right tool for the job now, and doesn't appear to be in the near future. Stop bringing it back into a conversation where it can't be used.

Long-Man commented 3 years ago

It is not about being nice or not being nice. Emby for Kodi, the addon from which Jellyfin Kodi is derived was not designed to be a regular Kodi Addon. In fact it Kodi does not have a category for such an addon. It was never designed to exist with other addons adding entries to the library. The design is to take over the Kodi library. Other Video addons could coexist for browsing and playing. Granted you and others may have different ideas today, However, that was not the design. In fact the plugin entries to the library that are mentioned were added to Kodi after Emby for kodi had matured. Even strm files from Addons wasn't a thing at that time. strm files were primarily for used for internet streams. That fact in of of itself is a part of the reason for some of the playlist problems in Kodi today. The strm files were treated as playlists as some of them were exactly that.

Content aggregation from multiple sources is what you are dealing with. If you decide to use strm files then you would be changing the design of Addon. That's not a problem. Situation changes and we and our solution evolve to accommodate.

With regards to MediaImport, as you say it is not mainline, true. That does not affect the fact that in it's current implementation of taking over the Kodi database, Jellyfin kodi does not allow for media aggregation. That limitation was seen by the original developers, hence Manage Libraries and reset kodi database in the addon settings. That's not my opinion, That is just the fact.

I can see that you are annoyed with my bringing up MediaImport. My argument is not so much about MediaImport as it is about Montellese. Here is a Core Kodi Dev who is willing to help to solve these problems. IIRC, he is one of the persons with the most commits to the Kodi repo and that does not include MediaImport. Whatever road you choose to travel is going to require some changes in Kodi. He has made many changes to Kodi which from which your version of Jellyfin Kodi has benefited. In your refactoring you may have encountered some code that seems, at best, inefficient now, but was the only way possible at that time. He will not be around forever. His life has changed. A lot of his contemporaries are gone. It is going to be extremely difficult to find someone with the knowledge, skill and time to make the changes. He has very little time now. In a short while he will not have any. Don't lose the opportunity to work with someone who can help. I could put names like JMarshall, Spiff etc but they would probably not mean much to you. Kodi is a beast and you need a BeastMaster to tame it. He did not spend his time on MediaImport because he has no better use for it. He did because you foresaw the problems that you and many other addon devs are having.

It would be sad if the devs commenting in this thread realize after it is too late.

Just My 2 cents.

oddstr13 commented 3 years ago

@Long-Man if MediaImport gets merged into Kodi 19, it would be of great help when we decide we're tired of supporting Python 2, and go full Python 3. Unless and until both of those conditions are met, MediaImport doesn't provide much if any value for us (no offense to @Montellese).

Don't try to fit a square peg in a round hole.

https://youtu.be/lPooEK-whTE?t=195

Long-Man commented 3 years ago

Unless and until both of those conditions are met, MediaImport doesn't provide much if any value for us

Fair enough. MediaImport will not be merged into Kodi v19. I have stated that previously.

Just note that the problem being discussed here occurs because the user has different expectations from the original developers. Original Expectation: Single Source, Single Server. Current Expectation: Aggregate content from multiple sources, server(s) and addons.

oddstr13 commented 3 years ago

And I see nothing wrong with the current expectation, as long as we can find a way to make it cooperate. We may not be able to get it working 100% in all cases, but that is also fine.

gersilex commented 3 years ago

While discussing this offline, @oddstr13 oddstr brought up a good point that writing thousands and thousands of strm files to the disk is likely to wear out sd card based systems and nuke any longevity of those systems

This shouldn't be a problem if those files are not deleted and recreated multiple times a day. Changes to the file's filesystem metadata like last-modified and last-accessed dates are aggregated by modern filesystems and don't cause many writes at all. It's comparable to the transactions that happen on Kodi's built-in SQLite database. But yes - if those files are freshly re-created often this might lead to problems in the long run.

mcarlton00 commented 3 years ago

In theory they're only written once when items are added to the library (and on re-syncs, of course). So in that regard it's not a huge impact. But for example in my tv shows library that would be writing 25,000 strm files. That's still a good bit of disk activity, and if my estimates are right would be ~45MB of disk space. Might be the best method available, I have a few other methods I want to experiment with still.

Ra72xx commented 3 years ago

Is there any progress on this problem? I had to give up on this plugin as I also rely on the Netflix plugin, and the database keeps on getting messed up. (My current workaround is mounting the media folders via NFS into Kodi and using "local information only" as a scraper, which kind of keeps in sync with the NFO-files generated by Jellyfin, but a native integration would be much more elegant...)

Long-Man commented 3 years ago

@Larx , my recommendation would be to add the strm files to your jellyfin library. The jellyfin addon will pass the url from the strm file to kodi which will play it as long as the netflix addon is installed on Kodi. Even for different Netflix accounts and over vpn too. I actually have a common share mounted on my LibreELEC boxes where I store the strm files from the netflix and Prime addons. This way I can add, remove or update them from any device. The server monitors the share for changes and updates accordingly, which is then pushed to all clients. This has allowed me to respect the Single Source, Single Server design of the jellyfin addon. The only downside that I have found is resume points. It asks to resume or start from beginning but does not respect resume. The info is there so I just skip forward to that point. Maybe the addon authors could look at that.

Hope that helps, LongMan

Ra72xx commented 3 years ago

Thanks for the input! Hm, I haven't thought of including the Netflix STRM-files in Jellyfin, interesting approach.

However, thinking about it, as I also use Kodi addons for other online sources which are not available in Jellyfin and which don't provide STRM files, I still couldn't go for the single source, single server approach.

Kodi offers more functionality than Jellyfin concerning unifying different media sources at the moment, so I am probably stuck also with using Kodi's way of handling its library.

Long-Man commented 3 years ago

@Larx, Adding the strm files to Jellyfin does not prevemt you from using other Kodi addons. You would only do this for addons that has the export strm files feature. This is a way to avoid the problem outline in this issue. All media addition and deletion are handled by the jellyfin server so there is no need to use the Clean Library function in Kodi. Using the clean library function in Kodi, while using Jellyfin Kodi, is what leads to the problems outlined here.

Long-Man commented 3 years ago

@oddstr13, @mcarlton00 and @CastagnaIT, a PR has been submitted with some improvements from his MediaImport work. I am not promoting MediaImport this time even though testing and feedback is always welcome. Could you give this build a spin with your addon and provide some feedback and suggestions for improvements. The build contains other improvements that may not be in that PR but could be added if it provides improved performance. Specific changes "In general any Python API method which takes a Python list / C++ std::vector or a Python dict / C++ std::map / Dictionary. The bigger the list / dict the more performance gain. In my performance measurements it was most prominent on ListItem.setCast(). "

CastagnaIT commented 3 years ago

at the moment my concern is entirely with the AddonSignals add-on because of Kodi has a memory leak due to the use of xbmc.executeJSONRPC with JSONRPC.NotifyAll and this is blocking the developments of my projects

for my addon "netfx" the integration of mediaimport is not so simply, there is a need to limit the amouth of requests to the server to prevent possible account ban and this require different works and precautions I think this feature is nice but not of immediate necessity (for my case), i will start studying it when will be fully integrated in Kodi 20

Montellese commented 3 years ago

@CastagnaIT I think you misunderstood @Long-Man's last comment. He wanted to point out that I made some performance improvements to the implementation of the Python API in Kodi which should benefit any Kodi add-on especially when heavily using ListItem methods like ListItem.setCast(). I'm trying to get these improvements into Kodi independent of my media import work because they are not specific to it. Unfortunately there's no Kodi test build available with just these changes so you would have to try it out using my media import test builds but you can safely ignore the media import functionality.

CastagnaIT commented 3 years ago

aah ok, needs a timing comparison between before and after the improvement

CastagnaIT commented 3 years ago

i tried use the mentioned installer for w64 but i think is broken all python scripts fails to install/update and if i copy an existing user data from a working copy of kodi, all addons are without logo and strange things happen when i run netflix...

Montellese commented 3 years ago

i tried use the mentioned installer for w64 but i think is broken all python scripts fails to install/update and if i copy an existing user data from a working copy of kodi, all addons are without logo and strange things happen when i run netflix...

That's odd. I just did a fresh install using the Windows x64 installer and everything is working fine for me. I can install / update add-ons, I can browse plugins and play video streams etc. Media import also works.

CastagnaIT commented 3 years ago

i have give a second try, now has worked, perhaps antivirus sandbox had broken something... boh

however in your Kodi version every time i run the addon or i browse it the loading spin window remains for ~tens of seconds without reason (so from the log the addon execution is ended but loading spin still waiting...)

these are the results for create 181 ListItems, and used: ListItem.setInfo, ListItem.setArt, ListItem.setProperties, ListItem.addContextMenuItems, ListItem.addStreamInfo, (offscreen=True)

Performance version: 34 ms Original Kodi matrix (last Matrix nightly): 40 ms

~16% improved

there is an improvement but in my case it will not be visible to the user, so take in account that i have an PC AMD 8core ~4ghz i think on low-end devices like RPI could make a much more marked difference!

Montellese commented 3 years ago

however in your Kodi version every time i run the addon or i browse it the loading spin window remains for ~tens of seconds without reason (so from the log the addon execution is ended but loading spin still waiting...)

Yeah I noticed that bug as well and it has been fixed in the latest test builds, see here.

these are the results for create 181 ListItems, and used: ListItem.setInfo, ListItem.setArt, ListItem.setProperties, ListItem.addContextMenuItems, ListItem.addStreamInfo, (offscreen=True)

Performance version: 34 ms Original Kodi matrix (last Matrix nightly): 40 ms

~16% improved

there is an improvement but in my case it will not be visible to the user,

Thanks for taking the time to test and measure this. Great to see that it really also affects other add-ons / plugins. ListItem.setCast() is probably the method which would benefit the most but if you already see ~16% without it that's obviously great as well. Considering that I only made small adjustments to 4 lines of code.

In my media importer add-ons I create thousands of ListItems so the improvement is more noticeable.

Long-Man commented 3 years ago

@oddstr13, @mcarlton00 would still love your feedback especially on music which takes notoriously long to import. Remember this is not about MediaImport, you can ignore that part. The interest is more in performance improvements in generating ListItems in Python Addons. Getting this into V19.1 would be good for all concerned.

mcarlton00 commented 3 years ago

We've made significant speed improvements to the music sync, but it's still the longest of the library types. That might just by nature of them typically being larger than video libraries.

I won't be able to test for a few days until I get back to my other rig and can compile it there. I've found compiling Kodi on my laptop is too slow to be feasible most of the time, and from a glance I don't see Arch compatible builds ready to download. Will report back once I've had a chance.

Long-Man commented 3 years ago

Understood. TBH @Montellese haven't done build for linux, I am not sure why. However, I run builds of LibreELEC, Generic and Ras Pi 4, for my testing. I could share those if you are interested.

Montellese commented 3 years ago

Kodi's buildserver doesn't support uploading test builds for Linux since there are so many distribution which require different handling.