Relevance (usefulness) of splecific plugins

[johnthecracker]

Here are my comments I made relevant to this topic:

https://github.com/streamlink/streamlink/issues/3218

It will be more visible If I open a new issue about this.

I have a suggestion:

Find out relevance of plugins by conducting a survey like Gyan does for FFmpeg.

You may have noticed that Zeranoe doesn't exist any more and you'll look at Gyan's site sooner or later.

https://www.gyan.dev/ffmpeg/builds/

For example, ask me what streamlink plugins do I use. I can give you the answer without survey. Hls and filmon.

To get good result you could educate voters about which are core streamlink plugin without which streamlink wouldn't always work, like hls plugin.

Then my answer would be very specific, I use only filmon plugin. 1000 voters like me and you would have an idea about which plugins are more and which ones are less in use.

But seriously,

Can you please tell me which are streamlinks CORE plugins so I can safely delete all except filmon plugin?

[mkbloke]

You can (if you wish) in most cases delete all the plugins you don't use. It is not necessary for even the hls plugin to exist if you don't use it, as it's not required by other plugins for parsing HLS streams. Some plugins do however import other plugin classes and in those cases you would need both plugins. These are currently:

bfmtv.py:from streamlink.plugins.brightcove import BrightcovePlayer
dogus.py:from streamlink.plugins.youtube import YouTube
nbc.py:from streamlink.plugins.theplatform import ThePlatform
nbcsports.py:from streamlink.plugins.theplatform import ThePlatform
teamliquid.py:from streamlink.plugins.afreeca import AfreecaTV
teamliquid.py:from streamlink.plugins.twitch import Twitch
tv5monde.py:from streamlink.plugins.common_jwplayer import _js_to_json

Having information on plugin usage could be useful, but it's probably more hassle than it's worth to try and collect it.

Regarding your issue #3218, there is now, it would seem, a reluctance to accept yet more plugins into the master branch of streamlink unless they are going to be popular. I have wondered if there might be value in not shipping the plugins, but downloading them when required and updating them as required from a repository on the Internet somewhere. That would involve work to set up though and perhaps something like a stub plugin that contains all the URL matches for the existing plugins that are not yet on the local system.

[johnthecracker]

Hi mkbloke,

You showed me how I don't know about how streamlink program works with plugins and I did right in asking before fixing something that ain't broken at least. So I leave it without changing something I'm not well acquainted about.

Technology changes as time goes and I have to learn more than I know to get i24 stream to play directly in player. If there were more members in this team somebody might be interested in plugin for a reason of his own. My request was closed after 1 day so I don't know if anybody had a time to see it at all.

Number of plugins and plugin requests are growing and how to organize it all that it works without being a burden? I proposed what I can think of and somebody else certainly can be smarter than me and in a way that makes streamlink grow rather than shrink.

In the future I see number of plugins increasing and say now is the limit what package can take and then, what is the next? In my opinion finding a solution that will make streamlink more attractive and not less attractive and see a few new guys like I see yourself here since not so long ago.

I will leave this issue open in case of somebody's possible good proposal about what to do with ever growing number of plugins.

[mkbloke]

The problem (as I understand it) is that with the current streamlink architecture, each plugin class needs to be loaded from file in turn and its can_handle_url() method called to see if it can handle the URL. The more plugins that are added, the slower this process can become. Utilising one stub plugin or perhaps a JSON formatted cache file (whatever is deemed more appropriate) that contains all the plugin URL matches could help to alleviate this performance issue.

[bastimeyer]

@johnthecracker

Find out relevance of plugins by conducting a survey Can you please tell me which are streamlinks CORE plugins

We don't need to have a vote. Plugins which are relevant are those which match the inclusion criteria that were added shortly after the fork to CONTRIBUTING.md, or those which don't match the criteria 100% and have only been kept for backwards compatibility and/or due to their simplicity. Those technically non-matching plugins are for example those which are based on other plugins, because it is redundant.

I think the confusion about "relevance" comes from the fact that we've been seeing a lot of plugin requests of sites which are way too specific without any proof of actual usage and adding plugins just for the sake of adding plugins is nonsense, as it adds maintenance burden. This basically falls into the "unmaintained" category of the inclusion guidelines. Maybe this should be worded a bit better.

So what you call "core" plugins are basically all plugins which are included in Streamlink. Every other plugin, either sideloaded from the default user path or a custom path, are "non-core" plugins and we don't care about them, because they are not part of the project. Simple as that. If you don't need plugins which are included in Streamlink, then you can delete them if you want for what ever reason.

And if you don't agree with what the Streamlink maintainers consider worthy of being included/merged/implemented, then there's the fork button and the two-clause BSD license. Remember that this is not a commercial product you've paid for. You are asking strangers on the internet to write code for you on a FOSS project every time you request a plugin.

finding a solution that will make streamlink more attractive and not less attractive

More plugin requests or submissions of plugins which are not actually being used or just in very special cases of only one or a handful of users won't make Streamlink more attractive, neither from the user nor developer point of view. It's the opposite. What needs to happen to make it more attractive for developers is to clean up the code base, remove cruft, improve the available APIs and add more/better documentation.

You may have noticed that Zeranoe doesn't exist any more

Oof, thanks for the info... That will require some changes on streamlink/streamlink-assets then if we decide to bump the FFmpeg version included in the Windows releases.

---

@mkbloke

with the current streamlink architecture, each plugin class needs to be loaded from file in turn and its can_handle_url() method called to see if it can handle the URL

See https://github.com/streamlink/streamlink/pull/2570#issuecomment-680032944

The plugin import logic needs to be rewritten, as the module currently being used for importing is already deprecated since a while. And as you've already said, it's loading every plugin first because it needs to rank and then match them.

A better implementation would be doing the URL matching without loading the plugins all at once. But how should this be done?

• Splitting up the plugins into matcher and plugin parts and then loading the actual plugin code on demand? That are still a lot of I/O operations for low-perf devices and the module structure would explode because of that.
• Moving all the regexes into a separate module, so that the plugins don't have to be split up? That monolithic approach will become unmaintainable very quickly.
• Much better would be implementing some kind of pre-build module which parses all plugins, reads the a standardized attribute name of the URL regex of each plugin from the AST and writes a list of all plugin names with their regexes into an import module which then gets used during runtime, similar to the second point I've mentioned. And then user plugins and custom plugins also need to be handled without such a pre-build step.

[mkbloke]

@bastimeyer, thanks for the link. I think I read that thread before.

implementing some kind of pre-build module which parses all plugins, reads the a standardized attribute name of the URL regex

Yes, that's the solution I had in mind. It's the most logical one for the reasons you have stated also.

I have been re-imagining how plugins could work. Essentially, you'd ship streamlink without the plugins or even the plugin-list file. On first run, finding no local plugin-list file it would be downloaded. After that, it would only be downloaded again when the age is beyond some point (automatic update) or unless forced by the user. In the former case, the plugin-list would be retrieved asynchronously so as not to delay the user.

Part of the concept would involve versioning, so the plugin-list for current streamlink might be downloaded from http://source.com/streamlink/v1/plugins/plugin-list and stored locally as something like plugin-list-v1. That plugin-list file would contain entries for each plugin, such as plugin name, url regex, an array of requires and a last updated value.

Plugins would be updated automatically and asynchronously when it's determined that locally installed plugins are out of date. Plugins that are not locally installed would be downloaded prior to use when a candidate plugin has been determined via the URL matching. It would probably be good to provide options that give users full control over the plugin-list and plugin update downloads of course, but I would tend towards defaulting them to on and letting users know what options they can change in a config file to disable them, perhaps upon the very first invocation of streamlink when it has no local plugin-list file.

You can then get at least some statistics on plugins too. If you appended the reason for the download to the URL at the time, then that would give a little more insight too, such as ?install, ?update and ?required.

Anyway, it's just a concept I've been mulling over...

[bastimeyer]

@mkbloke Streamlink is not a package manager and should not be turned into one. https://github.com/streamlink/streamlink/issues/1347#issuecomment-357467050

[mkbloke]

Streamlink is not a package manager and should not be turned into one.

A fair point. I'm not overly interested in arguing over policy, so I'll say no more about it.