search

nzbgetcom / nzbget

efficient usenet downloader
https://nzbget.com
GNU General Public License v2.0
302 stars 16 forks source link

Websocket API #67

Open brianmario opened 9 months ago

brianmario commented 9 months ago

Hi! So happy to see this project get new life. I know there are some other active forks as well. But I've chosen this one to follow, for now.

I have a couple of apps that act as a sort-of front-end to nzbget (among other things) and would love to have the ability to get closer to realtime updates from nzbget. Currently I poll the API to get updates, just like the official UI does. But if a websocket API were added, the official UI could be ported to it and it could also be used by 3rd party developers.

I think this (hopefully) won't require too many changes to the internals of the codebase. Maybe it could be done in phases:

  1. Make the existing RPC API available over websocket. I imagine this would be JSON only, but I don't see why XML couldn't also be supported just as it is today.
  2. Add a new "endpoint" to the websocket API for subscribing to events. Instead of making this complicated right away, to start with it could just be a firehose of everything the existing UI would need to drive itself. Download rates, status/state, etc.
  3. Possibly add more granular subscription types to the API. Things like state updates for a download (added, removed, paused, resumed, etc).

It's not my intention to say "hey could you add this for me", but realistically I don't have a lot of time for coding these days. I'd be happy to try and hack on it, and probably will with what time I do have. But if this is at all interesting to any of the existing maintainers, or even someone willing to take the work on - I think it would make a great addition.

luckedea commented 9 months ago

@brianmario Interesting issue and good ideas/suggestions too, that's very helpful. Can you tell more about your use-case - which apps are in your pipeline, what kind of delays do you see?

It is important to get NZBGet "friendly" with other tools in the chain so I'd consider this a high priority issue, but I really have to set it up myself and understand (or better - see) the problem.

brianmario commented 9 months ago

@luckedea to be honest, I'm not seeing any delays or anything at the moment. Polling is plenty fast, and there probably isn't a very valid use-case for making nzbget scale to millions of requests/sec.

So I think this is less about speed and more about getting closer to realtime updates. There's probably a case to be made about "efficiency" as well, but not in the context of any sort of scaling problem.

Personally, I don't think it needs to necessarily be a high priority. It would just be really nice to have :)

luckedea commented 9 months ago

@brianmario That makes sense. Just curious what other tools to test nzbget with and make sure it works efficiently. I still run some old self-written scripts and middlewares to glue some of the pieces together and it's time to drop all that :)

paul-chambers commented 8 months ago

I found the existing RPC-API code to be a little tortured, though that may be because I don't think like the original author :)

Making the information flow 'bidirectional' would certainly help clients like Sonarr and Radarr, who show download progress in their UI. It currently behaves more like 'we've reached this stage' than 'percentage complete'.

Reporting 'percent done' will take more rework than the more simple change of pushing the 'current stage' status rather than polling for it. But from a user experience perspective, 'percent done' is much more granular and 'feels' more responsive than a bar that 'jumps' a handful of times, with little 'progress' shown for indeterminate periods in between. I typically see at least half a dozen downloads shown in Sonarr 'stuck' at the same position on their respective progress bars for a while. All that really communicates is that it's started, but not finished yet. Doesn't really tell you when it will be.

There are some significant challenges to overcome to predict 'how much time is left' with any reliability. For example,

In short, it's not a single linear progression of '0-100%', more like 'stage 3 of 5 is 57% complete'.

There's also the necessity of collaborating with the respective client projects to ensure you deliver an API they see the benefits of adopting. Investing effort into a new API is pointless if the clients are not interested in adopting it. That's often not a decision based on the technical merits but on where their flames are currently highest. For passion projects, you need to find someone on their team who shares your passion.

Plus, there will definitely be a time period during which you have to maintain backward compatibility with the 'original' API, including its behavior.

Having said all that, I'm a big fan of event-driven, pub-sub style APIs personally, and I wholeheartedly agree you can build better clients with them. It can be a paradigm shift for some developers though. It helps if you can lean on existing protocols like MQTT since there are already prebaked resources (code and documentation) that help with the learning curve.