search

spl0k / supysonic

Supysonic is a Python implementation of the Subsonic server API.
https://supysonic.readthedocs.io
GNU Affero General Public License v3.0
263 stars 59 forks source link

Feature: Podcasts #188

Open thecarlhall opened 4 years ago

thecarlhall commented 4 years ago

I've started working on this, so wanted to file an issue for awareness. I'm open to other contributors if there are any.

Phase 1

These functions offer the base functionality for podcast features, and are the most straightforward to implement. They only read or write records in the db. All of the heavy lifting is done by the items in Phase 2. A deviation from the Subsonic/Airsonic implementation is that these functions do not handle any disc IO including downloading or deleting episodes. The daemon will need to be running, or refreshPodcasts called using the CLI. This keeps the web server from suffering resource constraint due to big downloads, and allows the long-running work to be free from the limits of a request/response cycle.

API

PR: https://github.com/spl0k/supysonic/pull/190 - closed; merged to spl0k/podcasts

Phase 2

These functions are expected to handle more data and take longer to run, and are offloaded to the CLI and daemon.

API

CLI

Daemon

Later

This should be delayed until API compatibility upgrades past 1.9.0.

Open Questions

Q1: Can episodes be streamed to the client (e.g. Jamstash) from the source and not be stored on the server? This would save HDD usage on the server, but relies on the source always being available. If streaming from source is not an option, refreshPodcasts should download episodes by default.

thecarlhall commented 4 years ago

@spl0k Given the size of this work, I'd like to commit it in multiple PRs. If you're cool with that, I can post the first one for getPodcasts, createPodcastChannel, deletePodcastChannel, deletePodcastEpisode. Otherwise, I'll post a bigger PR when I'm done with all of the work. The CRUD of the app is pretty straightforward. The offline/async/period stuff is getting more attention now. I'll have cli access and admin controls as well.

spl0k commented 4 years ago

Go ahead, submit multiple PRs đź‘Ś It'll be easier to review that way, especially when this podcast stuff spans over multiple use-cases.

spl0k commented 4 years ago

Hello.

Any progress on this front? Do you need help, advice or anything?

I just noticed your question. I guess that could be a configuration option. Either download the file or stream directly from source if the configuration doesn't allow the download of episodes. If streaming from source it might be a good idea to cache the file in the case of clients requesting the same episode several times. See supysonic.cache.Cache. If downloading and disk usage is a concern, we could leverage the aforementioned cache to store the files as it provides automatic cleanup if the total size exceeds a limit, but that might require some extra work to keep the database in sync with the cache state in the event of episodes being removed due to the size limit being hit. Maybe the correct course of action would be to not use this cache but something similar (with a max size and/or retention time defined in configuration) designed to allow users to manage the downloaded episode themselves through the CLI. With features like getting the used disk size (total / per channel / per episode?), deleting episodes matching a user-defined condition, etc.

thecarlhall commented 4 years ago

Hi, @spl0k. Work has taken over all of my time. My local repo has some near-finished changes in it (+330,-86). I'll take a look through my notes to make a plan. I'll hopefully start moving some commits upstream by the end of the month.

I've setup AirSonic to mimic it's behavior for some of this. If I remember correctly, I wasn't able to prefix the stream API like I wanted, so we may be forced to download before playing to maintain API consistency. I think default config retained 10 episodes and updated+pruned on a regular schedule. The first 10 episodes are downloaded async when a playlist is added.

Downloading the episode list will happen in the web request. I was thinking on-demand downloading episodes would happen in the daemon and cli (assuming large file sizes), unless there's a way to start an non-blocking job from a request?

spl0k commented 4 years ago

I wasn't able to prefix the stream API like I wanted, so we may be forced to download before playing to maintain API consistency

I'm not sure I understand to problem here (nor how podcasts work in Subsonic/AirSonic actually). Should refreshPodcasts download the episodes or just the list from the remote server? My first guess would be the list. If that's the case we could then add database entries from this list, even without downloading the corresponding episodes. This would assign ids to the episodes. When calling stream, check if the episode exists locally. If it does stream it from there, otherwise act as a proxy (potentially storing it locally at the same time). And since ids are UUIDs they are (almost) guaranteed to be unique across all the database, so you don't need to add a prefix for the stream endpoint. First try to get a Track with the provided id, if it doesn't exist look for a PodcastEpisode and if that still fails return an error; that's how it's done for getCoverArt where images can reside directly on disk or embedded in audio files (with the the exception that Folder.id is an integer). Unless I didn't understand what you meant by "prefix".

unless there's a way to start an non-blocking job from a request?

Not directly by the web application no. The only way would be to send a command to the daemon to tell it start the download. If you need an example on how to communicate from the web app to the daemon check the jukebox mode. Or for the CLI how scans are started.

thecarlhall commented 4 years ago

per AirSonic, refreshPodcasts updates the list and downloads the latest episodes (to the configured limit), and deletes episodes if there are more than the limit. I'll impl this in the cli first then the daemon scheduling will be quick. I was thinking of setting the config via the supysonic config file rather than building a UI for it just yet.

I think you get where I was going using a prefix. I'll dig into your suggestion instead, which sounds more sane than my idea now that I know that code a little better. I was thinking of handling the decision of "stream or local file" using podcast:<id> as the id passed to the stream api. This would allow the api to handle where to route things.

My preference for streaming would be to send an http 302 to the client to pull the stream directly. Otherwise it would stream through the server which partially defeats the point of streaming. Your idea of proxying would be a nice optimization to the downloading episodes. It might be that the first impl of this is "download-then-stream" and a later impl can tackle streaming directly. I'll test more to see if this works with clients, but Jamstash is fussy.

thecarlhall commented 4 years ago

(after thinking a bit more) I think I see now what you're saying about UUIDs. That's clever, and pretty easy to impl. Thanks!

spl0k commented 4 years ago

My preference for streaming would be to send an http 302 to the client to pull the stream directly.

I haven't thought about that. That might be simpler to implement, provided clients support it. One advantage of the proxy over the redirect is that you can provide transcoding if the client requests it. The middle ground could be to redirect if there is no transcoding parameter/setting, and proxy otherwise.

thecarlhall commented 3 years ago

Hey- just checking in to let you know that progress is slow but steady. I'm writing for tests for downloadPodcastEpisode, refreshPodcasts, and stream in the API, and download & refresh in the CLI. After these things make it through PR, I'll wire up the daemon to do the work on a regular schedule.