Improve playout file cache and reduce disk io

[jooola]

Based on the previous comment https://github.com/LibreTime/libretime/issues/307#issuecomment-967801897

Playout is currently downloading each file to play to a cache directory, which introduce a huge amount of disk IO.

I don't see the benefit of this feature, while playout could simply play the file remotely, directly from the storage. In addition Liquidsoap is also able to do this, maybe doing in it playout is somewhat redundant.

Decoupling the core/api and playout is a good idea, but maybe duplicating data is overkill. Some proper fallback could be setup, if connection between playout and the storage is lost.

Here is a screenshot of the disk Write IO Wait time when playout is running normally with a huge outage in the middle:

Having a constant >30ms write wait time is not considered as a good thing, I consider it as harmful.

[paddatrapper]

The benefit to caching comes in containerized applications. Allowing the playout system to download the file from an http endpoint allows the playout system to be containerized separately without needing to mount the media library in both containers. For things like kubernetes, this makes deploying much easier, as the two containers can then be on different pods, have different scaling configurations and be more resilient (a pod/node goes down and it restarts just the playout or just the api while leaving the other ready to accept connections).

Correct me if I'm wrong, but I understand playing the file remotely as streaming it continuously from the API server. This, to me, is very risky, as any network interruption between the two systems will lead to dead air.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 5 months. It will be closed if no activity occurs in the next month. Please chat to us on discourse or ask for help on our chat if you have any questions or need further support with getting this issue resolved. You may also label an issue as pinned if you would like to make sure that it does not get closed by this bot.

[jooola]

I think this should go in a config option to either download through the API or directly read from the local file storage.

Correct me if I'm wrong, but I understand playing the file remotely as streaming it continuously from the API server. This, to me, is very risky, as any network interruption between the two systems will lead to dead air.

Agreed, I think I was thinking about streaming from the local file system, and not necessarily from an external API.

[jooola]

We recently moved the icecast server to an external server, to de-duplicate it across the Libretime installs.

Our monitoring tool recently resolved the warning about high write waiting time, so it seem I made the wrong assumption with my initial description of the problem.

So this less a problem, and while we might want to reduce IO by reading from the storage directly, this might bring a set of other problems such as integrating a file lock, to prevent deletion and modification. This file lock might get implemented some day, but until then, we should probably wait and see if this is of any interest.