Call for Proposals: Future of LibreTime MVC

[hairmare]

Hi all,

The MVC based Web-Interface in airtime_mvc/ is based on Zend Framework 1.x which has recently reached its EOL date on 2016-09-28.

Due to this something needs to be done. We can probably get away with using the framework for a little while longer, but need to start finding a consensus on how to replace it ASAP.

If you have a proposal on how to proceed, please post a new issue with the proposal and link to it from here. I'm going to make a list of the proposals in here and maybe we can use GitHub reactions like 👍 and 👎 to vote on them.

Proposals

• TBD (I'll link to all of them here)

[Robbt]

In my opinion the most straight forward solution would be to migrate LibreTime to Zend Framework 2.x - here is a migration guide. https://framework.zend.com/manual/2.1/en/migration/overview.html

[hairmare]

As a former customer of Zend (and at the time potential user of that guide) I would rather not go down that route. It basically amounts to a complete rewrite of almost everything while not having any pointers that really help :( IMO Zend screwed paying customers with that guide.

Personally I'd rather rewrite to something less complex than ZF2/3, maybe Silex or similar, at least that's the proposal I was planning to flesh out when I find some time (likely post-3.0.0).

[TheCrealm]

Is there a trend for a specific Framework?

Should it be more simple like Flight or Silex or more powerful like Symfony or Laravel (my personal favourite)?

[hairmare]

I think those are the questions we need to find a consensus on. I'm not sure if there are any new trends in PHP community at large.

We should also ask ourselves if and how we would like to decouple the js/jQuery/angular parts from the PHP parts.

[bburton]

It goes without saying that eventually decoupling the UI from the backend would be a good approach for several reasons. I can't speak to pros and cons of the different frameworks but if one of them that's widely used and well supported has better support for REST API's then that should be a consideration if it would be helpful in completing REST API support.

[gattytto]

key features must be: long-term support and platform-agnostic (since it's a frontend), so I'd say to use js and go (golang) instead of php-based frameworks.

there's a big hype on virtualization+containerization of things and looking at kubernetes (google!) approach of things, one can see their "dashboard" divided in two parts, one is a golang backend that communicates with the real api, and then a pure js frontend, which looks and feels nice, and works pretty well.

To divide the frontend in backend+frontend might sound pretty silly, but not at all in terms of High Availability.

https://github.com/kubernetes/dashboard/blob/master/docs/devel/code-structure.md

just my 2 cents, keep up the good work, this is a great project

[Robbt]

Well I doubt that we will transition over to Go vs. Python because so much of the backend has already been written in Python and so rewriting everything is not likely. But I think a separation of the API that interacts with the Python backend that actually schedules the tracks and interacts with liquidsoap is probably a good idea.

In my mind it would be ideal for the frontend to be separated from the backend via a well documented API. The project appears to be moving in that direction but more work needs to be done on the API and more work needs to be done on the Python end of things as well.

But like hairmare said just decoupling the javascript/angular stuff from the PHP end of things would help simplify programming etc. At this point adding even a simple feature involves touching a lot of different codepoints and languages (php, javascript, python) and interacting with a few different javascript libraries.

[Robbt]

Well it's been almost a year and no progress has been made on this decision yet it is ever more apparent that if this project is going to have a future it needs to be rewritten with some kind of modern framework so that we aren't dependent upon a EOLed codebase.

It is going to be a ton of work rewriting everything regardless of anything else. The big question is whether there is enough coding labor available to really push this through. As we aren't in a position currently where anyone is employed to develop LibreTime it doesn't make sense to pretend that we can just make a decision and then have it implemented.

Another option that I just became aware of while investigating paths forward from Zend1 is Zend Expressive. It's evidently a micro-service framework by the Zend team and it has been suggested that it is an easier path migration wise than Zend1 to Zend2 or Zend3. So I'll investigate it as it might be easier to get some support from the Zend community when we run into questions about why things were setup the way they were.

[ned-kelly]

@Robbt - Why not focus on containerising the whole thing (so docker) and then adding new functionality as different microservices in whatever newer language(es) and/or frameworks you decide? - Keep the current code base as-is but try not to add new features directly into the PHP codebase (other then changing HTML templates etc) - This way the project will continue to evolve but you won't be as tied as you currently are to ZF1 - I realise it's not perfect, but it may be a happy medium (at least in the medium term).

[Robbt]

Hey Ned, I think modularizing it and building a proper API backend is probably a good way. This could definitely include containerizing the different parts of the system with docker. But the API is pretty fragmented at the moment and not very well documented. So yeah rebuilding this and documenting it with a new backend service while adapting the front end is the goal. So defining a real API is probably the first step. It could be a simple REST API but we'd also need to handle auth and write an API for interacting with and defining the calendar and scheduling tracks outside of the JS & PHP it's built in. This work hasn't been done before although there is a partial REST API for media items.

[Robbt]

When thinking about how we can refactor LibreTime into something that is easy to work on I think it might be helpful to take some queues from this tech talk - Simplicity Matters by Rich Hickey - https://www.youtube.com/watch?v=rI8tNMsozo0

As a general point, he suggests taking apart the design and figuring out how to avoid complexity, ie tangled components that make understanding and changing the system difficult. This represents LibreTime in its current state, not only do we have a bunch of libraries joined together because someone thought it was a good idea and wanted to do the easy thing, they also hacked the libraries.

We have various functions that pass around data types in absurd chains of procedural code. There are a lot of ways we could simplify LibreTime.

If we ask ourselves what we need the MVC to actually do and how we could simplify it then we can perhaps start analyzing potential solutions and find the one that allows us to create the simplest abstraction.

Library - a collection of audio tracks with metadata - currently accessible as a json file via the API at /rest/media . I think a REST API for interacting with the tracks is a good idea and we currently use to upload tracks and could use it to filter tracks. The MVC framework doesn't utilize the API but directly accesses the database and does a bunch of display stuff with the datatables jquery library.

Calendar - a chronological abstraction that describes what is planned to air, what aired and provides a means for "shows" to be ordered. It also has a jquery UI library that was hacked to make it work for LibreTime with various timezones etc.

Shows - a item that exists inside of the calendar, really it could mean a lot of different things but ultimately it exists at a metalevel independent of the tracks because that is a "show instance" and the show describes things like who "owns" the show and can login live etc. Automatic playlists are kind of a cludgy way of automating the programming of shows.

Dynamic Smartblocks, Static Smartblocks, Playlists - this is definitely a place where we can improve LibreTime and has been at the heart of where I'm planning on working next (paid for by the grant to my station). How can we simplify this ?

Without getting into the specificity of these things it would make sense if smartblocks were broken down into simpler components -

• Filtering - this functionality takes criteria and only displays tracks that meet this criteria
• Ordering - sorting whatever tracks are returned
• Limiting - returning tracks that fit inside of a specific constraint ie # of items or under a certain combined length (originally it was over a length).

This is in essence a query building tool but one of the problems with our current UI is SQL queries are inherently confusing to people who don't understand them. We also conflate all 3 of these aspects into a single form and we have no way of dynamically limiting or optimizing the order of tracks to try to fit a limit based upon the amount of time left unscheduled in a track. There is also no easy way to reuse a filter with different order or limit other than duplicating the smartblock and then editing it. This creates things like smartblocks called Newest Episode of Show and Random Episode of Show when perhaps it would be ideal to have a show filter based upon podcast automatically created.

Playlists - a user modifiable list of tracks that can be added to a show instance (once added they are no longer tied to the playlist and exist solely in the "show instance") - a manually produced list of tracks - in theory this could also be a useful input for the smartblock as a manual filter that tracks could get pulled from and arranged into a optimal order to meet the criteria

We have lists of tracks filtered by automatic criteria and we have hand-picked tracks put into lists - the way it currently exists in conjunction with the library is complicated from a human side of things because we have overlapping ideas and the actual show instances that would make up the playlist from a show aren't easily reviewable or commentable. Ie if someone uploads a mix as a single MP3 they can't add annotation that breaks it up into individual songs for listeners and/or display as metadata on icecast or for reporting. There is also no record of what portion of a show was from the library vs. live via show source or master source.

Reporting/Archiving - The log of Show Instances/Listeners - This is important for the various reporting requirements provided by the various copyright agencies and also for the purposes of letting people running a station know who listens to what show and/or liked what tracks.

Playout/Liquidsoap - This is the heart of the product of what we are doing but it is probably the most needlessly complex and least understood aspect of LibreTime where there are quite a few bugs and things that can and do go wrong - Basically airtime-playout reads from the web API and pulls tracks and feeds them to liquidsoap which creates its own copy of them that it plays out to the hardware and/or web streams it feeds - it allows for rebroadcasting of external webstreams, broadcasting live via icecast login based upon show or master login and there is a lot of potential for problems.

Remote broadcasts can lose IP connectivity for short periods of time resulting in either dead-air or "whatever is programmed" breaking up the broadcast. The web UI will schedule tracks from the DB even if they are broken or empty. Seen this recently - not sure why the track itself was bad but it had 0 bytes as a download yet the file on the filesystem doesn't appear to be blank.

Ok so how can this be simplified - why do we rename tracks to a #.mp3 when we copy them over with pypo - there is no correlated way of looking up this track in the database latest (or is that the track db ID). What happens when the description in the database of a file doesn't match the contents of a file ie silan errors or what not. (the bane of radio - dead air).

I would like this to be overhauled to be as simple and reliable as possible - the current system isn't ready for production because it doesn't have enough fallbacks and or checks - if a file is bad then dead air will occur without any notification/clean-up or otherwise means of fixing the problem. Maybe we should leverage liquidsoap more here and figure out how to do some of the stuff we are doing in the MVC layer on the liquidsoap level - at the very least we need to figure out how to have a more robust interplay between the two and more accessible records of what occurs for troubleshooting purposes.

This is becoming more of a tangent than a succinct description of ways things could be simplified. But hopefully it can help act as food for thought. Ideally we could create a simpler and more modifiable system for whatever new MVC we choose to use as the new backend.

[ned-kelly]

This is becoming more of a tangent than a succinct description of ways things could be simplified. But hopefully it can help act as food for thought. Ideally we could create a simpler and more modifiable system for whatever new MVC we choose to use as the new backend.

Agree on all your points - I think it's also worth looking at how we can better leverage existing open source projects to help "simplify" several of the aspects of the solution - Some of the improvements you've called out such as dead air detection for example, could be configured as part of a Jack audio pipeline (using something like slientjack) - Which also then begs the question, should shoutcast etc even be configured as part of the system/UI, or just one key audio pipeline is all that is used as the heart of Libretime, and you can just configure everything else off the back of that (i.e. Libretime --> Jack --> Jack (dead air detection) --> Jack (compressor/limiter & other post-processing audio tooling) --> Jack Icecast/Shoutcast etc).

What you're talking about is fundamentally a 'rewrite' of Libretime however and not a small task, Is there enough demand in the community (currently) to warrant a total rewrite & at what cost?

[JohnnyC1951]

I suggest we repackage up all the committed current fixes, and just a few 'low hanging fruit' additions, with an improved & more robust installer (more complete dependencies and improved tests internal) on one named platform, e.g. Ubuntu Trusty / Xenial ONLY and call that 'Beta #1', take a deep breath and take it from there. There are enough seasoned coders both here and in the old AT Forum to take this to Zend 2 -> 3 if properly organized.

[Robbt]

I think using jack could be helpful but I have always found it to add mor complexity when I've used it in the past. It has always required a bunch of setup and configuration that could be annoying to a novice user.

One goal I've had for LibreTime is for everything to be configurable via the web interface because anything else will create a dependency on the end user being comfortable navigating the command line. I also want us to support optional mods that leverage the power of customization but don't think we should create a dependency on this sort of knowledge for the end user.

As far as going to Zend 2 then 3. I think it would be as much work as switching to a new framework. If someone can find backported Zend 1 security patches etc we could stay with it but I don't think it's smart to plan on using something that is officially not supported. So yeah is there enough energy and incentive to port everything ? This remains to be seen, it would take a lot of time for even a team to do this.

[JohnnyC1951]

Roger that on Jack. And only useful on local servers. My vote is to maybe use the pulse interface, get it so works, I.E. make LT a Pulse Sink and put a 'Beyond here be dragons' notice on it. Re Zend. As I understand it: There are helper tools from Zen1 -> Zend2, but not from Zend1 -> Zend3. So most folks go to Zend2 first. I gather from Zend2->3 is easy.

[Robbt]

If you know of any helper tools feel free to post a link to them. I've seen guide but I've also seen posts that it'd be easier to switch to their microservice framework I mentioned above than go to Zend 2.

[Robbt]

So just to keep this topic alive - and for me to share a resource I found this article on how to port a Zend 2 app to Zend Expressive - it might also apply to Zend 1 to Zend Expressive. Basically it involves moving to PSR-7 middleware from the MVC framework. I haven't studied it indepth.

I guess it seems like we need to have enough experience with a framework and what would be involved in the rewrite to figure out the next steps.

[gusaus]

Is this something that can be done in parallel with other priorities and milestones https://github.com/LibreTime/libretime/projects/2? Both could be set up as goals on https://opencollective.com/libretime if/once the project scope is defined.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 5 months. It will be closed if no activity occurs in the next month. Please chat to us on discourse or ask for help on our chat if you have any questions or need further support with getting this issue resolved. You may also label an issue as pinned if you would like to make sure that it does not get closed by this bot.

[zklosko]

@paddatrapper continuing from #1122 discussion...

But like hairmare said just decoupling the javascript/angular stuff from the PHP end of things would help simplify programming etc. At this point adding even a simple feature involves touching a lot of different codepoints and languages (php, javascript, python) and interacting with a few different javascript libraries.

Decoupling helps build modularity and works great with containers, but creates complexity instead of simplifying. As we're planning the rewrite now (Vue frontend, Django API), we still have to change code in multiple libraries/sections, though only in two languages. If we're aiming to keep the project simple, we should minimize modularity in the project. I'd like to propose one of the following:

1. Switching to Node.js for both the UI and the API: templating (Node/JS), client logic (Node/JS), server logic (Node/JS), and backend (Python), or
2. Sticking with the Django API and rewriting the UI to use Django's server-side functionality: templating (Django/Python), client logic (JS), server logic (Django/Python), and backend (Python)

Either way, we cut back on the amount of libraries/sections. I think option 2 would be best because Django has functionality like user auth and i18n built in and we've already built the new API using Django. We can build Vue interactivity into pages by coding Vue into our HTML and JS files without needing needing to bundle our code with Webpack; the rest (i.e. most templating, user auth, i18n, basic forms) can be done in Django and rendered on the server.

I know decoupling is sexy but it puts a lot of pressure on the client to offer up a good experience for the user. Browsing LinkedIn usually lets me down because my browser has to request everything and build the page before I see anything, which makes me feel like I'm on dial-up because of how slow it is. Speed and durability are more important than a glossy UI for broadcast-grade software, in my opinion, and we should have that be our main focus.

Just my two cents. I'd really like to hear @hairmare's take on this since he's already started with the Vue frontend.

[paddatrapper]

I agree that 2 is better than 1. However, the issue I see with using server side rendering is that we are then defining everything twice - once for the API and once for the view. For example, if we want to add a view for an advertising media library, we have to design an API call, a view render function, template and javascript to make it all work. With a SPA, we implement the API call and template. Then the javascript logic comes out of the frontend templating work automatically.

[zklosko]

Okay, I think I see what you're saying.

[paddatrapper]

Though I agree that we want to try keep things as simple as possible. We already have 5 independent different services, which is a bit of a nightmare...