Rethinking the Approach To Everything (Website/API/Apps)

[covarianttensor]

(Brace yourself for a wall of text)

I've spent a lot of time thinking about the issues raised during the developer's summit. App developers for Jupiter Broadcasting and Chris both know what they want, but there is no way to reconcile having a static website AND have an API to simply app development on various platforms (Android/iOS/PLEX/KODI/Roku?etc..); that would require the website to be dynamic which is not what Chris wants since it complicates running the website - and adds extra overhead to the servers too.

So far, an API has been thought of as an after-thought, the main focus has been the website; that's understandable, but considering the range of the problems that need to be solved, it's the wrong way to attack the problem. A better way to approach the problem is by thinking about an API first and everything else second.

If you've ever attempted to write an app for Jupiter Broadcasting, the first problem you will run into is finding feeds for all the shows and keeping up when new shows come up with their own feed. This was mentioned during the summit, a proposed solution was a 'feed of feeds'. Another problem is determining how to organize the shows within an app; this is a problem me an cbojar have run into when developing a Roku TV app. The first problem has a solution, the second doesn't. An API can obviously solve these problems and would avoid 'hacky' solutions. These are just two examples, many other things come up like livestream urls suddenly changing on you.

As for the website, visiting jupiterbroadcasting.com today, it's clear that it's slow due to wordpress (more noticeable on mobile) and nothing that wordpress does on the site couldn't be done with a static website generator; a static website is, of course, desirable due to a lack of attack surface, and minimal server resource consumption. But this leaves the possibility of an API out of the picture. So far the only approaches to an API that have been proposed have either been to scrape the website or keep wordpress and use it to give out data and function sorta-like an API. Both these solutions are bad, web scraping is not reliable and breaks when the website changes it's layout; and keeping wordpress makes running the website EVEN MORE complicated.

So what if the API existed independent of the main website? What if an independent database existed that was always populated with things like show lists, rss feeds, show notes, app picks, calendar information, show order, announcements, etc... and this database feed the API. Doing this would, of course, require a lot of time and effort to maintain on top of a website. But what if only the API needed to be maintained? What if mobile and tv applications changed their layout based on data from the API? What if a static website generator could automatically generate the website layout based on data from the API as well? Then only the API would matter. The API would BE Jupiter Broadcasting, and the website, mobile and TV apps would simply be dump pipes, driven by the what the API tells them to do, which give you a window into Jupiter Broadcasting's content. This paradigm shift would give everyone what they want, a static website and an API.

Doing things this way presents a few challenges though, which I will list. But before that, please take a moment to review a diagram of this new paradigm shift I am proposing, to help you better understand what I am talking about. This is a '30,000 Foot View' of what the production workflow would look like. This diagram intentionally leaves out specific implementation details.

Challenges:

• The first challenge for developers will be to decide what kind of database the will API use. Is it going to be some sort of SQL server? A bunch of static files maybe?
• The second challenge is finding an easy way for the staff at Jupiter Broadcasting to populate the database with data (e.g. Show Notes, App Picks, Livestream URLs, Show Listing Order, Announcements, etc...) What ever dynamic website or piece of software that ends up accomplishing this task, would only be ever used internally for the sake of adding/editing data into a database.
• The third challenge would be to decide what types of output the API would generate (JSON/ XML/ RSS/ TEXT/ etc...). What sort of queries can you ask it? How to run searches against it?
• The 5th challenge would be to write good documentation for the API. An API is only as useful as it's documentation. A wiki for the API would probably have to be maintained on Github or something similar.
• The 6th challenge would be to find a static website generator (or possibly write one from scratch) that uses data it gets from the API to automatically output a completely static 100% HTML Jupiter Broadcasting website. Any dynamic content can be embedded from other websites such as a comment system. Searching the website can be handled by the API as well, much like No Agenda shownotes website does (http://search.nashownotes.com/). This would make it easier to change the look of the website, since you don't have to deal with a complicated theme template system like in wordpress. I believe a static website generator project (can't recall the name of it), mentioned during the developer summit, stated they would be willing to alter the project to meet the needs of Jupiter Broadcasting. It's possible we could ask them to implement a JASON or XML parser that can be used to automatically generate or update a static website based on data from our API. But, of course, we first new to HAVE an API.
• The 7th challenge would be for developers to rewrite their mobile and TV apps to use the API in order to update their user interface to always show the most up-to-date content from Jupiter Broadcasting in the order and in the style in which you can enforce via the API. A developer of such an app should also be encouraged to incorporate things like: a calendar, announcements, featured shows, featured NEW shows, app pick lists, IRC, etc... if technologically feasible within the application platform (Android, iOS, Roku, PLEX, KODI, etc...). This would avoid things like Allan having to contact every developer to update things like the livestream url; he could just update the link the API returns for it. Also, developers don't have to update apps just to make sure the newest shows show up within it; thus avoiding re-submission into app stores and lag between new shows coming out and them being available to view within the app.

Basically, I feel we should attack the API problem first before we attack the website problem. Even if everyone decides against a static website, we still will need the API.

Hopefully, I was able to successfully communicate my idea, but I know you will all have many questions and I will try my best to elaborate and answer in a timely manner.

[p10tyr]

Your points are extremely valid. Please bear with me since I am a .NET developer and I am not sure how this translates into PHP/nginx or Linux in general.

If JB came to me as a client and we went through all of what was said on the dev summit and taking @covarianttensor post into account I would suggest the following for setting off.

N-Tier Project

• Data
• WebApi 2 (JSON)
• ASP.MVC 5 (Front End)
• ASP.MVC 5 (Back End)

Data You never really want more than one database, ever. CDN for videos and other external files, yea thats a great idea. What database it is doesn't matter but a relational would be preferred like mySQL or SQLLite.

WebApi 2 *MSDN This links into the database and doesn't care what the front or back end does. Even if when you need to add even more layers, say another API for single page applications, it still doesn't care. All it wants is the Database layer. This layer needs to be speced out better, what do we want from the API, maybe it should only be readonly? What methods do we need, etc etc

FrontEnd To satisfy @ChrisLAS the front end will serve static pages and be as optimized as possible. But how you ask? (The pages will also be generated automatically be the help of a plugin.)

Backend This layer ties in everything! Admins log in, you can do your show notes in the admin, in markdown! Nothing is cached here so its the slowest part of the system, but really it wont be remotely as bad as WordPress. You create pages, you write in markdown everything. You tell it where all the links are etc and publish it. After a publish the back end goes and generates all the static files and links for the front end! Finished.

Considerations This is how we would do it for any clients that came to us and its what we have been doing for years for public sector and private. Rapid development in a single project, multi tier in Visual Studio (or Sharp Studio) - It easy to develop, its blazingly fast on IIS 7+... but its Microsoft so it doesn't fit this community. I may be incorrect but I doubt there is anything on Linux that does anything similar. The source can be hosted on GIT from VisualStudio.

If we had 2 or 3 people work on this for a week just in their spare time JB would have a fucking great, working framework.

Only want to say to stay, away, from Azure! Bloody expensive!

Conclusion Sorry I just woke up so I dont feel like making a flow graph, but trust me, its not that complicated.

[covarianttensor]

@ppumkin Your approach to the API might work just fine, but it's at odds with how @ChrisLAS would like to do things. ASP.Net applications might run well on windows, but on Linux you have to use something like mono to run them on and mono has it's own set of issues. A safer bet would be to find a framework for the API that is written in something like ruby or python, whose interpreters are officially supported on linux by the respective organizations that develop and maintain those languages. But whatever database/API framework we use will be highly dependent on how easy the backend for the people running Jupiter Broadcasting is to use. That being said, I love C#.

[p10tyr]

Yea, apart from the Linux only bit, which if fundamental, that would be great to run on MVC.

But look at it conceptually. If something like that can be done in an easy framework on Linux then the N-Tier can also be used. I heard of PHP MVC, but I never heard of N-Tier PHP so to speak, as implemented in .NET to loosly couple the framework.

.NET went open source and allot of time has been spent by MS to make .NET run anywhere... yes its on Mono, but its backed by M$ now :) Sooo... it might just run perfectly via nginx and MonoCGI if they get their way. Then, theres no stopping Visual Studio Devs from doing what they do best.

But, ye, I mostly was talking about the concept, to reinforce your requirements.

1- Requirements 2- Concepts 3-Big argument 4- Mock up

:)

[covarianttensor]

@ppumkin Yeah, but I'm more comfortable developing on linux and visual studio doesn't run natively on it. Either way, .NET is NOT going to be an option. I have a gut feeling there are more linux/ruby developers in the JB community than any other kind. Though, I'm more comfortable with python. But we have to chose whatever our community is going to be most familiar with.

[cbojar]

It seems you both have been putting a lot of good thought into this. I want to throw in my two cents as well.

I feel like @ChrisLAS's biggest goal here is simplicity. He wants to have a workflow that just flows out from how he is doing things/wants to do things into a pipeline that handles the rest. Right now, the target workflow is to produce video/audio files and a markdown file, then push them out into the world. Building and maintaining a large, possibly custom CMS system does not comport to this goal. Instead, we need to reconsider the fundamental assumptions we are making here.

A Jupiter Broadcasting API would be counter-intuitively non-dynamic for the most part. Things change when a show gets published, but then the data is static. An API can be designed such that it doesn't need to return different results per request. For example, a show feed is updated on a new episode, and then it can be served statically since all the data is now set until the next episode is published. A static set of XML/JSON files can be produced for apps to fetch and process.

Because the data is static and public, we really don't need SSL or a dynamic system to serve requests. Instead, we can create a build pipeline that processes show notes as they are published:

Notes Repo --- Build Process --- HTML
                    | | | \__RSS
                    | | \___XML
                    | \____JSON
                    \_____Etc.

It could be as complicated as a build system like grunt or ant, or as simple as a bash script that calls all the scripts in a directory. The notes repo can be an actual DVCS repository, allowing for the addition of hooks to fire off the build service as soon as new notes get pushed. The directory of scripts is nice because build jobs could be written in any language that can be installed (even .NET on mono?). That means that, as long as a script can be written to handle it, any output can be produced quickly and independently, and with as little developer overhead as possible.

This architecture echoes what @covarianttensor was describing when he was talking about using a flat-files data backend. In this case, the flat files would be meta-data-rich markdown files in a DVCS repo. Similar (or more manual for less frequent) workflows could be developed for other elements of the API.

There are drawbacks to such a flow. Because there would be no dynamic endpoints, server-side search would not be available. Search is probably not an immediate necessity, and this could perhaps be added as a service on top of the API further down the road.

[p10tyr]

@covarianttensor cross platform C# IDE. There are others too but this has .NET "Intellisense" VS Devs love so much.

http://www.monodevelop.com/

The only thing I am trying is to get a mock of my N-Tier concept running under FreeBSD. The .NET technology is actually called vNext and I am doing this for the first time, ever. I might try that digital ocean droplet thing now and deploy the site there for 2 months using the promo codes :]

[p10tyr]

I just published ASP.NET 5 MVC onto Ubuntu with nginx frontend... it took me 2 days to do, but its still Beta and MVC had some glitches. But essentially I have a production ready ASP.NET MVC site running on nginx...

OK- Not ON nginx but I bet you clever guys already know that I did a proxy-pass to kestrel... but, it works and the project code-base took me 5 minutes to create.

The nice thing here is that using nginx front end you can swap in/out any part of the n-tier model with any technology now.

http://pumk.in https://github.com/ppumkin/jbdev_aspnet50

Running on Digital Ocean free $10 credit - I also tried FreeBSD but there seem to be a few more issues there with support of Mono and integration of the new KRE. Thanks Techsnap, I finally got a good excuse to try it out

[covarianttensor]

@cbojar I like the flat file approach best. Not having to run any php, ruby, or python code for an api is ideal. All, as an app developer, I really need is a few feeds to organize my content.

Notes Repo --- Build Process --- HTML | | | RSS | | XML | ____JSON Etc.

The solution then would be to find a opensource static website generator project, like picoCMS, to add support for auto generating these kinds of feeds along with the website; or make it so that a plugin could be written to auto generate these flat files.

This kind of setup is closest to @ChrisLAS's original intention. Examples of these kinds of feeds app developers would need are: a feed of the feeds of the latest shows in the order @ChrisLAS would like them to appear in apps, a second feed of feeds of the latest shows with only the last 5 episodes for each show, a feed of feeds of archived shows in the order @ChrisLAS would like them to appear in apps, a feed of announcements, a feed of featured shows (new?), a feed of recent episodes of all shows, a calendar feed, a feed containing an updated list of live stream and audio stream urls, a feed of show notes, a feed of app picks. The feeds should also contain a links to thumbnails and icons of various shows of various sizes. I think that would suffice for an api. This is especially important as I am considering writing an Android TV app for Jupiter Broadcasting soon.

P.S. I'm almost tempted to write something to auto generate feeds to test this out, but it really needs to be integrated into Jupiter broadcasting's work flow, web scrapping to generate feeds is a hacky solution at best.