[RFC] Frontend Nirvana

[mbabker]

Historical Perspective

Joomla's UI in 2.5 and earlier was a bit haphazard at best. It wasn't really documented and to build layouts you basically had to reverse engineer whatever core templates you elected to support or hope that core was doing something pretty close to what you wanted. Theming wasn't fun.

In Joomla 3.0, via the JUX initiative, core was redesigned in full based on the newly released Bootstrap 2 framework. It created a new visual appearance for core (which at least for the admin had largely gone unchanged minus some color and icon changes, maybe some positions on pages), but doing so highlighted some flaws in the core architecture and (rightfully at the time) created the perception that core was vendor locked to Bootstrap. That perception has lead to a lot of improvements in core, such as the use of JLayout for rendering form field markup, but it also created some questionable practices like how JHtmlBootstrap renders tabs. The important thing to remember out of this, whether you agree with the decision or not, is that the core markup structure finally had some documentation and it could be expected that developers could build toward a common standard without having to do a lot of hard work figuring out what the standard was.

Fast forward to 2018. The framework ecosystem has blown up massively and the standards have changed drastically. Whereas once upon a time it could be said Bootstrap was the de facto standard, in today's market place it's next to impossible to say one framework holds such a title.

The Problem

Joomla's core design standards have always been based on the default templates in use in each application (Khepri, Bluestork, Isis, and Atum on the backend and Milkyway, Beez, Protostar, and Cassiopeia on the frontend). Each template has been designed with its own framework (either adhoc or an existing third party solution), and has inherently dictated how the ecosystem builds its UI layer. We have also created some bad misconceptions:

• The use of Bootstrap means any extension can be installed and easily integrated into any template
• Joomla templates must be Bootstrap based
• Joomla is vendor locked to Bootstrap (we definitely have a bias toward it but really we aren't vendor locked if you know how to plug into the API just right)

The truth of the matter is the default frontend markup shouldn't be heavily opinionated toward a particular framework. So, the default layouts in extensions shouldn't be designed against one template.

The Proposal

The System Template

Joomla ships with a default "system" template which provides the default component (modal), offline, and error pages. Notice it lacks an index.php file, meaning it actually isn't usable as a template on its own. We should make this a minimalistic template that can actually be used. In some ways this can be the underscores theme for Joomla.

With this template in use, the default layouts (tmpl/*.php in component views and modules) should be redesigned to use a semantic structure (as far as classes and markup goes). What about utility components like alerts, pagination, tabs, cards, and whatnot? We're already doing custom elements for some of these things, use those in the system template and provide minimalistic CSS rules for those elements and things that aren't really CE candidates as needed (so make a usable .card element, but we don't need to include 12 color variants as Bootstrap does).

A Fully Styled Frontend Template

Cassiopeia is a fully styled and fully usable template. It's opinionated toward the designer's tastes and workflow. There is nothing wrong with that. Let it keep being opinionated. Use Bootstrap for all the things? Go for it. CSS grid, CSS variables, custom element every single thing? Go for it. The fully styled template doesn't HAVE to be the core standard, and that's kind of the theme I'm aiming for in this issue (no pun intended).

A Decoupled Frontend Template

And all of that leads to this point. Core should not need to ship with the same default template for 6 years. And a core update really shouldn't force someone's favicon to reset because they're using the template we're shipping when they set up their site. So, we should work out a way to handle our packaging and distribution so that the template updates are standalone from the core release cycle and that should a day come in the 4.x lifecycle where a new template is designed in full we can distribute that without distributing Cassiopeia.

What About Cross-Application Resources

In general there's agreement that there isn't a problem with the admin template being more closely locked to Bootstrap but the frontend template shouldn't. This gets interesting when you start getting into things like form fields, since it can be argued our form APIs and uses are more dominantly geared toward use in the backend than the frontend. In the case of forms and theming, one way Symfony's component addresses this is through the concept of form themes (they have a hierarchical setup where they have a base theme which is basically semantic markup then child themes where you start getting framework (and version) specific structures). Maybe this is one part of the API where I have to concede that we need some kind of framework toggle and core has a way to support a library of layouts for Bootstrap, Materialize, Semantic-UI, etc. There's an issue to a lesser extent with the message container (alerts) and pagination but those has always supported markup overrides and are much smaller UI components.

Ecosystem Impact

Well, this one's to be determined. Maybe this means the J4 ecosystem isn't full of Bootstrap templates and other frameworks have a sane opportunity for consideration since you wouldn't have to bend over too far backwards to be able to use them. Maybe it adds burden to templaters because it means extension layouts aren't as plug and play into templates as they are in a purely Bootstrap environment. Considering the push to use custom elements and decouple Cassiopeia from the Bootstrap lock in, in some ways the things I'm writing here are already in motion but still highly tied to one template design.

[ciar4n]

I think this would be a move in a positive direction. To me this clearly defines that Bootstrap is a feasible option (or indeed any framework) rather than the 'default' Joomla UI framework. Wither we like it or not Joomla is opinionated towards Bootstrap for the majority of users. This change would clearly define that you use Bootstrap by your own choice rather than it been simply the convenient option when using Joomla.

I don't consider it that big of a burden to template developers. If anything it might encourage a wider diversity among template designs. Most importantly any wanna be templaters will be encouraged to learn and use overrides at an early stage.

[dgrammatiko]

@mbabker are you suggesting using the system template if no other template is available or?

[mbabker]

For a lack of better comparison, make the system template fully usable and make it not much more styled than the Bootstrap starter theme, leave "fancy" styling to a fully themed template (i.e. Cassiopeia)

• Add a index.php file to the system template that way it can actually be used as a very basic starter style template (again, think Bootstrap starter theme or WordPress' underscores)
• The layouts in frontend facing extensions (and preferably the layouts folder) be built using this template as the benchmark, opinionated style decisions should be "owned" by a fully styled template and be implemented as layout overrides
• This in essence makes the system template an adequate base to build templates from in general because the default layouts are less opinionated toward style related matters and more opinionated toward rendering semantic markup with basic integration of our UI utilities (custom elements if you're building a layout with tabs or alerts or whatnot, basic CSS utility classes for things like badges/labels, or to a larger extent markup for something like pagination)
• I want to force people to look at the frontend markup the way it should have always been; everything in the extension directories (*/tmpl) shouldn't be designed based on a fully themed template, yet every time the default frontend template is changed in Joomla all of the default layouts are rebuilt to suit that template author's design

[wilsonge]

I agree with this from the principle of making template development easier - the question is how to give extension developers a way to showcase their products. Because they can't be expected to develop something that requires full styling - it's going to lower the value of their product by quite a lot (most people buy them expecting something that more or less works out of the box).

I'm concerned we'll go back to the Joomla 2.x days where each extension developer shipped their css framework of choice

[mbabker]

There are two ways to manage core.

1) Status quo - Core and ecosystem are presented as vendor locked to whatever framework is chosen for the default frontend template, creates major hassle in major version upgrades and an inability to upgrade the UI framework

2) A semantic markup less coupled to a specific framework using some style of class naming schema (BEM fits well here) that developers can reasonably expect templates to adhere to

What does implementing 1 mean? Stop beating around the bush and just use Bootstrap for crying out loud, all this abstraction layer and CE and whatnot stuff just makes it harder to use the framework and if we're going to lock ourselves to it then just use it without adding complexity and bells and whistles for the sake of complexity and bells and whistles.

What does implementing 2 mean? Honestly, nobody's going to know unless we try. SolarFlare, Milkyway, Beez, Protostar, and now Cassiopeia managed to dictate the frontend UI standards because the entirety of the ecosystem is designed around what our default template looks like. The default markup structure of core should not change because we run a UPDATE #__template_styles SET home = 1 WHERE client_id = 0 AND template = 'foo'; query, yet that is essentially exactly what happens.

[phproberto]

This is pretty much my original plan for v3.5. Based on the investigations and discussions with frontenders then I suggest:

• Use default instead of system as this is a concept that's clear for users coming from other systems like drupal or magento.
• Move layouts/joomla to templates/default/layouts so all the default stuff is in the same place. Using default or whatever template we are going to use makes it possible to just replace the default template without having to change layout include paths, etc.
• Move all the frontend frameworks to their own libraries and move them outside the core repository. So we have libraries/bootstrap or libraries/foundation, etc. where all the logic and layouts for cards, tabs, etc. are stored. A frontend framework evolves faster than core so it's better to keep them outside so users can upgrade them at their own speed. We should maintain the bootstrap library if that's going to be used as default.

[brianteeman]

@phproberto is templates/default an actual working template or just a location for "shared" assets

[wilsonge]

This is pretty much my original plan for v3.5. Based on the investigations and discussions with frontenders then I suggest:

This elaborates further on @mbabker 's proposal i guess - but it still doesn't exactly solve the issue of what extensions are supposed to ship with. We can't expect extension developers to ship with no styling - although of course in an ideal world as a template developer you would expect that. And to an extent BEM would shove us even further into that box.

I'm not against this at all. But i need a viable proposal for how extension developers ship their frontend to consumers.

[ciar4n]

I'm not against this at all. But i need a viable proposal for how extension developers ship their frontend to consumers.

A defined class naming for the basic UI elements. Template can then style these classes (eg. map them to the CSS framework of their choice). As long as extension developers use these classes, they can be assured their UI element are styled by the template. Joomla could include some basic CSS for these UI elements as a fallback.

Custom element cover the more complicated UI elements. Layout styling should be done by the extension in their own namespaced CSS.

[mbabker]

I'm not big fan of "well look at what they do", but look at how the WordPress ecosystem deals with this.

WordPress core doesn't have a frontend UI framework. There are a handful of shell files left to show a header, sidebar, comments form, and footer. Just to make a theme work it has to have what we would equate to a component's default.php layout file; this can be empty but literally all your site will show is that header, sidebar, comments form, and footer.

The only major frontend facing plugins I've dealt with are WooCommerce and Gravity Forms. Woo's markup structure is pretty plain, but they also have a bit of a CSS architecture of their own, and some integration styling to some of WordPress' default themes. Gravity Forms I've not touched the frontend rendering of so I can't comment. Anything else I've touched has all been at the theme level and essentially how we treat layout overrides, nothing special there.

So how does this get pulled off in the context of Joomla?

Just as @ciar4n pointed out, we have to have a standard for class names for core UI elements. For templaters, taking this block from the article layout as an example, using some standard (BEM, joomla-badge-unpublished-content, I don't care just have a consistent standard) you identify the key elements of the container that would have some kind of styling. Yeah, it might mean there's a lot more class names in our markup that doesn't have practical use in the context of CSS or JavaScript, but it's a compromise we should make as it relates to a foundation for a system's UI layer that can be built upon and customized with ease. When you start getting into full on components (pure HTML only elements like a navbar or a pagination container or something akin to Bootstrap's alert component), you stick with the same semantic based markup with identifiable classes that should be used as the building blocks. When you start getting into the more advanced UI elements (accordion, tab, or something akin to Bootstrap's popovers), that's where you start getting into the custom element stuff. With all of that said though, forms are going to be something we really think about; JForm is used across applications and it's clearly pretty easy to let framework specific thinking slip into this API and its default output since it is used so heavily in the backend which is going to stick more closely to Bootstrap "pure".

Just as today, if you need something out-of-the-box that goes beyond what core ships, you have to include it in your extension.

I don't honestly see why we can't have a "clean" base markup that has basic styling rules in place (I'm definitely not advocating for the default/system/whatever-we-name-it template to be entirely naked, but stylistically it should be pretty unopinionated beyond implementing whatever we decide are the core UI components, yes that would mean we have some styling rules for simple things like badges/labels or that the pagination element is rendered nicely and not a vertical stack of unstyled links). Yes, it does create a bit more work because you either have to build a framework of layout overrides to just use your preferred UI framework without a mapping layer or your SCSS gets a bit complex to deal with mapping the core class name structure to your framework's classes.

Either way, all of this discussion has identified another thing we need in core. Template hierarchy. It'd be easy for developers to have their default markup based on the "clean" solution and a package of layout overrides based on Cassiopeia which are styled toward the template. But if you distribute that as a files extension, you can't update it without wiping out user customizations. We practically need something like WordPress' child themes logic. There has to be a better way of customizing the core template without basically forking it (which is what the duplicate template functionality in the template manager basically does), how many times over the years have we all dealt with support questions because a user customized something in the Protostar template and it got wiped out after an upgrade?

[phproberto]

@brianteeman it's a real template with a special name. So v5 can come with a total different template that means just rename old to XXX and use a new default.

@wilsonge I really don't see an issue shipping a default markup for a specific framework selected as default (i.e bootstrap). Extensions developers can keep shipping that as their default markup. Which is pretty much what he have now. If markup is 100% overridable things can be customised.

We should stop worrying about providing solutions for any website out there. We cannot do that. We cannot/should not maintain our own frontend framework.

Instead we should focus on:

• 100% overridable browser output
• Extension dependency management

Move frontend frameworks to libraries that users can upgrade at their own speed. Core can be shipped with bootstrap v3 but in the joomla-extension bootstrap repo you can find v4 and use it. Same for jquery, etc. Anything that evolves at a different speed than core needs to get moved outside even if it's served at a specifc version in core.

External libraries allow templates to have dependencies that are tested on install. If system does not meet them then install fails and users is requested to take a decision to solve the problem.

I've my own basic dependency manager that supports manifest sections like:

    <dependencies>
        <dependency type="library" name="joomla_entity" version="1.0.2" upgradeableVersion="1.[0-9].[0-9]" size="89336" hash="5a94f4ded855d34fba18c110ab1b1f23" file="dependencies/joomla-entity-v1.0.2.zip">
            Joomla Entity by phproberto
        </dependency>
    </dependencies>

So a extension can ship a zip (or use an URL) of a dependency that can satisfy some upgradeableVersion rule. If installed version is outside those requirements then installation fails and user needs to manually solve the issue.

This would be an example of extension shipping a bootstrap 4 version:

    <dependencies>
        <dependency type="library" name="bootstrap" version="4.1.2" upgradeableVersion="4.[0-9].[0-9]" size="89336" hash="5a94f4ded855d34fba18c110ab1b1f23" file="dependencies/twitter-bootstrap-v4.1.2.zip">
            Twitter bootstrap.
        </dependency>
    </dependencies>

Possible cases:

• System does not have bootstrap: it gets installed
• System has bootstrap in an upgradeable version (v4.0.0 to v4.1.2): it gets upgraded
• System has bootstrap in a non-upgrade version: installation fails informing the user about dependency.

This is nothing new. That's what composer, npm, etc. are already doing and seems to work....

[phproberto]

@mbabker I agree. Child themes can be a cool addition.

[mbabker]

👍 on also working out a dependency manager in Joomla. It is pretty commonplace these days for extensions to have cross-extension dependencies, we should do better to support checking more than compatibility with the currently running Joomla version.

[phproberto]

I'm already preparing a PR @mbabker as a starting point for that dependency manager.

[peteruoi]

Sorry for my intrusion. (probably this discussion out of my league) However, if i understand correctly the vast majority of (smaller) extensions that give some frontpage output will have to support many frontend systems? Something which is obvioulsy not easy to do... Or the templater has to support certain extensions... I think it's good (and probably profitable) for big joomla companies that will have to give a template (of the framework they want) together with the supported extensions but what happens when i want to use this gallery that is free and this template that uses different framework? I need an extra developer to customise the gallery output to this template? So maybe with this way we fragment the joomla ecosystem to these kind of templates go with these kind of extensions etc... It's a bliss for joomla that i can take free extensions and insert them to whatever template easily. Let's not destroy this! Ofc it's really good if templaters can use many frameworks suited to their needs. However, can't we have a sollution that this is possible and extensions work out of the box? If we can't have both i personally think this one drawback exceeds by much the benefits and will end in a catastrophy for the joomla extensions ecosystem.

Sorry if i didn't get well what you are saying :)

[dgrammatiko]

However, can't we have a sollution that this is possible and extensions work out of the box?

That is the proposal here, use namespaced css and style your output to your taste. The underline css framework of the template shouldn't interfere with your layout styling...

[mbabker]

It's a bliss for joomla that i can take free extensions and insert them to whatever template easily.

It's a blessing and a curse all at once. Having standardized on Bootstrap 2.3 for the development of Joomla 3, there's a hidden expectation that you should be able to install any extension and it for the most part just work with any template.

Some extensions do have functionality where they need to have their own out-of-the-box styling (and JavaScript) systems in place. That doesn't change here. What does change is the standard that extensions are developed against, which hopefully over the long term avoids UI layer B/C breaks simply because the default template shipped with Joomla is changed for all the reasons pointed out in this thread so far.

[brianteeman]

there's a hidden expectation that you should be able to install any extension and it for the most part just work with any template.

Not a hidden expectation at all - it was an advertised feature

[dgrammatiko]

... was ...

[brianteeman]

... if you remove it then we go back in time and lose even more users

[dgrammatiko]

@brianteeman that is not the point here, the point is to come up with something that:

• actually fulfils that promise
• doesn't break everything every couple year due to some external library change

In other words: decouple and conquer...

[brianteeman]

I know. I was just pointing it that it was an explicit expectation and that without that it will be a disaster

[mbabker]

You can only make such a broad guarantee if you vendor lock the entire ecosystem to one version of one framework (third party or self created) and block installation of anything that doesn't fulfill said framework requirements. Anything else is wishful thinking.

[Bakual]

That's not exactly true Michael. You can have a standard as we do today (call it vendor locked if you wish) where 3rd party extension are expected to work out of the box if the templates follows/supports the standards used. At the same time you can allow to override everything like J3 nowadays does quite a good job. If you have a template that is not following the standard (eg not a BS 2.3 template), then obviously you loose that "out of the box" feature, but on the other hand you've got free to do what you want. Risk vs Reward, your choice.

I'm sure it's very important that we still have this feature in J4 as it played a big part of the success of J3. How we achieve that is a different question and can be done in various ways. But we certainly need some sort of basic standard output. Using Bootstrap is just the simplest approach (with all its disadvantages).

[mbabker]

I'm not saying we should have zero standard. I'm saying the standard shouldn't be "Bootstrap 2.3.2", or "Bootstrap 4.1.1". At the risk of contradicting my past groans about us not inventing our own CSS framework, the standard should be something abstract enough that it fulfills the core UI requirements/features and can be implemented by templates using their framework of choice by whatever means they choose (be it layout overrides converting classes/markup to their preferred structure or using SCSS' extend capabilities when compiling CSS (or just copying a class' declaration and changing the selector if you're using CSS without a compiler)).

The problem is not in having a standard. The problem is in having a standard that changes simply because a new author came in with a new set of tools. What Ciaran's doing with Cassiopeia is majorly different than what Kyle did with Protostar which is majorly different than what was done with the frontend templates before that, in terms of workflow and tools used (J3 and J4 based on third party frameworks, 2.5 and earlier basically being the author's decisions).

There has been groaning for the last 6 years because we didn't upgrade to Bootstrap 3 because of the B/C breaks and now we're upgrading to Bootstrap 4 and saying the B/C breaks are OK because "new framework, new major version, SemVer suckers!". The presentation layer in Joomla is still treated as a second class citizen (HTML output has no assurances in dev strategy, non-HTML output supporting plugin system is laughable at best), if you notice the issues I've opened regarding it have been pushing for treating the presentation layer (arguably one of Joomla's strong suits) as a first class citizen.

[Bakual]

I'm just going to repeat what I always said: I'm fine with inventing our own standards. IF it is equally good documented as the Bootstrap standard. Because that is essential for us 3rd party extension developers. Otherwise the standard is a moot point since it will not be used. And that's where we failed pre-J3.

[mbabker]

There wasn't a standard pre-J3 and the same problems that existed then still exist now. The "standard" is whatever the default template is, it just so happens in J3 it was Bootstrap 2 and in J4 it's basically Bootstrap 4. It's not addressing the underlying problem of the entire UI framework having B/C breaks across major versions simply because we put a new template in the distro.

[Bakual]

Sure, but even with your proposal here, will 3rd party extension devs not try to make their extensions fit into the default (Cassiopeia) template? Or maybe I just don't understand the proposal, it sounds like the thing we discussed in PLT some years ago :smile:

[mbabker]

Remember over the last couple of years all the groaning and proposals to deal with B/C in the UI layer because J3 is built around Bootstrap 2 and that's not the case in J4? This proposal comes up with a way to address that, by having a DOCUMENTED STANDARD (gasp, a first for Joomla) that should be adhered to, and a standard that should NOT change because we decided to build a new template on a new framework (also known as what happens EVERY MAJOR RELEASE).

Or, we have this same round of discussions again if J5 ever comes around and its decided to build a new default frontend template on <accepted-hot-frontend-framework-of-the-time> which makes the UI layer practically incompatible with the previous version.

There is absolutely zero reason we should be having to change the markup in form fields between every major release. We are. Why? Because the default markup in core is designed around the default templates. THAT is the root problem. THAT is what creates the Bootstrap vendor lock perception. THAT is what we need to address.

[Bakual]

This proposal comes up with a way to address that, by having a DOCUMENTED STANDARD (gasp, a first for Joomla) that should be adhered to, and a standard that should NOT change because we decided to build a new template on a new framework (also known as what happens EVERY MAJOR RELEASE).

If you think we can pull that off, then that's awesome of course. No objections.

[mbabker]

We don't have the core resources to actually do this. We're having a hard enough time getting one template finished, no way in hell is it ever going to be practical to maintain a "naked" UI and a themed template.