Use a JavaScript framework for Skosmos GUI

Not a bug, so not using the template suggested for issues. Sorry if that's incorrect.

Duplicated effort to render HTML pages

While working on #846 , we had an issue where the concept mapping properties were loaded only when the concept page was opened directly in the browser.

Going through Hierarchy, Alphabetical order, or Groups, would not trigger some Ajax code created for the concept page document on ready function.

So when the concept page is opened in Skosmos, from what I understand, it is normally displayed via PHP, writing to the stream the rendered template. But when users press on the links in the hierarchy tree, for instance, it intercepts and cancels the event for the click.

Then there is an Ajax call to the backend which returns HTML. The HTML elements returned are accessed via jQuery, and updated on the screen.

The problem

There is an issue with that, where we have PHP code for rendering the template, that is replicated in JavaScript. This increases complexity for maintenance and development, as well as the risk for bugs.

The code used to create the hierarchy, groups, and even to choose how to display the concept are also a bit complicated, and could possibly be simplified with the patterns used in these frameworks, as well as benefit over ES6 (+webpack, babel, etc, which can be a con, as these tools are not so simple) or TypeScript.

Finally, the JavaScript code is tested manually only, as far as I could tell. With these frameworks there are multiple ways to unit test the code (Mocha, QUnit, Karma, etc), as well as functional tests - similar but better than old Selenium - like Cypress or Nightwatch.

Possible Solutions

Angular, Vue, React, even Backbone are frameworks that provide ways to load data - normally JSON, but all these should work with HTML - and update the UI.

Using Vue as an example, it is possible to use a history mode in the vue-router component, so that you don't have url's like skosmos#vocabulary/concept/123. The history mode allows you to keep the same URL you have without Vue, like skosmos/vocabulary/concept/123, without that # token. So nothing would change in the navigation for Vue.

Vue was used by GitLab because they could add it to their jQuery application without having to switch the whole app to Vue. So it should be possible to do it in incremental cycles (possibly with other frameworks too, just using Vue here because I'm using it for another project, and am more familiar with).

Visually, nothing would change. I suspect UI rendering would be faster, as Vue has a few optimizations for caching, modifying the DOM efficiently, etc.

New features for the group/hierarchy/etc section like #366 would be simpler to implement as it would not have to change the existing code (assuming the option for JavaScript was used in that issue), as well as issues like #768 , as there are many libraries that support it out of the box (in my case I'm using Vuetify, which has some nice internationalization options, including rtl).

Possible workflow

Assuming this is acknowledged as a problem in the current GUI of Skosmos, I think this could be approached in the following way:

[ ] Choose a framework
[ ] Choose language (ES6, TypeScript, etc), and also libraries
[ ] Choose a page to migrate
[ ] Report the improvements, challenges, pros&cons, etc, in a pull request
[ ] Decide whether to continue or not (I am optimistic that at this stage the pros would outweigh the cons)

I think once the initial framework, libraries and language have been chosen, the very first page to migrate could be the about. Just to show that everything is working.

Then the next page could be the Concept page. This is a very challenging page to migrate. Would require several base classes (models, services) as well as some components. This would cover writing these base classes, writing components, and also testing.

It is possible too, that there would be the need to add more methods to RestController to return the data as JSON instead of rendered templates. I suspect it could simplify the PHP code base too.

Then a comparison of the previous code and render time & performance, against the new page would be enough to decide what to do next.

Just my 0.02 cents :)

Cheers Bruno

I think this a good idea! 👍 It would be a huge improvement to the maintainability of the codebase, but it's a lot of work especially on the concept page. There's a lot of technical debt in that untested jQuery mess. :)

Sounds promising, but indeed this would be a lot of work.

However, I think it would be best to aim for minimal JavaScript. In any case Skosmos pages will be mostly server-generated in the future too, so the aim is not to become a "single page app" but more of a traditional website with some dynamic enhancements when they make sense from a UX perspective. Here's a good, recent blog post with some intriguing (though a bit extreme) ideas about how to avoid using JS in the first place: https://dev.to/winduptoy/a-javascript-free-frontend-2d3e

Sounds promising, but indeed this would be a lot of work.

+1 definitely.

However, I think it would be best to aim for minimal JavaScript.

In any case Skosmos pages will be mostly server-generated in the future too, so the aim is not to become a "single page app" but more of a traditional website with some dynamic enhancements when they make sense from a UX perspective.

That would be great! The intention was not to suggest to re-write Skosmos as a single page app (though adding JS code for the About page was probably a bit too much). We could do that, and have Skosmos as an API only, or we could mimic GitLab, migrate parts of the app to Vue (or elm, angular, etc), and stop there.

Or even try different approaches.

Here's a good, recent blog post with some intriguing (though a bit extreme) ideas about how to avoid using JS in the first place: https://dev.to/winduptoy/a-javascript-free-frontend-2d3e

I agree with most of what he said, though I think he could re-word a bit to be less extreme (and he would still be right in most of what he said). e.g.

Don't be scared. You can build it yourself! You don't need a framework!

That's not a good statement. Would be better if he said that if you have a simple enough case, a framework may not be necessary. But if you decide to implement your own code you probably will have to stay up to date to security, DOM, browser compatibility, HTTP compliance, etc.

I think if you are good with JS (looks like the author is very familiar with JS) then it won't be a problem. You may be able to recognize beforehand whether you will need a framework, or not. Which parts of your app will need JS or not, and even write a small JS code to take care of what is necessary.

with some dynamic enhancements when they make sense from a UX perspective

I think this is the current proposed design/approach in Skosmos, except it got a little out of control in some pages. But components like the search with suggestions, the concept page, these will probably need some JS code for UX.

A JS framework may not be the right way to solve the problems listed in this ticket. But maybe we could change the title of this issue to: "Investigate how to implement JS code for Skosmos UX with good quality".

That would allow discussion that could go in the direction of

using a framework that does what we need, but not going SPA necessarily
implementing a small code base in JavaScript, with tests (karma, mocha, qunit, there's a long list of libraries for unit testing the application JS, I'm sure even the author of that blog must support having tests for the JS code)
using more libraries to simplify the code like lodash, underscore, etc

And etc, but having as directive to produce code that provides good UX, and is simple to be maintained by Skosmos developers and also by new contributors.

WDYT?

I'm following the diffs to the draft Wiki page about SKOSMOS 3.0 :slightly_smiling_face:

The frontend will be rebuilt. We will still use Bootstrap CSS for layout. Partial page updates will be handled by a JavaScript framework (not jQuery); for this purpose, AlpineJS is currently the most promising candidate, while Vue.js is another option.

Besides AlpineJS and Vue.js, maybe it's worth considering Svelte and htmx too :+1: (I think Svelte tries to do the same as Vue, but without a virtual-dom, and htmx is more similar to alpinejs).

Thanks for chiming in @kinow ! We were actually going to ask for your input on the draft but you found it before we have even finished writing it...

Regarding JS frameworks, I think the core issue we have is the tension between server-side and client-side HTML generation. Whose responsibility is it? The most SPA oriented frameworks assume that the server just provides some kind of blank slate HTML and from there on everything is handled on the frontend side. We don't want to go there, but would like to mostly generate HTML on the server side (at least the core parts of the page, e.g. concept information), but then perform partial updates.

Hi @osma !

Thanks for chiming in @kinow ! We were actually going to ask for your input on the draft but you found it before we have even finished writing it...

I'm really looking forward to Skosmos 3.0, so I've been watching issues and Wiki :slightly_smiling_face:

Regarding JS frameworks, I think the core issue we have is the tension between server-side and client-side HTML generation. Whose responsibility is it?

I think there's no correct answer here :sweat_smile: There are some cases where client & side both perform the same action, e.g. security checks, validation. The templating is, as you said, normally handled in the UI/visualization side.

In the case of Skosmos 2.x, I think visualization is currently handled in both PHP with Twig, and also in JS with JQuery.

I have always found it a bit confusing, to be honest. Regardless of a JS framework or not, for me it'd be simpler to have PHP handling everything with Twig, and JQuery calling the server to send POST requests while showing a spinning wheel, for instance. Or JS handling everything, and the PHP side just sending the necessary data.

At the moment I believe there are Twig templates handled in PHP, and are then loaded and placed in the page with JQuery. Some updates also happen as you scroll the page (if I recall correctly), with JS and PHP working together to render the page.

The most SPA oriented frameworks assume that the server just provides some kind of blank slate HTML and from there on everything is handled on the frontend side.

I think it depends. There is the classical TODO example for an SPA framework where the HTML is a blank page, the server sends a bunch of JSON data, and the JS framework code is responsible for building the whole page.

When Vue was gaining popularity, one of the features that was always explained in videos was how easy it was to get started with Vue, as you didn't need to re-write your application. You could start from a part of your system (e.g. sidebar, just the menu, maybe the user area or the admin area only, etc). Letting you render HTML as it was received from the server side, while delegating certain <div id="user-app"></div> to a Vue app created like (simplifying) new Vue("#user-app"), and another <div id="menu-app"></div> as another app.

The best use case of what I'm trying to explain here, is the adoption of Vue by GitLab. They have several posts since they migrated (in ... 2015 or 16) with follow-ups in their blogs about how it was going, pros and cons, and I looked at their docs and they have a nice tutorial&docs for new devs.

https://about.gitlab.com/blog/2016/10/20/why-we-chose-vue/

It's interesting that in the link above, they have this part,

I remember back when I was using Backbone, I had to really force myself to stay DRY, because it's really a blank canvas. Vue.js does not make large assumptions about much of anything either. It really only assumes that your data will change.

GitLab was written in JQuery, like Skosmos. They could not ditch JQuery as it was a really large application. So, from what I remember reading somewhere (maybe this follow-up post)or watching a talk, maybe?), they started replacing parts of their UI with Vue to learn and get familiar with.

I believe you could do the same in React, maybe with a little more effort. So you don't really need to have 100% in the UI. Now with Vue 3 and its teleport feature you can even have HTML DOM elements outside of the top-level element provided to the Vue element, and Vue will render your Vue component (i.e. no need to have a separate application) there and it will be transparent for you.

We don't want to go there, but would like to mostly generate HTML on the server side (at least the core parts of the page, e.g. concept information), but then perform partial updates.

I think it would be good to give some leeway to the person doing the migration to verify if there are any drawbacks to this design decision when using a JS framework.

It's possible to have HTML being generated on the server side and displayed in the JS app as-is. However, having Twig as a template engine processing text, and sending HTML to the UI is, IMO, not the best approach.

I can imagine a few possible scenarios where a dev could perform to have more control over what/when/how is rendered in the JS code, e.g.

a new version of bootstrap is released and we need/want to upgrade (as happened recently); a UI dev is used to this type of tasks for Bootstrap, Vuetify, Quasar, tailwind, etc. The UI dev knows they just need to look at the pure HTML files, or the HTML templates in JS and update it accordingly to the Bootstrap docs; having parts of the in HTML, parts of that in JQuery/JS/Vue/etc, and parts in Twig can be really frustrating (e.g. someone reported a possible security issue rendering unescaped text - since Vue/React/etc all sanitize it, it should be OK, but now we need to check Twig/moustache/etc too if the server is rendering HTML to be displayed in the UI directly);
working on issues where a dev is not sure if the data displayed is being rendered in JS or PHP;
trying to control how long lists of elements are rendered in the UI (if you have server & client to coordinate, you may have to ask/wait for changes in the PHP code, whereas with everything being rendered being given to the JS to template/render it the UI dev has more control);
some heavy API calls have to be put in a service worker, here the UI dev probably wants simple JSON messages being transmitted between service worker and backend, and then accessed by the JS code to trigger templating/rendering via JS
a feature requires some big changes in the UI, like how things are displayed/rendered; in this case maybe having the data retrieved from server but being templated in JS makes things simpler to refactor the UI JS layer without having to touch the backend;
a new feature requires changes in the template, one part of the template is JS, another part is in PHP, now the change requires coordinating between the two changes;
someone decides to write a new UI for Skosmos because they need it heavily customized in a way that's not generic or worth asking the community to change Skosmos iteself; here if there's a clearly separation between UI and backend, the UI being JS, then this someone can just re-write the JS layer and consume the PHP code;
a feature requires a HTML component to be rendered in a part that is being handled entirely by the JS framework, but also in another part that is being handled by PHP Twig; that'd be pretty hard/impossible to achieve in a clean way, even with Vue teleport. More likely you'd have to have JS code with JQuery or pure DOM manipulation happening somewhere else, I think.

These are mostly imagined scenarios, although for some I use issues I worked on as reference. IMO, Twig could be completely removed (which would reduce dependencies in the PHP side too :medal_sports: ). It doesn't mean to do everything in the JS/SPA space. Render HTML when needed, like the top bar menu.

For the concepts part, there is very little performance penalty in sending JSON as response instead of HTML since the Skosmos UI is very simple. I think it'd be worth to do a quick benchmark comparing PHP+Twig with JQuery, version PHP with React/Vue/Svelte/htmx/etc (maybe just the concepts part, or the hierarchy/sidebar).

If there is ever the need to render things on the server side with Vue or another framework, most frameworks nowadays support server-side rendering too, although that can be more complicated as there are other restrictions when doing so, and you need Node between JSframework and the PHP code, or something like this for for instance to allow non-node ssr.

TL;DR: I think this is a great move in Skosmos and I think it's worth considering design changes in where/how the UI renders and handles templates :+1: in the end it could be confirmed that there's nothing wrong in keeping twig

Cheers Bruno

As an example of what I mean with

a new version of bootstrap is released and we need/want to upgrade (as happened recently); a UI dev is used to this type of tasks for Bootstrap, Vuetify, Quasar, tailwind, etc. The UI dev knows they just need to look at the pure HTML files, or the HTML templates in JS and update it accordingly to the Bootstrap docs; having parts of the in HTML, parts of that in JQuery/JS/Vue/etc, and parts in Twig can be really frustrating (e.g. someone reported a possible security issue rendering unescaped text - since Vue/React/etc all sanitize it, it should be OK, but now we need to check Twig/moustache/etc too if the server is rendering HTML to be displayed in the UI directly);

we have Bootstrap classes in

JavaScript with JQuery https://github.com/NatLibFi/Skosmos/blob/b13b2c4ae38e1114d31551e98f12e8dc
Twig in HTML elements https://github.com/NatLibFi/Skosmos/blob/b13b2c4ae38e1114d31551e98f12e8dc9068b1f0/view/concept-shared.twig#L22
Twig with Twig syntax using variables that contain values from/for Bootstrap elements https://github.com/NatLibFi/Skosmos/blob/ac64a056db52576c7537b49be1b19793af82709e/view/vocab.twig#L9

If instead we had PHP just sending the data in these cases, then only JS code would have to be updated. In Vue/React/etc normally you can combine SCSS with JS, making it easier too to write components in your own CSS style (something I think would be great for Skosmos too - again, the UI is not huge, it would make it independent of upgrades in Bootstrap - but needs to be done with care & thinking on future-proofing, and it's a separate issue).

Likewise, if we had everything in Twig, then only one place will have to be upgraded. But having the responsibility of compiling templates in two places causes issues like this with Bootstrap, but possibly with other libraries & code. This is the one issue I remember well wishing to have everything in a single place :+1:

-Bruno

Thanks a lot for your detailed thoughts @kinow - and the links to the GitLab blog posts, which seem very relevant for our situation! Reading those makes me think that Vue could certainly be a good option for our use.

Regarding server side vs. client side rendering, you make good points, but my opinion is still that we should try to render the "core" content on the server, while auxiliary elements such as navigation (at least the dynamic parts like the sidebar) would better be rendered on the client side - which is already pretty much the case, only it's messier than it should.

In this screenshot I've marked the server-rendered (PHP + Twig) parts with blue and the client-rendered (some kind of JS) parts with red:

This isn't 100% accurate for the current situation (the sidebar and the search widget are currently a mixture of server-side and client-side rendering) but I think we could try to enforce the separation better. And of course we should document it in diagrams like this one.

We don't currently have anything like a separation between frontend and backend developers, as seems to be the case for GitLab (they're obviously much bigger than us as well). It's true that having to master both sides and keep them in sync can be a burden, but I don't see this as a huge issue in the current situation.

NatLibFi / Skosmos