Why difference between core and custom ui nodes?

[bartbutenaers]

Description

Hi @joepavitt,

The old dashboard originally contained a limited set of core ui nodes. A few years later, fortunately there arrived a mechanism to contribute custom ui nodes. That was a great step forward. But the setup of the custom ui nodes was very different from the core nodes.

The new dashboard works very similar. There is a set of core ui nodes, and the setup of the custom ui nodes is different. Or at least for me (as a VueJs noob) they look different.

I was wondering why not all ui nodes are implemented the same way, i.e. like custom ui nodes. The only difference would be that the core nodes would be installed out of the box, in contradiction to the custom nodes which still need to be installed seperately. Although it would be ok for me if I had to install all core nodes seperately via the palette. But that is another discussion...

Because if the core ui nodes would have the same setup as the custom ui nodes, that would have some advantages for me as a ui node developer:

1. The contribution documentation for core nodes and custom nodes would be identical.
2. It would be easy to look at the code of core nodes to learn from it, in order to be able to create my own custom node. Now when I want to get started with this, it is very hard to determine if I can reuse some core stuff or not.
3. A custom node could become easily afterwards a core node, if there would be some need for that. E.g. for maintenance issues or whatever...

Perhaps this technically very difficult, or it has some disadvtages I am not aware of. Don't know. Just thinking out loud...

Thanks for reading! Bart

Have you provided an initial effort estimate for this issue?

I am no FlowFuse team member

[joepavitt]

This is absolutely what I wish to strive for, as a first iteration I've copied the pattern from Dashboard 1.0, where all 3rd party widgets are integrated into the client-side Dashboard via ui-template.

It all essentially boils down to needing to have a better way of completing this line: https://github.com/FlowFuse/node-red-dashboard/blob/89fb0644baab1e257c90c3fe8b663c23684a15b0/ui/src/App.vue#L64

Node-RED: We have a fixed set of config options, all get registered to the RED object, so we have complete freedom on how this is handled.

Client Side: This is where the difficulty lies. The main Vue app maps a component against each of the widget configurations received from the ui-config SocketIO event. Because the client-side code for any of the third-party widgets is not stored locally in the directory for Dashboard 2.0, we have no way of mapping the component at this point, hence the need to have a different structure.

That different structure we have, allows for us to pass in the template content, etc. into ui-template and build a component dynamically, but each of these pieces needs to be separate for us to do so (hence the current requirement for a template/script split).

Not sure how @TotallyInformation has tackled this problem with UI Builder actually?

[joepavitt]

We may have some luck with https://github.com/FranckFreiburger/vue3-sfc-loader but even then, if you have a dependency in your package.json for your third-party node for example, how does the Vue Component load that? It's a standalone package, this loader would get the .vue component, but not be able to npm install any of the dependencies that .vue file would have.

[TotallyInformation]

If I'm understanding this correctly, I think it would be a mistake to try to rely on the loader. Instead, you should require that each Vue component included in a new Dashboard node is built into a pure .js library. Then it becomes simply a matter of loading them - which can be done dynamically.

So you probably need an example D2 module repo that includes the right dev tools and pre-defined (npm) scripts to let people build their components up front. This would also allow for the components to meet some pre-set criterial as well by including some pre-sets such as minifying, adding .map files and even enforcing JavaScript versioning.

The sfc-loader is ok for simple tasks but it can't be fully relied on.

The above approach would even potentially allow custom web components to be included in a D2 module and over time, it might be possible to provide an example web component that was able to leverage communication with Node-RED without Vue being involved at all. And certainly, it would enable the inclusion of multiple Vue components in a single module.

As I say, I hope I've interpreted the discussion correctly?

[joepavitt]

Beautifully interpreted @TotallyInformation

Then it becomes simply a matter of loading them - which can be done dynamically.

How does the Dashboard client get access to these files though? Node-RED would need some way of serving these files up?

[TotallyInformation]

Yes, well that is something that Node-RED is already doing. All you need to do is add an ExpresJS path to enable the serving and something in D2 to add the URL to the Dashboard.

All 😁

The Express side of things isn't that hard since custom nodes already have the ability to reference Node-RED's user-facing ExpressJS server (UIBUILDER does this itself by default though it also has the option to create a separate one if needed). I think you will probably want to do that from somewhere inside the D2 controller though rather than forcing every node contributor to do it. Which might be a little harder but you will need to be tracking the installation of D2 nodes I assume? and maybe you already track when D2 nodes are added to flows?

I think it is this that you need to track, as soon as there is any instance of a previously unloaded component is added to a flow, you will need to add the ExpressJS path - perhaps use a router dedicated to D2 as it is much easier if you manually track additions to a router - removing paths from Express is surprisingly and annoyingly hard (in UIBUILDER, I have a number of routers and I have a dedicated variable that tracks everything that is added/removed).

Having added the route, you can then dynamically add the <script> to the D2 page.

---

From a custom node contributor's perspective, it will all be a lot easier if there is a pre-defined structure that they need to meet. So that each module has a set folder structure and file naming convention.

It goes without saying that the entire load process from D2's perspective will need to be wrapped in a try/catch or whatever so that you will always gracefully kick out any module that hasn't done things correctly.

[joepavitt]

From a custom node contributor's perspective, it will all be a lot easier if there is a pre-defined structure that they need to meet. So that each module has a set folder structure and file naming convention.

So, we have this now, and I've made https://github.com/FlowFuse/node-red-dashboard-example-node as an example of these. @bartbutenaers's very valid point though is that to build custom nodes is quite a different development experience to building core nodes, and I'm trying to work out, how can we get those to overlap, i.e. ideally, a single .vue file, with a fully defined Vue component inside should plug into Dashboard 2.0 nicely, which it currently does not.

[TotallyInformation]

I think the main difficulty from the developers view will be how to test. You will probably want to have something like a gulp watch function - this is what I do in uibuilder for builds since I have quite a few parts that now benefit from having a different structure for development than is needed for deployment.

As I remember it, Vue 3 has a pretty decent set of build processes that should be adaptable. However, its dev server won't be of much use to D2 node developers.

For uibuilder development, I not only have the gulp watch but I run Node-RED under PM2 with its own watch function such that any changes to key files being developed automatically restart Node-RED. It would be possible to adapt something similar to use with gulp (or any similar runner) and systemd - that could be provided as an example script in your example node repo.

[TotallyInformation]

Re the example repo:

• Why is there a CDN reference to D3?
• The nodes folder should contain the .js built version of the .vue component - or it should go in its own folder, /dist for example?
• package.json will need devDependencies for the build tools. And it should have scripts defined for running build and watch processes.

[joepavitt]

Why is there a CDN reference to D3?

Just to show an example of adding a <script> injection into the head of the third-party widget

The nodes folder should contain the .js built version of the .vue component - or it should go in its own folder, /dist for example?

Currently, there is no build step for the client-side .vue, it's passed exactly as is.

[bartbutenaers]

Hi @TotallyInformation, @joepavitt, Thanks for having the active discussion!! Not sure if I will be able to have valuable input, because this stuff is far away from my comfort zone...

A noob question: Why does the node developer need to do the build from Vue to Js? I have always thought that the dashboard loops all widgets that are registered, and then does a build of all of these. Or is that perhaps to slow to do that at runtime?

Without doubt I have a way too simplistic view about this. But my trigger to start with this issue was this: there is already a mechanisme to load custom ui nodes in D2. Why are the core ui nodes not designed and loaded in the same way as the custom ui nodes? Then everything works in the same way? Please illuminate me ;-)

[joepavitt]

because this stuff is far away from my comfort zone...

Once we start venturing into the build/packaging layer of JS, I'm out if my comfort zone too!

Why does the node developer need to do the build from Vue to Js?

They do not need to. I think the proposal here is that, by enforcing this, it would be way to ensure consistency.

I have always thought that the dashboard loops all widgets that are registered, and then does a build of all of these.

You're absolutely right, the gap here though is the Vue app, which does the loop needs to have a map of those widgets, then to the associate Vue component that renders that widget. The difficulty lies in getting third party widgets/components registered with the Dashboard 2.0 Vue App, and providing the App with the relevant dependencies, this is all compiled together in Dashboard 2.0 as part of the npm run build which then provides our distributed package, way before we have any context on what integrations exists, etc.

Why are the core ui nodes not designed and loaded in the same way as the custom ui nodes? Then everything works in the same way? Please illuminate me ;-)

The cleaner, easier to read and far more flexible way of writing the components and widgets would be how we have written them in core.

Ideally, the third party widgets would be the same. It's got to be possible, I just haven't worked out how yet.

I don't want to have core widgets bound by the same limitations that (currently) a third party widget would have. Ideally, third party widgets get the same developer experience as core. We're just still at that annoying early stage of this where I haven't quite found the answer yet.

Whilst I've used Grunt/Gulp/Webpack before, it's always been in teams where I have far more experienced heads to lean upon for assistance. I do not have that luxury here unfortunately.

[bartbutenaers]

Ah ok, so it would be better if the custom ui nodes could be loaded in the same way as the core ui nodes. Completely the other way around as what I was proposing. Makes sense! But this turns my feature request suddenly into a difficult one...

I do not have that luxury here unfortunately.

That is indeed far from easy if you have to figure out everything on your own... Very obvious that you called @TotallyInformation

I assume this cannot be of any help?

[TotallyInformation]

Why does the node developer need to do the build from Vue to Js? I have always thought that the dashboard loops all widgets that are registered, and then does a build of all of these. Or is that perhaps to slow to do that at runtime?

Well, you've probably gone into this more than I have - my knowledge of the workings is entirely theoretical at this point and likely to remain so given how hard it is to remember stuff about my own approach! 🙂

What I was trying to get across was that if you pre-build the component to js, all you need to do to use it is load it via a script tag. If you don't do that, you have to build at runtime which, as you say seems a bit bonkers since you'd be doing that every time. On a small Pi or similar that might be very slow. Especially with a lot of custom components.

Pre-build also gives you more flexibility and future opportunities. For example, using web components. Perhaps not something for now but maybe for the future.

Anyway, pre-built js versions of Vue components are simple to load and remove a lot of the runtime complexity. I think.

Ideally, the third party widgets would be the same. It's got to be possible, I just haven't worked out how yet.

I think I am proposing something that would work for both wouldn't it? Why would you want to build at runtime? pre-build during development - this is the industry standard way anyway.

Whilst I've used Grunt/Gulp/Webpack before, it's always been in teams where I have far more experienced heads to lean upon for assistance. I do not have that luxury here unfortunately.

Honestly, it doesn't matter what runner tool is used, you would only have to set it up once and then put the script in the example repo. And if you use the same approach for core nodes, everyone is following the same best practice. With the added advantage that you could give this to a dev with no Node-RED experience and they could still do it.

And, custom node devs don't need to "know" the runner since it is pre-configured. At worse, they might have a couple of simple things to configure and those could probably be put into the package.json file as variables to keep things even simpler. The package.json would also define the scripts to run for build and watch as well. As in this example from uibuilder:

  "scripts": {
    "preinstall": "node ./bin/uibpreinstalljs",
    "build": "gulp",
    "buildfe": "gulp packfe",
    "watch": "gulp watch",
    "listbin": "ls ./node_modules/bin",
    "docs": "docsify serve ./docs",
    "edit-docs": "%LOCALAPPDATA%/Programs/Typora/Typora.exe ./",
    "buildSidebar": "node ./bin/docsify-auto-sidebar.js"
  },

That is indeed far from easy if you have to figure out everything on your own...

Seriously, any decent VueJS tutorial will already have this defined. A runner to manage the build process is at the core of VueJS/REACT/etc development. And I'm sure between us, we can work it out even if you can't simply lift it from an example.

Very obvious that you called @TotallyInformation

Oh dear, scary if you are calling me to help with VueJS!! 🤣

I assume this cannot be of any help?

I have to admit that I never understood his approach, it seemed excessively complex to my small mind though I'm sure there must have been good reasons.

---

I know that setting up a build process seems complex but it should only need to be done once and it works everywhere. A standard pattern. And you end up with VueJS components that can be loaded at any time even by dynamically loading direct to the page. Just like any other JavaScript library.

Here, for example is the full code from the uibuilder client library to dynamically insert a script library:

    /** Attach a new remote script to the end of HEAD synchronously
     * @param {string} url The url to be used in the script src attribute
     */
    loadScriptSrc(url) {
        const newScript = this.document.createElement('script')
        newScript.src = url
        newScript.async = false
        this.document.head.appendChild(newScript)
    }

And that is vanilla DOM, VueJS probably has something to do this as well.

[joepavitt]

Seriously, any decent VueJS tutorial will already have this defined. A runner to manage the build process is at the core of VueJS/REACT/etc development

Worth noting, we do already have full build process for core, the complexity here is the step of third party widgets providing the relevant details to core, such that the third party .vue files could be included.

I'm on my phone at the moment, but will take a look at the FlexDash docs early next week. I'll also be in person with @knolleary and @Steve-Mcl then too, and would like to think we can solve this problem then

[joepavitt]

Oh dear, scary if you are calling me to help with VueJS!! 

I'm good with the VueJS part, it's the build/packaging part I'm struggling with. Particularly, what steps are taken, and most importantly when.

[TotallyInformation]

I wonder if you have any documentation or a diagram of how this works? Might be easier for everyone to be able to see where things could be improved or better integrated or whatever is needed? I can't help feeling that I'm missing something about the core of D2.

[bartbutenaers]

Hi @TotallyInformation, There is an event diagram but not sure if that helps you understand e.g. the build process.

[TotallyInformation]

Oh my! Documentation!! Nice. 😊

[TotallyInformation]

OK, I think this part is what I didn't understand: https://dashboard.flowfuse.com/contributing/widgets/third-party.html#defining-html-to-render

Feels like an odd way of doing things to be honest. To read in the .vue file as text and then process it. Kind of feels like it is ignoring the way that Vue would normally work. Though I'll admit that my Vue knowledge is limited.

I guess that this method allows a custom vue component to avoid having some Dashboard boilerplate? But then the result is this issue.

Again though, I might be way off mark.

[joepavitt]

Feels like an odd way of doing things to be honest. To read in the .vue file as text and then process it. Kind of feels like it is ignoring the way that Vue would normally work.

You're absolutely right, as a starting point, I just built from how Dashboard 1.0 does this, utilising the ui-template node, and sending the template as text/string.

Was the easiest first iteration, but definitely want to take up the feedback here and utilise a smoother build pipeline.

[joepavitt]

Just want to update that we had an in-person summit for FlowFuse this week, and this was actively started.

Work needs to still be done to get third party dependencies (e.g. ES6 modules) working, but we've made progress (work currently on external-widgets branch)

Just wanted to give you the heads up @bartbutenaers

As part of it, we will also be completely re-building the Dashboard build pipeline, so it's no small feat

[joepavitt]

This being address as part of #351 where we have ovrhauled the Dashbaord 2.0 (and external widget) build pipelines and integration workflow.

[bartbutenaers]

@joepavitt Thanks for the feedback! The hard work of you and the team is really appreciated!!

[joepavitt]

Closed by #351