flaviocosta-net commented 6 years ago

This issue is a transfer of the respective ESH forum topic to GitHub.

The idea is to propose a new Sitemap concept that would help overcome various problems seen on the current sitemap implementation:

It is represented on Xtext DSL, which is an unnecessary dependency for many scenarios
It is not editable via REST
Components cannot be easily reused or extended for different UIs

New or Renewed Concepts

Atom: the smallest composition unit of a sitemap. An atom cannot contain any other components, although it does have certain attributes that describe how that atom can be rendered. Atoms are usually grouped to form more complex components that users can interact with (e.g. a widget containing an icon, a label and a switch control, as typically seen on today's sitemaps).
Container: an ordered set of components that provides hierarchical structure to the sitemap. Containers may be visually rendered (e.g. as a frame on the current page, or as a subpage) or be completely seamless. The latter case is useful as a container may be used to specify attributes that are propagated to its child components, without displaying any visual cue to the user.
Component: an Atom or Container. All components may have at least three attributes: type, data and style. Type is a mandatory attribute that defines the semantic of the component for the UI. The other two attributes are optional, but most components should have at least one or the other. Containers have a mandatory attribute layout that is used by the UI to decide how the components inside the container should be laid out.
Sitemap: the root container for all components arranged to provide certain system views. As the sitemap is a container, it contains the mandatory "type" property of every component and the properties "data" and "layout" that can be specified on any container.
Current page: all components being currently displayed to a user on the screen, or that can be displayed by horizontal or vertical scrolling the current view. The current page is typically changed by clicking on a visible and enabled atom associated with another page. If a current page is not specified, the UI should display the contents visible on the topmost container.

Sample Sitemap

I'd like to start with the sample sitemap definition written in the current DSL that we find on the openHAB documentation:

sitemap demo label="My home automation" {
    Frame label="Date" {
        Text item=Date
    }
    Frame label="Demo" {
        Switch item=Lights icon="light"
        Text item=LR_Temperature label="Livingroom [%.1f °C]"
        Group item=Heating
        Text item=LR_Multimedia_Summary label="Multimedia [%s]" icon="video" {
            Selection item=LR_TV_Channel mappings=[0="off", 1="DasErste", 2="BBC One", 3="Cartoon Network"]
            Slider item=LR_TV_Volume
        }
    }
}

This is the expected visual rendering for this sitemap:

Demo sitemap

In this proposal, this same sitemap could be specified with the following JSON:

{
   smarthome: {
      id: "demo", label: "My home automation",
      components: {
         frame: {
            label: "Date",
            components: {
               text: {item: "Date"}
            }
         },
         frame: {
            label: "Demo",
            components: {
               switch: {item: "Lights", icon: "light"},
               text: {item: "LR_Temperature", label: "Livingroom", state: "%.1f °C"},
               group: {item: "Heating"},
               text: {item: "LR_Multimedia_Summary", label: "Multimedia", state: "%s", icon: "video",
                  components: {
                     selection: {item: "LR_TV_Channel", mappings: {0: "off", 1: "DasErste", 2: "BBC One", 3: "Cartoon Network"}},
                     slider: {item: "LR_TV_Volume"}
                  }
               }
            }
         }
      }   
   }
}

Except for the added "components" and "state" elements, this is almost a 1-to-1 mapping of the DSL to JSON. This approach can make it easier to understand the new concepts and how they can support backwards compatibility with the existing sitemap definition files. The system can easily parse a source file in the sitemap DSL and convert it to the JSON format, using the resulting structure for all further processing while still giving the user the convenience of the shorter DSL syntax.

The idea is to put most of the processing burden on the server side, so content and formatting are added on top of the input JSON (sitemap definition model) to generate a rendering model that is sent to the UI with the contents of the current page. The rendering model for the initial page of the above sitemap could look like this:

{
   sitemap: {
      type: "smarthome", data: {id: "demo", label: "My home automation", lang: "en"}, layout: "list",
      components: {
         container: {
            type: "frame", data: "Date", style: "font-weight:bold;font-size:large", layout: "listcontrol",
            components: {
               atoms: {
                  {type: "icon", data: {id: "classic:text", value: "20160801000000"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Today"}
                  {type: "text", data: "Monday, 01.Aug.2016", style: "font-weight:bold"}
               }
            }
         },
         container: {
            type: "frame", data: "Demo", style: "font-weight:bold;font-size:large", layout: "listcontrol",
            components: {
               atoms: {
                  {type: "icon", data: {id: "classic:light", value: "OFF"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Lights"},
                  {type: "switch", data: "OFF"}
               },
               atoms: {
                  {type: "icon", data: {id: "classic:temperature", value: "21.3"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Livingroom"},
                  {type: "text", data: "21.3 °C", style: "font-weight:bold"}
               },
               atoms: {
                  {type: "icon", data: {id: "classic:group"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Heating"},
                  {type: "group", data: "item:Heating"}
               },
               atoms: {
                  {type: "icon", data: {id: "classic:video"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Multimedia"},
                  {type: "group", data: "components:4"}
               }
            }
         }
      }   
   }
}

Sitemap rendering

In order to convert a definition model into the respective rendering model, a SitemapRenderer service needs the following inputs:

A well-formed sitemap definition model, usually provided as a text file created manually or via some visual editor.
A sitemap style definition, specified as CSS.
A sitemap prototype definition, specified as JSON with JSONPath references.

The style definition specifies how containers and atoms will be formatted. The approach used here is similar to the one adopted by JavaFX: while standard CSS syntax is used, it does not support some CSS layout properties such as float, position or overflow. The style actually provides a theme to format the sitemap components and it may be customized by the user. The disposition of the elements is determined by the "layout" attribute seen on the rendering model. This is different from the current sitemap solution, where colors, size, margins and fonts are decided by the client GUI. By following a style definition imposed on the server side, now the sitemap will have a more consistent look on the Basic UI, or on various mobile clients.

The prototype definition determines how the elements in the definition model will be restructured to form the component types in the rendering model. The prototypes can also be customized by the user, by editing the existing prototypes or creating new ones. This flexible abstraction layer allows component definitions reuse across the sitemap.

The following style file should generate output with formatting similar to the sitemap screenshot seen above:

/* Default theme */

container.frame {
   font-weight: bold; font-size: large;
}

atom {
   font-weight: initial; font-size: initial;
}

atom.text {
   font-weight: bold;
}

atom.icon {
   width: 16px;
   height: 16px;
}

The CSS class name (for instance, .frame seen on the first definition above) needs to match the component type. The SitemapRenderer on the server side adds these style definitions to each component where they apply, so the client doesn't need to bother with stylesheet inheritance - each component specifies on its "style" attribute any formatting options to be applied. If there is no style for a certain component, then the client UI can decide its default formatting. Usage of standard CSS syntax and semantics make it easier to implement web browser-based UIs, which already include built-in support for CSS.

Here is an example of a prototype definition file that would work for the sitemap concepts used in the sample above:

smarthome: {
   sitemap: {
      type: "$",
      data: {id: "$.id", label: "$.label", lang: "$.lang,system:language"},
      layout: "list",
      components: { "$.components..*" }
   }
}

frame: {
   container: {
      type: "$",
      data: "$.label",
      layout: "listcontrol",
      components: { "$.components..*" }
   }
}

seamless: {
   container: {
      type: "$",
      layout: "listcontrol",
      components: { "$.components..*" }
   }
}

implicit: {
   container: {
      type: "$",
      layout: "listcontrol",
      components: { "$.components..*" }
   }
}

text: {
   atoms: {
      {type: "icon", data: {id: "$.icon,item:icon,classic:$", value: "item:state"}},
      {type: "label", data: "$.label,item:label"},
      {
            metadata: {
                  if: "$.[?(@.components)]",
                  then: {type: "group", data: "components:seqno"},
                  else: {type: "$", data: "item:state"}
            }
      }
   }
}

slider: {
   atoms: {
      {type: "icon", data: {id: "$.icon,item:icon,classic:$", value: "item:state"}},
      {type: "label", data: "$.label,item:label"},
      {type: "$", data: "item:state"}
   }
}

switch: {
   atoms: {
      {type: "icon", data: {id: "$.icon,item:icon,classic:$", value: "item:state"}},
      {type: "label", data: "$.label,item:label"},
      {type: "$", data: "item:state"}
   }
}

group: {
   atoms: {
      {type: "icon", data: {id: "$.icon,item:icon,classic:$", value: "item:state"}},
      {type: "label", data: "$.label,item:label"},
      {type: "$", data: "components:seqno,item:id"}
   }
}

selection: {
   atoms: {
      {type: "icon", data: {id: "$.icon,item:icon,classic:$", value: "item:state"}},
      {type: "label", data: "$.label,item:label"},
      {type: "$", data: "$.mappings"}
   }
}

Let's analyze each line of the "sitemap" prototype to understand how it works:

smarthome: {

When the "smarthome" element is found in the sitemap definition model, the renderer expands this prototype.

   sitemap: {

The name of the resulting element will be "sitemap".

      type: "$",

The type property will be set to "smarthome". The "$" character is a JSONPath reference to the original element in the definition model, which happens to be "smarthome". We are not using an explicit reference to a JSONPATH transformation, but we may decide that it's better to have that either with a "FUNCTION()" syntax or as a "namespace:".

      data: {id: "$.id", label: "$.label", lang: "$.lang,system:language"},

The data property will be set to a JSON, containing an id and a label copied from the original definition element, also using JSONPath syntax. The lang property is passed to the client to indicate the language of the components in the container. It can be used, for instance, to change text alignment or even rearrange the atoms to better suit a language written from right-to-left, such as Hebrew or Arabic. Based on the specification in the prototype, the renderer will first try to read a lang attribute from the source element in the definition model. If absent, then it retrieves the language from the system settings (i.e. using the system: namespace).

      layout: "list",

The layout property does not come form the definition model, it is fixed in this prototype. The value "list" is a reference to the Material Design concept of a list, so this is informing the UI that it should arrange the components of the container as a list:

Material Design list layout

      components: { "$.components..*" }

The recursive descent operator (..) followed by the wildcard (*) tells the renderer to recursively retrieve all members contained under components (and expand the relevant prototypes as needed).

There are also some atoms under the "text" prototype using a special syntax that is worth explaining here:

      {type: "icon", data: {id: "$.icon,item:icon,classic:$", value: "item:state"}},

The icon data will have an id and a value. The id will be the icon attribute in the definition model, or if it's missing then it's the icon attribute on the associated item (via the item: namespace), else it will be the icon "text" from the classic iconset (via the classic: namespace). The value attribute is always the state of the associated item.

         metadata: {
            if: "$.[?(@.components)]",
            then: {type: "group", data: "components:seqno"},
            else: {type: "$", data: "item:state"}
         }

Once the prototype expander meets an element named "metadata", it knows that the elements immediately below it should not simply be sent to the output, put interpreted in a certain way to decide how the prototype is expanded. On the example above, if there is a member "components" on the definition model, the third (rightmost) atom will be rendered as a "group" atom; if absent, we render it as a regular "text" atom.

We can notice that the "Multimedia" element on our example sitemap does contain components, so it was rendered as a group with data "components:4" (meaning the fourth occurrence of "components" in the definition model). When the user clicks on the arrow on this component, the client makes a REST API call to the server which returns the sitemap view for that specific page, to display the contained components:

{
   sitemap: {
      type: "smarthome", data: {id: "demo", label: "My home automation", lang: "en"}, layout: "list",
      components: {
         container: {
            type: "implicit", layout: "listcontrol",
            components: {
               atoms: {
                  {type: "icon", data: {id: "classic:screen", value: "0"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Channel"},
                  {type: "selection", data: {0: "off", 1: "DasErste", 2: "BBC One", 3: "Cartoon Network"}}
               },
               atoms: {
                  {type: "icon", data: {id: "classic:soundvolume", value: "15"}, style: "width:16px;height:16px"},
                  {type: "label", data: "TV Volume"},
                  {type: "slider", data: "15"}
               }
            }
         }
      }
   }
}

On a very high level, the main work performed by the SitemapRenderer is:

Load the definition model and maintain a Java representation of it.
Load the default style and prototypes that match the sitemap type.
Convert the definition model into the rendering model by expanding the prototypes until the elements are resolved to the final component types. In this process, it should resolve any references based on the specified namespaces with a separate service that could be called something like NamespaceReferenceResolver.
Augment the resulting components with the applicable CSS styles.

The namespaces mentioned in this example are:

smarthome: any types that have no explicit namespace use the sitemap type as the default namespace (so type: "frame" internally resolves as type: "smarthome:frame").
classic: icon in the classic iconset - it is actually the only namespace actually sent to the client, since all other references are resolved on the server side. The resolver for this namespace knows that it should not be replaced with a value on the server side, but sent as-is to the client. Then the client can make a separate HTTP request to the server to obtain the image represented by the icon.
system: information about the system (e.g. default language or position).
item: attributes from the item associated with the component.
components: references to components in the sitemap. Components are a mandatory member of containers.

Now let's demonstrate "dynamic sitemap" features that match the existing capabilities of the sitemap:

{
   smarthome: {
      id: "dynamic", label: "My dynamic sitemap",
      components: {
         selection: {item: "LR_TV_Channel", mappings: {0: "off", 1: "DasErste", 2: "BBC One", 3: "Cartoon Network"}},
         slider: {item: "LR_TV_Volume",
            visibility: {
               or: {
                  {value: "item:LR_TV_Power.state", eq: "ON"}
                  {value: "item:LR_TV_Channel.state", noteq: "0"}
               }
            }
            labelcolor: {
               {gteq: "50", color: "red"}
               {gteq: "15", color: "orange"}
            }
         }
      }
}

Below we can see the resulting rendering model - you will notice that the visibility and labelcolor definitions are implemented by the renderer as specific CSS styles (e.g. "display:none;color:orange"). Since style processing cannot be implemented via prototype expansion, we would need the Java implementation for the NamespaceReferenceResolver to apply the required logic once it sees smarthome:visibility and visibility:labelcolor elements.

{
   sitemap: {
      type: "smarthome", data: {id: "demo", label: "My home automation", lang: "en"}, layout: "list",
      components: {
         container: {
            type: "implicit", layout: "listcontrol",
            components: {
               atoms: {
                  {type: "icon", data: {id: "classic:screen", value: "0"}, style: "width:16px;height:16px"},
                  {type: "label", data: "Channel"},
                  {type: "selection", data: {0: "off", 1: "DasErste", 2: "BBC One", 3: "Cartoon Network"}}
               },
               atoms: {
                  {type: "icon", data: {id: "classic:soundvolume", value: "15"}, style: "display:none;width:16px;height:16px"},
                  {type: "label", data: "TV Volume", style: "display:none;color:orange"},
                  {type: "slider", data: "15", style: "display:none"}
               }
            }
         }
      }
   }
}

Extension and reuse

One virtue of the proposed solution is the fact that definitions can be added or changed declaratively.

Styles can be adjusted by editing the style file. Alternative style files can also be provided, giving the user the option to select in the UI which theme to use. The API endpoint that lists the available sitemaps can also return the available styles or themes available for each sitemap, for instance:

smarthome.style: Default theme.
smarthome-dark.style: Theme with dark background and white text.
smarthome-tablet.style: Theme with bigger icons and text, optimized for big tablet screens.

New prototypes can be created, and they may also be based on existing prototypes. For instance, I may want to have all my smart bulbs to have the same icon and with intensity changed by predefined mapping options:

lightselection: {
   selection: {
      item: "$.item", icon: "light", mappings: {0: "Off", 50: "Medium", 100: "Bright"}}
}

When the renderer identifies that "lightselection" is mapped to another prototype "selection", it takes the intermediate structure and apply the second prototype to it, which will then generate the atoms and stop the processing of this element. Now we can use our new prototype on the sitemap definition model:

   lightselection: {item: "GF_Hall"},
   lightselection: {item: "GF_Dining_Room"}

The rendering model will have fully expanded containers and/or atoms:

   components: {
      atoms: {
         {type: "icon", data: {id: "classic:light", value: "0"}, style: "width:16px;height:16px"},
         {type: "label", data: "Hall"},
         {type: "selection", data: {0: "Off", 50: "Medium", 100: "Bright"}}
      },
      atoms: {
         {type: "icon", data: {id: "classic:light", value: "50"}, style: "width:16px;height:16px"},
         {type: "label", data: "Dining Room"},
         {type: "selection", data: {0: "Off", 50: "Medium", 100: "Bright"}}
      }
   }

Another possible customization via the prototypes would be to convert the standard sitemap layout into a dashboard-like interface: instead of "list" layout on the sitemap and "listcontrol" for the inner containers, the layouts "cardcollection" and "card" could be used respectively to display widgets in a very different way:

Card collection layout

The prototype expansion functionality is actually generic enough that it could even be used for other purposes not in the sitemap scope, such as allowing the creation of reusable Item definitions.

Other possible new features that are not supported by the current sitemap could be:

Defining component visibility based on more than one condition using "and" (the current sitemap only supports "or").
Creating a vertical navigation sidebar consisting only on Icon atoms, or keeping a component always visible on top regardless of the page the user is in (e.g. an intrusion alert generated by a home security system).
Decoupling the icon from the item state - the presented prototypes links dynamic icons to simulate the current sitemap implementation, but they can be changed to determine the icon to be displayed based on the state of an item not displayed in the sitemap.
Disabling a component instead of hiding it.
Adding a "Webcam pictures" to a sitemap which, once clicked, would open a page with "gridlist" layout to display all photos captured by a webcam in the last 24 hours.
Associate an URL with a component, so once the user clicks it a web page is displayed on a new window.
Create custom component types for specialized UIs (e.g. a date picker atom generated declaratively by a new prototype, it's the client that would have the burden to determine how this component should be displayed).

This proposal is still on a very abstract level, and some adjustments would likely be needed once certain issues became more obvious during an implementation attempt. Topics such as SSE and notifications are not addressed here yet. There are also some interesting features that I've left aside here, such as the possibility of using an indoor positioning system (e.g. FIND) to automatically set the current page that corresponds to the room the user currently is in.

Implementation

The core steps/modules to accomplish the proposal above could be:

Sitemap definition model parser: reads the definition model in JSON, and creates Java instances for Containers and Atom.
JSON Prototype parser and expander: reads prototypes and applies them to the definition model.
Sitemap CSS parser: reads the style file applying relevant rules such as CSS selectors.
Namespace reference resolver: resolves references on the server side.
Sitemap renderer: produces the rendering model using the 4 components above.
REST API/SSE and sample client implementation

There is also some additional work required for a complete solution:

Sitemap DSL to JSON converter: for DSL support for backwards compatibility on the server side.
Rendering model converter for legacy clients: for backwards compatibility on the client UI side.
VS Code support for new sitemap: syntax highlighting and autocomplete for definition model, and maybe for CSS and prototypes.
Sitemap editing REST API: create/update/remove atoms and containers.

kaikreuzer commented 6 years ago

Many thanks for your elaborate concept, @flaviocosta-net! Just ftr, I have put this issue on our backlog, so it is something that will be followed up on soon!

sjsf commented 6 years ago

Wow, that's a pretty thorough concept! I very much like this approach. As you said, it's still on an abstract level, but on that level it fully makes sense from my point of view. And judging from the likes it got, I think it is safe to say I'm not the only one 😉

Especially the separation between the definition model and the rendering model is something which will be of great benefit and although we are missing it today, there are similar aspect buried deep in the o.e.sh.ui bundle and the corresponding ui implementation bundles.

I only see a few minor details (like e.g. giving enough styling guidance while leaving enough room for UIs to stay responsive, icon set handling with "classic" being only one example) right now which might be worth discussing, but as I also expect that some more will surface as soon as it goes into the implementation phase, I don't think it makes sense to spend any time with them right now.

I also spend some time thinking about whether it's good to create "our own" template engine/language, but so far I tend to agree that it is smarter to simply to it here - it's only a very small and limited but at the same time rather specialized subset of what all the available templating frameworks deliver. So with the goal of keeping the overall footprint of ESH as small as possible, I currently agree that it is best like you proposed it.

All in all I'm convinced that it's worth a try to implement the basics of this proposal. How should we proceed with this? You outlined some pretty clear steps, of course. Ideally I would like to keep it as simple as possible and rather have multiple iterations on it in order to minimize the overall risks. Limiting the focus on the feature-set of today's sitemaps might be a feasible approach in order to reduce the scope in the first iteration, what do you think?

Would you be willing to also start with the implementation? That clearly would be awesome, of course! My apologies, if this sounded like a stupid question - as you don't have a contribution record here I simply can't tell whether you are into coding at all, nor if you are willing (and allowed?) to dedicate the time on it 😄 Either way, I'm happy you spent the time on this concept, it by far supersedes the initial attempt from the ESH Forum. I'm really looking forward to seeing the legacy sitemaps to getting such a great overhaul!

flaviocosta-net commented 6 years ago

Hello @sjka, thanks for the feedback! As I was writing down these ideas, there were many uncertain issues that came to mind: whether icon set handling was being properly considered, if/how we should support sitemap elements with "Default" type or deprecate that, what is the best way to handle languages/i18n, which changes are needed for notifications/SSE, but I assumed that during a first implementation attempt these and other issues would become clearer, and then decisions would have to be made.

For the template engine used on prototype expansion, this is something that could also be done completely on Java: we could provide an abstract Java class with a default SmartHome implementation/subclass that basically performs the expansion logic outlined above. Developers can later create their own subclasses to customize the default behavior, and one possible subclass would read the expansion rules from a json file, rather than having them hard-coded on Java, so this declarative expansion customization can be added a bit later if the value becomes more evident.

I agree that we should start focused on building the feature set that is equivalent to today's sitemap, and once this is mostly working then we expand to new improvements (the prototype expansion layer would probably be discussed at that moment).

I can start working on the implementation (I actually have a binding I've developed running well on my server, soon I should be creating a fork to have it eventually contributed to ESH). Anyway, there is definitely a lot of implementation work to be done for this sitemap concept, and it is essentially composed of two main parts:

Client-side: UI implementation will be so different that a significant portion of the code on existing clients would have to be rewritten, so it would be good to start with a brand new implementation to support freely testing the new concepts and refactoring it as much as needed in the process. I first considered using the native Windows client for that purpose as it is not fully functional yet. However, since it is not cross-platform and I have no experience with C#, my idea is to create a new JavaFX, so it will work on Windows, Linux and Mac, provide a first finished desktop client for openHAB and serve as a reference implementation without disturbing the existing clients for now.
Server-side: the challenge here is to reuse the existing code as much as possible (do not reinvent the wheel), while implementing any required changes without impacting the existing sitemap support until the new code is 100% ready and mature. While I can dig into the current implementation and figure out how it works, I have no prior knowledge of "similar aspects buried deep in the o.e.sh.ui bundle" and I may easily miss something. I think someone else (you? 😃) should start it, or propose a draft architecture, or give me close mentoring/tutoring at least on the initial phase to define a solid foundation upon which the development will happen.

What do you think?

sjsf commented 6 years ago

Sure, that makes sense. I also don't have perfect answers for all those detail questions yet. Just some initial thoughts, but overall I share your opinion that it might be easiest to have a look at them individually while working on a PR on code:

"Default" element: I personally tend to drop it and only keep in in the old sitemap DSL for compatibility reasons. During import it can be mapped to the corresponding concrete element.
I18n: For sure makes sense for the templates, ideally using the existing TranslationProvider. Maybe that's also something we could postpone to a second iteration?
Changes in eventing/SSE wrt to sitemap related events: I don't expect that the new implementation will leave the SitemapSubscriptionService untouched, therefore also the PageChangeListener might not be usable anymore as is. As you already pointed out: The existing UIs need to be adapted anyway, therefore don't bother too much about them and let's just see what the concrete requirements will be with the new design.

Using a completely new UI for show-casing the client-side implementation sounds like a good idea to me. It will have to be "close" to you though, because we might require to change stuff incompatibly in the early stages. Therefore I wouldn't let other "external" UIs adapt to it in the beginning. The next ones indeed should be the ones in ESH, i.e. the Basic UI (and maybe even the Classic UI?) because changes there are easiest to synchronize.

With the stuff "buried deep in the o.e.sh.ui bundle" I mainly meant ItemUIRegistryImpl.java. It's been growing a lot over time and has proven to be quite prone to regressions and collateral damages. So I wouldn't be too sad if we will have to rewrite major parts of it 😉 I'm happy to help with this. Also I'd think that @resetnow and @lolodomo might be happy to join in at some point.

flaviocosta-net commented 6 years ago

Sounds good, I will start putting some work on the client side and once I have something that would allow some initial testing then I will see how I can help on the server-side, thanks!

flaviocosta-net commented 6 years ago

To give an update on this, I finally have some working code! As expected, it was a very helpful exercise to refine some ideas and get closer to actual implementation.

Here is the rendering model being used as an input:

{
    "type": "smarthome", "data": {"id": "demo", "label": "My home automation", "lang": "en"}, "layout": "list",
    "components": [
        {
            "type": "frame", "data": "Date", "style": "font-weight:bold;font-size:large", "layout": "listcontrol",
            "components": [
                {
                    "type" : "widget", "components": [
                        {"type": "icon", "data": "icon:text#20160801000000", "style": "width:16px;height:16px"},
                        {"type": "label", "data": "Today"},
                        {"type": "text", "data": "Monday, 01.Aug.2016", "style": "font-weight:bold"}
                    ]
                }
            ]
        },
        {
            "type": "frame", "data": "Demo", "style": "font-weight:bold;font-size:large", "layout": "listcontrol",
            "components": [
                {
                    "type" : "widget", "components": [
                        {"type": "icon", "data": "icon:light#ON", "style": "width:16px;height:16px"},
                        {"type": "label", "data": "Lights"},
                        {"type": "switch", "data": {"selected": "ON"}}
                    ]
                },
                {
                    "type" : "widget", "components": [
                        {"type": "icon", "data": "icon:temperature#21.3", "style": "width:16px;height:16px"},
                        {"type": "label", "data": "Livingroom"},
                        {"type": "text", "data": "21.3 °C", "style": "font-weight:bold"}
                    ]
                },
                {
                    "type" : "widget", "components": [
                        {"type": "icon", "data": "icon:group", "style": "width:16px;height:16px"},
                        {"type": "label", "data": "Heating"},
                        {"type": "group", "data": "item:Heating"}
                    ]
                },
                {
                    "type" : "widget", "components": [
                        {"type": "icon", "data": "icon:video", "style": "width:16px;height:16px"},
                        {"type": "label", "data": "Multimedia"},
                        {"type": "group", "data": "components-4/"}
                    ]
                }
            ]
        }
    ]
}

It produces the following UI:

demo

When you click on the Multimedia group, it loads the following rendering model:

{
    "type": "seamless", "data": {"label": "Multimedia"}, "layout": "listcontrol",
    "components": [
        {
            "type" : "widget", "components": [
                {"type": "icon", "data": "icon:screen#0", "style": "width:16px;height:16px"},
                {"type": "label", "data": "Channel"},
                {"type": "selection", "data": {"mappings": {"0": "off", "1": "DasErste", "2": "BBC One", "3": "Cartoon Network"}, "selected": "1"}}
            ]
        },{
            "type" : "widget", "components": [
                {"type": "icon", "data": "icon:soundvolume#15", "style": "width:16px;height:16px"},
                {"type": "label", "data": "TV Volume"},
                {"type": "slider", "data": "15"}
            ]
        }
    ]
}

This is how it looks like in the application:

multimedia

I will just give a final review to the code, especially making sure Javadocs are accurate and help understand the concepts and implementation. I should have it ready to be put somewhere on GitHub soon.

Once this is done, I will start having a look on the server-side code, since the client application is just simulating the sitemap with local files and it will need to start talking to REST APIs in order to validate and refine the concepts.

sjsf commented 6 years ago

Cool, thanks for the update! This indeed looks like a really helpful exercise to validate the requirements towards the rendering model.

I'm looking forward to seeing the next steps! I'd reckon that will be the definition model and application of the prototypes/templates on the server-side, right?

flaviocosta-net commented 6 years ago

Here is the link to the repository: https://github.com/flaviocosta-net/openhab-javafx

I am not sure it is organized the best possible way, as it's the first time I push code on GitHub, so please let me know if there is anything to improve on that. For the code itself, it's obviously still very basic and there should be a lot more progress when integration with the REST API is started. It mostly uses plain Java with generics, there are no extra dependencies such as EMF that we see on the server-side model. Comments on the source code itself include more details about the implementation, ideas and pending items.

For the next steps, we now need to load the sitemap (DSL?), process it and deliver the rendering model through the API.

My understanding is that http://hostname:port/rest/sitemaps should be the entry point for the API, but it should not list new sitemaps for existing (legacy) clients, and new clients should have the option to ignore legacy sitemaps. There are multiple ways of doing this:

Use the Accept header for API versioning, with a custom MIME type such as Accept: application/vnd.eclipse.esh+json;version=2, and serve the new content only to compliant clients. Legacy clients would be served only legacy sitemaps.
Still have a custom MIME type, but always serve the latest version by default. Existing clients should be informed in advance to start using Accept: application/vnd.eclipse.esh+json;version=1 if they don't want to see the new sitemaps (for now).
Add a version attribute to SitemapDTO and let clients do whatever they want with the information.
Adding versioning information as an optional parameter on the endpoint, so clients could call something like http://hostname:port/rest/sitemaps;version=X and then we would apply one of the first two approaches listed here.
Adding the new sitemaps with a different structure in the API response, so legacy clients would just ignore the information and new clients would look specifically for it.

What would be the recommended solution for that?

Also, on the code that reads the sitemap definition (SitemapProviderImpl?), how will we add support for the new format? Will we eventually augment the DSL to support the new sitemap functionality? Can we create a demo.sitemap file including the definition model as a JSON file instead of DSL, and ModelRepositoryImpl will have to detect the file format by its contents in order to parse it properly as I assume we don't want sitemap files to have different extension depending on its actual format?

flaviocosta-net commented 6 years ago

By the way, how can I use ESH classes on a client application? I would like to use SitemapDTO to deserialize the API response, but I can't find an example on how to have the required dependency on a plain Maven/non-Tycho project.

This is the pom.xml I am playing with, it doesn't seem to help resolve any ESH dependencies:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>org.openhab.ui</groupId>
    <artifactId>javafx</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>openHAB JavaFX UI</name>
    <description>Sitemap client desktop implementation based on JavaFX</description>

    <build>
        <sourceDirectory>src/main/java</sourceDirectory>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <modules>
        <module>bundles</module>
    </modules>

    <distributionManagement>
        <repository>
            <id>repo.eclipse.org</id>
            <name>Eclipse SmartHome Repository - Releases</name>
            <url>https://repo.eclipse.org/content/repositories/smarthome-releases/</url>
        </repository>
        <snapshotRepository>
            <id>repo.eclipse.org</id>
            <name>Eclipse SmartHome Repository - Snapshots</name>
            <url>https://repo.eclipse.org/content/repositories/smarthome-snapshots/</url>
        </snapshotRepository>
    </distributionManagement>

    <dependencies>
        <!-- Material Design -->
        <dependency>
            <groupId>com.jfoenix</groupId>
            <artifactId>jfoenix</artifactId>
            <version>8.0.3</version>
        </dependency>
        <!-- REST API -->
        <dependency>
            <groupId>javax.ws.rs</groupId>
            <artifactId>javax.ws.rs-api</artifactId>
            <version>2.1</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-client</artifactId>
            <version>2.27</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.inject</groupId>
            <artifactId>jersey-hk2</artifactId>
            <version>2.27</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
            <version>2.27</version>
        </dependency>
        <!-- JSON deserialization (ESH-recommended library) -->
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.2</version>
        </dependency>
        <!-- Eclipse -->
        <dependency>
            <groupId>org.eclipse.jdt</groupId>
            <artifactId>org.eclipse.jdt.annotation</artifactId>
            <version>2.1.0</version>
        </dependency>
    </dependencies>
</project>

kaikreuzer commented 6 years ago

how can I use ESH classes on a client application?

Unfortunately, not that easy, see https://github.com/eclipse/smarthome/issues/3126. What you would currently roughly need is this.

flaviocosta-net commented 6 years ago

@kaikreuzer, thanks for the reference to smarthome-packaging-sample, I adapted my POM and it works like a charm! Indeed .core and .io.rest.core load a lot of unnecessary packages, but it serves well for testing purposes.

The good news is that this new sitemap implementation will probably help reduce some of this coupling, as the information on the rendering model (i.e. what the client sees) is all expressed in terms of Components (Containers and Atoms), there are no direct references to extraneous entities such as Items (these should be fully resolved on the server side). In fact, I must start working on server code as some of the code I put on the client actually belongs on the server - for instance, icon:soundvolume#15 should be syntactic sugar on the definition model, the rendering model should see it replaced by /icon/soundvolume?state=15&format=png&iconset=classic.

After considering these points and reading the linked discussions, I think a good implementation approach would be:

Support for the existing DSL must be implemented since the beginning. It is needed for backwards compatibility with thousands of sitemaps already created everywhere. At a later moment, the DSL will have to be extended to add support for the new sitemap features.
The definition model will be stored in the JSON DB, it can either be converted from a file.sitemap in DSL or generated by an external tool (e.g. openHAB Home Builder). The Java model will also be maintained on the memory cache.
At client request, the definition model will be parsed, with relevant references expanded or replaced, to generate the rendering model for the client.
Even the /rest/sitemaps endpoint is dependent on the existing legacy model, it does not make sense to keep this moving forward except as a deprecated feature for clients that have not been adapted yet. Ideally, the new implementation should coexist with the existing one, without changing it. By default, the REST API should return the most compact representation of the new rendering model, but clients should be able to add matrix path parameters (;version=1) to request the backwards-compatible format while it is still supported. All clients will need to be adjusted, but it may simply start by adding the parameter to make sure it keeps work, until support for the new standard is implemented.
As a final note, features that must be implemented include multi-language support based on the Accept-Language header, but this is part of the improvements to be added once the basic functionality listed on the points above is implemented.

yveshanoulle commented 6 years ago

The definition model will be stored in the JSON DB, it can either be converted from a file.sitemap in >DSL or generated by an external tool (e.g. openHAB Home Builder). The Java model will also be >maintained on the memory cache. What happens if people want to update their sitemap?

I'm working with two machines for the same building. (For testing purposes) Im storing the full configuration in a source database. I commit changes that I then download on a machine. Having a history of the configuration changes is important.

I don't want to have to make changes manually on a machine. I want to download the latest changes from my source control system.

yveshanoulle commented 6 years ago

I understand that multi-language is something to do after the basic features are done. There is a risk that then it's harde to implement the use of multi-language.

I don't understand the sentence:

multi-language support based on the Accept-Language header

flaviocosta-net commented 6 years ago

I don't want to have to make changes manually on a machine. I want to download the latest changes from my source control system.

A scenario with two servers is not a usual one, and retrieving definitions from a remote location is a completely separate topic.

I don't understand the sentence: multi-language support based on the Accept-Language header

This is a standard content negotiation mechanism for HTTP, and it will also require clients sending language preferences to the server. This should be part of the new proposal, it just cannot be the first requirement being implemented...

yveshanoulle commented 6 years ago

A scenario with two servers is not a usual one

Anyone who wants to (re) install a new server will run into same issue. I just do this every week, hence I bought me a seconds server to make it easier to install the latest version of openhab, without less risk to have an unstable server.

and retrieving definitions from a remote location is a completely separate topic.

For me it's not. This is how sitemaps work these days. What I read in this proposal, is that the way sitemaps will work in the future, you will break this. When I would bring this up in the future, it will already be broken and Openhab will become useless to me.

it will also require clients sending language preferences to the server.

Do I understand correctly you will work with a language of the browser/ phone That is a bad idea for most multi-language users. That was designed by people who work only in one language. It forces multi-language people to always work in the same language. Most of the time I work in English, in some applications I work in Dutch, others in French. That is how many people who speak more then one language regularly work. Forcing us to choose one main language is bad for us.

Moreover linking the localisation to the language is very bad. Even when I work in English, I want my localisation settings be Belgium.

The document you linked to says something similar:

"It is not a good idea to use the HTTP Accept-Language header alone to determine the locale of the user. If you use Accept-Language exclusively, you may handcuff the user into a set of choices not to his liking."

flaviocosta-net commented 6 years ago

For me it's not. This is how sitemaps work these days.

Keeping the sitemap in the local memory cache is the way it's already implemented today, so maybe I didn't completely understand your use case. The idea is to make the new sitemap implementation a superset of the existing one: everything that works today should keep working with no or minimal changes. Before the new implementation is released, please feel free to try it and flag any currently supported features that may be missing.

Do I understand correctly you will work with a language of the browser/ phone

No, on most standard web browsers the language settings are configurable by the user, the same should apply to sitemap clients. What the link indicates, however, is that multi-language support is just one part of the features required for localization/internationalization, and it should take some good amount of work to get full support for that especially on a multi-user environment that you describe. This doesn't mean that it won't be eventually implemented.

yveshanoulle commented 6 years ago

Keeping the sitemap in the local memory cache is the way it's already implemented today, so maybe I >didn't completely understand your use case. So it will keep working with a sitemap file?

Before the new implementation is released, please feel free to try it and flag any currently supported features that may be missing. I'm already testing new versions every week, hence I have two servers. I don't like to complain after something is done, I'm trying to be proactive help think about a useable direction.

No, on most standard web browsers the language settings are configurable by the user, Yes and what I'm saying is that as a multi-language user that is very annoying for websites. Annoying but workable for websites. For web applications (which is the case which smarthome/openhab) that is not workable. Most people when they think about multi-langual applications think from the fact that every user has one language and the application has to support multiple languages. What I'm saying, is that there is a big part of the world, that speak multiple languages and will work in different languages in different applications.

I use the same browser to acces:

local news sites: for this my browser has to have as main language Dutch, because French and English part of the local news contains much less content.
international news sites: for this my browser main language has to be set to English, because the international sites don't have any Dutch content and the French content is limited.
The Web application for my kids school does not support Englsih seriously and crashes when the main language is set to English
Amazon, google and a lot of other sites direct me to local versions of their sites, that limit me in what I can buy. I need to use amazon.de to make use of amazon prime. When my main language is set to Dutch it sends me to amazon.nl (same for French and amazon.fr)

This general means that my main language is English on most of my browsers.

My tablet and my phone are also used by my kids so the language is set to Dutch. In my office I work with both Dutch and French speaking people, although we are a country with offically three languages, most don't speak or read all three languages. (Last time I tried to speak German, I was told the person did not understand Dutch. ) So people using office tablets should be able to swicth to a different language.

Yes that can be done using different sitemaps, and that is fine for now, yet when you implement multi-language in sitemaps, i'm telling you it's not usable when it's based on the browser or tablet language setting. (In some companies, people don't even have the right to change these settings)

y

flaviocosta-net commented 6 years ago

@yveshanoulle, I will take that into consideration, although:

On web-based clients (e.g. Basic UI), language settings rely on browser configuration. You may need different user profiles on the browser or on the OS if you want to serve content in different languages (Firefox), or use some plugin/extension to switch between these (Google Chrome). This is mostly a client problem, not something to be fixed on ESH.
On general sites, you should have more problems with that because, as you say, content availability is different depending on the language. On sitemaps, the content will be the same, it's only a matter of providing the preferred translations, proper date/time formatting, etc.

For now, I have created a fork to start working on it and see where it goes.

yveshanoulle commented 6 years ago

1)a) We would like to use tablets in our facility to control openhab. Asking users to log in as different users would remove the opportunity to reuse the same tablet for everyone in a room/ building.

b) It's only a client problem because you make it one. Many websites/ web applications offer to switch the language and ignore the browsers language. I'm probably more used to these kind of websites because I'm from Belgium where we do have three official languages (and most Belgium alos speak English which is not even an official language, then people who's country only has one language.

2) I agree with two, this is why I requested multi-language in the first place because right now, we have to have create multiple sitemaps to solve this.

Thanks

flaviocosta-net commented 6 years ago

I have done some preliminary work on this: now I have a minimal DSL that generates what would be the definition model (with Xtext) and I have some ideas on how the SitemapProvider can be adjusted to provide sitemaps both in the current and the new formats. The big piece missing is what should happen between those two parts: the generation of the rendering model (that is sent to the client) from the definition model (either from a file written in DSL, or as a JSON that can be edited via the REST API). Either way, only the definition model is actually editable, the rendering model is read-only and regenerated whenever a client requests a sitemap view.

I still think we need some kind of engine doing that, because it would be used both to customize and reuse component definitions. The idea I originally drafted above is more or less what we see on JSONT, but the concept is somehow old and still I could not find any working Java implementation. Moreover, we probably should not be transforming a definition JSON to rendering JSON, that transformation should ideally be done directly on the Java model, which is an operation that may be performed with an existing Eclipse tool:

VIATRA: It seems to be mature, actively developed and well-documented. If it indeed fits the technical requirements, it could be a strong option.
Epsilon: It also seems to be mature and flexible, although the latest release is from 2016.
ATL: Some people seem to like ATL, but perhaps it has a more limited scope than the two options above, but it is part of the Model to Model Transformation (MMT) project which is still in incubation phase.

It someone here has experience with any of these technologies, that might help guide the selection of the most suitable option... otherwise I will see when I have the time to try out these tools and see which one would be the best fit.

maggu2810 commented 6 years ago

I don't understand the discussion about the language HTTP header. Isn't that what we currently are doing in all our REST resources? The language the response should be translated to is controlled by the respective header in the request. Why should it be done different here?

Just to link a few:

Only if there is no one defined we should be fallback to the system specific one.

flaviocosta-net commented 6 years ago

Hello @maggu2810, the idea is to apply the existing solution used in REST resources, that you mention above, to the new sitemap implementation as well.

However, ultimately Yves was raising another, related requirement: to allow users to select the sitemap display language via some component of the sitemap itself - for instance, with a Select element that will dynamically offer all available languages, so once the user selects one of the options, the sitemap is reloaded and the new language becomes effective for that session. This is something easy to do on an HTML web site, but not so much on a REST API which is stateless and has a much more strict contract with its clients.

flaviocosta-net commented 6 years ago

To provide an update on this, I have a proof-of-concept implementation showing a basic end-to-end processing almost ready. The new DSL looks very close to what could be its final shape, the transformation from definition models into rendering models has a robust design, and the data to be sent to the client is already accessible on SitemapResource.

I started testing based on the demo sitemap that comes with ESH, since it already has the things and items defined to support it. This is how the same sitemap is expressed with the new DSL:

sitemap newdemo label="New Demo Sitemap" {
    Frame label="Demo Items" {
        Text item=DemoContact label="Contact" value=[transform-map("en.map")]
        Text item=DemoString label="Message"
        Text item=DemoDateTime label="Date" value=[format("%1$tA, %1$td.%1$tm.%1$tY")]
        Group item=DemoSwitchGroup icon="icon1"
    }
    Frame label="Magic Test Items" {
       Colorpicker item=MagicColor
       Switch item=MagicOnOff
       Slider item=MagicDimmer
    }
    Frame label="Location" {
        Mapview item=DemoLocation style="height: 10"
    }
    Frame label="Weather" {
        Text item=Weather_Temperature
        Text item=Weather_Pressure
        Text item=Weather_Humidity
    }
}

String values may be specified either as a "fixed string" (between quotes), or via an [expression] (within square brackets) that is calculated dynamically. Styles can also be specified via expressions, so they can be calculated and updated based on any combination of item states and functions.

For the generation of the rendering model from the definition model, I first experimented with VIATRA - while it has some very cool features such as incremental model transformation, its transformations are specified with graph pattern semantics that is not easy to learn, and the framework works best with off-line/pre-compiled model transformations. Eclipse Epsilon, however, was a great match for what is needed here. Simple syntax and flexible runtime features work helped implementing the required transformation logic.

These are the new bundles created so far:

org.eclipse.smarthome.model.sitemap.definition.runtime Just contains the SitemapDefinitionProvider interface and its default implementation.

org.eclipse.smarthome.model.sitemap.rendering Object model (POJOs) for sitemap rendering models.

org.eclipse.smarthome.model.sitemap.rendering.runtime Base implementation to execute the transformation of definition models into the rendering model.

org.eclipse.smarthome.model.sitemap.standard.definition Xtext grammar and generated model for the standard (a.k.a. SmartHome) new sitemaps.

org.eclipse.smarthome.model.sitemap.standard.definition.ide Generated by MWE as part of the framework.

org.eclipse.smarthome.model.sitemap.standard.runtime Transforms standard sitemap definition models into the rendering model.

There are very few changes to existing code on other bundles, the exception being org.eclipse.smarthome.io.rest.sitemap: that requires significant changes to be able to support both old and new sitemaps in the REST API.

The next step is to decide how the rendering model will be serialized and sent to the client. I started adjusting the DTOs, but the work got tedious and possibly not even necessary - unlike the other models in ESH, the rendering model is not an EMF model with all the "overhead" added by EMF. It only contains the essential data that needs to be sent to the client, so it already fulfills the purpose for which apparently DTOs were introduced for the other models.

Another aspect I am considering is to move away from the "paged navigation" structure of the current API. Instead, the client could retrieve the whole sitemap rendering model once its home page is accessed, and all navigation between pages would happen on the client-side without any additional communication with the server. The subscription mechanism would be used to incrementally notify all subscribed clients of any changes to the sitemap contents. These would be updated asynchronously, so the user on the client application would (almost) always have an up-to-date view of the sitemap being served from the local cache.

If this approach is considered acceptable, I will proceed with this part of the implementation as explained above.

sjsf commented 6 years ago

That's indeed a big leap forwards! Thanks for sharing the update.

Regarding the drop of the "paged" navigation: I personally don't see an issue with that. The amount of data transferred for the rendering model is so minimal that this kind of optimization is not necessary. It makes things much simpler if we don't have it.

Do I understand you correctly that you are using Epsilon currently to transform the current, Xtext-based sitemap definition into the rendering model only? I don't know many details about the project, but it sounds like it makes sense. However, the only thing important to me is that we are able of having other sources for sitemap definitions which then won't require any dependencies on the Xtext/EMF-family. They are great technologies, but in terms of footprint we need solutions to be able to run with without, if they decide against using the DSLs. Just to be on the safe side, I thing that's worth mentioning.

There are very few changes to existing code on other bundles, the exception being org.eclipse.smarthome.io.rest.sitemap: that requires significant changes to be able to support both old and new sitemaps in the REST API.

That reflects my expectation. I actually don't think it is worth maintaining backwards-compatibility within io.rest.sitemap - major parts of e.g. Basic UI will need to be rewritten anyway. So it it makes things easier for you, don't hesitate to put another, new REST API next to it.

PS: I'm curious to see the some code also - even if it's work-in-progress and still a mess, don't hesitate to push an intermediate state to a branch in your repository fork.

flaviocosta-net commented 6 years ago

@sjka, thanks for the update, very appreciated!

Regarding the drop of the "paged" navigation: I personally don't see an issue with that. The amount of data transferred for the rendering model is so minimal that this kind of optimization is not necessary.

Indeed, the rendering model for the demo sitemap has about 2.1 Kb, even if the sitemap was 100 times bigger it would still be only 210 Kb, still small enough to be completely transferred in one shot without any significant delays.

In fact, I finished implementing that part - you can see below the (old) sitemap on the Basic UI and the (new) sitemap as currently displayed by the JavaFX UI, on the right:

sitemaps

The new implementation allows the user to navigate between the pages without requesting any data from the server, so the pages are displayed immediately. Any updates (to atom data or component styles) will be pushed to the client asynchronously, via SSE/streaming API - this is probably the next step in the implementation.

Do I understand you correctly that you are using Epsilon currently to transform the current, Xtext-based sitemap definition into the rendering model only?

It is there to convert the new (also Xtext) definition model into the rendering model. The only bundle where Epsilon is used is org.eclipse.smarthome.model.sitemap.standard.runtime, as it contains the code required to perform this transformation from the standard (new) sitemap implementation into the rendering model. Another bundle can be written to convert the old sitemap model into the rendering model; the client would consume it without really knowing if the respective definition model is specified in a file in a different DSL or JSON stored in a database.

The current ESH dependency on EMF is actually hard-coded into the ModelRepository, which explicitly returns models as EObjects. Coincidentally, Epsilon has its own class also named ModelRepository which makes no reference to anything related to EMF, as it is designed to be independent from specific back-end representations.

I actually don't think it is worth maintaining backwards-compatibility within io.rest.sitemap - major parts of e.g. Basic UI will need to be rewritten anyway.

Lots of existing code will eventually be removed - yes, ItemUIRegistryImpl will become obsolete, being replaced by an Epsilon transformation script such as the one attached here: SitemapRendering.zip

Still, there are other important clients such as the ones for Android and iOS, not sure how quickly these will be adjusted, so the existing sitemap REST API may need to keep running for some time.

Currently there are no unit tests yet and Javadoc are missing here and there, but otherwise the code developed so far should be relatively presentable already. I will check how to properly put this in a branch, as I want to avoid a mess with that and I don't have much experience with Git.

flaviocosta-net commented 6 years ago

@sjka, I have pushed the code into a new branch on my fork repository. I hope it was done more or less correctly, please let me know if you see something that needs to be fixed on that.

Obviously, in case you have comments, questions and suggestions regarding the code in its current shape, I would look forward into hearing these so any improvements can be done sooner than later, As mentioned above, I should now proceed with the required SSE implementation.

Misiu commented 5 years ago

@flaviocosta-net I just found this issue and I must say that this already looks amazing, awesome work! How is the implementation going? Is there something I can help? I'm not a Java developer, but I can help with tests. Does the new sitemap concept mean that it will be easier to add new item types? I really would like to see new items that look similar to sensor (https://www.home-assistant.io/lovelace/sensor/), history graph (https://www.home-assistant.io/lovelace/history-graph/) or thermostat (https://www.home-assistant.io/lovelace/thermostat/)

OpenHab is awesome (I'm in process of migrating to 2.4) but it is missing UI features that could be used to build UI for demanding users (for example for wifes 😊 )

P.S. that dashboard proposal from Your initial post looks awesome!

yveshanoulle commented 5 years ago

Keeping the sitemap in the local memory cache is the way it's already implemented today, so maybe I didn't completely understand your use case.

I was talking about how the sitemap is transferred to the server.

Today I create all my configuration in files on a local computer, that get uploaded to a source control system. Then I download them on the openhab server, either using a button openhab when I'm not at home, or when I'm home I SSH'ed into the openhab machine and pull it in.

The server then gets the files, openhab sees the changes and reloads them in memory. And then I reload the sitemap in my browser.

y

kaikreuzer commented 5 years ago

@flaviocosta-net There is quite some interest in the community to drive the new sitemaps forward. As ESH has been terminated, could you port your code over to https://github.com/openhab/openhab-core and possibly directly create a WIP PR with it?

eclipse-archived / smarthome

Proposal for a new Sitemap concept #5337

New or Renewed Concepts

Sample Sitemap

Sitemap rendering

Extension and reuse

Implementation