Improve Decoupling of API & UI

[kahboom]

tl;dr

Following from our discussion in the retrospective, there is an inherent problem in the application development lifecycle that I think is directly impeding our ability to keep up with feature requests in a timely fashion. I know this is a LONG description, but please take a moment to read, as I think everyone is affected by this. It's an opportunity to improve the development process. If we don't solve this soon, technical debt will continue to increase and the overall product roadmap will be jeopardized.

The two items to address are 1) decoupling the API & UI; 2) making a more parallel work flow rather than linear and blocking. Solutions proposed are at the bottom. We should collaborate and discuss any potential tradeoffs or alternatives.

Introduction

At the beginning of the project we made a huge effort to decouple the UI from the backend by creating a Node.js API (now in Java). As features continue to be added to the iPaaS roadmap, we are finding it more and more challenging to develop these features without getting blocked by one or more areas.

Theory

When you are spending as much time trying to get the local build of the REST API working properly (to be able to work with real data) as you do actually implementing a feature, it's a problem. It almost feels like we need yet another layer of abstraction to be able to work completely decoupled from the backend. From the UI perspective, which sits kind of right in the middle, it boils down to two factors:

1) separation of concerns and true decoupling 2) timing per sprint to develop these features (linear vs parallel work flow)

Current Work Process

Here is a rough outline of our work process:

1. Discuss requirements for the sprint.
2. Assign feature owners. Feature owners are usually backend folk (not sure if this is intentional, but seems like a very important consideration we should be consistent with and have a philosophy it supports).
3. Feature owners write a design proposal, which includes the data structure the UI should adhere to.
4. UXD collaborates with feature owners to create designs.
5. UI receives designs, looks at design proposal for the REST API, and begins work.
6. Confusion about whether to mock out data or use existing REST API work ensues. Neither option is smooth, as using existing resources is risky for reasons stated below, and mocked data is great, but when it comes time to wire up the API, there are often huge discrepencies.
7. Scrambling to get things working properly together, whether using maintainable approaches or not, in time for the demo and sprint end.

Testing the Theory

The problems described earlier manifest in a few different ways that I've noticed impede development:

Separation of Concerns

• The REST API sometimes has continuously changing requirements anywhere in steps 3-5 that result in adjustments to the data it feeds to the UI (and its structure), possible changes in endpoints, sometimes the structure is not known at all, etc. This is often out of necessity due to potential issues that are later discovered in the design proposal (naturally, this is much easier to discover on implementation rather than in theory on a PR, which in and of itself is a problem). Other times they are dealing with CI issues.
• The UI tries to satisfy design requirements but they sometimes do not match the data structure provided by the API. Improvements have been made here by increasing communication between UXD and the backend, but sometimes, as stated earlier, structural issues and discrepencies arise mid-implementation, which is normal. Backend issues are discovered, such as a bug or server issues with using the REST API. Also, a lot of the data returned from the API is not relevant and the structure is inconsistent across features.

Timing

• The REST API & UXD teams are essentially trying to work in parallel with the UI team, which puts an unnecessary amount of pressure on the former teams to get things completed in time for implementation in the UI, and early enough for QE and Docs to also make adjustments (a deadline the UI often can't meet). Parallel should be our goal, but execution is often linear and blocking.
• Since much of the sprint time in the UI is spent creating components and templates for the design implementation, this is often discovered at the very end of the sprint when wiring up the API.

The UI should be able to operate strictly with a known data structure and rough designs. Whether or not that has been implemented in the REST API is irrelevant. The data structure should be known before a single line of code is written on either side.

But who dictates what the structure should be? At the moment it is usually the feature owner, which is usually a backend engineer. There is nothing wrong with that, but if we choose to go that route, some adjustments need to be made.

Proposed Solutions

Feel free to add to this:

• Decide where the data structure should come from. Considering that iPaaS is heavily user-focused and technically SaaS, to me, it makes sense that it either comes completely from the front end or is mediated between the API and UI (more on that below).
• Decide how that data structure is decided on:
  • Consumer-driven contracts, as @zregvart suggested. (Concept & example for Spring). A pattern to adhere to an evolving service, and can come in many forms, as simple as a spreadsheet.
  • UI-driven data models, to which the API then adapts. This doesn't mean UI engineers need to be feature owners, but just that the UI portion would dictate the structure. This would allow us to mock out data knowing that when we switch to the REST API it will be a seamless transition (in theory).
  • Using an intermediary solution, like GraphSQL and Apollo (here is a full stack tutorial). Not a huge fan, but it could work.
  • Putting a freeze on design proposals, regardless of how bad the structure is, and reconvening the next sprint?

---

@syndesisio/all - Let's discuss benefits and tradeoffs, other possible solutions, and any considerations below!

[chirino]

Perhaps waiting for UXD to be complete before doing UI part of the problem.

If we build a 'skeleton UI' (one with out any styling or any thought paid to the user) in conjunction with the backend PoC. Then:

1. you get a conceptual ui demo available quicker.
2. defines the data structures in the ui land.

Then UXD can look at that, and polish it up, UI would then take that feedback and refactor that initial 'skeleton UI' into something actually usable.

[deeleman]

For me the way forward is crystal clear:

Considering that iPaaS is heavily user-focused and technically SaaS, to me, it makes sense that it either comes completely from the front end or is mediated between the API and UI

and...

UI-driven data models, to which the API then adapts. This doesn't mean UI engineers need to be feature owners, but just that the UI portion would dictate the structure. This would allow us to mock out data knowing that when we switch to the REST API it will be a seamless transition

Are the way to go IMO.

Same as you guys, I've been deeply involved in the design and development of large Angular, Backbone and Knockout enterprise applications consuming microservices over the past years and the decoupling suggested above has worked extremely well for me. On the contrary, every time the UI/UX setup had to conform to a data schema designed by a team other than the FE team, the end user experience turned into a endless spiral of issues most of the times, leading to unsustainable applications upon time.

Please let me point out some arguments on this:

1. The data schemas provided lacked relevant data or its format did not conform to the final visual UX purpose, forcing the UI devs to create convoluted mechanisms to digest, reformat or remap data. This obviously will turn into more boilerplate in the frontend.
2. The UI devs will want to ensure DRY patterns by creating specific machinery to handle server error messages, CMS-driven data pulled from the backend, i18n features, etc... This will turn into a common set of properties and interfaces that the REST API will have to enforce across its different I/O messages to ensure consistency across all its endpoints. It's usually the UI's responsibility to define and inform of such standards. When that responsibility is removed from the UI devs, the API messages will lack relevant information or will have to build additional systems to infer it.
3. Components, services, and template bindings and filters will be designed with reusability in mind. The idea is to build as many components and reuse those as much as we can as a sort of LEGO blocks wherever needed. Those entities will be opinionated about how data must be presented in order to reuse persistence patterns whereas available (Flux, Redux, etc). Simply put, reusable components do require reusable API messages observing reusable sets of properties of its choice, and not all the way around. This does not compromise Agility, but rather contributes to it.
4. Component-driven apps are all about navigational reusability and interaction, so conversations between UI and UXD need to happen on a early stage to better lay out the underlying data schemas that fuel the UX. This way components can be better designed with scalability in mind from day one. Otherwise, changes in the product scope will eventually turn into dramatic changes in the API design, since that initial data abstraction conforming to the reusable nature of components was not observed at first.
5. The more we declutter the information exchange between the client and the server, the cheaper the bandwidth bill gets by the end of the day. And it is not all about moentary costs, but memory and resource costs as well, mostly when implementing low level browser APIs (such as Service Workers or Web Workers for caching, offline support, etc).

Bear in mind that modern UI applications are not mere presentation layers anymore. Modern SPAs deliver nowadays complex navigational and interactive flows that also rely on functional, persisted, reactive data models and, as such, UI should take the initiative in the design of the API interfaces IMHO. Ultimately, Syndesis UI is a user-centric product, so the UI layer will want to have its say on how the data exchange between the user and the product occurs.

[gashcrumb]

Perhaps waiting for UXD to be complete before doing UI part of the problem.

This advocates for a backend-first approach though. I thought one of the fundamental approaches with ipaas is that it's supposed to be designed with user experience in mind first...

[deeleman]

If we build a 'skeleton UI' (one with out any styling or any thought paid to the user) in conjunction with the backend PoC

In order not to waterfall, the use of quick wireframes is a standard practice, but no UI can be envisioned without the user in mind. It's the idea of what expectations does a user persona have in regards of content and navigation preferences which will help us lay out the interaction that each feature needs to present to the user, and it's such elements which will tell FE & BE what's the way forward.

Simply put, pushing back the product cosmetics to the very last stage of each feature development is all fine, but the content needs to be defined as far as we can right from the beginning.

IMO, an ideal workflow would be: UXD will want to have its say about the content and navigation flows, UI will have to translate that to I/O messaging specs and backend will have a final call about what's possible and what's not from a time/resources POV when it comes to build the corresponding API interface.

[gashcrumb]

IMO, an ideal workflow would be: UXD will want to have its say about the content and navigation flows, UI will have to translate that to I/O messaging specs and backend will have a final call about what's possible and what's not from a time/resources POV when it comes to build the corresponding API interface.

+1 I definitely think we could define the API objects and interface up front and define a mock backend and start UI development,the backend would then just need to worry about implementing what's defined then.

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub https://github.com/syndesisio/syndesis/issues/540#issuecomment-348312255, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVdrHbEu3vmmm40483EXfOERHhvuu5Uks5s7xDIgaJpZM4QxAfo .

[kahboom]

@chirino - We are already doing that to an extent. We get the designs relatively quickly and just implement the bare UI, leaving minor aesthetics for the end, and the user experience is in mind from the beginning when UXD creates the designs. So, in that sense, it is already user-driven from the presentation perspective, but not in terms of the data.

Having said that, designs have not been blocking me, personally, as of lately--which I know was more of an issue before and I think UXD has made huge improvements here (not sure how? but good work! 👍 ). On the other hand, dealing with the virtual machine and local REST API has really slowed the team down (i.e. nip.io has been down for over 12 hours now, errors with the MiniShift instance). We keep losing days of work when it involves the API.

EDIT: I want to emphasize that this is not the fault of any single engineer. It's a process issue. For instance, I don't think @zregvart could possibly work any harder/faster than he already has on the API Connector REST API stuff.

Using the staging API is terrible practice, but I kind of miss it because it would allow me to work when it wasn't down. And it's a lot of work and pressure for the API team I think, especially for one person to take on a whole feature in the backend, so I sympathize with them as well. I guess this is a good time to reconvene.

I may have preference, but personally don't care who creates the data structure. What I care about is that the API needs to be completely decoupled to not inhibit UI development and reaching project goals on time.

[chirino]

Switch to xip

[chirino]

I'm saying on way to break down this process issue is to get folks working on both sides of the the app.: rest and UI.

That way it does not matter on which side you start on. In the end you will end up having to do both sides

[zregvart]

I think we have agreed on trying out consumer driven contracts in place or in addition to design documents for the next feature/sprint. I can do a demo and if we agree that this is a way to go a workshop around this example feature to get up to speed.

What I think CDC gives us is precisely what we are discussing here: having the UI drive the data model and the API endpoints through defined executable specification. Being executable we get stubs for the UI to develop against and tests for the backend to verify the implementation with.

see #136 and #509

[deeleman]

I'm saying on way to break down this process issue is to get folks working on both sides of the the app.: rest and UI.

That will occur no matter what. So we better make it happen through a logical workflow. ;)

That way it does not matter on which side you start on. In the end you will end up having to do both sides

Well, it does matter and that's the point of this thread, precisely. Otherwise we will wind up duplicating efforts, as if we were trying to build a tunnel by having two folks digging from opposite sides of the hill. In other words, we don't want frontenders bugging backenders everyday to get changes on the API because we want to inject a new text string as a form title and we don't want backenders to have to rebuild the API interface everyday.

[deeleman]

What I think CDC gives us is precisely what we are discussing here: having the UI drive the data model and the API endpoints through defined executable specification. Being executable we get stubs for the UI to develop against and tests for the backend to verify the implementation with.

That sounds great. I'm a bit familiar with the CDC paradigm but I'm quite curious about how can we implement it on the frontend while keeping the separation of concerns between UI and API. All the FE libs I know require a lot of boilerplate code for generating the contract schemas or are either too outdated. However, I guess you have other tooling in mind for putting this into practice, which I'm really keen to learn in detail. All in all I'm definitely looking forward for that demo @zregvart :)

[deeleman]

@zregvart Although I have no direct experience with it, perhaps it might be interesting to assess the open source version of http://www.krakend.io/

[amysueg]

It's not clear to me that this issue is focused on the UX aspects of the process - seems to be related more to the connection between APIs and UI front-end coding - but since this seems to be a broad "process" discussion I'd like to add a few points.

The UX team is committed to providing the highest-level design work as quickly and efficiently as possible, without creating bottlenecks for development. To do so, we rely on product management for well-defined requirements, user stories, and acceptance criteria. We also rely on timely reviews with stakeholders. And we are keenly aware that once designs are approved, during implementation, things do change and so we keep in close touch with the front-end devs as implementation proceeds in order to make changes. We have a new UI developer (@seanforyou23) on the UX team who is working with UX and UI-devs to ensure consistent implementation of Patternfly UI components, to find opportunities to contribute designs back to Patternfly, and to provide coding assistance for HTML/CSS code. Finally, our work includes user research as well. We have a survey running right now to gather feedback on TP2 for example.

I believe that all of these processes are working reasonably well (and I think I saw @gashcrumb note that previously). The biggest issues for us are

• prioritization of work (including 'small' UX issues that aren't related to major features)
• a clear understanding of when UX designs are needed within the sprints/TPs
• clear acceptance criteria

We are working on a new Git Page to link to designs and status. We will present that to the team when it is ready to go.

[rhuss]

I agree that the user of the API should define it.

However, for the backend the domain model is important as this is how the business logic is implemented. This does not necessarily and typically does not match 1:1 to the UI driven API.

The solution to this is also very old and proven: Using ViewObject (or Data Transfer Objects as their also often be called) with a translation layer in the backend to the Domain objects.

This would give us the following flexibilty:

• API can be defined that only the data needed is transferred. There can be many different View Objects per Domain Object, its not a 1:1 match obviously (think of IntegrationListItemVO, IntegrationDetailsVO, ...)
• The backend can model the domain that it fits the business logic (e.g. how to persist integration, how to create runnable artefacts from the integrations).
• The API can be easily mocked out with affecting the domain model.
• Both API and Domain can change independently (just adapt the translation layer).

But I think I said this already several times.

This implies that we have to change our current model of giving through the domain objects directly from the database to the UI (and get away from the document oriented Jsondb which would put to much of a burden to the translation / api layer to manually extract UI relevant parts without using a proper query).

Regardless of how we define the API (I'm all for trying ou CDC but I have zero experience with it), I think this is the even more important point to decouple UI from backend. It's a huge change, though. (see https://github.com/syndesisio/syndesis/issues/42 , an evergreen)

[dhirajsb]

Currently we have a single deployment.json for connectors, which configures the frontend and backend. We should split that up into separate configurations for UI and backend rest services. There should be a middle tier for decorating/adapting the backend config to UI config.

It might even be possible to create an opinionated but configurable engine for generating the UI config from the backend config. I think that in a way the current rest services do that, but we need to pull that behavior out and have separate well documented schemas for the UI and backend model.

[deeleman]

It might even be possible to create an opinionated but configurable engine for generating the UI config from the backend config

I do not seem to understand this. What would be the purpose and scope of this config setup? Could you please elaborate on it? Thanks :)

[gashcrumb]

We have newer issues discussing this...