search

opensearch-project / OpenSearch-Dashboards

📊 Open source visualization dashboards for OpenSearch.
https://opensearch.org/docs/latest/dashboards/index/
Apache License 2.0
1.61k stars 827 forks source link

[RFC] Dashboards Modernization #4306

Open seanneumann opened 1 year ago

seanneumann commented 1 year ago

1. Introduction

Hello there, fellow contributors of the OpenSearch Dashboards community!

I've done a lot of thinking, writing, and talking about OpenSearch Dashboards with a lot of really smart people. I've been taking notes and want to share my thoughts. Let's be honest... it's time to update the look and feel and make it easier to develop and maintain. I would be genuinely thrilled to have the opportunity to collaborate with all of you on enhancing and modernizing Dashboards.

Before I delve into the details of this proposal, I want to emphasize that it is not intended to be an exhaustive roadmap of all development efforts I think are needed. I deeply appreciate the incredible contributions made by countless developers over the past decade, and my intention is to provide an honest evaluation of what I think our goals should be for the upcoming decade. Together, let's establish a strong foundation for a successful project, building upon the achievements of the past.

In this proposal, I present an opportunity to revamp OpenSearch Dashboards, or Dashboards for short. My primary objective is to create a user-centric experience that aligns with the evolving needs of the OpenSearch community. Here are three key areas that I believe need our attention:

  1. Look and Feel Redesign: It's high time we all give Dashboards a complete makeover that will elevate the user experience to new heights.
  2. Cohesive Plugin Experiences: I recognize the significance of integrating plugins seamlessly within Dashboards, ensuring a unified and streamlined experience for all users.
  3. Technology Modernization: Embracing the latest technologies is crucial for the long-term success of Dashboards. I am eager to explore opportunities to modernize the underlying technology stack.

Your individual input and contributions truly matter! As an active participant, I encourage you to engage with me and share your thoughts. If you feel that something is missing or have specific ideas you'd like to contribute, please don't hesitate to leave comments, raise issues, propose new features, or even contribute code.

2. Tenets and Goals

First and foremost, let's set the tone. What better way to do that than to agree on some tenets and goals. If you have better ones, please let me know.

2.1 Tenets

2.2 Product Goals

2.3 Platform Goals

3. Current State

Let's talk about the current state of Dashboards. Specifically, what I think are problems.

When I say Dashboards, I mean everything from the core platform (e.g. application chrome, data and saved objects APIs, etc), to native plugins (e.g. Home, Discover, Dev Tools, Stack Management, etc.) and feature plugins (e.g. Observability, AD, Maps, Alerting, etc.). Dashboards is assembled by the work in many repositories that have well over 1M+ lines of code. Dashboards suffers from systemic usability issues rooted in disjointed workflows, visual inconsistencies, and lack of getting started help. The developer experience is overly complex and has a high barrier of entry discouraging contributions. The technology under the hood is antiquated, fraught with technical debt, difficult to maintain, and requires us to constantly chase down emerging security issues.

3.1 Usability and Cohesion

Users express concerns with usability. They have highlighted the need to simplify the UX for users attempting to create a visualization, add it to a dashboard, and share it with their team. They noted broken menus, complex integrations, and unpredictability of what the UX would show.

The lack of cohesion is largely due to feature plugins being built differently than the native experiences forked from Apache 2.0 licensed Kibana 7.10. As you move between core experiences and feature plugins there are visual inconsistencies in the fonts, colors, layouts, workflows, and charts. Feature plugins are mostly siloed experiences, are not integrated into golden path user journeys, and render styles and visualizations in different ways.

Why is this? Impacting visual consistency, Dashboards offers multiple UI component frameworks and visualization rendering libraries with overlapping functionality each with their own authoring experiences and implementations. In other words, there is no standard way to develop a plugin. Impacting workstream integrations, plugins often don’t leverage existing core interfaces like Saved Objects to store UI-configured metadata or Embeddables to render their visualizations to co-exist on user dashboard pages. This is a remnant of the constraints in Open Distro where the plugins were separate products built for Kibana.

3.2 Developer Experience

Developing cohesive experiences for Dashboards is challenging. The effort to bootstrap the plugin development process and get it bundled into the current version-coupled releases is far too high of a barrier. The multiple UI component frameworks, half a dozen visualization libraries, lack of an SDK with consolidated interfaces, and poor developer documentation creates high cognitive burden.

A major pain point for developers is the amount of time and work it takes to bootstrap the plugin development process. A developer must pull down the OpenSearch Dashboards codebase, gain some knowledge of it (which is not yet well documented), and build it, just to get started. This alone can be a daunting set of prerequisites. Secondly, plugins have a contract with Dashboards directly. As Dashboards evolves there is a high potential to break plugins because of the direct dependency. This means there is a lot of burden to constantly keep their plugins up to date which may not be sustainable.

3.3 Convoluted and Antiquated Technology

A lot has been learned from forking Kibana and maintaining Dashboards. The 11+ year old architecture inhibits rapid debugging and frequent enhancements. There are nearly 1.1 million lines of source code and over 8800 source files. Keeping up with security vulnerabilities and updates to nearly 2500 dependencies (over 265 direct dependencies and over 2215 indirect dependencies) is untenable. The technical debt is high, as we continue to use deprecated or abandoned packages from a decade ago.

Many of the dependencies are ancient; some were abandoned many years ago and some deprecated recently. Some cannot be upgraded without also upgrading several others and some need major rewriting to upgrade or replace. For example, the upgrade of Node.js from v10 to v14. This was a big, disruptive effort that required upgrading several other dependencies, coordinating with plugin developers, addressing a long list of breaking changes, and going through many rounds of testing. Once that effort was complete, 64 security issues were resolved immediately. There are many more dependencies that need upgrading with similar effort.

Another example is EUI (Elasticsearch User Interface), a UI component framework built by Elasticsearch. As part of iteratively improving Dashboards, EUI to be OUI (OpenSearch User Interface) was forked and refactored, closing over 200 security issues in the process. But even after all of that work, OUI is not as well supported or robust as design systems like Google Material Design System, an open source design library supported by hundreds of developers.

To render different types of visualizations, Dashboards uses six different charting libraries, some nearly a decade old. This use of many different charting technologies provides an inconsistent user experience and results in duplicative design and engineering effort. This has made it difficult to respond to community requests for new charting types.

The most urgent technical debt issue is AngularJS, the web framework Kibana was first built on. Long Term Support for AngularJS was discontinued in December of 2021. Staring in Kibana 6.0 (released in 2017), Elastic has iteratively been migrating their AngularJS code to their new React platform. As of Kibana 7.10, the last open source version, there were still large amounts of AngularJS code (144K/1.1M or 13%).

4. Future State

4.1 Look and Feel Redesign

Redesigning the Dashboards experience will require focused effort in several areas: (1) sharing a future vision of the application experience, (2) codifying design system guidance, (3) evolving the component library, and (of course) (4) heavy refactoring in core Dashboards and plugin codebases.

4.1.1 Future Vision of the Application Experience

@kgcreative just published #4298 as a companion to this document, a proposal for future vision of Dashboards. I'll let him walk through that.

4.1.2 Codify Design System Guidance

The OpenSearch Design System (OSDS) is a set of standards intended to manage design at scale using reusable components and patterns. It provides pattern guidance, branding information, and web components (via OUI) for visual consistency of fonts, colors, layouts, and interactions. It supports the OpenSearch Project themes of Ease of Adoption, Usability, Accessibility, Cohesion, i18N/Localization, and Developer Experience.

To set the project up for success, the following must be codified:

This is actively being worked on by @KrooshalUX and @kgcreative.

4.1.3 Evolve the Component Library (OUI)

The OpenSearch UI framework (OUI) is the component library is used at the OpenSearch Project to build projects that share our aesthetics. It distributes UI components and static assets for use in building web layouts. OUI should adhere to the guidance described in the design system (see 4.1.2).

Changes to OUI should start with updated styles, colors, and fonts to adhere to the design system. Once defined in design system, we can make changes to the Sass variables with updated styles for both dark and light modes. Sass (Syntactically Awesome Style Sheets) lets you assign a value to a variable and then refer to it whenever you need to use that value.

I'd like to modernize the CSS stack alongside web components and align on modern front-end practices. For example, I eventually want to move from SASS to CSS customer properties. This will let developers redefine the CSS via JavaScript, which will be useful for real time re-theming.

If there are new components that need to be made (e.g. a new data table or chat box), those will need to be built directly into OUI. Any application (including Dashboards and plugins) that want to leverage the components can consume the latest OUI via NPM.

For current documentation on OUI, please visit oui.opensearch.org

4.2 Refactor Dashboards and Plugins

I'd like to do a complete end-to-end redesign of Dashboards and Plugins. In order for this redesign to be scalable and consistent, it must leverage OUI.

4.2.1 OUI Compliance

Today, Dashboards and plugins don't always use OUI, or use it properly (as documented in the usage guidelines). There are many UI implementations that use overrides or custom styles or components. In order for theming updates to be applied correctly and consistently, a prerequisite is to remove any potentially conflicting styles and components from Dashboards and plugins. They must be made OUI Compliant. The goal is to make styling and component definition the sole responsibility of OUI. An idealized outcome would be to remove all custom inline styles, classes, and stylesheets from Dashboards and plugins.

Achieving and maintaining OUI compliance will have several work streams. You can track them and get involved via the Dashboards & Plugins OUI Compliance #4065 meta issue and the OUI Compliance project board

4.2.2 Applying the future Vision

As clarity around the future vision for Dashboards materializes, we'll need to continue iterating on OUI changes and refactoring Dashboards/Plugin pages with updated flows, layouts, etc. A “Future” Playground was setup that takes the latest main branch to show incremental and demonstrable progress. See Future at future.playground.opensearch.org.

4.3 Cohesive Plugin Experiences

4.3.1 Cohesion through Interfaces - OUI, V12i, and SDK

To bring cohesion to Dashboards we should consolidate and expose the most important interfaces through an SDK. Three areas to focus on: 1) OUI, for UI component and style consistency, 2) V12i, for a single charting visualization rendering interface backed by Vega-Lite, and 3) core interfaces, for plugin integration into golden path work streams.

Users Benefits

Developer Benefits

Consolidation of UI component and visualization rendering libraries into an SDK reduces maintenance and cognitive burden. We should introduce a configuration and template system and a style integrator so that common default preferences can be set once, instead of on every visualization. We should standardize and modularize configuration options so that UI components are consistent and saved visualizations are cross-compatible. We should clearly separate data source configurations and fetching from visualization configurations. We should define capabilities and behavior that visualizations should satisfy, so that it’s quicker and easier to build new visualization type definitions that are still fully featured.

For more information on V12i and SDK:

4.3.2 SDK Docsite

As the list of interfaces required to support extensibility are identified, we should consolidate and rewrite poor developer documentation, like Saved Objects, Embeddables, Expressions, etc. This will help plugins to onboard to cohesive interfaces until an SDK is available. We should evolve the documentation for the SDK and make them available via a docsite.

4.4 Technology Modernization

4.4.1 Upgrade Node.js

Node.js 14 has ending support in April and will no longer receive security updates. An upgrade to Node with long-term support is required. As of Release 2.8, Dashboards now supports Node.js versions 14, 16, and 18. Users have the flexibility to choose from these versions to run the web application. Previous 2.x versions were distributed with Node.js version 14. OpenSearch 2.8.0 includes version 16.

4.4.2 Remove AngularJS Dependency

Dashboards has a direct dependency on AngularJS 1.8, a discontinued open source JavaScript-based web framework. Long Term Support for AngularJS was discontinued in December of 2021. This dependency must be removed. To achieve this systematic removal, refactoring, and replacing of major areas of the Dashboards codebase is required. This includes the Discover plugin (47K), Timeline (25K), opensearch_dashboards_legacy (17K), and parts of the Dashboard plugin.

4.4.3 Client Consolidation

Today, there are a half a dozen clients being used under the hood, including a legacy Elasticsearch client. To maintain backwards compatibility with OpenSearch, I propose moving to a single client, specifically the latest version of the OpenSearch JS client.

4.4.4 Versioning: Decoupling the rigid dependency

When Dashboards is run, it compares its version with those of the configured OpenSearch instance and all plugins. If they are not within the same major.minor version, Dashboards is marked red and exceptions are thrown. This inhibits Dashboards from releasing independently and is a blocker for the OpenSearch Extensibility Technical Roadmap.

The rigid versioning dependency should be decoupled. Ultimately, the goal is for Dashboards to be backwards compatible with at least 2 major versions back of OpenSearch and for extensions to be supported by our SDK which will minimize the potential for breaking changes. Extensions should not need to be updated each time a new minor version of Dashboards is released.

4.4.5 Ensure Backwards compatibility and Relax Version Checks

Today, when versions don’t match, Dashboards is marked red and/or exceptions get thrown. As backwards compatibility is better supported, I propose relaxing the version checks. Dashboards should ensure its backwards compatible with OpenSearch 1.3.x and relax those version checks.

4.4.6 Define Major.Minor.Patch to be SemVer compliant

Item one of the SemVer specification is that “Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. Which ever way it is done, it should be precise and comprehensive.” As Dashboards has never official declared an API, pretty much everything is considered a breaking change (e.g. Node 10 to 14 upgrade). The APIs declared will inform the interfaces prioritized for the SDK.

See Appendix: Dependencies to Consider Modernizing below for other more thoughts on modernization.

5. Feedback and Contributing

I welcome your thoughts in the comments below.

Additionally, you may communicate with me and the Dashboards dev community in the #dashboards Slack channel. Come say "hi!".

Thanks for reading!

6. Appendix: Dependencies to Consider Modernizing

Technology Function Problem Proposed Solution
Lodash Lodash is an open-source JavaScript utility library that provides a wide range of functions and tools for simplifying common programming tasks. OSD uses lodash, a library for the previous era of JS. For fewer dependencies, lesser memory usage, and faster execution, lodash should be abandoned in favor of native built-in methods. Additionally, numerous CSP blockers are tied to these libraries.- _root.js, core.js, template.js, and lodash.js evaluate strings as code. There is hardly anything lodash offers that is not natively suported by all browsers. There are some common ones that we could easily port to a utility function of our own to reduce dependency.- This could be "Small" effort after de-angularization.
jquery jQuery is an open-source JavaScript library designed to simplify client-side scripting of HTML. It provides an API for interacting with HTML documents, handling events, animating elements, and making AJAX requests. OSD uses jquery, a library for the era before the previous era of JS. For fewer dependencies, lesser memory usage, and faster execution, jquery should be abandoned in favor of native built-in methods.- Some jquery-based plugins are used by OSD Mostly used in legacy code, there is hardly anythything used that is not also natively supported. The dep should be dropped.
Bluebird Bluebird is a open-source JavaScript library for working with asynchronous code. It is a Promise library that provides a number of useful features and enhancements to the built-in JavaScript Promise object, including performance optimizations and additional methods. Eliminate Bluebird in favor of native async functionality; it is slower and impacts memory usage, but also contributes to the CSP-related vulnerability of OSD. Bluebird needs to be replaced with await and async which have been natively available for over 5 years in all relevant environments. Promises have been available for longer.
Handlebars Handlebars is an open-source JavaScript templating engine that allows developers to create reusable HTML templates with dynamic content. Replace Handlebars which is used for in-browser compiling templates at runtime; it contributes to the CSP-related vulnerability of OSD. We need to reconsider how this library is being used completely. Server-side sanitization and compilation is one option but maybe a simple custom code can solve the problem that created the need for handlebars.
Numeral, @elastic/numeral Numeral is an open-source JavaScript library for formatting and manipulating numbers. It provides a simple and easy-to-use API for formatting and converting numbers into different formats, such as currency, percentages, and time durations. OUI and OSD use Numeral.js extensively for all the number formatting needs. This was abandoned 6 years ago and has several bugs impacting users. numbro, d3-format, or even a super-lean cutom implementation that serves our use-cases.- Since this would impact existing index patterns and other saved objects, it would be best to provide everything numeral.js offers using modern techniques via a custom implementation or consider translating saved-objects during upgrade.
Selenium Selenium is an open-source testing framework used for automating web browser interactions. It allows developers to write scripts in various programming languages such as Java, Python, Ruby, etc. to automate testing of web applications. The testing ecosystem uses Selenium and an installation of Chrome, along with Jest’s Jasmin2. Testing with Selenium is notorious for being flaky and we have had to patch Jasmin2 and other Jest components to partially fight this flakiness. Additionally tests take a long time to run and modern choices are shown to be 25% faster; that sums up to 1.5 hours faster tests. Jest’s Circus with Puppetier or Playwright would not only save developers time, they will eliminate the flakiness of Selenium too.- Preferece is to leave this choice to the functional tests' repo but there is agreement to move them out of OSD.
TypeScript TypeScript is a superset of JavaScript that provides additional features and capabilities for developing large-scale, complex applications. TypeScript has had breaking changes in every 4.x minor release which has made it very hard to keep up with latest releases. OSD and OUI have had to use older releases of some dependencies as result of not upgrading. Considering the blast radius, this could be a big effort. Upgrade to the latest version of TS and fix everything that breaks. Also convert JS files to TS.- rip out TS-dependant build scripts and rebuild them with simpler logic
Webpack Webpack is an open-source module bundler for JavaScript applications. It is used to bundle and package JavaScript code and other assets such as stylesheets and images into a single file or a set of files that can be deployed to a web server or used in a browser. Babel and Webpack belong to the old era where the JavaScript ecosystem was very different. The in-browser payload is gigantic because it contains a lot of bloat added by Babel despite any optimized code written. - @osd/optimizer is a custom bundling solution around webpack- @osd/optimzer is undebuggable With the new native module system and using newer bundlers to replace Webpack, development speed increases and bundle sizes reduce. Directly using TypeScript’s compiler and bundling with Parcel or using SWC’s ecosystem are good options. Newer solutions would also save time dealing with numerous plugins and complex configurations of both Babel and Webpack when they release major versions.- This is also a dependecy management problem
Babel Babel is an open-source JavaScript compiler that allows developers to write code in modern versions of JavaScript (ES6 and later) and transpile it into an older version of JavaScript (ES5) that can be used in current and older web browsers. This allows developers to take advantage of the latest features of JavaScript while still ensuring their code is compatible with a wide range of browsers. Babel belongs to the old era where the JavaScript ecosystem was very different. The in-browser payload is gigantic because it contains a lot of bloat added by Babel despite any optimized code written. - Start with replacing babel- Have a more bleeding edge browser supprt (simplify maintainance for higher velocity)- Replacing with tsc
Moment Moment is a popular JavaScript library used for parsing, validating, manipulating, and formatting dates and times. It provides a range of features for working with dates and times in JavaScript OUI and OSD use Moment.js, another library created for the previous era of JavaScript ecosystems. It adds a huge amount to the in-browser payload to provide functionality that is now natively available in browsers. The library maintainers called it “done” about 30 months ago. Options:- Wrapping built-in Intl (Node 18+ and all browsers)- Using modern libraries based on Intl like Luxon- Using modern libraries with their own implementations like date-fns
Tinymath Tinymath is a lightweight and fast JavaScript library for performing mathematical operations in JavaScript. It provides a range of mathematical functions for working with numbers, such as addition, subtraction, multiplication, division, and more. Tinymath is designed to be lightweight and easy to use, making it a good choice for projects that require simple mathematical calculations. Replace tinymath, a library for evaluating arithmetic functions in strings and abandoned 4 years ago; it contributes to the CSP-related vulnerability of OSD. tinymath is used only in timeseries and that too only its evaluate method. mathjs offers an evaluate method that can be used.
js-yaml js-yaml is a JavaScript library that allows developers to parse and serialize YAML (YAML Ain't Markup Language) data in their JavaScript projects. YAML is a human-readable data serialization format that is often used for configuration files, data exchange between systems, and other tasks where data needs to be easily readable by humans. Upgrade js-yaml; even though it is a single major version bump, the effort might be huge. The version used contributes to the CSP-related vulnerability of OSD. While this is the most popular YAML parsing library, it hasn’t really been maintained in the past 2 years; maybe an alternative should be identified. Upgrade to js-yaml >= 4
node-fetch and whatwg-fetch node-fetch is a popular JavaScript library that provides a lightweight and flexible interface for making HTTP requests in Node.js. It is designed to be easy to use and has a simple API that is similar to the browser's Fetch API. Multiple ways of making web requests are used by OSD like node-fetch and whatwg-fetch. Node 18+ has native fetch support and browsers have had it for 6 years now. Replace uses of node-fetch with native offering- timeline visualization- telemetry plugin- fetching snapshot artifacts- node biary downloader Remove whatwg-fetch which is just a polyfill altogether.
deep-freeze-strict deep-freeze-strict is a JavaScript library that provides a way to freeze JavaScript objects and prevent any modifications to them. It works by recursively freezing all nested objects, making them read-only and immutable. Drop the usage of libraries like deep-freeze-strict and resize-observer-polyfill in favor of more performant native implementations.
React React is a JavaScript library for building user interfaces Dropping React is a controversial subject but needs to be discussed. AngularJS was killed in favor of Angular and Kibana and OSD have struggled for years to replace parts that use AngularJS. Additionally, React is fat, a memory hog, and slow. A web-component-based templated event-driven vanilla JS solution would be leaner, faster, and easier to develop and maintain.
dristanta-silwal commented 1 year ago

The future state section highlights the challenges faced by Dashboards in terms of usability, developer experience, and technical debt. The author identifies specific problems and proposes solutions, such as a complete end-to-end redesign, OUI compliance, and the consolidation of interfaces through an SDK. These initiatives aim to address usability issues, improve developer productivity, and reduce technical debt by adopting modern practices and technologies.