Closed seanneumann closed 1 year ago
CEHENKLE
commented
2 years ago Although I'm clearly biased towards loving Drag and Drop and EUI Anywhere, I gotta say "Dashboards Anywhere" is just so cool. I can't wait to see what you guys are gonna cook up this year :)
seraphjiang
commented
2 years ago There are two more projects doesn't mentioned in this issues
Dashboards Service Infrastructure project, which we will track at below repo https://github.com/opensearch-project/dashboards-anywhere
Dashboards Desktop https://github.com/opensearch-project/dashboards-desktop
seanneumann
commented
2 years ago There are two more projects doesn't mentioned in this issues
Dashboards Service Infrastructure project, which we will track at below repo https://github.com/opensearch-project/dashboards-anywhere
Dashboards Desktop https://github.com/opensearch-project/dashboards-desktop
Thanks @seraphjiang!
zengyan-amazon
commented
2 years ago An initial step for data source adaptor https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1388
seraphjiang
commented
2 years ago Add proposal for security out of box
https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1215
seanneumann
commented
2 years ago @zengyan-amazon @seraphjiang I've updated content of this issue to link to your issues.
seanneumann
commented
2 years ago @mihirsoni adding your metadata storage issue: Proposal: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/413
seanneumann
commented
2 years ago @seraphjiang Adding Desktop issue from Desktop repo: https://github.com/opensearch-project/dashboards-desktop/issues/3
Hagaygur
commented
2 years ago Hey, any chance to see Autocomplete on the road map? I am seeing loads of new and exciting things here which differentiate from Kibana, but this super annoying issue to anyone who isn't a cluster maintainer still isn't up to par.
vvvprabhakar
commented
2 years ago Want to contribute not able to access this
CEHENKLE
commented
2 years ago Want to contribute not able to access this
Hey @vvvprabhakar Are you still having problems? I've double checked that it's public, so you should be able to access it?
seanneumann
commented
2 years ago Want to contribute not able to access this
Hey @vvvprabhakar Are you still having problems? I've double checked that it's public, so you should be able to access it?
@seraphjiang can you help follow-up on this one?
seraphjiang
commented
2 years ago sure @vvvprabhakar let me know if you still have problem to access
peternied
commented
2 years ago @CEHENKLE OpenSearch Dashboards has been so useful, it effectively serves as the management interface for OpenSearch. We might want to make sure that how OpenSearch is managed (maybe its a new Dashboards Plugin?) is still possible as/after this decoupling work takes place. Should we create an issue for this, or do you think this has been captured elsewhere?
seanneumann
commented
2 years ago @CEHENKLE OpenSearch Dashboards has been so useful, it effectively serves as the management interface for OpenSearch. We might want to make sure that how OpenSearch is managed (maybe its a new Dashboards Plugin?) is still possible as/after this decoupling work takes place. Should we create an issue for this, or do you think this has been captured elsewhere?
Considering we want Dashboards to focus on data visualizations from multiple datasources and databases, any management system should be an extension (plugin). I'd love to think more with you around the best way we can support that type of extensibility in core Dashboards.
peternied
commented
2 years ago @seanneumann I've created https://github.com/opensearch-project/OpenSearch/issues/3160 maybe we can dialog more in that issue?
CEHENKLE
commented
2 years ago @peternied I've been kicking this around with @elfisher a bunch -- I think you're right that we're missing a whole set of admin tools for on prem management of OpenSearch. I suspect this will be an area we'll pick up the gauntlet for in late 2022/2023.
Anyone else feel like this is a hole in OpenSearch?
tmarkley
commented
2 years ago Research and planning for plugin decoupling (versions, build, development, etc.) is now tracked here (I've repurposed this issue since the conversation was going in this direction): https://github.com/opensearch-project/OpenSearch-Dashboards/issues/287
kavilla
commented
2 years ago [Groom]:
@seanneumann async search seemed blocked based on the discussion to role in? Can we replace this with the de-Angular project?
pergus
commented
2 years ago I love the idea of "Datasource Adapter", it would make Opensearch Dashboards a superior alternative to Grafana. Are there any plans for using Trino as a data source?
joshuarrrr
commented
1 year ago @seanneumann Can we close this issue with a summarized year in review? Excited for all the 2023 initiatives to come!
seanneumann
commented
1 year ago @joshuarrrr and all, I just posted a summary of 2022 as a preface above. Let me know what you think.
seraphjiang
commented
1 year ago Thanks @seanneumann for summarizing the great 2022, Your support for Dashboards Anywhere and excellent leadership for OpenSearch Dashboards are much appreciated.
Thank you @zengyan-amazon @zhongnansu @kristenTian @noCharger @mpabba3003 @yibow98 for bring the first dashboards anywhere feature - Multiple DataSource to users.
Thanks @Flyingliuhub @BionIT @huyaboo for building the Playground
Thanks @elfisher @ahopp @kgcreative @KrooshalUX @vagimeli @natebower for great partnership on dashboards anywhere on product, user experience and documentation.
Thanks all maintainers for the amazing support.
Looking forward to vision and strategy for 2023
ahopp
commented
1 year ago @seanneumann Do you think we're ready to close this issue? Seems like it's time.
seanneumann
commented
1 year ago Yep, I'll close it.
seanneumann
commented
1 year ago Closing out.
Update: end of year summary
For the 2022 year, the OpenSearch Dashboards team identified key initiatives we wanted to go after, which primarily included Dashboards Anywhere, Visualizations experiences, and a modern design system. Our goal working with the community has been to be the most open, secure, and versatile visualization application for analytics data. As we look back on 2022, we wanted to provide an update on the progress made.
Authentication Out-of-the-Box
To run independently (i.e. not on an OpenSearch data node), Dashboards will require out-of-the-box security solutions. We proposed to move security APIs into the Dashboards core, enable security by default, and release various modes of authentication as extensions (e.g. basic auth, OIDC, SAML, etc). In 2022, we launched support for concurrent multiple authentication methods in OpenSearch Dashboards via the Security Plugin (not in the core). This enhancement to the Dashboards security plugin provides a more unified and user-friendly login experience to Dashboards users, who are now able to choose a preferred option from a login UI that integrates basic authentication (username and password) and multiple single sign-on (SSO) providers for OpenID Connect (OIDC) and SAML. Looking forward, we will need to expand our authentication capabilities and move them into core Dashboards to remove reliance of running on an OpenSearch node.
Metadata Storage
Currently, Dashboards stores its metadata configuration inside an OpenSearch index. Metadata availability depends on OpenSearch cluster availability and other cluster parameters such as cluster health, state, and version. To mitigate this problem and unblock future extensibility of Dashboards, we proposed decoupling metadata storage from OpenSearch. This will allow us to configure metadata extensions to use additional databases such as MySQL, Postgres, DynamoDB, Serverless (S3+Athena). Outside of POCs and design brainstorming, this capability was never prioritized. As we continue to see the need to decouple, this will move up on the priority list. If this is a feature you'd like to see or contribute to, please let us know in the comments.
Multiple Data Endpoints
Dashboards is designed to work with OpenSearch as the only data source, the versions must match, and it can’t communicate outside the cluster. We proposed creating the capability for Dashboards to securely query data from multiple data sources that support OpenSearch DSL and APIs. In 2022, we launched the Multiple Data Sources feature. Users can add multiple data sources to a single dashboard, allowing you to dynamically manage data sources, create index patterns based on those data sources, and execute queries against a specific data source and then combine visualizations in one dashboard. This launch was critical to moving Dashboards to becoming decoupled. See our launch blog for more details.
Playground
Playground wasn't in our original proposal, but over the year we saw an overwhelming amount of support for a live demo environment of OpenSearch and OpenSearch Dashboards. In 2022, we launched OpenSearch Playground which provides a central location for existing and evaluating users to explore features in OpenSearch and OpenSearch Dashboards without installing or downloading anything. Give it a try at playground.opensearch.org and see our launch blog for more details.
VisBuilder (aka Drag 'n' Drop)
We proposed a new experience that will allow users to create data visualizations and gather insights without preselecting the visualization output and with the flexibility to change visualization types and index patterns on the fly. In 2022, we launched experimental versions of VisBuilder to simplify the process of creating visualizations in OpenSearch Dashboards. This included the foundational drag and drop ability, table vis with field summaries, ability to switch between types, and vis types Metric, Bar, Line, and Area. We're still an experimental phase so your feedback is important. Check it out under the visualization options. See our VisBuilder documentation for more details.
OUI
Elastic UI (EUI) and Elastic Charts are the UI and Visualization frameworks that comprise the bread-and-butter of the core Dashboards experience. Since the latest versions of these libraries are no longer Apache 2.0 licensed, we proposed forking EUI. In 2022, we forked EUI to use as a starting point for the OpenSearch User Interface (OUI). Dashboards is using this today. This positioned us to address existing and upcoming security vulnerabilities and start to work on a new experience vision. Download it today via NPM. And checkout our documentation site via oui.opensearch.org.
Maintainership
In addition to the above releases, the maintainers have been hard at work keeping the software up to date with the latest security patches, upgrading major dependencies (e.g. Node 10 to 14 bump), shipping the OpenSearch JS client, and launching 17 versions of Dashboards (1 major, 5 minor, and 11 patch releases).
So what's next? In 2023, our northstar is a refreshed cohesive experience across OpenSearch Dashboards and plugins backed by powerful interfaces. We'll be posting details on this in March.
Thanks everyone! We appreciate all the amazing support to evolve Dashboards to be the most open, secure, and versatile visualization application for analytics data.
Original Post
Hello there!
We’re the team at Amazon that maintains the OpenSearch Dashboards repo. As part of planning for 2022, we wanted to share with you how we’re thinking about the upcoming year.
We'd love to work with you to build out these ideas. This isn’t designed to be a comprehensive list of everything getting built in the next year. OpenSearch is a shared community project, so we're excited to see the ideas and projects that come from the community. These are just the areas where this group of people plans to put its time and attention given what we know now. So if there’s something missing that you want to see, that’s great! Let everyone know by writing issues/feature proposals/code for the thing you want to see.
1. Introduction
OpenSearch Dashboards (aka Dashboards) is an open-source (Apache 2.0 licensed), Node.js/React/TypeScript based, analytics and search application for the OpenSearch engine. Dashboards is the primary exploration and visualization tool for data in OpenSearch. Dashboards provides a platform for developers to build, test, and deploy extensible experiences, such as fine-grain access control, alerting, and index state management.
As a community, we work with the aim of being the most open, secure, and versatile visualization application for analytics data. The growing expertise of contributors is rooted in application and service based architecture, working with diverse data sources, and rich user experiences.
2. Tenets
3. Key Initiatives
These are the initiatives we’ve identified for 2022. As issues are opened for each, we'll link them here.
(1) Dashboards Anywhere
Dashboards is tightly-coupled with the OpenSearch engine: it depends on its indexes for persistence, and runs on and shares resources with OpenSearch data nodes (i.e. when OpenSearch is out of service, so is Dashboards). Dashboards can only visualize data from a single OpenSearch cluster (i.e. it has no capability to connect to other data sources or mix data sources into a single view). Customers continue to ask for a “single pane of glass“ across Dashboards use cases. Not only do they highlight their inability to triangulate issues as a major frustration, but they continue to highlight the need to use multiple tools to manage single use cases. Finally, the community has also highlighted that some patterns only emerge when data is combined and cross-examined with other datasets.
To grow beyond these constraints, we will launch a more modular Dashboards that can run independent of OpenSearch resources to support a more stable and extensible user experience. This initiative will position Dashboards to run anywhere, whether it be on-prem, in a managed cloud service (e.g. AWS, Azure, etc.), a serverless environment, or even on a user’s Desktop.
Critical pre-requisites to this initiative include out-of-the-box authentication, metadata storage, and the ability to connect to multiple data endpoints (e.g. OpenSearch, Elasticsearch, Prometheus, etc.).
(1.1) Authentication Out-of-the-Box
To run independently (i.e. not on an OpenSearch data node), Dashboards will require out-of-the-box security solutions. Currently Dashboards has no security features out-of-the-box that make it easy to connect to disparate data sources.
We’d like to make interfaces to become first class citizens and various options of authentication to become extensions. We propose to move security APIs into the Dashboards core, enable security by default, and release various modes of authentication as extensions (e.g. basic auth, OIDC, SAML, etc).
(1.2) Metadata Storage Adapter
Currently, Dashboards stores its metadata configuration inside an OpenSearch index. This coupled design approach works well for OpenSearch customers to get up and running quickly, but it introduces challenges while operating at scale and providing high availability for Dashboards. Metadata availability depends on OpenSearch cluster availability and other cluster parameters such as cluster health, state, and version. To mitigate this problem and unblock future extensibility of Dashboards, we are decoupling metadata storage from OpenSearch. This will allow us to configure metadata extensions to use additional databases such as MySQL, Postgres, DynamoDB, Serverless (S3+Athena).
(1.3) Datasource Adapter
Dashboards is designed to work with OpenSearch as the only data source, the versions must match, and it can’t communicate outside the cluster. If an administrator needs to view data from different sources, they need to use their respective monitoring software. For example, if the administrator needs to view both OpenSearch logging data and AWS CloudWatch metrics, they need to use both Dashboards and the AWS Console.
We will create the capability for Dashboards to securely query data from multiple data sources that support OpenSearch DSL and APIs. Our first priorities will be the ability to query OpenSearch 3.x, 2.x, 1.x. We will build the adapter to be extensible so users who need to connect to other disparate data sources can leverage our platform.
Once Dashboards can securely connect to these endpoints, we’ll provide users the ability to create a dashboard that composite the multiple data sources into one view, moving us closer to a “Single Pane of Glass'” future.
(1.4) Dashboards Desktop
Download Dashboards to your computer, configure your connections to your data sources, and enjoy all the same benefits of a cloud visualizations app experience locally.
(2) Dashboards Visualizations and Experiences
The Dashboards team now includes core Visualizations and Experiences. We are working closely with both product management and design to explore where we can make the most meaningful impact to usability, accessibility, and internationalization/localization. Today, many of the experiences, especially when plugins are involved, are disjointed and inconsistent. Ultimately, we want to reduce the number of steps it takes for customers to complete their tasks and delight them along the way.
(2.1) Easy Start and Configuration
The current out-of-box-experience (OOBE) was created for commercial up-selling and not designed for ease of getting started or making configurations. There is no integrated experience for users to enable and configure plugins. In order to install a new plugin you have to dive into the filesystem to make configurations. We will provide the ability to surface all configurations in a centralized UI with the community in mind. Both application and plugin capabilities can be easily enabled and customized. Additionally, we'll provide an OOBE to get users up and running quickly, which includes comprehensive tutorials with templates for both developers and non-developers.
(2.2) Drag and Drop
Today, visualization and exploration in Dashboard is harder than it needs to be. Users need to preselect both the chart type and index pattern they will use to create a visualization. Additionally, users may need to navigate between multiple screens (e.g. discovery → visualize → discover → visualize, etc.) to find the right feature, the right chart, and the right index to include in their dashboard. Not only is this more work then it needs to be, it prevents the serendipitous discovery that more accessible exploration allows.
Our proposal is to allow users of Dashboards to create data visualizations and gather insights without preselecting the visualization output and with the flexibility to change visualization types and index patterns on the fly. Users would be able to switch visualization types, index patterns, and data fields quickly and easily in a simple Drag & Drop experience. More specifically, we are proposing a number of key features for OpenSearch Dashboard Drag & Drop: (1) Users can create visualizations by dragging a data feature or field and dropping them on the visualization canvas. (2) Users can change the data features and the visualization within the Drag & Drop CX. Eventually this will also include the index pattern. (3) Users will be able to see suggested visualization based on the selected data features.
The Drag & Drop experience we're proposing would be created as a first party plugin (i.e. within the Dashboards repo and part of the default Dashboard experience) and accessible via the “New Visualization” option within the Visualize page.
(2.3) Accessibility
Accessibility means that everyone who uses a product can receive the same benefit, regardless of any condition or disability they may have. For OpenSearch Dashboards, accessibility will focus on identifying barriers that exist in the product and working to remove them. OpenSearch Dashboards will seek to be accessible to, and delightful for, all of our customers, specifically including the ~15% of the world population that has a significant disability. This includes accessible color palettes, compatibility with screen readers, meaningful keyboard navigation, and adherence to visual design guidance.
(2.4) Internationalization
OpenSearch Dashboards is already being used globally, in many countries and languages. Our north star is a localized experience in the customer’s language of preference with a design following the language conventions.
(2.5) EUI & Charts
Elastic UI (EUI) and Elastic Charts are the UI and Visualization frameworks that comprise the bread-and-butter of the core Dashboards experience. While there was room for improvement, they have clear contribution guidelines, component development guides, component design docs, and style guides. Since the latest versions of these libraries are no longer Apache 2.0 licensed, we have forked their respective repositories and started folding them into the Dashboards repository. As a stepping stone, we will upgrade Dashboards to use the latest Apache 2.0 versions of EUI and Charts to address known security issues and stabilize any breaking changes. The decisions on rename/branding these libraries is TBD.
(2.6) Dynamic Loading of Extensions
Currently plugins are tightly coupled with Dashboards making it complex to independently ship enhancements. Running plugin as a separate process will uncover advantages like dynamic loading of plugins. We should adopt a "VSCode Extensions" model. Details on this are still being researched, but we know this will be critical.
(2.7) Asynchronous Search
Searching large volumes of data can take a long time, especially if you’re searching across warm nodes or multiple remote clusters. Asynchronous search in OpenSearch lets you send search requests that run in the background. You can monitor the progress of these searches and get back partial results as they become available. After the search finishes, you can save the results to examine at a later time.
4. Next Steps
There’s a lot on this list of goals and realistically they may take multiple years to fully achieve them. But we look forward to making big leaps ahead on all of them in 2022 with you. Over the year, we will also continue to add links to this post for projects, so please watch the post if one of them interests you.