Closed ashwin-pc closed 2 years ago
Im only seeing index pattern as an option for the data panel. I've mentioned several times before that saved searches should be there as well. Especially since allowing the user to choose a saved search would have no impact to the generic design I had hoped was going to be built. Removing this ability would make this the only visualization that can't use saved searches just like Kibana did with Lens and Maps and seems needlessly limiting. Is the user going to be able to choose from a saved search as well?
The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.
Wait, maintain visual parity with existing visualization? The majority of OpenSearch Dashboard visualizations DO NOT use EUI charts. When this was forked from Kibana, EUI Charts was barely used. I believe, if I remember right, only Discover and some of TSVB used EUI Charts. If you mean to say visual parity with latest Kibana, okay I can buy that. But is this visual parity requirement an Amazon need or a community need?
There's a lot to consider when bringing in another Elastic built repository now behind non open source licenses besides where the library will be hosted (this repo or separate). For instance, what are the number of bugs/regressions that are being inherited by creating EUI Charts from the version that can be forked? What's the plan to fix all of these regressions and bugs that have been inherited? Has the team considered the missing features in Lens that are missing because of its use of EUI Charts, which in turn limits this tools potential in the future? I'm curious, what alternative charting libraries were discussed with the community and team?
Thanks for the feedback @JacobBrandt! You have brought up some very good points here. Hopefully this addresses your concerns.
Im only seeing index pattern as an option for the data panel. I've mentioned several times before that saved searches should be there as well.
Sorry, I totally missed saved searches while writing up the design. Yes, using saved searches is definitely something that we should have and will introduce in future iterations. I’m hesitant to include it in the first iteration though because it does have some challenges that we need to solve for. I’d love to hear your thoughts if you disagree and think that we need this feature in the first iteration of this experience.
Wait, maintain visual parity with existing visualization? The majority of OpenSearch Dashboard visualizations DO NOT use EUI charts. When this was forked from Kibana, EUI Charts was barely used. I believe, if I remember right, only Discover and some of TSVB used EUI Charts.
Shoot you are right! Thanks for catching that. TSVB also happens to be the chart type that I looked into while writing this. I will correct that text. What i meant was that we will use the same chart components used by the visualize plugin today (which i had incorrectly assumed was EUI) to maintain visual parity.
But is this visual parity requirement an Amazon need or a community need?
I mention visual parity because the main goal of the drag and drop experience is to offer a much better way to explore the data and create visualizations, and not to immediately change the end visualizations created. This will help the user use this tool as a drop in replacement for the existing visualization plugin. That being said, this decision does not limit us in future from moving to a different visualization library since the visualization layer will be abstracted away from the core of the experience.
I'm curious, what alternative charting libraries were discussed with the community and team?
We did consider alternatives such as Plotly.JS and i’m pretty certain we will migrate away from the default charting libraries in future. But for reasons mentioned above I don’t recommend us doing so initially.
But is this visual parity requirement an Amazon need or a community need?
To add on to what @ashwin-pc mentioned, we're trying to make it consistent with the current experience from an iteration perspective - I am personally advocating the "progress not perfection" path and we saw this as a good first step. This is not driven by any particular vendor.
Especially since allowing the user to choose a saved search would have no impact to the generic design I had hoped was going to be built.
Nothing is preventing this from being added, but we're trying to prioritize an initial version. But maybe this would be a good place to plug in on the dev side @JacobBrandt?
@ashwin-pc @ahopp Thanks for responding, and sorry for not reaching back out until now.
RE: Saved searches I disagree with waiting to have this feature because I agree with this statement.
This will help the user use this tool as a drop in replacement for the existing visualization plugin.
Since we all agree with this statement above perhaps we can figure out what challenges are causing this feature to get cut from the design that is presented here. If it's not there from the start you've cut away half of the capabilities from all the other visualizations and makes adoption much harder for end users. From experience, using saved searches in visualizations has been relatively easy and straight forward. For example, the only difference from a search standpoint between using just an index pattern is that the SearchSource making the request should include the SearchSource of the saved search as its parent.
I honestly think visual parity will present more challenges than saved searches and I'm confident that we can figure something out to have saved searches right away.
RE: Visual parity Ah ok that explains it. My response to this was related to the design saying this tool will use EUI charts to have visual parity. I get why the design would want visual parity but was confused by that statement which is why I was pointing out that using EUI charts doesn't really achieve visual parity and was wondering if there was some other reason I was missing for why it was going to be used. Thanks for clearing that up.
RE: Saved Searches
@JacobBrandt yes, saved searches should in theory be relatively straightforward. I'll admit that I've limited experience with it which could also be the reason for most of my concerns, however there are two challenges it introduces with our current design that I would ideally like to solve before introducing it to the experience.
After reading your response and thinking about this a bit more, I believe that both of these concerns are relatively minor and can easily be solved. I think the question now is how important is this feature and should it be prioritized over some of the other features mentioned in the design. @ahopp what are your thoughts?
@ashwin-pc @ahopp One thing that is not clear here is how long external contributors will have to wait before providing enhancements and modifications to the baseline. As I was talking with another colleague about this I realized that this is the reason why I've been concerned about saved searches and brought it up.
I might be misunderstanding this, but right now it seems like external contributors would have to wait until this design, as outlined, is finished. I say this because the design and the team talks about all these things being necessary in the first iteration/initial version and other future work, including saved searches, could come later when it's complete. I take that as everything as outlined in the design will become the first publicly accessible baseline to end users and external contributors. There's a lot there to try and accomplish in one iteration before external contributions. Ideally, I'd rather have a more iterative approach with the community where the generic abstractions get pushed in place sooner rather than later so that external contributors don't have to wait. Thus mitigating the risk of waiting indeterminate amount of time to help with development and provide additional capabilities like saved searches.
@ahopp You mentioned could saved search be a good place to plug in. Yes it could, but because of the unknowns I mentioned above I was concerned about when that would be possible? If contributors don’t have to wait for everything as outlined in the design to be finished than this sounds like a great way to implement this feature and we could move forward that way.
@JacobBrandt I see your concern. You are right about the end user having to wait the design to be completed, but external contributors will not have to wait for that. We will be opening a branch on the Dashboards repo soon for the design and will have all the development happen directly on it so that it is accessible external contributors immediately. We will only hold merging it to the mainline until we are confident that we have an experience that we are comfortable sharing with users, but this should not limit contributors from either playing with it or contributing to it right away. We will also be opening the issues for each feature here. Does this alleviate your concern?
@ashwin-pc Perfect. That all sounds great. Sorry for my misunderstanding.
In the list of Available Fields to select from it would be extremely useful to replicate the Popular section as used on the Discover tab:
This is an extremely useful feature for both novice and advanced users as it helps them easily locate the fields they require most often very quickly and easily.
My comments/questions:
Should we prioritize converting visualizations to work between the current visualizations experience and the proposed experience
- My short answer would be yes. If our long-term plan is to
Replace the most existing visualization tools and make this the default
(also, what do you mean by "most"?) then I think migration options are required.
How can we provide a preview or beta.
- @kgcreative has mentioned wanting to work on defining how to release a "preview" or "beta" features and what the process should be for them. I'm happy to contribute as well. It'd be great to have a proper mechanism to gather feedback from the community and iterate over time (while still remaining backwards compatible).
Should this be a part of OpenSearch Dashboards, the Dashboards-visualizations plugin, or be a separate plugin
- If the goal is to have this become the default option then wouldn't it be best for this to exist in the Dashboards repository? I think that's what you're going for here, but just wanted to confirm.
Can we create a meta issue that links each of the remaining issues to track the work that needs to be done and provides the community with a way to view progress and pick up work themselves? I know the label exists but I think it would be worthwhile to have a dedicated issue.
Sure! @tmarkley is there a example issues you can share that we can use as a template to what you'd like to see?
Can we clarify what features we want to be in and out of scope for the initial release (this could be as simple as two lists)?
I think you're describing a list of in-scope features plus a backlog? If so, I think we can do something like that. There are few features we'd be happy to add for the initial release if someone wants to build them, but are prioritizing other features with available resources so the backlog might be more like aspiration P1/P2 features rather than a traditional backlog. Where would you want to see this? The meta issues?
It'd be great to have a proper mechanism to gather feedback from the community and iterate over time (while still remaining backwards compatible).
100% agree. We should propose something soon.
Also as an FYI we're planning on closing the design on 12/17. We can obviously continue to iterate but I think the high-level design has been resolved and we're already working on development.
Sure! @tmarkley is there a example issues you can share that we can use as a template to what you'd like to see?
It doesn't have to be anything fancy, a prioritized list of tasks and regular status updates would be great. #920 is the first thing I thought of that was similar.
I think you're describing a list of in-scope features plus a backlog?
Yeah that would be great.
... if someone wants to build them ... Where would you want to see this? The meta issues?
Yes I think that would work to have it in the meta issue(s). I think we should make sure community members understand what is being prioritized and what else they can pick up and contribute to for the project.
Thanks for the feedback everyone. To summarize the conversation:
Background
Today, visualization and exploration in OpenSearch Dashboard is harder than it needs to be. Users need to preselect both the chart type and index pattern they will use to create a visualization. Additionally, users may need to navigate between multiple screens (e.g. discovery → visualize → discover → visualize, etc.) to find the right feature, chart, index to include in their dashboard. Not only is these more work then it should be, it prevents the serendipitous discovery that easy exploration allows. This document outlines the high level design recommendation for a visualization experience that makes exploring data and creating visualizations much easier.
Tenets
Out of Scope
Recommended Design
The recommendation is to create the Drag and Drop experience as a first party plugin within the Opensearch Dashboards repository (i.e. explain to layman). It will act as an additional way to create visualizations alongside the exiting tools within the current visualizations plugin. The tool will be incremental to the visualization tools available to the user in OpenSearch Dashboards today. Due to the large nature of the design, I will divide it into 4 major sections:
While we considered many options for each of our choices, we ultimately recommended this approach so that we could have minimal disruption to existing users workflow while also providing a seamless experience to the users who wish to create visualizations easily. It also lets us iterate and evolve over time.
The user flow
There are 3 main ways the user can reach the new tool.
For each of these flows the user is navigated to a full page view of the drag and drop tool that lets the user create or edit the visualization. After creating the visualization, the user can either choose to save the visualization or add the visualization directly to the preferred dashboard (existing or new).
Alternatives
Tool features
This tool features:
Architecture
Workflow
The app uses the fields selected or dropped into the chart context by the user to determine the best aggregation logic (outlined in a later section) to generate the appropriate DSL Query(s) and fetch the data to render the visualization.
Saved Object & Embeddable
The tool will introduce a new saved object type and embeddable to render its visualizations. This means that existing visualization will not be compatible with the new tool and separate factory. This is done to prevent breaking existing visualizations while allowing new visualizations to be created easily without much overhead.
Saved object structure
The one big drawback to this approach is not having the ability to edit existing visualizations using the new experience. Since the existing experience has a lot of visualization types that it already supports and a lot of customizations that we will not support for the initial launch and in future, reusing the existing schema will be rather tedious.
If the ability to edit existing visualizations is important, we can optionally provide an api/interface to selectively convert or migrate exiting and compatible visualization saved objects to the new saved object type. This will duplicate the existing visualization to the new format so that if there are any issues with the migration, the existing visualization remains unaffected. This approach also allows us to selectively enable migration for the visualizations that can be safely supported.
Alternatives: Reuse existing visualization schemas: This approach lets us keep the existing schema and all new visualizations created using the new experience uses the existing
visualization
saved object type and embeddable. Supported visualization types will open the new experience when edited while the old experience is used for unsupported visualizations. This allows existing visualizations to use the new experience easily and allows wider adoption with no migration required, but not all options supported by the existing experience will be supported in the new experience. This may break workflow for current users. Additionally, this would break existing visualizations when edited and some existing visualizations may never be supported by the new experience (e.g. Vega). Lastly, inheriting a schema that is not optimized for the current experience may limit future expansion.Charting Library
The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.
While the default visualization types use EUI, there is no restrictions on what charting library can be used since each visualization type will be responsible for rendering itself and defining the fields and config types it supports it need to render correctly. In future this method can be used by other plugins to register their own visualization to exist alongside the default visualization types.
Drag and Drop
For the initial release there will only be 2 primary drop targets:
The field cards can be dragged from the Data panel into either drop target and the page will automatically provide context to the user based on where the user is dragging the field.
Based on the dropped region an appropriate aggregation strategy will be used to fetch the data and render a visualization is possible. When a particular field cannot be dropped into a particular region, the drop cue does not display.
Aggregation Logic
One of the key problems we will need to address with the tool is how different interactions render a corresponding visualization that is both useful and makes sense to the user. Each existing visualization tool out there today does this in their own way and have their own strengths and drawbacks. To simplify this problem for our initial release and to get better user feedback on the approach we need to take here, my recommended is to allow the user to only be able to drag and drop a field into the canvas for the first field they wish to visualize. The Visualization generated will be based on a simple lookup below.
The Visualization Config panel will update with the appropriate fields and aggregation functions and the Visualization canvas will no longer act as a drop target. All subsequent fields need to be added directly to the Visualization Config Panel which assumes an appropriate aggregation function for the Chart axis and field type (similar to the existing visualize tool today). The User will also still be able to configure the visualization by Dragging and dropping fields directly onto the Visualization Config Panel. This approach makes the initial discovery of the data easy while keeping the reason for why a particular visualization was rendered intuitive.
Alternatives
The recommended solution is a meets both these approaches in the middle.
Future scope
Open Questions