[Discuss] Improve "New visualization" menu to differentiate between types of visualization building methods

[KrooshalUX]

Is your feature request related to a problem? Please describe.

The current UI for 'new visualization' is not scalable to account for different visualization building interfaces. Currently, it is designed to accommodate individual visualization types (bar, area, pie, etc). Until there is one unified build experience that encompasses different data sources and/or query langauges, we will need to address this when presenting users with a menu of choices within the UI (what are you trying to build and how do you want to build it (index pattern or PPL query or SQL, etc). VisBuilder is the first change to this menu, but there are more coming with deep links into Observability visualization build methods.

Describe the solution you'd like

A redesign of the New visualization modal that clearly communicates 1) build method and potentially 2) visualization output. I put emphasis on potentially because the bonus challenge here is that VisBuilder and Observability visualization methods allow the user to swap between visualization types within the experience. Ideally we would survey some users on what they might expect to see when clicking "Create visualization" - do they already have a chart type in mind? do they want to see what best represents the data they are querying? I think some of this may have been investigated as part of VisBuilder previously.

Note - as stated above, my long term desire/recommendation would be to provide users with one UI for building visualizations regardless of data source/ query language.

Additional context

I would like to hear from @anirudha @joshuarrrr @ashwin-pc specifically about requirements around this potential redesign from their perspectives

[ashwin-pc]

Firstly, love the idea. A few things i want to make sure we keep in mind. We already have 2 types of Visualizations that we can register here:

1. Visualizations that use the Visualize App (Everything in core except VisBuilder uses this, even Vega and Timeline)
2. Aliases (They just show up as icons in the picker but redirect the user elsewhere). VisBuilder uses this.

On the UI right now there is no way to distinguish between the two and I think thats for the good. So now the challenge becomes, how do we add more to this list while still being able to group them is a meaningful way.

One approach that i can think of is to add group and priority to the API. By default all Visualizations that dont have this will behave like they do today and be grouped as such, but you should be able to add any visualization type to either a predefined set of groups or no groups. And the new UI should be able to handle a variable set of visualizations.

[joshuarrrr]

I'm excited we're now ready to tackle this UX flow. There are a couple of different lenses to view the problem:

1. When a user wants to create a new visualization, which choices do they need to make before they can begin exploring and building?
2. How can we avoid decision paralysis for users who don't want to make any choices?

For question 1, current state is that users must either:

1. Choose a visualization type to create in the "default editor" (aka legacy) experience. This editor additionally requires choosing a data source before navigating to the creation screen:
   1. Area
   2. Coordinate map
   3. Data Table
   4. Gannt Chart
   5. Gauge
   6. Goal
   7. Heat Map
   8. Horizontal Bar
   9. Line Chart
   10. Pie
   11. Region Map
   12. Tag Cloud
       2. Choose an alternative visualization authoring experience (or "build method", as @KrooshalUX put it)
   13. Vega - a highly flexible, general purpose tool for text-based (via Vega/Vega-Lite schemas) visualization authoring
   14. VisBuilder - intended to replace the default editor, with a focus on data exploration and easy switching between chart types
       3. Choose a specialized visualization authoring experience based on primary data modality
   15. Maps - a new and improved geospatial authoring experience with many features. While it supports many types of layers, it is also capable of the following legacy visualization types
   16. TSVB - intended to provide an easier-to-use UI to author time-series visualizations
   17. Timeline - intended to provide a text-based (via timeline expressions) alternative for authoring time-series zvisualizations
       4. Choose a non-visualization "type"
   18. Controls - An alternative UI (can be embedded as a dashboard panel) for specifying filters
   19. Markdown - A general purpose tool for adding static HTML content to dashboard panels

Of course, in reality, it's hard for any but the most experienced users to understand that the UI is mixing all of the above into a single interface. In addition, despite all their differences. In https://github.com/opensearch-project/oui/issues/637, the new visualization types don't necessarily even fit into the 4 categories above, because they're primarily distinguished by the data query language.

From the technical end, I have a few suggestions that I think may help us navigate the near future where we still have a heterogeneous mix of visualization creation experiences

1. We need more metadata fields than "chart-type" which is currently insufficient and overloaded. We should allow tools like VisBuilder to identify as a visualization editor, and also to list it's capabilities - the visualization types, data sources, and query languages it supports. This gives the UX team flexibility to adapt and improve the flow incrementally, and even provide a more guided experience.
2. We have an overall plan to consolidate visualization creation. But, as part of that journey, we can provide more interoperability between experiences, to reduce the cost of a reversed decision.
3. We can incorporate data source selection into the default editor. While this may seem like wasted work, we already have a proof-of-concept PR: https://github.com/opensearch-project/OpenSearch-Dashboards/pull/1132, and it would allow UX to focus on the existing modal and flow without the additional hurdle of the second modal screen for selecting a data source.

[anirudha]

Let's get a mock ux for this in place. Even if it's a napkin mock for clarity

[kgcreative]

Mocks are on our radar, but @anirudha , it would help to hear your thoughts and some of the needs from your team before we produce that.