[Proposal] Consolidated Dev Tools

[brijos]

Is your feature request related to a problem? Please describe.

There are multiple ways to query OpenSearch within OpenSearch (e.g. Dev Tools, Query Workbench). To make it less complicated and drive adoption for a variety of querying methods offered in OpenSearch, we want to build a central location to make querying for users in OpenSearch easy.

Describe the solution you'd like

To have one querying workbench experience where users can query OpenSearch in the language that they are most familiar with.

• When troubleshooting an OpenSearch cluster or data within the cluster, an admin wants a central place to quickly troubleshoot and develop queries using languages they are familiar with, so that they can quickly find out what is going on with their system.
• When learning about a new query interface, users will want to use a search and results interface which feels similar to other query interfaces in the tool, so that they don’t have to learn a new way of doing things.
• When revisiting a query interface, an user wants to select/save x previously run queries, so that they don’t waste their time recalling popular queries.
• When someone from compliance wants to see what queries have been run on OpenSearch, a user of OpenSearch wants to be able to show what queries have been run.
• When coming upon schema structures which are not familiar, an user will want to explore the possible data structures, so that they can build the right query with confidence.
• When constructing new queries, a novice user will want quick access to documentation/examples/help on how to construct queries, so that they can learn as they go and feel confident in the project.
• When constructing a query incorrectly, all users want to be told why the query isn’t executing, so that they can fix their mistakes and move on (e.g. EXPLAIN)
• When constructing queries, performance conscious users and cluster administrators want to understand how the query could be written more efficiently, so that they can avoid taking up critical cluster resources.

Describe alternatives you've considered

Leaving the different ways to query OpenSearch as is.

Additional context

Mockups to come soon.

[kavilla]

Is the proposal removing the console plugin in favor of query workbench? Or enabling the console plugin to utilize SQL? Or nothing to do with the query workbench plugin.

Right now I would say the dev tools really lets you hit the REST APIs of OpenSearch and OpenSearch Dashboards and copy those as a curl if wanted.

[dtaivpp]

@kavilla I think it is just trying to remove some of the duplicated efforts. Right now, I think there are:

• Discover
• DevTools
• Search Relevance
• Query Workbench

That can all issue queries in different formats. What I could see is a single place where people can experiment with any of the N ways to query. It should probably support autocomplete to help people get more familiar with the languages easier. I'm not too opinionated on where it ends up landing but like you mentioned I think we should make sure we don't limit the tools that we currently have like devtools but rather just make them all easier to find and switch between.

[brijos]

Exactly what @dtaivpp said. This is a consolidation effort to made the various tools more findable and usable.

[dcarbone]

It would be really nice if any incarnation kept the ability to write rest queries directly without the need for interpolating what I want into yet another query language. I can definitely see the query workbench being useful to others, perhaps those used to working with graphql and the like, but for old folks like me rest + json is simple and works.

I do wonder if perhaps it may be better to try to leverage a tool like postman. Perhaps provide a way to export an openapi schema file that I can import as a collection into a more purposeful tool, thus removing the onus of maintaining this section?

[dtaivpp]

I think that would be great (generating the postman collections from spec that is). I've though this would be great as well.

We do have the opensearch cli which I think was supposed to serve a similar function but it hasn't seen the adoption I'd like. I feel like everyone is really familiar with postman though so that would probably get a ton of traction.

The thing I really like about our web interface for doing this is the auto completion and how it can work kinda like a notebook.

[dcarbone]

the opensearch cli tool works very well, but from personal experience it has some shortcomings:

1. can easily list profiles by name, but cannot view actual profile config
2. cannot edit a profile
3. possible paths in curl command are not hintable
   • this could potentially be helped with the availability of an openapi-like schema file that could be loaded on-demand to produce autocomplete hints.
   • also i recognize this is a massive unit of work to implement. just saying it'd be nice for newcomers :)
4. cli tool is not readily available in any repos, including the rpm repo maintained for the actual opensearch service itself

those are just the ones i've run into recently. i do appreciate its existence, however :)

i'd also like to point out that simply rectifying point 4 would be really nice. in particular, i'd like to be able to easily embed the cli tool in container images for other developers to use locally in a cross-arch environment (mixed x86 and arm), without having to fuss with downloading it manually.

[seraphjiang]

Dev Tools

As a OpenSearch Developer, I'd like to keep all existing Dev Tool functionality, that is primary tool people use to learn and debug OpenSearch Query. I'm not sure if people has need to learn other query language.

As a developer, when I learn OpenSearch API, DSL, dev tool is my best bet. Other query tool or query lanauge are nice to help.

Learn how to use SQL/PPL might only use for query some data, it doesn't help to learn and debug OpenSearch API. How many OpenSearch user will use SQL/PPL to run POST/PUT/DELETE api , which are important to ingest/manage data.

Admin persona might represent some usage, and scenario, but could not cover major use case for me to make decision to have one centralized tool to replace existing tools. Love to hear more input.

Observability (Log Analytic)case

I saw there is overlap between Discovery app and Workbench , but i'd like to have more comprehensive side by side functionality comparison between these two to understand their existing usage, and what are required if we do want to merge, what could be removed.

Search Relevance

The need and use cases for Search are very different compare to Log Analytic. @macohen might provide more input.

@AMoo-Miki @seanneumann @anirudha @elfisher @ahopp @kgcreative @zengyan-amazon @macohen @xinlamzn @dblock

[brijos]

Made updates to extend the user stories based on users and Admins. The initial focus was admins, but power users will also benefit from having a single location to query from.

[macohen]

For Search Relevance:

• Search Comparison Tool As a search relevance engineer, I want to compare different search results given a user query as a variable and two different sets of QueryDSL. (this is the search comparison tool today in the Search Relevance plugin)
• Do we have any research or users interested in SQL or PPL for search applications? My hypothesis is that for search applications most users are using QueryDSL, would be interested in SQL, and probably not so into PPL.

As a search relevance engineer, I want to tune search queries over multiple sessions, saving them, and send them to others for their input.

Human Judgement Tools

• As a search relevance engineer, I want to rate a set of search results for a given query as good or bad so I can save that rating and use it for input into manual tuning of results or automated models for ranking.

These are basic; @macrakis, do you have more to add?

[macohen]

Can we look at any consolidation from the perspective of the user? A search relevance engineer is going to want a different set of tools than someone monitoring systems and applications, but there is overlap in the usefulness of each tool. I can imagine a set of plugins that are great for observability only, some that are great for search and some that work for both.

[vagimeli]

See PRs:

• https://github.com/opensearch-project/documentation-website/pull/4874

Doc team creates the Dev Tools content when doc issues are created. Please prioritize the topics and add me to the doc issues as the point of contact.

[ashwin-pc]

Just want to add here that we are confusing devtools with console. Devtools is simply a wrapper plugin that allows a plugin developer to register any developer tool under it. Right now only the console plugin makes use of this registration, but was built with the intention that more plugins would do the same. The easiest approach here might just be to have all the other dev tools related apps like search relavance and query workbench to do the same.

here is where the console application registers itself as a dev tool: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/console/public/plugin.ts#L58C1-L87

[kgcreative]

Yes -- i think the idea here is not to combine all these tools, but rather to make them all available as tabs in devtools. Devtools I think was intended to house all the different tools that devs and folks working more specialized advanced tasks could need. To me, Devtools housing Console | Query Workbench | Search Comparison makes sense, as those are intended to help developers in their day to day. This is orthogonal to Log Explorer + Discover being merged into Data Explorer. I wouldn't equate data explorer with dev tools. (If that makes sense)

[macohen]

For Search Relevance, the goal is to expand this beyond what I now think I understand are developer tools. Thanks @ashwin-pc. The audience may be non-developers rating results, interest in search analytics, as well as relevance tuning, A/B testing, more. Ideas in the space need to evolve, but I would not want to move the comparison tool in and out of dev tools.

[macrakis]

Agreed with Mark. Many business users and product managers who don’t write queries will want insight into the performance of search. Search Comparison is just the first (small!) step in that direction.

Cf. Aparna’s different personas.

From: Mark Cohen @.> Reply-To: opensearch-project/OpenSearch-Dashboards @.> Date: Thursday, September 14, 2023 at 13:44 To: opensearch-project/OpenSearch-Dashboards @.> Cc: Stavros Macrakis @.>, Mention @.> Subject: Re: [opensearch-project/OpenSearch-Dashboards] [Proposal] Consolidated Dev Tools (Issue #3033) Resent-From: @.>

For Search Relevance, the goal is to expand this beyond what I now think I understand are developer tools. Thanks @ashwin-pchttps://github.com/ashwin-pc. The audience may be non-developers rating results, interest in search analytics, as well as relevance tuning, A/B testing, more. Ideas in the space need to evolve, but I would not want to move the comparison tool in and out of dev tools.

— Reply to this email directly, view it on GitHubhttps://github.com/opensearch-project/OpenSearch-Dashboards/issues/3033#issuecomment-1719882500, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAGO6CX7XTGQUZX72VW43MTX2M65HANCNFSM6AAAAAASU2FWRE. You are receiving this because you were mentioned.Message ID: @.***>

[kgcreative]

I'm ok with Search Comparison staying together with other search relevance tools, as this will quickly outgrow the devtools context

[AMoo-Miki]

As a non-tech user, complicated interfaces with a ton of tabs and knobs confuse me; in fact, too many options are entry barriers for people like me. Discover is that uncomplicated interface for the likes of me to just get the data we need; I am with @seraphjiang on that.

Plugins, on the other hand, are free to create specialized interfaces or much more complex multi-usecase interfaces, even duplicating parts of existing OSD functionality.Discover cannot afford to do that because it would just alienate everyone who needs simple easy-to-use interfaces.

With the Dev Tools, like @ashwin-pc said, plugins can choose to add themselves there and the Console, being targeted at technical folks, can enrich itself with richer UX but complicating Console with numerous querying techniques will need a spectacular UX to not be a pain to use.

[kgcreative]

I propose we close this issue for now.