search

opensearch-project / documentation-website

The documentation for OpenSearch, OpenSearch Dashboards, and their associated plugins.
https://opensearch.org/docs
Apache License 2.0
67 stars 466 forks source link

[PROPOSAL] Overall Documentation Structure/Restructure #419

Closed ahopp closed 1 year ago

ahopp commented 2 years ago

Background

Since the launch of OpenSearch, there has been some great progress in documenting the project. So far, the Open Distro documentation has been updated and moved to OpenSearch, and there have been continued additions to content coverage on OpenSearch plugin features, OpenSearch search, fundamental OpenSearch language clients content, OpenSearch APIs (e.g., CAT, bulk, DSL), and other ad hoc content based on basis address community requests (e.g., updating security content, updating docker configs, and more).

Despite the significant progress, we haven’t established a plan we can work backwards from specific set of needs. Additionally, the current documentation is still structured in a manner better suited for a peripheral product (e.g. Open Distro).

Proposal

To resolve this, I am recommending an update to the basic hierarchy of content as well as the overall content subjects I think should be covered by the documentation site. In this proposal I tried to consider the following:

While I framed this as a table of contents and described what sections I think we should have in our documentation, I also want to acknowledge that there are many more decisions to be made (e.g. how we title the content, what interlinks we have, what navigation looks like throughout, etc.). Those topics are important and need to be discussed. However, the structure and documentation content seemed to be the most composable, logical place to start. Once we feel good about the initial organization schemes and structures, we should review proposals for labeling, navigation, search, etc.

Finally, while there is some importance in ordering (e.g. overview should probably be high on the list), I’ve focused on coverage for this proposal. I think we can make efforts optimizing the ordering once we have aligned on the content.

Recommended Structure

Below is the general proposal for the section and structure of the content needed for the OpenSearch project.

What are remaining open questions? While I tried to focus on what documentation users would be looking for based on feedback on the forums, GitHub, and looking at examples of other OSS projects, I know there is still a big opportunity for optimization. Please poke holes, provide feedback, identify gaps, and make proposals! From there we can use this issue as a living roadmap we all can add to and plan our documentation efforts.

ashwin-pc commented 2 years ago

Amazing breakdown! I don't see much here about development docs. Will that be covered separately in the OpenSeach and OpenSeach dashboards developer guides?

ahopp commented 2 years ago

Amazing breakdown! I don't see much here about development docs. Will that be covered separately in the OpenSeach and OpenSeach dashboards developer guides?

Great callout! I think it would make sense to add them in the developer guides but we can also link them in the main overview. What do you think?

Naarcha-AWS commented 2 years ago

@ashwin-pc: Still learning everything of course. Could you describe the distinction between the overall OpenSearch documentation and the "development docs". I assume the development docs you are referring to are these: https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#developer-guide.

If so, should we make a distinction between "Development" and "Contribution"? Either way, adding the links to the various guides in the main overview make sense to me. I think we could extend that principle to all external links.

Naarcha-AWS commented 2 years ago

On the troubleshooting section: 

Troubleshooting: This section will focus on resolving common challenges such as troubleshooting configuration issues and fixing security issues. Topics in this section include;

    Common issues
    securityadmin.sh
    TLS
    SAML
    OpenID

I'm conflicted about making that section it's own. Would it be better to have individual "Troubleshooting" subsections within each section. For example, the security plugin docs could include a page for troubleshooting TLS, SAML, and security admin.

Furthermore, we could make "Common Issues" it's own top-level page, to account for issues that might not fit into the Use Case sections.

Any thoughts?

ashwin-pc commented 2 years ago

@ashwin-pc: Still learning everything of course. Could you describe the distinction between the overall OpenSearch documentation and the "development docs". I assume the development docs you are referring to are these: https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#developer-guide.

If so, should we make a distinction between "Development" and "Contribution"? Either way, adding the links to the various guides in the main overview make sense to me. I think we could extend that principle to all external links.

By dev docs I meant documentation that new contributors can use to familiarize themselves with the internals of our systems. Looking at Andrews list, I see a lot that helps new users understand OpenSeach but for new contributors, there are still a lot of gaps.

For example, for OpenSeach Dashboards the only section I see is a link to the dev guide. Which makes sense for an OpenSeach user. But if someone wishes to contribute to Dashboards, the existing documentation is quite sparse. And the developer guide doesn't do much more than give us setup instructions and the style guide to follow.

My comments are biased to Dashboards but I think the same applies to OpenSeach.

ashwin-pc commented 2 years ago

Amazing breakdown! I don't see much here about development docs. Will that be covered separately in the OpenSeach and OpenSeach dashboards developer guides?

Great callout! I think it would make sense to add them in the developer guides but we can also link them in the main overview. What do you think?

I might be wrong, but I don't think we have a good developer guide. In dashboards we have a single developer doc and that doesn't say much. It's just installation instructions and a style guide. We need a whole new set of docs for that. Ideally generated directly from the code.

mattweber commented 2 years ago

IMO, it would be great if docs could be moved into main repo like Elastic does it. It is very useful to have docs live with the code so that contributors can add/update/remove the relevant docs in the same pull request. It can be a requirement that changes are documented before a pull request is merged.

ahopp commented 2 years ago

IMO, it would be great if docs could be moved into main repo like Elastic does it. It is very useful to have docs live with the code so that contributors can add/update/remove the relevant docs in the same pull request. It can be a requirement that changes are documented before a pull request is merged.

This is interesting. I feel like making /doc (or /docs) a top-level directory in the "main" repository a good mechanism to signal the importance of documentation but I'm not sure how it would work given the distributed nature of the repos in the OpenSearch project specifically.

@CEHENKLE @dblock what are your thoughts?

ahopp commented 2 years ago

I'm conflicted about making that section it's own. Would it be better to have individual "Troubleshooting" subsections within each section. For example, the security plugin docs could include a page for troubleshooting TLS, SAML, and security admin.

Furthermore, we could make "Common Issues" it's own top-level page, to account for issues that might not fit into the Use Case sections.

Any thoughts?

My first instinct is why not both? We can have a general troubleshooting section where people can go to in addition to individual "Troubleshooting" subsections within each section. The content only needs to be written once and can them be discovered in either use flow (e.g. problem -> troubleshooting -> specific topic OR problem -> specific topic -> troubleshooting).

ahopp commented 2 years ago

I might be wrong, but I don't think we have a good developer guide. In dashboards we have a single developer doc and that doesn't say much. It's just installation instructions and a style guide. We need a whole new set of docs for that. Ideally generated directly from the code.

This makes sense to me. The question then becomes: Where should they live in the above structure?

keithhc2 commented 2 years ago

My first instinct is why not both? We can have a general troubleshooting section where people can go to in addition to individual "Troubleshooting" subsections within each section. The content only needs to be written once and can them be discovered in either use flow (e.g. problem -> troubleshooting -> specific topic OR problem -> specific topic -> troubleshooting).

So the general troubleshooting section would just link off to the individual troubleshooting sections, where people can find the actual content?

Naarcha-AWS commented 2 years ago

I might be wrong, but I don't think we have a good developer guide. In dashboards we have a single developer doc and that doesn't say much. It's just installation instructions and a style guide. We need a whole new set of docs for that. Ideally generated directly from the code.

This makes sense to me. The question then becomes: Where should they live in the above structure?

Maybe a top-level section labeled "Contribution".

Contribution: This section provides instructions and best practices for developing the OpenSearch project.

keithhc2 commented 2 years ago

By dev docs I meant documentation that new contributors can use to familiarize themselves with the internals of our systems. Looking at Andrews list, I see a lot that helps new users understand OpenSeach but for new contributors, there are still a lot of gaps.

For example, for OpenSeach Dashboards the only section I see is a link to the dev guide. Which makes sense for an OpenSeach user. But if someone wishes to contribute to Dashboards, the existing documentation is quite sparse. And the developer guide doesn't do much more than give us setup instructions and the style guide to follow.

Ultimately, this is going to depend on how we define what the docs site's purpose is. This is how we're currently organizing the information:

Repos:

What is this thing? How do get started with development and build it from source? How do I contribute changes or bug fixes?

Documentation:

What is this thing? How do I install it? How do I get started with it? Reference stuff: supported operations, configuration, tuning, best practices, REST API, etc.

So if we want to include developer guide content in OpenSearch documentation, then I'm not sure what information we'd put in the repos themselves, and we don't want duplicated information anywhere. My thinking is if first-time contributors are looking to contribute to the projects, then they're on the repos already, so why not stick the developer guide information like how to contribute and more in-depth explanations of systems in the repo's README? If the docs site's purpose is to document how to use OpenSearch, then it doesn't make much sense to document how to contribute to OpenSearch on the docs site.

Naarcha-AWS commented 2 years ago

By dev docs I meant documentation that new contributors can use to familiarize themselves with the internals of our systems. Looking at Andrews list, I see a lot that helps new users understand OpenSeach but for new contributors, there are still a lot of gaps. For example, for OpenSeach Dashboards the only section I see is a link to the dev guide. Which makes sense for an OpenSeach user. But if someone wishes to contribute to Dashboards, the existing documentation is quite sparse. And the developer guide doesn't do much more than give us setup instructions and the style guide to follow.

Ultimately, this is going to depend on how we define what the docs site's purpose is. This is how we're currently organizing the information:

Repos:

What is this thing? How do get started with development and build it from source? How do I contribute changes or bug fixes?

Documentation:

What is this thing? How do I install it? How do I get started with it? Reference stuff: supported operations, configuration, tuning, best practices, REST API, etc.

So if we want to include developer guide content in OpenSearch documentation, then I'm not sure what information we'd put in the repos themselves, and we don't want duplicated information anywhere. My thinking is if first-time contributors are looking to contribute to the projects, then they're on the repos already, so why not stick the developer guide information like how to contribute and more in-depth explanations of systems in the repo's README? If the docs site's purpose is to document how to use OpenSearch, then it doesn't make much sense to document how to contribute to OpenSearch on the docs site.

That is in line with how other Open Source tools operate, keep contribution instructions in the code base where contribution occurs. I do think there is value in having a hub in which we route users towards our repos though. For example, Elastic has a Community Contribution Program, a hub for users which to contribute to Elastic. They gate their instructions behind a web app that requires you to sign-up, which I don't love. However, the page and app does encourage contributors by promising recognition for those that make significant contributions.

ashwin-pc commented 2 years ago

Ultimately, this is going to depend on how we define what the docs site's purpose is. This is how we're currently organizing the information:

Repos:

What is this thing? How do get started with development and build it from source? How do I contribute changes or bug fixes?

Documentation:

What is this thing? How do I install it? How do I get started with it? Reference stuff: supported operations, configuration, tuning, best practices, REST API, etc.

So if we want to include developer guide content in OpenSearch documentation, then I'm not sure what information we'd put in the repos themselves, and we don't want duplicated information anywhere. My thinking is if first-time contributors are looking to contribute to the projects, then they're on the repos already, so why not stick the developer guide information like how to contribute and more in-depth explanations of systems in the repo's README? If the docs site's purpose is to document how to use OpenSearch, then it doesn't make much sense to document how to contribute to OpenSearch on the docs site.

So where would the details for building say a plugin go? To build a plugin today all we have is a blog post. But it does not for example contains the details of how to generate one, what are the different functional libraries available to a plugin developer, different key concepts that one should know before building one etc.

I guess what i am looking for in the documentation is something like this: https://developer.android.com/docs

Its not really a guide on how contribute to Android for example but how to build on top of it. And most of the contents are also very relevant to someone who's contributing.

The actual style guide and contribution specific details can and should remain in the repo, but guides, api, and other core concepts make more sense here

keithhc2 commented 2 years ago

Ultimately, this is going to depend on how we define what the docs site's purpose is. This is how we're currently organizing the information: Repos: What is this thing? How do get started with development and build it from source? How do I contribute changes or bug fixes? Documentation: What is this thing? How do I install it? How do I get started with it? Reference stuff: supported operations, configuration, tuning, best practices, REST API, etc. So if we want to include developer guide content in OpenSearch documentation, then I'm not sure what information we'd put in the repos themselves, and we don't want duplicated information anywhere. My thinking is if first-time contributors are looking to contribute to the projects, then they're on the repos already, so why not stick the developer guide information like how to contribute and more in-depth explanations of systems in the repo's README? If the docs site's purpose is to document how to use OpenSearch, then it doesn't make much sense to document how to contribute to OpenSearch on the docs site.

So where would the details for building say a plugin go? To build a plugin today all we have is a blog post. But it does not for example contains the details of how to generate one, what are the different functional libraries available to a plugin developer, different key concepts that one should know before building one etc.

I guess what i am looking for in the documentation is something like this: https://developer.android.com/docs

Its not really a guide on how contribute to Android for example but how to build on top of it. And most of the contents are also very relevant to someone who's contributing.

The actual style guide and contribution specific details can and should remain in the repo, but guides, api, and other core concepts make more sense here

Right, and that's why I'm saying we should decide what the purpose of the OpenSearch docs site is, and whether that should just be how to use OpenSearch VS everything OpenSearch-related. I would say that if users are looking at the plugin's repo to download, clone, do whatever with a plugin, they should be able to look at that repo's README to get the information they need, rather than jump through another hoop to the docs site.

I agree that guides and APIs should be in the OpenSearch documentation, but I think it makes more sense to have breakdowns on what libraries are available in a plugin/what to know when thinking about contributing to a plugin within the repo, where they would actually make contributions and where the code actually lives.

I like that Android development link because it educates people on how to use Android to build custom apps. But they don't mention anything about contributing to Android, which is at an entirely different link: https://source.android.com/setup/contribute

nateynateynate commented 2 years ago

I'd like to call out what I feel is a gap in our existing docs - In the situation where someone might be learning Docker and OpenSearch at the same time, I found the amount of guidance just a bit short. We should probably provide a generous dose of help for some basic docker familiarity. Some examples:

Installation and Configuration -

asafm commented 2 years ago

Regarding contribution - from someone who just enters OpenSearch as a developer (not a user), there is a huge gap of How OpenSearch Works: Internal Architecture / Design

A lot of excellent content in that area has been contributed as blog posts. Today to learn those fields, I have to google, find all sorts of blogs posts, and piece it together.

I think it makes sense to have a section in the docs about the architecture and design of OpenSearch.

Here is an example of an issue I opened which has been closed since there is not such place to add that content to: https://github.com/opensearch-project/documentation-website/issues/426

ahopp commented 2 years ago

@asafm I've added a "How OpenSearch Works" section to the proposal let me know what you think.

ahopp commented 2 years ago

I'd like to call out what I feel is a gap in our existing docs - In the situation where someone might be learning Docker and OpenSearch at the same time, I found the amount of guidance just a bit short. We should probably provide a generous dose of help for some basic docker familiarity. Some examples:

@nateynateynate I think we could include some of this in the install guides for OpenSearch via Docker (under "Installation and Configuration" section) and link out to Docker content for any command deep dives. What do you think?

ahopp commented 2 years ago

I agree that guides and APIs should be in the OpenSearch documentation, but I think it makes more sense to have breakdowns on what libraries are available in a plugin/what to know when thinking about contributing to a plugin within the repo, where they would actually make contributions and where the code actually lives.

I like that Android development link because it educates people on how to use Android to build custom apps. But they don't mention anything about contributing to Android, which is at an entirely different link: https://source.android.com/setup/contribute

@keithhc2 So, where are we landing on the structure? I can make an update if we feel we're aligned.

asafm commented 2 years ago

@asafm I've added a "How OpenSearch Works" section to the proposal let me know what you think.

@ahopp Looks good 👍

stockholmux commented 2 years ago

I really like this. It's long over due and it's solving some long-standing concerns I've had.

Overall feedback: There are a lot of sections. I think, as a user, I might get overwhelmed by this list. Laterally, I also don't like how several dissimilar things are is bundled together ("Analytics, Dashboards, and Machine Learning" seems especially potpourri). I would suggest an additional layer of organization. Maybe three big areas - "Setting up & running OpenSearch," "Using OpenSearch for your data" and "OpenSearch internals & contribution" and then a few other general ones (maybe Overview, external links, API reference).

Other nits and picks:

"Security" section: I would definitely break this up. This seems to cover the footprint of the security plugin which bundles a lot dissimilar functionality. I would say that access control needs its own category. Encryption might also be a good area to cover. (SSL is listed as a query language which is amusingly incorrect).

"Search and Query" section: needs work. SSL is encryption - should it be PPL? No mention of DQL. SQL is listed twice.

"Clients and tools" section: break this up. Languages and clients are apples and oranges. I think this historical more than anything.

nateynateynate commented 2 years ago

@nateynateynate I think we could include some of this in the install guides for OpenSearch via Docker (under "Installation and Configuration" section) and link out to Docker content for any command deep dives. What do you think?

Perfect. I sure would like to not have any reason to have to visit a different website to get the practical information needed. I'll take it.

ashwin-pc commented 2 years ago

@asafm I've added a "How OpenSearch Works" section to the proposal let me know what you think.

Shouldn't we have a similar section for dashboards?

Btw this is what I was referring to when I mentioned developer docs being different from the developer guide on the repository

brijos commented 2 years ago

"Clients and tools" section: break this up. Languages and clients are apples and oranges. I think this historical more than anything.

I agree. Below is how I see it.

Clients: This section will provide light documentation and links to download clients. Clients include: Python, JavaScript, Java, etc.

Tools: This section will provide documentation for tools that are part of the OpenSearch project. Topics in this section include:

OpenSearch CLI Ingestion tools (e.g. Data Prepper, Logstash, Fluentd, Tika, etc.) Data collection agents (e.g. Beats, Fluent Bit, etc.) Analytics tools (e.g. Grafana, Power BI, Tableau, etc.) OpenTelemetry OpenSearch Benchmark

dlvenable commented 2 years ago

This is a big change and there are lots of https://opensearch.org/docs links out on the internet. I think it would be valuable to support redirecting old links to new links. Perhaps not every link could get a redirect, but a lot of the major pages should. As a user of documentation, I find it highly frustrating to follow an old link and then have to rework my way into the correct page.

ahopp commented 2 years ago

support redirecting old links to new links

This is a great callout, thanks @dlvenable I believe there was previous work done to ensure there was redirecting from old docs (e.g., when the fork happened) so I imagine that can be extended to this restructuring. I'll add above to make sure we have it called out.

ahopp commented 2 years ago

I agree. Below is how I see it.

@brijos I've updated to reflect your input - let me know if that works!

passbt commented 2 years ago

I'm struggling to find any information on DQL in the documentation. When in the openseach UI in Discover page the default is syntax is DQL. If I click the DQL link in the search bar, I'm taken to this url, which gives me no relevant information that I'm aware of. It would be very helpful to link to a page with examples or update this page to include examples of how to use the DQL search bar. For now, I'm left searching using Lucene, because I find syntax documentation for Lucene.

I added my comment to this issue because I saw @stockholmux mention lack of DQL documentation in one of his comments.

ahopp commented 2 years ago

Thanks @passbt I've added it explicit under query languages as well as the Dashboards section!

ahopp commented 2 years ago

@macrakis I've moved k-NN and search plugins under "Optimization" (which itself is under "Search and Query"). What's your thought here - is this the right taxonomy?

Naarcha-AWS commented 2 years ago

Here's a potential reorder for the major side-nav topics, as discussed by @hdhalter and I.

Any thoughts?

dblock commented 1 year ago

I think "Developer Guides" should be "Contributing", be fairly high level, and we should kick any actual developer guides out of the doc and move those entirely in READMEs.

REST API Reference is probably just "APIs" and has a tree of APIs that are all versioned.

hdhalter commented 1 year ago

Hi all, thanks for all your thoughtful feedback! I invite you to take a look at the recent version of the documentation at https://opensearch.org/docs/latest/. We've not only created A TON of content over the last year, but have restructured the TOC. We've arranged the topics into logical categories based on use case, rather than plugin. We've also added a feedback mechanism where visitors can give us direct feedback on the content or structure. Feel free to give it a try.

But this is only the beginning! We have a lot more enhancements in store. We are continuing to add content where there are gaps, we are improving the content that is there, and we are further categorizing the content. Also, we will decouple the content that doesn't follow the OpenSearch versioning.

We are excited to continue to helping to make the community successful in using OpenSearch, and I'll keep this issue open as we continue to enhance the documentation.

ahopp commented 1 year ago

@hdhalter Do you think it's time for me to close this issue? Specifically since https://opensearch.org/docs/latest/ addresses the bulk of the proposal. I might suggest that it is time to close on the 2022 proposal and open a 2023+ proposal/plan.

Thoughts before I close?

hdhalter commented 1 year ago

Sure, @ahopp , I don't mind if we close this. Good idea to start a new one with the changes that are in flight and planned.