[Discuss] Central repository for all design documents

[Bukhtawar]

Is your feature request related to a problem? Please describe. Most of the times what is being designed is put up as an RFC but over the course of time designs do evolve and RFCs become stale. Today the design details are scattered over issues/PRs/blog posts and in some cases non-existent which makes contributions by new contributors all the more harder and equally intriguing for a maintainer to review changes

Describe the solution you'd like Have a directory inside the same repo to host all design documents in a single place and ensure same gets updated as designs evolve over time.

Describe alternatives you've considered Use documentation repo to document all design changes

Additional context Add any other context or screenshots about the feature request here.

[dblock]

This was brought up in https://github.com/opensearch-project/.github/issues/28 about a year and a half ago, you might be interested in that discussion.

[Bukhtawar]

I like the idea of wikis as well thats how some orgs like Kafka are managing it https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Kafkaasalong-termstorageservice

However I would prefer design docs and their implementations to be co-located so as to ensure any modifications in the code is able to update the design docs as well

Tagging more folks @backslasht @andrross @sachinpkale @mch2 @gbbafna @reta for close out on this discussion(giving it one more try)

https://github.com/opensearch-project/.github/issues/28 was making tagging harder and folks might not be subscribed for notifications as well

[anasalkouz]

Related: https://github.com/opensearch-project/.github/issues/181 If we are going to use another tool for design proposals, then may be a shared google doc folder could help.

[andrross]

However I would prefer design docs and their implementations to be co-located so as to ensure any modifications in the code is able to update the design docs as well

@Bukhtawar I agree that having designs as close to the code as possible is good, but I'm skeptical that it will truly solve the problem of designs drifting from the actual implementation. That's not a reason not to do it though. I have no objection to creating some sort of "design" folder within the core repo and adding markdown docs to it. If they turn out to be unhelpful or unused we can just remove them. Are there existing designs you're looking to backfill into this hypothetical directory, or just looking to establish a precedent going forward?

Another related thought is to make more use of package-info.java files for providing high level details of what the package does and how all the pieces within it fit together. Lucene, for example, makes use of this. Though usually this information is more code documentation than design information.

[andrross]

Related: opensearch-project/.github#181 If we are going to use another tool for design proposals, then may be a shared google doc folder could help.

@anasalkouz I think we need to be pretty disciplined here about using other tools only to facilitate collaboration and that completed version of the thing being reviewed needs to be captured back into a GitHub issue or a markdown doc in the relevant repository. Otherwise the "scattered over issues/PRs/blog posts" problem only gets worse.