[New Website Feature]: Improved Standards Infusion Process: Self-Service Model and Better Docs

[riverma]

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Related problems

Our "standards-as-code" infusion process is a big innovation in how we infuse best practices into projects. However, it could be improved through better documentation and tools.

Specifically: (for docs) its not really clear for end-users or developers about how this really works and how to engage with it, and (tools) an end-user is not able to infuse easily themselves.

Describe the feature request

• Better documentation:
  • Overview of infusion process for end-users, how it works, and how end-users can "join" or "subscribe"
  • Better directions for developers on how they should aim to design their best practice solutions to be infusion-ready
• Better tools:
  • End-user tools allowing SLIM community folks to infuse best practices on their own easily
  • 🤖 Automated customization of SLIM best practices for end-user repositories (i.e. using LLMs)

The proposed feature would satisfy the following:

• Tool
  • Command-line based
  • Fetches the latest SLIM best practices as specified
  • Customizes best practices (to the extent possible and optionally) for a given checked out repository
  • Creates a new branch and proposes a pull request accordingly
• Docs
  • Improved pages within SLIM's "Join" and "Contribute" pages regarding the above points

[riverma]

Here's a proposed usage statement of an idea that came about through discussion with @ingyhere on having a "SLIM CLI" tool for self-service purposes:

Usage: slim-cli [OPTIONS] COMMAND [ARGS]...

  This tool automates the application of best practices to git repositories.

Options:
  -h, --help           Show this help message and exit.
  -d, --dry-run    Generate a dry-run plan of activities to be performed.

Commands:
  list  Lists all available best practice from the SLIM.
  repos           Initialize / list git repositories from a file or a single repository from the command line.
                       Usage: slim-cli repos add [FILE_PATH | REPO_URL]
                       Usage: slim-cli repos remove REPO_URL
                       Usage: slim-cli repos list
  apply  Applies a chosen best practice to a new git branch in the loaded repositories.
                       Usage: slim-cli apply [OPTIONS] BEST_PRACTICE_ID [--use-ai]
  deploy          Pushes the new branch to the remote and creates a pull request.
                       Usage: slim-cli deploy [OPTIONS] BRANCH_NAME
  apply-deploy          Automatically applies a best practice, pushes the new branch, and files a PR.
                       Usage: slim-cli apply-deploy [OPTIONS] BEST_PRACTICE_ID [--auto-customize]

Options for 'apply' and 'apply-deploy':
  --use-ai     Enables an intelligent system to customize the best practice for each repository

Examples:
  # List all available best practices
  $ slim-cli list
  ID.                 NAME            DESCRIPTION
  SLIM-123    README        A template that can be used to help developers and users understand your repository's project code concisely and clearly.

  # Load repositories from a file
  $ slim-cli repos add /path/to/repo_list.txt
  Loaded repo 1
  Loaded repo 2

  # Load a single repository
  $ slim-cli repos add https://github.com/myorg/myrepo
  Loaded repo myrepo

  # List loaded repositories
  $ slim-cli repos list
  https://github.com/myorg/myrepo
  https://github.com/myorg/repo1
  https://github.com/myorg/repo2

  # Remove repositories
  $ slim-cli repos remove https://github.com/myorg/myrepo
  Removed https://github.com/myorg/myrepo

  # Apply a best practice to repositories loaded from 'repo_list.txt'
  $ slim-cli apply SLIM-123 --use-ai
  AI features enabled
  Applied best practice SLIM-123 into git branch slim-123 in repo 1
  Applied best practice SLIM-123 into git branch slim-123 in repo 2 

  # Push the branch 'feature-branch' to remote and create a PR
  $ slim-cli deploy slim-readme-pr
  AI features enabled
  Pushed branch slim-123 to repo 1 as a pr https://.../path/to/pr

  # Automatically apply best practice 123 with customization to loaded repositories and handle PR
  $ slim-cli apply-deploy SLIM-123 --use-ai
  AI features enabled
  Applied best practice SLIM-123 into git branch
  Pushed branch slim-123 to repo 1 as a pr https://.../path/to/pr

[yunks128]

@riverma This will be a great tool!

Just a couple of minor suggestions/thoughts:

1. auto-apply should be auto-deploy for consistency.
2. For potentially destructive actions such as pushing to remote or creating PRs (especially LLM-generated text or code), we may need to consider adding confirmation prompts.

[riverma]

@yunks128 - thank you for your excellent feedback! I've made a correction for the auto-apply consistency above. Also, good point about destructive actions. I've added a proposed argument for a dry-run mode, that runs the tool in a planned setting to see what will happen before it happens. This may be better than a prompt confirmation, as having a prompt can interfere with automation use cases with the tool.

[riverma]

Development will be done here: https://github.com/NASA-AMMOS/slim-cli

[stirlingalgermissen]

This reminds me of starter kits, which are very useful for new projects. For example

• https://github.com/kriasoft/react-starter-kit
• https://github.com/ngx-rocket/starter-kit

Perhaps we can learn from these and others. Ideally adapting a default set of SLIM templates is doable with a single command

[yunks128]

@riverma and @yunks128 will have a meeting on July 1 at 9:10am (PST) to discuss the next steps for slim-cli . If anyone is interested in joining the discussion, please use the following Teams meeting link: Join the meeting now Meeting ID: 299 187 890 148 Passcode: nKGmgG

[riverma]

Proposed next steps:

• @yunks128 focus on the use_ai feature. For each SLIM artifact, we want to do two things:

  1. Identify the most useful context that we want to provide to an AI along with the artifact itself (or maybe everything we know about repo?)
  2. Support calling AI agents, such as GPT-4 (API) or a local model (e.g. llama3, etc.) with specialized prompts to customize the artifact for the given repository

  Some additional thoughts on an interface:

  def use_ai(artifacts_uri, choice_of_ai_model, repository_uri) 
      # use AI model to customize artifact using context from repository
      # ...
  
      return customized_artifact_text_blob

• @riverma focus on the infrastructure of the slim-cli tool. Specifically:

  1. Support the listing and categorization of all SLIM artifacts
  2. Add support for multi-gitter tool for the Git related operations

[riverma]

Next steps for slim-cli:

• Ensure that the tool has safety measures for AI and general usage:
  • No inner source repository should be able to use cloud AI models
• Ensure tool's README is fully up-to-date with features available
• Clean up the code base to separate and ensure tool is maintainable
  • AI module (factory model recommendation from @hookhua)
  • interface (CLI) module
  • Core
• Prompt engineering to ensure AI-based template infusion works in a valid way
  • Make sure all best practices supported by slim list are accurate. Each may need to have their own custom prompt handling.