search

Azure / azure-sdk-tools

Tools repository leveraged by the Azure SDK team.
MIT License
110 stars 174 forks source link

Provide specs PR reviews Teams support channel auto-responder message #7659

Open konrad-jamrozik opened 7 months ago

konrad-jamrozik commented 7 months ago

Per this conversation:

If a person asks for help on the specs channel (https://aka.ms/azsdk/support/specreview-channel) (see last step in https://aka.ms/azsdk/pr-getting-help), they should get auto-response from a bot to tell them to follow the PR process.

This bot could be AI-powered, properly prompted bot, that knows the most common scenarios and guidance.

For example

If you are asking for ARM review, look at the PR diagram [URL] step 2

If you have problem with LintDiff, then... [link to ci-fix]

If you are waiting for final merge, then... [link to merge queue]

If you have breaking changes, then... [etc.]

These are prompts, so internal bot database, so the bot should reply with the relevant info to the user based on that internal knowledge, not show them a huge if/else table.

The bot should be smart enough to evaluate posts to the Teams channel and detect if they pertain to specs PR. Reply only if they do and what the bot can say is of value. I.e. if this is some very exotic case, it should not reply. If the post is not providing PR URL, this is the first thing the bot should ask about.

The bot likely should mimic human behavior, making it hard to distinguish, to untrained eye, this is a bot. For example, if the PR author asks for help with LintDiff failure, the bot could say "I see you have problem with LintDiff, have you consulted aka.ms/ci-fix troubleshooting guide for it? If you have any trouble with the instructions there, let me know".

The bot could be integrated with the PR: calling GitHub API to obtain the status of Next Steps to Merge comment and Automated merging requirements met check and telling the user what those things already say.

We could potentially use the history of https://aka.ms/azsdk/support/specreview-channel as training data, in addition or instead of custom prompts.

Related work:

raych1 commented 7 months ago

Most of these proposals are supported in current AI bot.

The bot has already integrated with the "Automated merging requirement met" check and displays the link to it. We can explore integrating the bot with other sources, like 'Next Steps to Merge' to provide even more relevant and helpful information to the user.

The bot can try to understand the user query and identify the review stage of the spec pull request or identify the tool the user wants to get support. With this classification, the bot can prepare the corresponding information and return to the user.

konrad-jamrozik commented 7 months ago

@raych1 is the bot deployed anywhere? If I want to use it, how can I do it? Any developer docs you could link me to? @weshaggard told me you folks were working on some CI/CD pipeline for the bot, but I am not up-to-date on what is the status on this.

konrad-jamrozik commented 4 months ago

I am going to work with @JonathanCrd and @mikeharder to determine some basic non-AI-powered auto-responder for Teams channel.