Change Validation Severity for an individual or global API Setting (md)

[Mgranfie]

As a user I want to be able to customize the severity for problems, when I have an individual API open and have the ability to change the severity for more than one API.

Must have Requirements: (P1)

A validation profile is something that is predefined by us or configured by the user.

A validation profile consists of:

• Name
• Description
• Severity (low/med/high/ignore) settings for each validation rule

For user defined/custom validation profile, description is optional.

The product provides 3 built-in Validation Profiles:

1. None: don't use any validation
2. Basic or Standard: validated against the open API specification rules
3. Strict: apply all known rules at high severity (what about the others?)

• A user can create their own validation profile
• There is no limit to the number of Validation Profiles that can be created. Average is 5 (total)
• There is no limit to the number of Rules within a Validation Profile (all validation profiles have the same Rules, but with different assigned severities to those rules)
• Severities are customized for user defined profiles
• Users cannot define their own validation rules

The design work for this issue will focus on verbiage, layout and consider scalability and a global Settings page, in the future.

Settings Page Requirements

• List all Validation Profiles (built-in profiles and custom profiles, if any)
• Choose one of the Validation Profiles to be the "default" profile - used when no other profile has been set (for a given API)
• Create a custom Validation Profile
  • Configure the name and description of the profile
  • Configure custom severity settings for all validation rules for the profile
• When creating a Validation Profile, allow the user to clone an existing profile as a starting point for the new one
• Edit an existing custom Validation Profile to change its name/description/severities

Nice to haves/Future capabilities (P2)

• Custom name the severities (does this introduce troubleshooting and maintenance issues?)
• Ability to customize a description for each severity. User specific meaning to them only. Stay generic or needs explanation. (can they provide their own definition of what the severity means to them.)
• Ability to associate a Validation Profile with an "Action" such as "Generate Project...", preventing the action from being completed if "high" severity problems are found. In other words, we would start doing validation prior to allowing the user to complete certain actions (Generate Project, Publish to Git, etc).

[EricWittmann]

@Mgranfie I have updated this story for accuracy and content. I've fixed some things that weren't quite right and I've added some additional content. Please review and let me know if anything is not clear! :)

[Mgranfie]

@EricWittmann just reviewing the updates and wondering if the Severities (High/Med/Low/None) map directly to the Validation Profile None, Basic, Strict….

Can the Validation profile be changed at any time, are there any restrictions around changing the profile assignment?

What is the source of the validation rules?

Strict is applied, does this apply all the rules including the high severity or just high severity?

[EricWittmann]

From these followup questions I think we may not yet be on the same page. However, answers to your questions below (to the best of my ability).

@EricWittmann just reviewing the updates and wondering if the Severities (High/Med/Low/None) map directly to the Validation Profile None, Basic, Strict….

No. It happens that the "None" profile sets the severity of every rule to "Ignore". But that is unusual. Typically any given validation profile would have some of the rules set to "Ignore", some other rules set to "Medium", and yet other rules set to "High" (for example). The difference between the profiles is which of the rules are ignored vs. medium vs. high, etc.

Can the Validation profile be changed at any time, are there any restrictions around changing the profile assignment?

Custom Validation Profiles (created by users) can be modified at any time. Built-in profiles cannot be changed.

There are no restrictions around changing which of the profiles the user is applying to the API while editing.

What is the source of the validation rules?

Right now the rules are all included embedded in the software. Users cannot invent their own rules right now. I will be organizing the rules into a catalog of some sort once we build the Validation Settings page, because we'll need to list out all of the rules (with name and description) and at that point I'll need such a catalog.

Strict is applied, does this apply all the rules including the high severity or just high severity?

This sounds like a question specific to the "Strict" built-in validation profile? Right now the "Strict" validation profile sets the severity level of all rules to "High". Does that answer the question?

[Mgranfie]

@EricWittmann confirming some updates to the requirements and have a few questions. Will roll these into the original story once sorted out and confirmed. Thanks.

1. Profile rules, provided by the application are NOT editable.
2. Only severities for rules can be edited within a custom profile.
3. A user should be able to bulk edit severity assignment on rules within a custom profile.
4. A user should be able to filter on Profiles and rules ?? within all profiles by (Severity + Keyword?)
5. A user should be able to filter on rules with a profile by name, keyword, severity, metadata(?) (would description be covered by Keyword? what are Categories and Groups? Will the Metadata show with the return results and is there some baseline metadata, that is important and should also display on the rule, maybe when opened?

[EricWittmann]

1. Profile rules, provided by the application are NOT editable.

Correct.

2. Only severities for rules can be edited within a custom profile.

Correct.

3. A user should be able to bulk edit severity assignment on rules within a custom profile.

Correct.

4. A user should be able to filter on Profiles and rules ?? within all profiles by (Severity + Keyword?)

This is a stretch goal, IMO. The feature is interesting but not core to the problem. I think for iteration 1 we should have NO FILTERING on the list of profiles. We should have advanced filtering options on the list of rules.

5. A user should be able to filter on rules with a profile by name, keyword, severity, metadata(?) (would description be covered by Keyword? what are Categories and Groups? Will the Metadata show with the return results and is there some baseline metadata, that is important and should also display on the rule, maybe when opened?

I think that every rule COULD have the following meta-data:

• ID
• Name
• Description
• Keywords
• Category/Type
• Entity (what sort of thing the rule applies to - e.g. Data Type or Security Scheme)
• OpenAPI Spec Version (2.0 or 3.0.x or both)
• OpenAPI Spec Mandated

We should probably discuss this more because it's complicated and we could model the meta-data and the UI in a few different ways. The challenge is to display a large number of rules (over 150 and growing) and be able to organize them in a variety of ways. For example, I might want to show only the rules that relate to "description" fields. Or I may want to only show the rules that apply to OpenAPI 3.0.x APIs. Or maybe I want to see all rules that apply to Data Types. You get the idea.

[Mgranfie]

Notes...

• ID (does this need to be included to id with issue UI or not?) Where should the stop showing me this be exposed in the UI? In the Issues pane or in the Validation UI (low pri).
• How would user find the issue that is no longer in use?/turned off? becomes none. Maybe use the None... in the command.
• Keywords
• Categories - ie: a piece of info is required. Filter by this "category"
• Stacy- good work for "Entity"
• Indicate if it is 2.0, 3.0 or both. (default will be 2.0)
• Tag rules that are mandated by the spec (differentiate 2.0 vs. 3.0, is this a thing? Eric will check)

How to apply this meta data and filter on the rules. How to present this information in the list of rules. Bulk edit off a filtering event. Select and bulk edit.

[Mgranfie]

Mockups located here