The Reviewer struct now instantiates a new Application instance, which handles all the logic for parsing the GitHub issue into a structured application that we can extract the data from and validate.
This process takes the issue's body Markdown and attempts to break it down into sections that it can parse. It relies heavily on the format of an issue body, which is largely reasonable given our usage of issue forms, but we may want to make more tweaks and safeguards in the future
The Application instance instantiates its own Validator instance, which makes a bunch of content parsing and validation utilities available.
This struct collects validation errors.
All validation and parsing utilities following the ValidationCallback type, which will also return 1) does the validation pass, 2) what is the new value to pass to the next validator, and 3) an error message. This means that these callbacks have the ability to modify the value along the way.
After breaking down section content, Application validates across all application sections, optionally using the return value of the validations to assign a property value. For each section it will only proceed to the next validation step if the first one succeeds.
Once this is finished we apply the validation errors to the application problems, which we can use to determine if the application is valid or not.
Testing
This PR sets up tests for Application and Validator, with more to come, and also sets up a new workflow that executes the test suite when changes to ./scripts/** occur.
This PR also adds various test issues we can use to test and debug the application processor manually:
git checkout jh/application-validation
make install_deps && make build_processor
./processor review --test-issue <issue-name>
This command will print the application input it received, the parsed data, and whether or not there were problems with the application.
Example output
```
[DEBUG] Using test issue 'valid-project'
[DEBUG] Skipping GitHub Init
[DEBUG] Parsed input data:
{
"Account URL": "testdb.1password.com",
"Additional comments": "Thank you!",
"Age confirmation": "- [X] Yes, this project is at least 30 days old",
"Can we contact you?": "- [X] Yes, you may contact me",
"Email": "wendyappleseed@example.com",
"Event application": "- [ ] Yes, this application is for an event",
"Homepage URL": "https://github.com/wendyappleed/test-db",
"License URL": "https://github.com/wendyappleed/test-db/blob/main/LICENSE.md",
"License type": "MIT",
"Name": "Wendy Appleseed",
"Non-commercial confirmation": "- [X] No, this account won't be used for commercial activity",
"Number of team members/core contributors": "1",
"Profile or website": "https://github.com/wendyappleseed/",
"Project name": "TestDB",
"Project role": "Core Maintainer",
"Repository URL": "https://github.com/wendyappleed/test-db",
"Short description": "TestDB is a free and open source, community-based forum software project.",
"Team application": "- [ ] Yes, this application is for a team"
}
[DEBUG] Application data:
{
"account": "testdb.1password.com",
"project": {
"name": "TestDB",
"description": "TestDB is a free and open source, community-based forum software project.",
"contributors": 1,
"home_url": "https://github.com/wendyappleed/test-db",
"repo_url": "https://github.com/wendyappleed/test-db",
"license_type": "MIT",
"license_url": "https://github.com/wendyappleed/test-db/blob/main/LICENSE.md",
"is_event": false,
"is_team": false
},
"applicant": {
"name": "Wendy Appleseed",
"email": "wendyappleseed@example.com",
"role": "Core Maintainer",
"id": 38230737
},
"can_contact": true,
"issue_number": 6,
"created_at": "2023-07-12T19:49:35Z"
}
[DEBUG] Application has no problems
```
Across all applications:
A project title is required
The applicant must confirm that the project is non-commercial
Regular text field are stripped of all formatting and HTML elements, with the exception of newlines, and cannot contain URLs, emails, or emojis
Applicant name, email, and project role are required
The following test issues are available:
valid-project
./processor review --test-issue valid-project
This is what we expect to be the typical application data. With this application:
The Age Confirmation checkbox must be checked
The License Type and License URL fields must be set
This issue should have the output "Application has no problems".
valid-team
./processor review --test-issue valid-team
This is another common application type, where a team is working on multiple open source projects. With this application:
The applicant has indicated that they are a team in the application form
The Age Confirmation, License Type, and License URLs are not required
This issue should have the output "Application has no problems".
valid-event
./processor review --test-issue valid-event
This is a less common application type, but we still see a few of them, where the applicant is applying on behalf of an event they're a part of. With this application:
The applicant has indicated that they are an event in the application form
The Age Confirmation, License Type, and License URLs are not required, like team applications
This issue should have the output "Application has no problems".
This is an example where someone has managed to trigger the application processor with no issue body whatsoever. It also serves as a test for when an application's section is missing, which should never occur unless someone was to manually edit their issue body. Note that this is different from when a section is present but incomplete or invalid.
This issue should have these problems
```
- Account URL: was not completed for application
- Non-commercial confirmation: was not completed for application
- Team application: was not completed for application
- Event application: was not completed for application
- Project name: was not completed for application
- Short description: was not completed for application
- Number of team members/core contributors: was not completed for application
- Homepage URL: was not completed for application
- Repository URL: was not completed for application
- License type: was not completed for application
- License URL: was not completed for application
- Age confirmation: was not completed for application
- Name: was not completed for application
- Email: was not completed for application
- Project role: was not completed for application
- Profile or website: was not completed for application
- Additional comments: was not completed for application
- Can we contact you?: was not completed for application
```
This is an example of someone creating a new application issue without filling in any of the fields.
This issue should have these problems
```
- Application title: missing project name
- Account URL: is empty
- Non-commercial confirmation: must be checked
- Project name: is empty
- Short description: is empty
- Number of team members/core contributors: is empty
- Homepage URL: is empty
- License type: is empty
- License URL: is empty
- Age confirmation: must be checked
- Name: is empty
- Email: is empty
- Project role: is empty
```
This is a test issue containing various issues, such as missing project name, URLs in description, invalid URL, emoji usage, unchecked checkbox, and more.
This issue should have these problems
```
- Application title: is missing project name
- Account URL: is invalid 1Password account URL
- Non-commercial confirmation: must be checked
- Project name: cannot contain emoji characters
- Number of team members/core contributors: could not be parsed into a number
- Homepage URL: is an invalid URL
- Project role: is an invalid value
```
/dev/web/developer.1password.com/-/issues/1065Summary
We are working on improvements to the 1Password for Open Source program. This PR continues from https://github.com/1Password/1password-teams-open-source/pull/915 and adds application parsing and validation.
What's changed?
Reviewerstruct now instantiates a newApplicationinstance, which handles all the logic for parsing the GitHub issue into a structured application that we can extract the data from and validate.Applicationinstance instantiates its ownValidatorinstance, which makes a bunch of content parsing and validation utilities available.ValidationCallbacktype, which will also return 1) does the validation pass, 2) what is the new value to pass to the next validator, and 3) an error message. This means that these callbacks have the ability to modify the value along the way.Applicationvalidates across all application sections, optionally using the return value of the validations to assign a property value. For each section it will only proceed to the next validation step if the first one succeeds.Testing
This PR sets up tests for
ApplicationandValidator, with more to come, and also sets up a new workflow that executes the test suite when changes to./scripts/**occur.This PR also adds various test issues we can use to test and debug the application processor manually:
This command will print the application input it received, the parsed data, and whether or not there were problems with the application.
Example output
``` [DEBUG] Using test issue 'valid-project' [DEBUG] Skipping GitHub Init [DEBUG] Parsed input data: { "Account URL": "testdb.1password.com", "Additional comments": "Thank you!", "Age confirmation": "- [X] Yes, this project is at least 30 days old", "Can we contact you?": "- [X] Yes, you may contact me", "Email": "wendyappleseed@example.com", "Event application": "- [ ] Yes, this application is for an event", "Homepage URL": "https://github.com/wendyappleed/test-db", "License URL": "https://github.com/wendyappleed/test-db/blob/main/LICENSE.md", "License type": "MIT", "Name": "Wendy Appleseed", "Non-commercial confirmation": "- [X] No, this account won't be used for commercial activity", "Number of team members/core contributors": "1", "Profile or website": "https://github.com/wendyappleseed/", "Project name": "TestDB", "Project role": "Core Maintainer", "Repository URL": "https://github.com/wendyappleed/test-db", "Short description": "TestDB is a free and open source, community-based forum software project.", "Team application": "- [ ] Yes, this application is for a team" } [DEBUG] Application data: { "account": "testdb.1password.com", "project": { "name": "TestDB", "description": "TestDB is a free and open source, community-based forum software project.", "contributors": 1, "home_url": "https://github.com/wendyappleed/test-db", "repo_url": "https://github.com/wendyappleed/test-db", "license_type": "MIT", "license_url": "https://github.com/wendyappleed/test-db/blob/main/LICENSE.md", "is_event": false, "is_team": false }, "applicant": { "name": "Wendy Appleseed", "email": "wendyappleseed@example.com", "role": "Core Maintainer", "id": 38230737 }, "can_contact": true, "issue_number": 6, "created_at": "2023-07-12T19:49:35Z" } [DEBUG] Application has no problems ```Across all applications:
The following test issues are available:
valid-projectThis is what we expect to be the typical application data. With this application:
This issue should have the output "Application has no problems".
valid-teamThis is another common application type, where a team is working on multiple open source projects. With this application:
This issue should have the output "Application has no problems".
valid-eventThis is a less common application type, but we still see a few of them, where the applicant is applying on behalf of an event they're a part of. With this application:
This issue should have the output "Application has no problems".
invalid-empty-bodyThis is an example where someone has managed to trigger the application processor with no issue body whatsoever. It also serves as a test for when an application's section is missing, which should never occur unless someone was to manually edit their issue body. Note that this is different from when a section is present but incomplete or invalid.
This issue should have these problems
``` - Account URL: was not completed for application - Non-commercial confirmation: was not completed for application - Team application: was not completed for application - Event application: was not completed for application - Project name: was not completed for application - Short description: was not completed for application - Number of team members/core contributors: was not completed for application - Homepage URL: was not completed for application - Repository URL: was not completed for application - License type: was not completed for application - License URL: was not completed for application - Age confirmation: was not completed for application - Name: was not completed for application - Email: was not completed for application - Project role: was not completed for application - Profile or website: was not completed for application - Additional comments: was not completed for application - Can we contact you?: was not completed for application ```invalid-no-responsesThis is an example of someone creating a new application issue without filling in any of the fields.
This issue should have these problems
``` - Application title: missing project name - Account URL: is empty - Non-commercial confirmation: must be checked - Project name: is empty - Short description: is empty - Number of team members/core contributors: is empty - Homepage URL: is empty - License type: is empty - License URL: is empty - Age confirmation: must be checked - Name: is empty - Email: is empty - Project role: is empty ```invalid-examples-1This is a test issue containing various issues, such as missing project name, URLs in description, invalid URL, emoji usage, unchecked checkbox, and more.
This issue should have these problems
``` - Application title: is missing project name - Account URL: is invalid 1Password account URL - Non-commercial confirmation: must be checked - Project name: cannot contain emoji characters - Number of team members/core contributors: could not be parsed into a number - Homepage URL: is an invalid URL - Project role: is an invalid value ```