GitHub App Integration

[NouberNou]

I've built a GitHub App that integrates Doorstop. Currently it provides:

• Installation on repositories for users or organizations, public or private
• Automatic parsing of requirements in repositories across branches
• Generation of document trees and requirements items with easy navigation between documents and linked items (including all the markdown support in the Doorstop HTML generation)
• Tracking of pushes and branch creation to update above generated documents and items
• Display of relevant committer information and links back to GitHub commits and related information (repo, user/organization, etc)
• Asset hosting
• Display of Doorstop validation issues
• GitHub check suites/check runs based on Doorstop validation issues
• A basic JSON REST API for retrieving parsed Doorstop information

In the near future I plan to provide support for:

• Tagging releases via tags
• Parsing GitHub issues/PRs/comments with links based on document prefixes and UIDs if present in text
• Diff viewing and a dedicated PR view for reviewing requirements changes when a GitHub PR is made
• Basic editing support for creating docs, creating items, and editing items in-band with the rest of the app

Long term goals include:

• Integration with CI apps and processes to automatically validate linked test case requirements
• Actual listing on the GitHub App market
• Support for more than GitHub as a integration platform
• A standalone self-hosted version of the software

The overarching goal though is to somewhat evangelize Doorstop by providing a more friendly integration and use experience, and to push requirements capture as part of good open source development (and good software development in general). I hope that having something like this provides a platform for that, at least on GitHub.

I am not ready to share links publicly but if anyone is interested in seeing something before the initial beta roll out let me know.

[NouberNou]

Few images though just to give some context:

Install dashboard for user's accessible installs:

Repository root page showing doc tree:

Needs more info probably, this is my least favorite page so far.

Specific document page:

The TUT document from the Doorstop repo:

Markdown output needs a bit more CSS work, so I won't scroll down for now (headers not having enough margin, tables need borders, etc).

[NouberNou]

Also the name obviously is a continuation of punnery.

[NouberNou]

Additionally I have set up a full AWS platform for the actual deployment, and working on using GitHub actions for the rest of the devops flow (not that that is important really to this discussion, just might be interesting).

[NouberNou]

Check Run Output

[jacebrowning]

Wow, this looks amazing!

[NouberNou]

Thanks! :smile:

[stanislaw]

@NouberNou what you have on the screenshots seems pretty close to what I had in mind when I opened this ticket: https://github.com/doorstop-dev/doorstop/issues/400. I was going to prototype something similar with JS on the frontend side and doorstop-server on the backend side as suggested here by @jacebrowning.

I need to finish a few other pressing issues on the other projects that I am working on to actually do this but for now I would like to ask if you plan to open source your frontend work? I have not worked with GitHub Apps before so I it is not clear if the skeleton of the html GUI on your screenshot is something that is provided by GitHub Apps or if it is something that you wrote from scratch.

Thanks.

[NouberNou]

I guess to explain a bit about the architecture:

GitHub Apps are basically just a set of authentication methods that let you authenticate as an accessor of repositories that a user/organization installs the app on. The app can then subscribe to events that GitHub will send you and also let the app make GitHub API requests within the permissions the app has requested. This is similar to how their OAuth scheme works, but it is a more structured approach. GitHub apps don't provide anything beyond that in terms of display (you could run everything in the background with no user interface to people that installed the apps if you wanted).

Doorframe is written in Python and uses Django as it's backend to handle events from GitHub, and users of Doorframe. Events propagate to workers in the background that then handle all the API requests for pulling repositories and parsing Doorstop. The parsing of Doorstop creates static JSON files that are a bit more digestible by the Django Views, which display content via rendering of templates. JSON files also serve as a nice easy way to provide a basic GET REST API for Doorstop data.

The front end is rendered by Django, and is standard HTML5 with Semantic UI as the CSS framework, along with JQuery, and related Semantic UI JS for any of the front end rendering elements. I've been hesitant to use something like React or a more full fledged frontend framework as... well it just seems like a lot of work to learn (I stopped doing web development as my day to day 10 years ago before getting into aerospace/defense, so I am still mostly familiar with a lot of older web development styles).

Ultimately I don't think open-sourcing the front end would be particularly useful as its pretty bog standard HTML and CSS cobbled together from the elements provided in Semantic UI. Also it is pulling content together from the Django template system which is pretty specific to Django (though a lot of Python template system share similar syntax), and Doorframe specific interpretations of the Doorstop data structure. Also its something I plan to iterate on and it will probably become more application specific as I add inline editing of requirements and other features, which would then demand significantly more interaction with the backend (though most likely through websockets or AJAX interfaces, so it'd still be pretty decoupled).

tl;dr

Yes, it is written from scratch, GitHub apps only provide authentication and API access, no actual UI or user features. Open sourcing the front-end wouldn't have much value since it is pretty application specific in terms of how it is generated and is pretty basic HTML/CSS using Semantic UI.

If you need help or advice on doing better HTML rendering or serving within Doorstop let me know though!

[NouberNou]

I've low key released the first iteration of Doorframe: https://doorframe.io/

Here is the Doorstop requirements in it (forked): https://doorframe.io/install/IDI-Systems/doorstop/develop

[JustinW80]

Long term goals include:

• A standalone self-hosted version of the software

Forgive my ignorance about GitHub Apps. And I understand it's far off on the roadmap. But how do you see licensing working for running the self-hosted version of Doorframe?

Tool evaluation at my organization takes awhile, and I'd love to show off Doorframe, but hosting proprietary data on GitHub is a nonstarter.

[dapperfu]

How difficult would it be to make doorframe host agnostic?

Corporate git servers are very non-homogeneous. GitLab CE/EE, Atlassian BitBucket, Gogs/Gitea, etc. They all have a runner concept

As @JustinW80 said this is a non-starter for a lot of companies. (Companies that use requirements for functional safety are very conservative).

[Zupp61]

@NouberNou would it be possible to get beta access? I'd be keen to see how Doorframe can help make my Doorstop files more accessible to our wider team.

[yoyodyn]

Would also like to see this "ported" for Gitea.

[NouberNou]

Unfortunately, Doorframe is sort of on the backburner as other work has kind of taken off at my company. Annoyingly it's stuff that I had originally developed Doorframe in anticipation of but now do not have the time to rapidly iterate on additional features that would really help (like creating/editing requirements in Doorframe).

We didn't have any funding for it (nor were we seeking it out, this was basically me, the owner, on a tangent while my other developers got annoyed that I wasn't working on our core products) so it's been running as a sort of public IR&D project with maybe the intent to expand it further.

Though at this point the idea of a standalone application, something that can run internally is getting more persuasive, especially since we've gained contracts and moved a lot of our projects internally to self-hosted GitLab.

I'd love to expand Doorframe, or rewrite it to be Git platform agnostic (it basically is already, but it's definitely not geared for self-hosting) but unless I can find the manpower (or the time for myself) its sort of just sitting in its current state for now.

[neerdoc]

@NouberNou I suggest we close this issue since it is not actually part of doorstop.

[jacebrowning]

Yeah, perhaps https://github.com/doorstop-dev/doorstop/edit/develop/docs/examples.md could include a list of 3rd-party clients.