RFC: Doorstop GUI usability

[stanislaw]

Hello,

Please excuse me for the following rather long description, but I think that the topic is rather relevant to the possible development course of Doorstop.

Background

Our team is considering to use Doorstop as a tool for writing specifications that can be traceable to source code.

Everyone seems to be happy with a feature set of Doorstop especially the collocation of specs close with source code and being able to put stamps on the links between REQs and REQs and files.

The only thing that seems to be a blocker for us is how to make it easy and convenient to work on a specification like a document but at the same having it stored as multiple YAML files via Doorstop. The biggest opponent for Doorstop in our case is not Doors or Jama which, as everyone agrees, have to be Stopped, but Confluence. Our team mostly works in Confluence and there it is natural to work with its Pages which are exportable documents, having rich formatting, table of contents, plugins, etc.

We are wondering, what is your experience of working with Doorstop on some bigger projects? Do you actually use the Doorstop GUI (tkinter) or do you use something else? We can hardly imagine that it becomes practical to maintain REQ001-REQ100 in the folder of YAML files and having to export them to HTML every time a tiny change is made and then go back to a file like REQ042 to do another small iteration.

GUI redesign

The way I see the Doorstop GUI could work for our case would be something that looks like a serious redesign of the current UI layout with an accent made on making it possible to edit a Single document - the wide central region of the app's screen, see the table of contents on the collapsible left side of the screen, see the details of the selected Item on the collapsible right panel. The central panel could look like a normal document split by cells:

• 1 req statement per cell.
• Levels are the first-class citizen: a cell can be a parent for a group of children cells at the next level. Next level is indicated visually by indentation and maybe some additional (unobtrusive UI) guidelines.
• Cells are visible but not annoying, each statement is still a part of a single document
• Creating a new statement equals to creating a new cell at a given level
• Each cell allows embedding diagrams and tables (markdown?)

With this idea in mind, I have looked into the current GUI based on tkinter and I think I can conclude that it's not something that can scale up to accommodate all of the features described above.

GUI Frameworks

I have discussed this with the team and our opinions are settled around the following options of implementing and new GUI for Doorstop:

• PyQT or C++ Qt
• Electron
• Atom plugin
• Sublime plugin

I ended up being in the Qt camp with an idea of also using PyQt in order to build on top of existing Python implementation. The other options were suggested by my colleagues who think that implementing UI in Electron can be easier, faster and more accessible to the OSS community but this would mean a JS version of Doorstop has to be written which I rather don't like.

The idea behind Electron and Atom/Sublime plugins is based on the idea of jumping on a bigger fish in order to not introduce something completely custom and detached.

Additional input

Two more inputs for possible implementation:

1. I think the power of Doorstop, if applied to the real-world projects, will be higher if the REQs can be reviewed as a single document in the Pull Requests. Reviewing single YAML files is not convenient so I imagine a better way is to always include a one-file export to the Pull Request so that people make their reviews on a single markdown file generated from REQs rather than on individual YAML files.

• Weak subpoint here: have you ever considered to keep all of the items in ONE markdown file? The idea here would be to use some meta-information around markdown headers like it is done in static website engines that generate html from markdown files based on the meta-information also stored in those markdown files. The downside here is of course redundancy of the Doorstop-related meta information in the technical document because each level header will have a chunk of such meta-information.

2. Our team does not have resources to contribute any serious work to Doorstop so, for now, it is mostly my spare time interest if described above can be implemented. I am having the "Worse is Better" approach in mind: https://en.wikipedia.org/wiki/Worse_is_better - experiment with the frameworks and have that 3-column layout as simple as possible and see if it produces a value that can be iterated on later.

Thanks for attention.

[jacebrowning]

One existing way to get the cell workflow you describe is to work in your spreadsheet editor of choice:

$ doorstop export REQ req.xlsx

# make changes to 'req.xlsx'

$ doorstop import req.xlsx REQ

I've used this approach when needing to make changes to lots of requirements at once.

[JustinW80]

@stanislaw I continue piloting Doorstop in small areas of my team and pitching it to scale up across others. I agree that the biggest challenge to adoption is how comfortable people are working in a different workflow, exposed to the "bittynesss" of individual plaintext files. There are some people still resisting moving away from MS-Word docs to DOORS, so this is often a bigger leap.

In my pitch, I do the round-trip-to-Excel demo that @jacebrowning shows. Trying to make it easier, you can script the "generate spreadsheets" and "sync sheets back to Doorstop" commands.

Working in the doorstop reorder --manual's generated index.yml is also pretty nice as an outline editor.

I think the one advantage that the current GUI has is that you can get the items sorted in order and ready for editing without any intermediate steps.

About separate files for each requirement: I also follow the zettelkasten method (smart notes) at zettelkasten.de, where there is a similar discussion: should you keep notes in individual files or as blocks in one big file? Most there strive for atomicity and have each note stand alone on its own, and I think that's a good goal for a Doorstop item, especially for tracking changes in the version control system. Having that atomicity baked in to the tool encourages you to write better requirements.

Doorstop has a lot in common with a zettelkasten, actually, with a big emphasis on tracing your knowledge. See @renerocksai's sublimeless_zk for an example of a popular cross-platform GUI implementation. I think he uses PyQt. Maybe there are some ideas there for the Doorstop GUI enhancements you seek.

I like Electron apps (except for their speed), but there are a lot of people who find them gross and non-native. The tkinter GUI already looks non-native, so that's not a differentiator.

[jacebrowning]

Yet another option to explore would be adding edit functionality to the web interface (doorstop-server):

1. Run server locally
2. Edit items via the (bottle) UI
3. Changes are reflected in one or more files on disk
4. Stop server
5. Commit changes

[michaelnt]

I have a branch with edit functionality added to the web pages. It works reasonably well however there were a few problems almost entirely caused by us allowing multiple people to edit via a single we server:

• Each change is committed to the version control system, i.e. no UI to commit a set of changes. The use is prompted a log message but this was often left blank.
• How to authenticate the user against the version control system, we had the web UI prompt for a username and password and then used that to try authenticate to the version control.
• No UI for updating the workspace or handling conflicts.
• No UI for tables etc., although there is good support in JS for editing markdown.

I edit the yaml files directly and have a script which watches for changes and automatically re-publishes the html and causes my web browser to reload the page.

I also use the reorder command as an outline editor and excel export for editing say links for a lot of requirements at once.

[JustinW80]

See also @akuker's doorstop.net concept, with screenshots, in this comment in #301. It looks closer to DOORS's user interface.

[stanislaw]

It turns out that there was some PyQt work done for Doorstop. Adding it here for completeness: https://github.com/nickvanderstel/doorstop-qt.

[sevendays]

I hacked this (pyside2) to display and edit the (existing) requirements in the whole tree:

However I cannot add new requirements or reorder them... If anyone is interested to have a go at that I can remove the "design" and "analysis" tabs and make a repo.

It is super-fast because it only edits the requirements (saving the reqfiles after editing), it doesn't do any check.

(Design is automagically produced using regexes on requirements, but that's very confidential).

[a348680]

edits

Hi .. Is this GUI is available in Git to have a look at it

[sevendays]

edits

Hi .. Is this GUI is available in Git to have a look at it

I promised - I coded - I published.

https://github.com/sevendays/doorhole

It's pretty raw but it's working. If you have issues running it please start the discussion there.

[sevendays]

Just to give an update to whoever may be using this GUI: I finally made creation and deletion of requirements possible through a context menu. Now the tool is way easier to work with!

[ownbee]

I've been working on a doorstop GUI during Christmas (wanted to test out PySide6) and I think it turned out quite good so I decided to publish it and also throw in a mention here. It has a bit more functionality and a different layout then yours @sevendays . I've been working with doorstop for a little bit now and threw in some stuff that I really missed in a GUI.

Repo: https://github.com/ownbee/doorstop-edit

[sevendays]

Hey @ownbee , that's nice and definitely a huge step forward wrt doorstop-gui!

I like the treeview and the ability to link to other requirements.

There is one thing I dislike: the entire requirement document is not visible at once. When you write requirements, you go back and forth to check, copy and paste.

[ownbee]

@sevendays I think different people and organization have different process when writing requirements and how to structure them. I usually don't need to see everything at once and prefer to focus on one thing at a time, but that is because how my current requirements are written.

However, there is a function for sectional overview which you can toggle:

But I think it could be improved. I think a more compact sectional HTML rendering could help and I am planning to add one. The current doorstop style markdown is a bit "airy" to be efficient. The styling is also a thing I plan to improve.