Aligning speccing & designing apps / feature across the front end teams

[Mohammer5]

@Bekkalizer provided an excellent overview over new specifications & designs for the capture app during the last milestone demo meeting (27 May 2020). The design and specs are kept in a google doc and linked to from the respective Jira issues.

Team apps (platform?) started to use feature files for specification and tests, mainly in the ui libraries and sporadically in some apps and we can already see the benefit/impact of having these type of tests. So I think it'd be quite beneficial if other teams used feature files & cypress as well.

I can see that gathering and consolidating information in a google docs file is way simpler and quicker though and although we're not covering design in the feature files (so far?), @cooper-joe has mentioned in the past that the development of specifications and design is a circular process and can't be separated.

So I think that existing workflows that use google docs for gathering features and designs shouldn't be altered in a way that removes these parts from the workflow, but I think we can enhance them.

The current workflow, I assume, is as follows (please correct me if I'm wrong @Bekkalizer):

1. An epic in Jira is added
2. Very high level requirements are gathered and stored in a google doc
3. Over time the requirements become more precise and @cooper-joe can work on the design, which then produces further requirements or increases the precision of the existing ones
4. Once the requirements and the design are final, the different tasks are defined and added as issues in Jira, linked to the epic created in step 1

The two possibilities I currently have in mind are:

• add another step after step 4 and create feature files for the different tasks and only once they've been added the issue can be worked on by the developer
• create all the feature files after step 3, add them as drafts to the repository and then create Jira issues

Although fairly similar, the two approaches I described above would result in different Jira issues produced in step four. Fore example depending on how the feature files are designed, for each feature file a Jira issue could be added. Some companies use user stories as a feature file, in the ui libraries we've uses one feature file for every behavior a component can have.

[varl]

I really like this proposal.

Practically, I have always struggled with having "someone else" writing the feature files for developers to consume. In my experience it is the utopia we cannot have.

However, what I have seen work first-hand is having the functional people essentially jotting down coarse features on post-its (JIRA issues in our case), handing those to developers to turn into cold, hard, machine executable specs, and then getting those feature files stamped as the "this is how it actually works" spec.

It is up to the developer to judge when a feature is straight-forward and when to use the feature files to further discuss with the functional architect and decide exactly how something works.

So I'd say that the point 5 should be:

5. The developer breaks down the JIRA issues into feature files as she works on the implementation.

[Mohammer5]

That's a good proposal! Maybe it'd make sense to have a jira task for this ->

Convert DHIS2-XXXX into feature files

Which blocks the actual issue the developer will end up working on. Then feature files could go through our usual review process and stakeholders can have a final check before they get merged in (into the draft folder on master)

[HendrikThePendric]

I like this proposed workflow a lot as well, and also like @varl's suggestion for step 5, because I agree that the developer producing the feature files is the most realistic/pragmatic and efficient approach.

However, by defining step 5 that way, we introduce some room for misinterpretation, which could go unnoticed until the feature is actually implemented. @Mohammer5 already addressed this point in his comment:

[...] stakeholders can have a final check before they get merged [...]

This point is vital, to ensure that the developer doesn't implement the functionality in a different way than the functional architect intended.

🤔 perhaps the point above is important enough to be formalised into its own step number 6?

[varl]

A concern is that there is a significant risk we introduce a bad choke-point because as developers we outnumber the functional architects (the product managers) by 5:1.

Among the reasons we are investing heavily into automating our test processes on many fronts is that developers outnumber the amount of testers by a ratio of 5:1 as well. With developers vs. designers we are 50:1. We would be wise to not introduce another choke-point like that by requiring a rubber-stamp before we start to work on a feature.

Another consequence of this added step is that we are asking the product manager to "approve" our interpretation of the feature she has already provided the description for in JIRA. This is at least twice the amount of work for the PDM. We are effectively passing the buck of responsibility to someone else -- even if that is unintentional -- the added workload and context switching for the PDM makes this a tough adoption for them.

To make this a bit thornier, our feature files are going to change a lot during all phases of development:

• during planning the feature files are a technical approximation of the initial use case
• during implemenation the feature files capture edge cases, unforeseen and unspecified details, and cross-feature implications and interactions, of the new feature
• during testing the feature files are appended with bugs, more edge cases, etc.

Since these feature files are living documents under constant change, and serve different purposes during the various phases of development, the value of the initial "approval" of the specs is insignificant when put against the added workload for the product manager and the delay before the developer can work on a feature.

Consider the fact that technically we do have the "rubber stamp of approval" right there in the JIRA issue that the product manager has provided, that explains "what" the feature is and "why" we need it. If we need more information when writing our feature files, we should ask for it right there in the JIRA issue.

What we must do is maintain a link between the JIRA issue that describes the feature and the Feature file itself, so it is possible to look at what the original feature request was, and how it has evolved during the various stages, even long after the implementation has been completed.

[Mohammer5]

Oh yeah I definitely agree. I didn't describe what I meant properly:

I would like the feature files to be reviewed. If that's done by a developer, that's good enough. But we'll give PMs and other steakholders (that typo is on purpose :laughing:) the chance to have a look if they're interested / it's a very-high-priority feature or cruicial/high-value addition.

[HendrikThePendric]

Maybe I'm going into the nitty gritty a bit too much here, but I'm going ahead anyway...

I would like the feature files to be reviewed. If that's done by a developer, that's good enough

I personally disagree with this. If another developer approves the feature files, we have not verified that each feature has been understood correctly, we are only verifying that both developers share the same interpretation. IMO this type of approval represents very little added value and we could simply do without it.

And with regards to what @varl wrote, I don't really agree with "passing the buck of responsibility to someone else" part. If the balance between PDM and developer staff was more balanced, I think that the step I suggested is perfectly valid, and not about dodging responsibility, but simply about aligning expectations. Suppose I bought a plot of land and hired some building firm to build a house on it for me... I would expect me to deliver a list of requirements, and I would not expect the build to begin before I have approved every point of the planned implementation. Just to confirm all of my requests have been understood correctly.

That being said, since we are such an unbalanced bunch (PDMs VS devs), and also because the specs are likely to change whilst we are building the implementation, I can see why this last step is not feasible. So in the end, I'd personally prefer to just leave it at the step 5 that @varl suggested.

[ismay]

Maybe I'm going into the nitty gritty a bit too much here, but I'm going ahead anyway...

I would like the feature files to be reviewed. If that's done by a developer, that's good enough

I personally disagree with this. If another developer approves the feature files, we have not verified that each feature has been understood correctly, we are only verifying that both developers share the same interpretation. IMO this type of approval represents very little added value and we could simply do without it.

And with regards to what @varl wrote, I don't really agree with "passing the buck of responsibility to someone else" part. If the balance between PDM and developer staff was more balanced, I think that the step I suggested is perfectly valid, and not about dodging responsibility, but simply about aligning expectations. Suppose I bought a plot of land and hired some building firm to build a house on it for me... I would expect me to deliver a list of requirements, and I would not expect the build to begin before I have approved every point of the planned implementation. Just to confirm all of my requests have been understood correctly.

That being said, since we are such an unbalanced bunch (PDMs VS devs), and also because the specs are likely to change whilst we are building the implementation, I can see why this last step is not feasible. So in the end, I'd personally prefer to just leave it at the step 5 that @varl suggested.

Workflow

I agree with @HendrikThePendric here. I think in our expected workflow here we are assuming a very linear process. From management -> design -> dev -> etc. (don't pin me down on the specifics here, merely meant as illustration).

In my opinion that causes all problems caused in earlier stages to end up with the last person in the chain. We already see that for example in our UI lib., where something that would have been more adequately solved by different requirements from management, or a slight tweak in the design, ends up having to be solved in our components.

That is often very inefficient and timeconsuming, because it's not solving the problem at the stage where it's best solved. I think that we should assume some feedback cycles. I've personally found that checking with backend/design/management for the scheduler has greatly simplified the app and solved the complexities that were solved very inefficiently (or even incorrectly) in the current scheduler app. My argument here is that it might very well be worth the time investment in the end.

Jest vs cypress

My second point here is that I think that we need to maybe formalize allowing features to be tested in both jest and cypress. When we started using feature files for UI, all tests almost automatically ended up as cypress tests, because that's how we set up the pipeline from feature -> test. We've ended up with a very top-heavy testing suite for UI now, and I think we should move most of the tests to jest and retain just very high level tests in cypress.

To my knowledge we've not really formalized that (maybe we have and I just don't know it). But it seems important to me. Having to move a ton of cypress tests to jest after the fact is a rather costly endeavour.

[HendrikThePendric]

What @ismay wrote above made me realise that we have probably touched a point here that is not merely about optimising a single process but rather an entire organisation 😱 . Suppose we all agree that having some sort of iterative process with a feedback loop leads to the minimal amount of code that needs to be redone before it's even been released, that should be the preferred solution.

However, because we realise that process would not work for us given the current distribution of roles within the organisation, we develop a preference for a sub-optimal solution, which takes workload away from PDM/UX and increases the workload for the developer. This is perfectly reasonable and pragmatic in the short term.

But by choosing this sub-optimal solution, we sort of obscure the underlying problem and this could then result in that problem never being addressed (i.e. we need more PDM/UX people at DHIS2, but we keep hiring developers). And the result of this could be that we never reach our potential level of efficiency as an organisation.

[amcgee]

1:1:5 PM:UX:Dev is actually the standard ratio for small organizations. We need more of everyone, but particularly more UX. We are never going to have 1:1 PM to Dev ratio, and we need to work within our current framework.

This is a non-sequitur though. I agree with @varl's point about iterative development and @ismay's request for less top-heavy testing (in the UI case particularly). It seems there's a general consensus (among developers, notably) for steps 1-4 and I'll second @varl's proposal for step 5 to come out with:

1. An epic in Jira is added
2. Very high level requirements are gathered and stored in a google doc
3. Over time the requirements become more precise and @cooper-joe can work on the design, which then produces further requirements or increases the precision of the existing ones
4. Once the requirements and the design are final, the different tasks are defined and added as issues in Jira, linked to the epic created in step 1
5. The developer breaks down the JIRA issues into feature files as she works on the implementation.

If this sounds reasonable, 👍 this post and we'll bring it up in the FE dev meeting next week, then loop in the product stakeholders for ratification.