Inconsistencies in SRS Non-Functional Requirements

[BilalM04]

The Non-Functional Requirements (NFRs) seem to have small inconsistencies across several examples, as well as examples not including some NFRs. Additionally, for some examples, the NFR labels end in "-able". Ex. maintainable, understandable, usable, etc. While for others, it ends in "-ability". Ex. maintainability, understandability, usability, etc. @balacij pointed out that the NFRs in the examples should be fairly consistent and should include the same NFRs and labels, apart from the descriptions which are generally example specific. For example:

Single Pendulum:

Game Physics:

PD Controller

Is this something that is intentional, or does it need to be fixed?

[JacquesCarette]

This is definitely something for @smiths .

[smiths]

@BilalM04 you are correct that the NFRs have inconsistencies. Actually, many of the NFRs aren't particularly good. There are two reasons for this: 1) we've prioritized functional requirements; 2) NFRs are challenging.

The challenge for NFRs is that they should be abstract, verifiable, and unambiguous. This is challenging for all software but particularly challenging for scientific software. For instance, for the quality of correctness, how do we really guarantee this for scientific software? Let's say we are trying to solve an ordinary differential equation for which there is no closed-form (analytical) solution available. How do we know we have the right answer? Making it more challenging, what do we even mean by the right answer? We are nearly always using floating point numbers, which are VERY different from real numbers. If we somehow know the right answer is 1.0 and we calculate 0.9999999. Are we close enough? How do we define the error (relative error? absolute error?)? What if the calculated result is a multi-dimensional vector? How do we define the error? (We would normally use different norms of the residual vector, but that in itself involves making decisions that are potentially arbitrary.) I could go on with the challenges. :smile: My solution in my grad class on developing scientific computing software is for projects to write a verification and validation plan. Drasil does not currently generate any documents related to verification and validation. (@samm82's work will hopefully make a contribution in that direction.)

@BilalM04, if we hit a roadblock with your Haskell coding and you need another task to fill your time, we could discuss working on manual versions of a VnV plan for our various examples. (There is some initial work on this in the case studies repo.) I don't think right now is the right time for this though. You are making progress with writing Drasil code, and a VnV plan would be a big tangent at this time.

Instead of trying to fix the NFRs at this time, let's work on incremental improvement, like you did with the maintainability example. @BilalM04 please create some issues for things we can fix with the NFRs. You should put the issues in a project board so that we can more easily track them. Here are some issues that we can start with, feel free to add your own:

• Change all of the labels to be consistent with the original Smith et al template for requirements. That means we should be using the ilities (Maintainability, Usability, etc.)
• Fix the portability requirements. The current requirements are ambiguous. We shouldn't say the code can be ported to multiple operating systems, we should say which ones. In the future we might want to talk about the version of the big three OSs, but for now, let's say the code should be portable to work with Windows, Mac OSX and Linux. You could parametrize this as you did for maintainability so that different examples can pick a subset of the three OSes.
• For performance requirements, let's add a specific competitor that we can compare for execution time. Specifying performance in absolute terms isn't ideal. It can be unambiguous to say that the software will complete task X in Y seconds, but what if it completes the task in Y*1.1 seconds? Does that mean that the software has failed? In some safety-critical applications, it very well could mean that, but for most scientific computing it isn't particularly noteworthy if the software is a little bit slower than expected. Rather than an absolute comparison, a relative comparison is very handy. We need our verification and validation plan to give us the specific test cases, but we can gloss over that for now. Stating the performance requirement relative to other software, like for the game physics example is a good start. We should tighten up the language though and not give the competitor as an example, but rather as the target for the requirement. The competitor can be a parameter that is provided to each example.

[smiths]

The original question has been answered, but let's leave the issue open until @BilalM04 has had a chance to create some associated issues. @BilalM04 please close this issue once the others are created. You should trace back to this issue in those other issues.

[BilalM04]

@smiths the performance requirement only appears in the Game Physics example. Is it worth creating a function and parametrizing the competitor if it only appears in one example? Additionally, if future examples have this requirement, can the performance metric vary, or will it always be execution time?

[BilalM04]

I cannot create and link a project board to this repository. The repository owner, @JacquesCarette, will have to create a project board, then I can add the related issues.

[smiths]

I haven't created a project in a while, so I'm not 100% sure if this is good advice, but the instructions I found have a category of User Projects.

Ideally we don't want the constraint that @JacquesCarette has to create every project. Others have created projects before and I don't believe that @JacquesCarette had to do something, but I might be wrong. @balacij or @samm82 might have some thoughts on the best way to create new projects.

[JacquesCarette]

Projects have changed (I liked the 'classic' ones better...) So now individuals can create projects for themselves, and then I have to link them in.

[BilalM04]

@JacquesCarette I created the project and have invited you to be an Admin. Apologies for taking so long, got caught up in my work with mdBook.

[BilalM04]

@smiths You may have missed my previous comment, but I had a question about your point on the performance NFR:

the performance requirement only appears in the Game Physics example. Is it worth creating a function and parametrizing the competitor if it only appears in one example? Additionally, if future examples have this requirement, can the performance metric vary, or will it always be execution time?

I understand your point about tightening up the language; however, looking back, I may have misunderstood your point about parameterizing the competitor. How does this structure for the performance NFR sound:

"The (metric) for (feature) shall be comparable to (competitor)."

Where the item in the parenthesis will be parameterized.

[JacquesCarette]

Sigh - it looks like I can't. For repos created by an individual, as far as I can tell, only projects by that person can be linked to a repo. So it looks like I'll have to create it.

[JacquesCarette]

Ok, done.

[BilalM04]

@JacquesCarette I am unable to see the project on the repo. It might be the case that it is private, as all projects are private by default and need to be changed to public.

[JacquesCarette]

Should be public now.

[smiths]

@smiths You may have missed my previous comment, but I had a question about your point on the performance NFR:

the performance requirement only appears in the Game Physics example. Is it worth creating a function and parametrizing the competitor if it only appears in one example? Additionally, if future examples have this requirement, can the performance metric vary, or will it always be execution time?

I understand your point about tightening up the language; however, looking back, I may have misunderstood your point about parameterizing the competitor. How does this structure for the performance NFR sound:

"The (metric) for (feature) shall be comparable to (competitor)."

Where the item in the parenthesis will be parameterized.

@BilalM04 you are right that this question was lost in the mix of different conversations. I almost like your parameterized performance requirement. I don't think the second parameter should be characterized as feature, but rather task. The task might use a feature, but the performance is measured by doing something (the task). We also want to quantify the comparison. It is still ambiguous if we say "comparable to".

"The (metric) for (task) shall be within (target)% of (competitor)."

Although game physics is the only example that currently uses a requirement of this form, I could easily see more like this in the future, even with the current examples. Remember, we didn't pay that much attention to the NFRs. There is room for improvement in all of them.

The challenge with parametrizing is coming up with the English to put around it that will work for all cases. I don't think we'll be able to do that, especially without having a long list of potential performance requirements we could use to deduce the pattern. Anything you do in this direction will be an improvement at this time.

[BilalM04]

@JacquesCarette Could I get edit permissions for the NFR Project, I am unable to link new issues.

[JacquesCarette]

Done - I hate these new-style projects, the old ones were way better!

[smiths]

@BilalM04 when you close an issue, please give an explanation of why it is being closed. I believe in this case you are closing the issue because there are more recent issues that supersede it. If that is the case, you should explicitly point to those issues.

[BilalM04]

Question has been answered. New issues have been created for the problems in the current NFRs, as discussed in this ticket. The new issues are #3807, #3756, and #3757. These issues are grouped together under a new project: Fix Inconsistencies in SRS Non-Functional Requirements