How to improve the bug report template?

[milancurcic]

Here's what the bug report form looks like: https://github.com/fortran-lang/stdlib/issues/new?assignees=&labels=bug&template=01_bug.yaml

and the PR that implemented it: #513

In #561, @certik wrote:

This form is so hard to fill out that I almost gave up on reporting this.

I would suggest to either get rid of it, or simplify this.

Ondrej, can you explain why was it so hard to fill out? I'm looking at the #561, and it seems like if it weren't for the form you'd still write out all the same information. Do you agree?

I opened this issue to find out how we can improve it. As I see it, this form is as simple as it can get. I think that any issue that contained less than the required information would be incomplete. Let's look at it field by field:

• Title: Required by GitHub
• Description: We don't have an issue without this field.
• Expected behavior: This could be redundant in scenarios where the issue is that "it doesn't build", and the answer is the obvious "it should build". So, perhaps we could make this one optional.
• Version: Necessary for reproducing. Many newcomers are not aware that this is important.
• Platform and architecture: As with version.
• Additional information: Of all, this one seems the most redundant but it's also optional. I think we could get rid of it, as any additional information could be written in the "Description" field.

I've opened issues in in many terribly long templates, and even walked away from some because they were too much work. But this template is by far among the simplest I've seen. Nevertheless, we can still improve it and we should lower the bar as much as we can while still requiring the info we need. To that end, I suggest to:

1. Make "Expected behavior" an optional field
2. Remote the "Additional information" field.

What do you think?

[certik]

I had few minutes to report the bug. I went to open up an issue and it gave me options to choose from. I chose a bug report. The form gave me all kinds of fields to fill out. And it didn't allow me to submit it without filling them. I eventually just put "xxx" in them just so that I can submit it. I feel I provided all the information needed to track the bug.

The experience made me feel unwelcomed and inadequate to fill it out "correctly". And then based on the feedback I got in #561, I feel my feedback was not appreciated.

So I didn't have a good experience reporting the bug report. If you agree with me, then let's improve it.

If you don't agree with me, then we can leave it as is.

[milancurcic]

Thanks Ondrej. What do you think about my suggestions for improvement above?

[certik]

Yes, I think we should make all of the fields optional. If somebody is trying to report a bug, I think we should make it easy to do so. I would not even do a template honestly. But I also recognize that my experience might not generalize, so if you and others think the experience of the bug reporting is great, I am fine with it.

[milancurcic]

I'm in favor of keeping the bug report with modifications I suggested in the top post. It's the info we need and it's common for newcomers to omit this kind of info (version, compiler, platform). Maybe not in this repo because it's early and there have only been few bug reports, but in busier repos I see it all the time.

And we should keep the free-form issue for those who don't want to fill out the form.

Let's hear what others think.

[awvwgk]

The expected behavior can be redundant in case of a build failure or their like, we can make it optional. I would keep the stdlib version, compiler and platform required, those are the information that are most important to reproduce the issue, even if this information might already be contained in the description.

We could also mention the bug-report form and free-form issue in the contributing guidelines explicitly: https://github.com/fortran-lang/stdlib/blob/master/CONTRIBUTING.md#reporting-a-bug

[LKedward]

I'm also in favour of keeping the template and agree with the suggestions in Milan's first comment. The template isn't excessively long IMO, and it makes sure all the necessary information is posted upfront. (This is useful for both newcomers as well as experienced developers since it's easy to forget this information when focusing on the bug.) If it really is too restrictive, then the free form option is available right below the bug report button.

[certik]

It looks like all three of you (@milancurcic, @awvwgk and @LKedward) agree to have a template, so let's have a template.

I personally disagree. Let me explain why.

My own opinion is that at this point in the project we should make it as easy as possible for users to report bugs. Once we have thousands of issues and tons of users and are well established, we should offload some of the burden to the users (with requiring a template).

When reporting bugs, 20 years ago one of the most popular systems was Bugzilla, where you had to fill out tons of forms and the process was very tedious. Then I believe Google came with Google Code Hosting (now defunct) and allowed to just open an issue and you (as a user) just put whatever info you feel like. That made it so easy to report bugs and I really liked it. Then GitHub came and allowed the same with the issues. I really like it as a user. It takes just a few seconds to record a bug. Then other users can add more info and we can collectively figure out the problem and find a solution.

I see GitHub issues not as a way to report a bug from the user to a maintainer, and for them to fix it. Not at all. I see GitHub issues as an interface to record a problem from a user, for the other users. If I am the only one having an issue, it might be low priority. If there are several users having the same issue, we can come together to collaborate on a fix. There is no shame in having lots of open issues.

Here is a blog post that expands on this idea:

http://web.archive.org/web/20211028135121/https://drewdevault.com/2021/10/26/stalebot.html

(The original site seems down today, so I posted a wayback machine.)

I agree with the premise.

My goal is to remove friction and to lower the bar for users to use our software and to engage with us and to report bugs. All our bug reports so far seem very high quality. So making it simple, easy to report bugs, and to encourage users to report bugs and issues and ides they have for our software is my priority.

Also, I personally do not like the selection you have to make to report a bug, here is how it looks like for me:

I didn't even see there is "no template" option at the bottom. It seems like something you should not do and is not encouraged.

It seems I am in the minority at the moment, so I respect that, and let's continue to have templates for now. I also want to apologize to @awvwgk, I should have written the above points much more carefully and in length. I also encourage all of you (@milancurcic, @LKedward, @awvwgk) to think about the above points. We can also discuss it at our monthly call.

[milancurcic]

Thank you Ondrej for the detailed explanation. I agree with some points and disagree with others.

I agree that it should be as easy as possible to report. I also agree that the project may be too young for requiring detailed reporting. I'm usually in favor of implementing things only as problems arise, rather than preemptively. At the same time, others were considerably more active than me at triaging incoming issues, so if they say issue triaging should be fixed or improved, I trust them.

It's also okay for there to be many open issues (no shame about it, nor I think anybody suggested we should be ashamed about it), however I do want us to eventually address them and don't want to treat the issues as an infinite mailbox of unopened letters. Though I really like Drew DeVault's blog and agree with it (stalebot made me feel unwelcome many times), I think it's distracting here from the main point, which is how to improve or whether to keep the bug report form. I don't think anybody so far has suggested that we want fewer rather than more open issues. I want as many open issues as possible, but I want issues to be considerate as to not waste anybody's time. For example, if somebody writes a hasty complaint that something doesn't work and doesn't provide needed information, we have no choice but to either ignore it or ask for more information. Granted, we haven't had many such reports, and as you say, most issues have been high quality.

And, I want duplicates removed so that it's easier to search through the open issues. I'm curious, did you feel like your issue was unwelcome because it was closed as a duplicate?

What hasn't been mentioned yet is that a structured bug report helps readers find the information exactly where they expect it, rather than having to scan every issue. This becomes more of a problem as the project scales, I don't think it's a big issue yet.

I think your criticism of the bug report is 100% valid and helpful feedback. At the same time, it's also the first time we heard about it. These forms were introduced relatively recently so we're experimenting to find what works best.

Indeed, the form selection page is a friction point--one more thing to make a decision about and one more page to click through. I believe the original rationale for it was to stream open-ended discussions away from issues and into Discourse or GitHub Discussions. Personally I'm fine with all stdlib discussion being in issue format rather than streaming it elsewhere. But I'm also not opposed to the current approach. To help alleviate this friction point, I'd further suggest two more modifications:

1. Remove either Discourse or GH Discussions. In other words, offer one preferred choice rather than two.
2. Remove the mailing list choice (mostly inactive)
3. Place Free Form higher up on the list, perhaps number 1.

So perhaps the choices could be:

• Free form
• Feature proposal
• Bug report
• Discussion

Now that I type this out, I'm wondering if we can reduce it to 3 options by bundling the "Feature proposal" into "Free form" or "Discussion"? And under "Free form" we could write a description "For feature proposals, questions, or other general inquiries" or similar.

What do you think?

[awvwgk]

I edited a few post in this thread, I hope you are okay with the change, if not feel free to revert.

---

One important aspect in this discussion is where to direct certain input. While it is possible to use GitHub issues for discussions, I don't think they are the right format for longer threads, because we have better alternatives, like our discourse. Now we also have GitHub discussions, but it still suffers from the same technical issues, mainly that long discussions get hidden at some point and become difficult to navigate, which deteriorates the user experience unnecessary IMO. Now might be a good occasion to revisit the external links and prune some of the choices.

[certik]

To move forward, doing something like:

• Free form
• Feature proposal
• Bug report
• Discussion

Would work. Or even simpler.

One aspect that hasn't been discussed yet is that at this stage of the project, it is also extremely valuable to just hear feedback from users that "it doesn't work on my system" and then work with them to figure out the details. Once (or if) we let go of the idea that every report must be (immediately) acted upon, suddenly it's not a big deal. From time to time we can prune issues and close those that just don't have enough info for us to act on.

There is no right or wrong answer here. All I am suggesting is to move more in the direction of "free form". I think having a template as an option is fine. As long as it is fine to also just use the free form, even for bug reports.

To answer some specific points:

• @awvwgk's edits to my post are fine

• I'm curious, did you feel like your issue was unwelcome because it was closed as a duplicate?

No, that was totally fine. In fact, that is another argument for free form --- given that it will not be the issue we use to track things down, it's ok to have a lower bar.

• nor I think anybody suggested we should be ashamed about it [issues]

I agree that nobody suggested it

• In general, if I was suggesting an idea and then addressing it in my post above (or any of my posts), I didn't mean that anybody suggested it. I am doing it to immediately address some of the more obvious points, so that we can move the discussion forward. There is nothing personal in it, nor any implied criticism of anyone.

• I agree with @awvwgk and others that it is very good to direct users where the discussion should happen. I think we use the Discourse forum, GitHub discussions and issues. I am not sure if I like GitHub discussions, they seem more like issues. So perhaps we can even simplify and just recommend Discourse (for discussion) and GitHub issues for more specific (permanent or technical or actionable) feedback.

• Another aspect that was hinted (but not fully said I think) by @milancurcic and that I agree with is that "those that do the actual work should ultimately choose, both the design and how to work with issues / workflow". I think Linus Torvalds is pretty vocal about exactly this point. If we look for the past two years, the top three contributors based on https://fortran-lang.org/community/ are @certik, @jvdp1, @milancurcic. If we reduce it to some more recent period, say last year, it is @awvwgk, @milancurcic, @jvdp1. So those are people that should have a say. I think we all agree that we need to strike some happy medium, these people should not rule over the community, but at the same time, as @milancurcic said "At the same time, others were considerably more active than me at triaging incoming issues, so if they say issue triaging should be fixed or improved, I trust them.".

@milancurcic it seems that you agreed with everything I wrote. Can you point out specifically to what you disagree in my post?

[milancurcic]

@certik, the argument is evolving in a good direction, and some things that I thought you implied are not an issue/disagreement anymore. It took us a few times to iterate toward an agreement. I don't want to go back to nit-pick specific statements in posts that I thought I disagreed with, but rather I'd like to write if there's anything I disagree with in the present.

If there's anything now, it's this premise:

I see GitHub issues not as a way to report a bug from the user to a maintainer, and for them to fix it. Not at all. I see GitHub issues as an interface to record a problem from a user, for the other users.

and supported by Drew DeVault's article.

In general, for a mature and busy open source project with many contributors, I like that and think it works. The snowball is big enough that eventually users who hit the same issue come together, triage, clarify it, refine it, and come up with a fix. It's how I think it should work and is our long-term goal. However, for a young project like all Fortran-lang projects, I don't think it works yet. I think you need dedicated and clear leadership/maintainers to do the heavy lifting until there is critical mass (snowball is big enough to roll on its own).

But since these are merely your and my perspectives, it's not really a hard disagreement (i.e. you can't disagree with somebody's perspective, only with a factual statement), so that may have been a too strong of a word for me to use here.

It seems to me we all agree where we want to go, we may just not have completely converged on how to get there. I think all arguments in this thread have been compelling.

Another aspect that was hinted (but not fully said I think) by @milancurcic and that I agree with is that "those that do the actual work should ultimately choose, both the design and how to work with issues / workflow".

Yes, exactly, thank you for crystallizing it for me. I 100% agree, and we can't look at just https://fortran-lang.org/community/, but also at https://github.com/fortran-lang/stdlib/graphs/contributors, and who volunteered as a maintainer at #484, and who has been lately active at triaging. And we need to solicit feedback and listen to everybody else if they have an opinion to share. That's why I agree we should bring this up at the monthly call.

[ivan-pi]

Good discussion. I'm short on time, but I'll try to give my two cents:

• When the templates were introduced, I was quite confused and open a blanked issue instead on a few occasions. I've taken a look at the latest templates and they look more helpful now.
• Personally, I find the GitHub discussions more suitable for talk between developers than Discourse, and also as more direct channel for users seeking help with installation/usage. I would not like to see too many of such issues "polluting" Fortran Discourse (at least not in the current state, where the landing page does not separate based upon sub-topics). Having such discussions close to their respective projects instead of lost among dozens of Discourse threads seems better to me.
• I don't know if I'm subscribed to the mailing list, but I dislike having it among the templates. I'd prefer to use the mailing-list for announcements from fortran-lang, or as an open channel to receive mail from other organizations/communities.
• I'm worried that querying version could be a barrier for new users. For fpm, the template could be amended to suggest the fpm --version command. For stdlib, I am not sure where to look for the information. If I cloned and built stdlib myself, I might think of doing git log and give the last commit, but for someone using stdlib via fpm or extracting the release package it could be another barrier.
• Merging the Expected behavior / Additional information, and making the single box optional, would increase the simplicity of the template.
• The description of the templates should use verbs to clearly convey the expected action. Looking at Scipy, it has three templates:
  • Bug Report: Create a report to help us improve SciPy
  • Feature Request: Suggest an idea for SciPy
  • Report a security vulnerability: Please review our security policy for more details
• Similarly in Numpy we find among templates:
  • Bug report: Report a bug. For security vulnerabilities see Report a security vulnerability in the templates.
  • Documentation: Report an issue related to the NumPy documentation.
  • Feature request: Check instructions for submitting your idea on the mailing list first.
  • Post-install/importing issue: Report an issue if you have trouble importing or using NumPy after installation.
  • Two more links are given for Question/Help/Support and Development-related matters, which is what our fortran-lang projects currently use GitHub Discussions for (due to their small size).

[certik]

@milancurcic ok, looks like we are in agreement now.

Regarding:

I see GitHub issues not as a way to report a bug from the user to a maintainer, and for them to fix it. Not at all. I see GitHub issues as an interface to record a problem from a user, for the other users.

My view was actually that for a large project with a lot of users and lots of reports, you can actually afford to ask your users for more via a template, because you have a lot of reports, so you trade higher quality bug reports for a smaller number of them (since some people will be discouraged, but you will still get plenty).

But for a small project, like stdlib, we don't want to push anybody away who is willing to open up an issue.

[milancurcic]

@certik, I understand, and that's not the point I was addressing (I agree with it). I interpreted this:

I see GitHub issues not as a way to report a bug from the user to a maintainer, and for them to fix it. Not at all. I see GitHub issues as an interface to record a problem from a user, for the other users.

as:

Let the GitHub issues go unmanaged and eventually they'll triage themselves by the community.

which I don't think is a good strategy for a young Fortran project. It's quite possible that you didn't mean it in that sense, but that's how I understood it.

[certik]

I see. That's not what I meant by it.

I don't believe issues "fix by themselves". I don't believe an open source project "develops by itself".

Somebody has to be in charge and be pushing it. Others join. But if there is nobody leading it, it will fail. That's what I believe.

What I meant by:

I see GitHub issues not as a way to report a bug from the user to a maintainer, and for them to fix it. Not at all. I see GitHub issues as an interface to record a problem from a user, for the other users.

Is a suggestion to consider not treating issues as a "free order from the community" to the maintainers that they must fix (for free and quickly). But rather to treat them as means to communicate with our users. To lower the bar to report as much as possible. If the report has not enough info, the maintainers should ignore it (they can ask for more info, but don't even have to do that). If users want maintainers help, then maintainers can ask for more info.

[milancurcic]

Thanks all for the productive call. Here's my understanding of our takeaway for going forward:

1. Reduce issue options to:
   • Bug report
   • Feature proposal
   • Other (blank issue)
   • Discuss on Fortran Discourse (link)
2. For Bug report and Feature proposal, use templates instead of forms. Templates can include instructions on, e.g., how to get the version or commit hash of the project that the report refers to.

This still maintains the friction point of having an additional page where the user has to choose. One way we could further help minimize this is to place prominent links in the README.md at the top. For example, like this:

---

Fortran Standard Library

Report a bug | Propose a feature | Open a general issue | Discuss stdlib on Fortran Discourse

Clicking on one of these links bypasses the issue selection page.