Submission 2 (LearnExp/Regular)

[opencert]

Title: Challenges faced by students in an Open Source Software Undergraduate Course

Abstract: The Open Source Software (OSS) development is gaining popularity from year to year, however, entering the OSS community still remains a challenging task. In this work, we describe challenges faced by a beginner OSS code-developer during the first contribution. Additionally, we analyze our experience and offer hints for potential newcomers. Whole work was done as the project of the Open Source Software undergraduate course at the Computer Department of Nazarbayev University. OpenCERT_2019_paper_2.pdf

[opencert]

Assigned PC members:

• Yannis Dimitriadis
• Andreas Meiszner
• John Noll
• Marinella Petrocchi
• Lucia Rapanotti

[sksowe]

Still space for review? I would like to review this paper

[andreasmeiszner]

Feedback to the authors: The paper is informative and written well considering the stage the writers are at current. What would have been nice to see is a comparison of the authors "lessons learnt" against the existing literature so to understand what seem to be common lessons, and what might have been different in the case of the authors. In case the authors still find the time then this should be added.

[yandim]

The paper describes a learning experience with an OSS course and especially with the independent work involving working in an OSS project and reporting back to the class. The paper nicely presents the experience with three projects, and especially the reasons why the author withdrew from the first two projects. The paper also tries to link the community and government structures with the experience of a newcomer in an OSS community, as well as his intentions and preferences. Finally, it provides some interesting lessons learnt. The description of the experience is smooth and structured. Also, some of the links to OSS community features are interesting, although the evidence is not always strong and it is not directly related and compared with similar work in the literature. The paper could be more useful for the community, if such connection to the literature is provided. Some specific comments and recommendations are given below: • P2: "qualitatitive documentation": What is the meaning of this expression? • P3: "figure 1 clearly demonstrates": Please provide concrete links to the arguments- drawbacks for withdrawal. • P3-4: There is a significant amount of "duplicate info" in the firs and second attempt for project selection. The info could be synthesized and presented in an aggregate way (not following strictly a historic account... • P6: Any reference to support or not the claim that a hierarchical structure is beneficial for newcomers? • P10: You should reflect more regarding the number of contributors and "popularity", given that Jarvis is a much younger project. Also, the type and level of contributions should be taken into account. • P10: How have you assessed-estimated the starting complexity? • P12: More discussion could be provided with respect to the rest of experiences expressed in the OSS course. Thus, evidence from a wider set of experiences might be richer and more useful for conclusions for the community. • P13: More discussion is needed with respect to the connection between the lessons learnt and the formal literature review stated in [12].

Most probably, this reviewer will not be available for further discussion on the eventual responses – revised versions by the author, given the vacation period starts on August 29

[Hermioni]

The author reports on their learning experience in relation to an OS software development undergraduate module. The paper describes the OSS project selection and contributions made, and reflects on lessons learnt.

A large proportion of the paper is descriptive of the work done as part of the module: this could be shortened in favour of expanding the reflective part of the work (section 4), which should be of greater interest to the audience. The reflective part could be structured around recommendations both for teachers (i.e., how the overall module pedagogy could be improved to support students better) and for learners (i.e., some advice for peer students).

There is hardly any attempt to relate the work to the literature or to reflect on whether something new has been discovered as part of the learning process.

The paper is generally well structured and organised and the standard of English is good, although not always idiomatic. It is unclear, however, what the screenshots included actually add: some are not described at all in the paper.

Some detailed comments:

the introduction should introduce the paper and its contribution, before getting into the background of the module generally, the past tense should be used throughout as the paper describes a past experience (January 2019) the names of the contributors to each project should be removed, and only the nicknames which appear in the figures should be used p.5: it is not clear what ‘monarchical’ means in relation to the Coala governance structure; also not clear is the meaning of ‘hierarchical’ in relation to the Coala community structure p.8: similarly, it is not clear what ‘federal’ means in relation to the Jarvis governance structure; or ‘anatchist way’ in relation to the community structure p.10: please explain what you mean by ‘resistance for making the first contribution’: resistance by whole? how does it manifest itself. Perhaps the author means ‘reluctance’? The conclusion statement at the end of section 2 seems to come out of nowhere and perhaps should be moved to the end of the reflection section (section 4) p.12: the attempt to compare the personal reflection in section 4 with what was experienced by other students is very weak. It boils down to one sentence: “a large amount of them faced similar problems.” p.13: what is the significance of Steinmacher et al.’s categorisation which is cited here?

[qwireq]

Feedback to the authors: The paper is informative and written well considering the stage the writers are at current. What would have been nice to see is a comparison of the authors "lessons learnt" against the existing literature so to understand what seem to be common lessons, and what might have been different in the case of the authors. In case the authors still find the time then this should be added.

Thank you for your feedback! In "lessons learned" part we used the paper of Steinmacher et al. called "A systematic literature review on the barriers faced by newcomers to open source software projects". This paper is basically the systemized digest of different articles which investigated various types of barriers. It is possible to use these articles to compare an alien experience with each given "learned lessons".

[qwireq]

The paper describes a learning experience with an OSS course and especially with the independent work involving working in an OSS project and reporting back to the class. The paper nicely presents the experience with three projects, and especially the reasons why the author withdrew from the first two projects. The paper also tries to link the community and government structures with the experience of a newcomer in an OSS community, as well as his intentions and preferences. Finally, it provides some interesting lessons learnt. The description of the experience is smooth and structured. Also, some of the links to OSS community features are interesting, although the evidence is not always strong and it is not directly related and compared with similar work in the literature. The paper could be more useful for the community, if such connection to the literature is provided. Some specific comments and recommendations are given below: • P2: "qualitatitive documentation": What is the meaning of this expression? • P3: "figure 1 clearly demonstrates": Please provide concrete links to the arguments- drawbacks for withdrawal. • P3-4: There is a significant amount of "duplicate info" in the firs and second attempt for project selection. The info could be synthesized and presented in an aggregate way (not following strictly a historic account... • P6: Any reference to support or not the claim that a hierarchical structure is beneficial for newcomers? • P10: You should reflect more regarding the number of contributors and "popularity", given that Jarvis is a much younger project. Also, the type and level of contributions should be taken into account. • P10: How have you assessed-estimated the starting complexity? • P12: More discussion could be provided with respect to the rest of experiences expressed in the OSS course. Thus, evidence from a wider set of experiences might be richer and more useful for conclusions for the community. • P13: More discussion is needed with respect to the connection between the lessons learnt and the formal literature review stated in [12].

Most probably, this reviewer will not be available for further discussion on the eventual responses – revised versions by the author, given the vacation period starts on August 29

Thank you for your detailed review! Regarding the comparison, please, look to my previous reply. While here are the answers regarding your comments:

• P2: By "qualitative", I mean good structured, well-written and detailed documentation.
• P3: Yes, maybe I should state the drawbacks explicitly.
• P3-4: Your point is clear, I will work on it.
• P6: This assumption was taken from my own experience and experience of my classmates, of course, no documented survey was conducted.
• P10: Could you, please, elaborate on this point? I am not sure I clearly get your idea.
• P10: The starting complexity was assessed by the following criteria: the effort spent for understanding the code of the project, the amount of knowledge of the project needed to perform a contribution, time spent on learning the documentation to get familiar with the project and its order of contribution, the effort spent for getting assigned for an issue.
• P12: Do you mean the educational part of the course (lectures, presentations, etc.) by a wider set of experiences?
• P13: The discussion could be broadened by making more comparisons with similar works in literature and subsequently connecting it with the review paper and its categorization.

[qwireq]

The author reports on their learning experience in relation to an OS software development undergraduate module. The paper describes the OSS project selection and contributions made, and reflects on lessons learnt.

A large proportion of the paper is descriptive of the work done as part of the module: this could be shortened in favour of expanding the reflective part of the work (section 4), which should be of greater interest to the audience. The reflective part could be structured around recommendations both for teachers (i.e., how the overall module pedagogy could be improved to support students better) and for learners (i.e., some advice for peer students).

There is hardly any attempt to relate the work to the literature or to reflect on whether something new has been discovered as part of the learning process.

The paper is generally well structured and organised and the standard of English is good, although not always idiomatic. It is unclear, however, what the screenshots included actually add: some are not described at all in the paper.

Some detailed comments:

the introduction should introduce the paper and its contribution, before getting into the background of the module generally, the past tense should be used throughout as the paper describes a past experience (January 2019) the names of the contributors to each project should be removed, and only the nicknames which appear in the figures should be used p.5: it is not clear what ‘monarchical’ means in relation to the Coala governance structure; also not clear is the meaning of ‘hierarchical’ in relation to the Coala community structure p.8: similarly, it is not clear what ‘federal’ means in relation to the Jarvis governance structure; or ‘anatchist way’ in relation to the community structure p.10: please explain what you mean by ‘resistance for making the first contribution’: resistance by whole? how does it manifest itself. Perhaps the author means ‘reluctance’? The conclusion statement at the end of section 2 seems to come out of nowhere and perhaps should be moved to the end of the reflection section (section 4) p.12: the attempt to compare the personal reflection in section 4 with what was experienced by other students is very weak. It boils down to one sentence: “a large amount of them faced similar problems.” p.13: what is the significance of Steinmacher et al.’s categorisation which is cited here?

Thank you for the review! Also, thank you for your idea about the reflective part, it is a good point. Could you, please, elaborate on your point about illustrations? I think I am not getting your argument. The following are the answers to your comments:

• Fair point, I will broaden the introduction.
• Thank you, will fix past tense issues.
• I will also fix the naming issues.
• p5: by monarchical, I meant that most of the issues connected with the administration of the project are solved by a single person. Hierarchical community structure in case of Coala means that not every developer has equal rights. In order to have more rights, one should follow the stage rule of the project: "to solve issues with a particular level of difficulty, you need to solve at least one issue and review at least one contribution that is one level of difficulty below".
• p8: by federal, I meant that the questions about new features or connected with current code are solved by a group of people using GitHub forum as a discussion place, while by 'anarchistic way of contribution' I meant that anyone could contribute to any issue without any restrictions in contrast to Coala with its rules.
• p10: by 'resistance', I meant the following: as time passes and a project becomes more adult, the amount of easy to solve bugs decrease, as well as, the requirements on the knowledge of the project to make the contribution grow. In other words, as the project gets older it becomes more polished and its issues become more sophisticated. So a developer needs to spend more effort to contribute. So, now is the conclusion of section 2 more reasonable?
• p12: Fair point, I will work on this issue.
• p13: The significance is in the message that despite the growth of open source movement, the quality stays the same so that contributors face the same issues again and again. This leads to the idea that the organization of OSS should be enhanced in order to overcome the barriers. For example, by creating some common organizational criteria which are mandatory in order to a project be part of the OSS community.

[qwireq]

Here is the revised version of the paper. Thank you for your feedback @Hermioni, @yandim, @andreasmeiszner! Revised-submission_OpenCERT_2019_paper_2.pdf

[opencert]

Camera-ready version for the post-proceedings. paper_2.pdf