[PRE REVIEW]: Nashpy: A Python library for the computation of Nash equilibria

[whedon]

Submitting author: @drvinceknight (Vincent Knight) Repository: https://github.com/drvinceknight/Nashpy Version: 0.0.14 Editor: @RobertTalbert Reviewer: Pending

Author instructions

Thanks for submitting your paper to JOSE @drvinceknight. The JOSE editor (shown at the top of this issue) will work with you on this issue to find a reviewer for your submission before creating the main review issue.

@drvinceknight if you have any suggestions for potential reviewers then please mention them here in this thread. In addition, this list of people have already agreed to review for JOSE and may be suitable for this submission.

Editor instructions

The JOSE submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @labarba it looks like you're currently assigned as the editor for this paper :tada:

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof :page_facing_up: <--

[labarba]

@whedon assign @RobertTalbert as editor

[whedon]

OK, the editor is @RobertTalbert

[labarba]

Hi @RobertTalbert -- I've assigned you to edit this new submission, in all its mathy goodness! You can email me if you have any questions. Note that if you want our bot to create checklists in the Review issue for two reviewers, you have to assign then add reviewers here, and then start the review. I discovered this the hard way!

[labarba]

@drvinceknight -- Please have a look at the discussion we had in this pre-review issue https://github.com/openjournals/jose-reviews/issues/11#issuecomment-392291689 -- We are expecting the JOSE paper to touch on a few points that will help others know how they might adopt this for teaching, or for self-learning. We will also be looking for a narrated worked example, so please tell us where to find that in your repo.

[willingc]

@drvinceknight Probably makes sense to add to your example the more modern name "Bach or Stravinsky" along with 'Battle of the Sexes'.

[drvinceknight]

Thank you both for your comments.

OK, the editor is @RobertTalbert

Disclosure: Robert and I know each other and I consider him a friend. (I don't think this is a problem here.)

We are expecting the JOSE paper to touch on a few points that will help others know how they might adopt this for teaching, or for self-learning. We will also be looking for a narrated worked example, so please tell us where to find that in your repo.

@labarba after your suggestion on the JOSS review I have added an example use case to the paper. I also discuss the documentation and in the paper point to the url (http://nashpy.readthedocs.io/en/stable/ - this is also signposted on the github repo) which gives a thorough walk through not just of the software but also of the background algorithms. I'd happily modify this if it's missing something.

Please have a look at the discussion we had in this pre-review issue #11 (comment)

Regarding these I'm sure I am probably again missing what it is that needs to be improved but if you could be a bit more specific I'd happily address them:

what need does it satisfy? how is it eligible for JOSE?

The third paragraph I believe captures this: highlighting the portability of the library and the documentation.

how has it been used? how would someone else use it?

The last paragraph says this: detailing that it could be of use to researchers but also (importantly for JOSE) to courses in a number of fields. I also specify that it is currently being used by one such class.

what's the story behind it? what's the effort the author(s) have put in and their expertise to teach this?

This is also covered by the third paragraph (the number of algorithms implemented and the documentation).

Probably makes sense to add to your example the more modern name "Bach or Stravinsky" along with 'Battle of the Sexes'.

Thanks @willingc, I have addressed this in 97da11b by removing the name of the game entirely (it is not necessary).

[RobertTalbert]

Hey there, I was on vacation this past week and am just now getting around to this. There'll be some movement on this starting tomorrow.

[RobertTalbert]

@drvinceknight -- Do you have any suggestions for potential reviewers? I can just pick ~one~ two from the list if you don't.

[drvinceknight]

@RobertTalbert one sensible suggestion would be the maintainer of gambit: https://github.com/tturocy but otherwise I suspect picking any from the list would be good. :+1:

[RobertTalbert]

@drvinceknight -- Again I apologize for not getting to this earlier. I am going to be recruiting two reviewers to your paper, one being the person you suggested and another that seemed a good fit from our editor database. These folks have to agree to do the review, of course.

Additionally, I had a few recommendations based on the Author Guidelines.

1. Everything in the submission should be open-source; the code in the submission here it MIT licensed, so that's good; the documentation, however (nashpy.readthedocs.io), has a straight Copyright on it (in the footers of the pages) and the docs should instead be Creative Commons licensed. At least, it was a little ambiguous because the entire repo is MIT licensed but it says "Copyright" on the docs.

2. Although the "paper" part of the submission should be < 1000 words, this one might be a little too short, and in particular it needs a clear "Statement of Need" (clearly stating the need for the module and who the target audience is).

3. The documentation part, too, could use a little more in the way of pedagogical content in terms of the examples present (right now there's just one example, Rock-Paper-Scissors) and the pedagogical theory involved in the module and its use. There are clear "how-to" instructions in the docs but they could use more in the way of explaining how the module would be used in a real instructional setting and the learning issues that might be involved in doing so (cognitive load, etc.).

In short the docs and the paper need more fleshing out for how instructors might use the module in the context of a course (or self-teaching) in addition to the already-clear instructions on how to install and execute the code.

Feel free to follow up on those comments in the thread. I'm out of town all next week but will be checking in regularly.

[RobertTalbert]

Also @drvinceknight do you happen to have @tturocy 's email or twitter handle? I need to contact them to ask about reviewing with some details, the length of which might not be appropriate for this pre-review thread. :)

[drvinceknight]

Additionally, I had a few recommendations based on the Author Guidelines.

Everything in the submission should be open-source; the code in the submission here it MIT licensed, so that's good; the documentation, however (nashpy.readthedocs.io), has a straight Copyright on it (in the footers of the pages) and the docs should instead be Creative Commons licensed. At least, it was a little ambiguous because the entire repo is MIT licensed but it says "Copyright" on the docs.

I'm afraid I don't understand. Copyright and license are two different things https://opensource.stackexchange.com/questions/297/whats-the-difference-between-copyright-and-licensing having an open source license with copyright is pretty standard (a lot of open source modules for example include the copyright of specific authors on each page).

Although the "paper" part of the submission should be < 1000 words, this one might be a little too short, and in particular it needs a clear "Statement of Need" (clearly stating the need for the module and who the target audience is).

I believe that I'm clearly stating the need with statements like:

"The state of the art in terms of software implementations of these algorithms is Gambit [@mckelvey2006gambit]. Gambit includes a python wrapper to its core C functionality however is not currently portable (for example Windows is not supported). Nashpy is a Python library with all dependencies being part of the standard scientific Python stack (numpy and scipy [@scipy]) thus it is portable. "

"Nashpy is designed to be used by researchers and students in courses in the fields of mathematics, computer science and/or economics."

I am clearly misunderstanding what is expected here as twice now I've been invited to improve this.

The documentation part, too, could use a little more in the way of pedagogical content in terms of the examples present (right now there's just one example, Rock-Paper-Scissors) and the pedagogical theory involved in the module and its use. There is clear "how-to" instructions in the docs but they could use more in the way of explaining how the module would be used in a real instructional setting and the learning issues that might be involved in doing so (cognitive load, etc.).

In short the docs and the paper need more fleshing out for how instructors might use the module in the context of a course (or self-teaching) in addition to the already-clear instructions on how to install and execute the code.

I am beginning to believe that perhaps I have misunderstood the scope of JOSE and before proceeding it would perhaps be worth I clarify this in my mind. My assumptions are based on reviewing for JOSE (the paper I have submitted I would suggest is very similar to one I have reviewed) and reviewing for JOSS where the idea is to write a very short/brief paper that accompanies the software.

I had originally submitted this paper to JOSS but was invited to move it to JOSE, if that requires a major rewrite with a lot of pedagogic theory and duplication of documentation then I think I would rather find time to write a paper about my actual course (which I am thinking about doing). For example including a discussion of cognitive load in the paper doesn't necessarily apply here as it's "just" a piece of software that's easy to install that computes equilibria. However a broader discussion about the course within which I use it would of course have such a consideration.

Perhaps JOSS was the correct place for this work. (Or indeed perhaps somewhere else altogether.)

[drvinceknight]

Perhaps JOSS was the correct place for this work. (Or indeed perhaps somewhere else altogether.)

Looking at http://jose.theoj.org/about

and the section on "educational software tools" which applies here (it is not an "open-source educational materials").

What do you mean by "educational software tools"?

Open-source software that serves as educational technology; examples include (but are not limited to) alternatives to learning management systems, autograders, cloud systems for lesson delivery, student collaboration tools. For these tools, peer review will follow a similar process as JOSS.

this does make me believe that perhaps Nashpy's place (it is a piece of software that does what is taught in game theory classes not anything of the kind of an autograder, student collaboration tool etc...) is not here (but I'd suggest that if that's the case it's also true of a paper that I've reviewed and has been accepted) and it was better suited to JOSS https://joss.readthedocs.io/en/latest/submitting.html#what-should-my-paper-contain.

[labarba]

I'm afraid I don't understand. Copyright and license are two different things https://opensource.stackexchange.com/questions/297/whats-the-difference-between-copyright-and-licensing having an open source license with copyright is pretty standard (a lot of open source modules for example include the copyright of specific authors on each page).

The issue is that you have not put a license (e.g., CC-BY) on the documentation. MIT License for code, CC-BY for non-code content.

[drvinceknight]

The issue is that you have not put a license (e.g., CC-BY) on the documentation. MIT License for code, CC-BY for non-code content.

Thanks that makes sense. Is that not covered by the general license of the github repo? (The source documentation rst files are under the MIT license with the rest of the code) https://github.com/drvinceknight/Nashpy/blob/97da11b74d6d230c7c58793309762bdbd56d7b2a/LICENSE#L6:

"Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"),[,,,]"

[labarba]

Regarding the scope, I agree that this submission is similar to Pynamical, and not properly a learning module, as we originally envisioned for JOSE. If you explore a bit the conversations in the Pre-review and Review issues of the few first submissions, you'll see we've been having a conversation about this type of submissions, which don't fit in our original categories.

Like Pynamical, and also PyLJ, under review, this can be thought of as "the software part" of a learning module. In the Pre-review issue for PyLJ, I tell the author: "Your software may be useful by itself as a teaching tool, but it will be much more useful with full worked-out examples." The same applies here.

[drvinceknight]

Yeah completely realise this is early days for JOSE and that the scope is perhaps evolving. Not a problem at all, just trying to understand :)

Sounds like I should take the software elsewhere (perhaps back to JOSS or indeed a game theoretical journal perhaps) and as intended https://github.com/openjournals/joss-reviews/issues/761#issuecomment-398056671 I can put together a more involved piece of work on the actual course itself which I'd be delighted to submit here (although it'll take me a little while to get to that) :) Does that sound about right?

[labarba]

Regarding eligibility for JOSS, see the discussion on the Pre-review for Pynamical. I think this submission would be in a similar position.

[labarba]

In sum, I think this submission IS appropriate for JOSE, but you are being asked to include a complete worked-out example that would illustrate to another instructor how they could use this software in their own teaching, and would give a self-learner an entry point.

The Statement of Need is a required section of the paper (under its own heading). It should say who the intended audience is, how someone could adopt the software, if it's been used in the classroom already, and what's the "story" of this software.

We're not asking you to do "a major rewrite with a lot of pedagogic theory and duplication of documentation". But we do ask you to write a paper that is the "advertisement" to your software, for the purposes of other teachers to know how to teach with it.

[drvinceknight]

Yes you pointed that conversation out to me when you suggested I bring Nashpy here. I must admit that I'm confused (similarly to https://github.com/openjournals/joss-reviews/issues/761#issuecomment-398056671).

I reviewed the Pynamical paper in a very similar fashion to the way I've reviewed for JOSS, I believe this paper is very similar to the Pynamical one but perhaps I made a mistake when reviewing the Pynamical one.

I think I would like to withdraw this paper from JOSE as it does not fit in the scope of the journal (I think?).

In terms of justifying if the package is a research package I'd be happy to have that discussion if and/when I submit it to JOSS.

When I get a bit more time I intend to write a paper about my actual game theory course and perhaps by then there will be more papers in JOSE that I can compare to to see if it fits within the scope :) (Nashpy is actually quite a small part of that course, as it uses a number of other bits of software glue to deliver a flipped class using jupyter notebooks :)).

Thanks again for taking the time to explain and walk me through, apologies if I've been slow to understand.

[drvinceknight]

We cross posted:

In sum, I think this submission IS appropriate for JOSE, but you are being asked to include a complete worked-out example that would illustrate to another instructor how they could use this software in their own teaching, and would give a self-learner an entry point.

I will take some time to think about this point (I'm clearly misunderstanding as it's been asked of me a few times) and either withdraw or attempt to satisfy the request (an actual heading on the statement of need is easy enough :+1:))

If it is within the scope of JOSE would this be as a "open-source educational materials" or an "educational software tools" ? http://jose.theoj.org/about (Still trying to understand.)

[RobertTalbert]

Just to add to this, I'd echo what @labarba said, that I think making this a great JOSE submission is a matter of reorganizing and highlighting what's already in it (e.g. the statement of need) and adding a small amount of practical detail on classroom usage. If you'd rather do that at a later date and focus on JOSS for now, I think we all respect that.

Also it's definitely the case with JOSE that we are putting the wings on the airplane while it's currently in flight, so situations like this are good data. Thanks for all your responses.

[drvinceknight]

Also it's definitely the case with JOSE that we are putting the wings on the airplane while it's currently in flight, so situations like this are good data. Thanks for all your responses.

I am hoping this exchange is helpful and my tone is coming across in the correct way (always tricky by text). I'm super excited about this journal and appreciate you all doing the work you do for it :+1:

[RobertTalbert]

If it is within the scope of JOSE would this be as a "open-source educational materials" or an "educational software tools" ? http://jose.theoj.org/about (Still trying to understand.)

@labarba might have more to say about this, but my take is the following: JOSE submissions are intended to be a combination of tools, materials, and writeups that explain the basis for those tools and materials and how they should/could be used in the classroom. For example with NashPy, submitting the code is awesome and needed; what also would need to be there is some significant guidance for how it could be used, based in sound pedagogical theory.

For example if you had a use case for someone using NashPy in a flipped classroom situation where students play around with the module as part of their pre-class experiences, describing that fully (how students should approach using it, what cognitive or learning issues students might encounter while using it before class, etc.) and/or providing some instructional materials that could be deployed before/during class that use the module would be ideal. I think the gist is that JOSE submissions should be complete packages that give instructors or learners both the tools and the materials wrapped around those tools to use in a learning situation. That's my take at least.  ¯_(ツ)_/¯

[drvinceknight]

I think the gist is that JOSE submissions should be complete packages that give instructors or learners both the tools and the materials wrapped around those tools to use in a learning situation. That's my take at least.

Are you saying that in this case for Nashpy it would be something like:

• Game theory: what it is
• Nashpy: what it is
• Multiple use cases possible but the following is one approach:
  • explaining the theory of Nash equilibria,
  • inviting students to use Nashpy to verify their own answers to similar problems
  • also show students the implemented algorithms which refers back to the theory
  • include a reference or two on the subject of active learning and repetition
• discuss documentation and other aspects of the software itself
• statement of need

Is that along the right lines?

[RobertTalbert]

Just a suggestion and not anything official, but here's what I could see:

• Description Nashpy
• Statement of need (Who needs this? What problems does it address?)
• Multiple use cases that mirror actual instructional situations and (maybe) provide sample materials, such as:
  • Pre-class assignment for a flipped learning environment where students use Nashpy to discover things about Nash equilibria -- along with something brief about what the assignment is, principles involved in its design, expected learning outcomes, and possible pitfalls
  • In-class activity where students are working in pairs or small groups to use Nashpy to learn something or accomplish some task about Nash equilibria -- again along with something brief about what the assignment is, principles involved in its design, etc.
  • Extended student work intended to be done following a class meeting, like a miniproject, that uses Nashpy, with the same explanation and background as above
• Then end the submission with technical stuff like documentation, aspects of the software etc.

Again this is not official, just kind of what I see as ideal submissions to JOSE. Others can correct me on that.

[RobertTalbert]

Also I don't think there would be any need to describe the mathematical background here -- no detailed description of game theory necessary (i.e. you could mention it, but otherwise assume that the reader has the background knowledge necessary to put Nashpy to use) .

[drvinceknight]

Multiple use cases that mirror actual instructional situations and (maybe) provide sample materials, such as:

Pre-class assignment for a flipped learning environment where students use Nashpy to discover things about Nash equilibria -- along with something brief about what the assignment is, principles involved in its design, expected learning outcomes, and possible pitfalls
In-class activity where students are working in pairs or small groups to use Nashpy to learn something or accomplish some task about Nash equilibria -- again along with something brief about what the assignment is, principles involved in its design, etc.
Extended student work intended to be done following a class meeting, like a miniproject, that uses Nashpy, with the same explanation and background as above

OK so that definitely feels like something a lot more involved than I intended for this submission as it would start to encroach on the pedagogic paper I have in mind about the actual full course (of which Nashpy is a small component). That paper is essentially going to be about this https://github.com/drvinceknight/gt: collection of Jupyter notebooks, automated tests, lesson plan documentation (with a lot of the type of activities you're alluding to), code to render the site, reflections etc.

[drvinceknight]

FWIW, my interpretation of your expectation of an " ideal submissions to JOSE" clearly fits under the "open-source educational material" scope of http://jose.theoj.org/about :)

[RobertTalbert]

Well, that's a relief. 👍

[drvinceknight]

Well, that's a relief. +1

Hah :) ... but I do believe that that means a paper about Nashpy does not fit under the "open-source education material" scope. (a paper about a course that happens to use it? perhaps?)

[drvinceknight]

Thanks again for taking the time to respond to me on this issue, as an editor myself I know that these discussions can be tiresome and add to an already heavy workload (congratulations by the way @labarba, looks like SciPy was a great success!).

I have taken some time to think about this and I'm pretty clear in my mind at present that a paper on Nashpy does not fit with JOSE. Nashpy is a piece of software for computing Nash equilibria.

• It is likely to be useful in research when needing such calculations.
• It is likely to be taught in game theory classes in a manner similar to how numpy could be used in a linear algebra class to show how one would compute the inverse of a large matrix in practice (to be clear: I'm not daring to compare Nashpy to numpy, it's far more modest and niche :)).

As such, I believe that the short paper I have written on Nashpy does not fall under the two categories of papers that JOSE caters to:

• "open-source educational materials": this is not a paper about a course. Nashpy could be used and/or taught in any pedagogic context.
• "educational software tools": this does not seem to be appropriate for the paper I have written either: Nashpy is not directly a piece of software that supports teaching.

It of course can be used to support teaching and when I have time, I plan to write a paper submit it to JOSE. This would fall under the "open-source education materials" criteria that will also detail how Nashpy is used in this class. However, this would be equivalent to a paper discussing how any piece of software was used in a pedagogic context.

I'm not sure how to "officially" withdraw the paper, please let me know if you would like me to close this issue or otherwise.

I look forward to continuing to see JOSE grow and will be happy to support it in the future, by submitting my own work and also reviewing if called upon. Thanks all for your time with this.

[labarba]

@drvinceknight -- I was unable to keep up with this conversation, due to being at SciPy (and flying back yesterday, and triaging email all the while).

Two other JOSE editors were at SciPy, and we had a discussion about the general issue of "software submissions" that are not infrastructure software (edtech), but software whose functionality is meant to give learners a vehicle to explore some technical content. We confirm that we do want these submissions for JOSE, and we'll need to update the Author Guide to reflect this, and give some guidance.

In sum, we want the educational-software submissions to include (1) sufficient examples for an independent learner or instructor to get started and be able to navigate the package, and (2) description of how it would be used or has been used in the classroom (or for self-learning).

I see that you've decided to withdraw and re-submit to JOSS, and that's up to you of course.

[drvinceknight]

@drvinceknight -- I was unable to keep up with this conversation, due to being at SciPy (and flying back yesterday, and triaging email all the while).

I completely understand :) Congrats again on what looks to be a very successful Scipy.

In sum, we want the educational-software submissions to include (1) sufficient examples for an independent learner or instructor to get started and be able to navigate the package, and (2) description of how it would be used or has been used in the classroom (or for self-learning).

This sounds like a great way to go and it's more or less what I had understood the scope to be from the conversation here. The paper I have written presenting the Nashpy library does not fit in this scope but I look forward to submitting such papers in the future on other pieces of software.

Thanks again for your time.

[labarba]

To be clear, what I say above about what we expect of educational-software submissions is supposed to be found in the submission repository, and not in the paper. The paper is only the "advertisement" and should briefly describe what the repo includes and where to find it. We're still talking about a two-page + references paper.