We need more clear language around author submission and adding new features

[lwasser]

Specifically this is a discussion happening in the rdata package review where the author is making changes in between the package being submitted , accepted and then fast tracked to joss. we need to clarify in our policies that

1. no code changes should be made in between the timing of submitting the package /being accepted and then submitted to JOSS - in the same way that you wouldn't edit an academic paper while it was in active review.
2. that packaging should not submit to us until their API is somewhat stable (i feel like we have / had that language somewhere but i can't find it now.

[vnmabus]

Just to be clear about the case with the rdata package:

1. It is not very reasonable to request that no code changes are made while the software is being reviewed. As opposed to an academic paper, software is never "finished", so changes should be expected. What it might work is to associate the review with a particular commit/tag, noting that changes after that are not reviewed.

2. What happened in the rdata package case is that the API was considered stable by the author (me), given that it provided a useful and complete functionality: reading binary .rda and .rds files in Python. I did not expect to do any big change to it, as that functionality covered my use cases. However, as it happens in open-source software, a user wanted to be able also to do some features that fit perfectly in the package, and thus proposed several PRs during and after the review, that is:

   • Faster routines for reading binary arrays
   • The ability to read also .rda and .rds in text (non-binary) format.
   • The ability to write some Python objects as .rda or .rds files.

   Given that these features were very useful additions to the package, I though it was better to wait for them to be merged before submitting it to JOSS, and to propose that author to coauthor the JOSS paper. Of course, that would mean that there are some PRs not included in the pyOpenSci review, so fast-tracking might not be possible. If that is the case I would be happy to let the JOSS reviewers re-review the code (or just the diff, as they prefer).

[lwasser]

@vnmabus i hear you. the problem is that this is still a review process. It's too hard to ask a review team to review a moving target (ie an API that is evolving and changing).

Our process:

• The author gets the package to a stable place and submits -
• the review happens, author responds but doesn't add major functionality during a review.
• then it goes to JOSS . Typically there is not a big gap between an acceptance (rdata was accepted in Feb from what I can see) and the package moving to JOSS. The JOSS process is generally fast ( a few weeks) once pyOpenSci has reviewed), as such we wouldn't expect major changes during that time.

My understanding from your comment here is that things got busy for you (i totally get it!). And then you decided to wait to submit to JOSS until a pr was merged. I totally understand the reasoning it just conflicts with our process.

That approach conflicts with our process because JOSS accepts our review as theirs when they publish - they don't re-review. A rereview of code is not a part of our fast track partnership. As such we are now potentially fast tracking new code that hasn't' been reviewed.

JOSS can do another full review. but that is also more resources (reviewer time) invested in reviewing a package! It's hard to find reviewers as they are volunteers and are busy too. So the ideal scenario is that there is one single review process on a somewhat stable API with 2 reviewers. Then things are fast tracked by JOSS. I hope that makes sense.

[vnmabus]

Yes, I totally get it. What I meant, mainly, is that the author may think 2 holds and submit it, but that can change as other people propose PRs with new functionality. Thus it is not really possible to promise that the API will not change in the future, even if the author thinks that.