Scope and structure of SciPy 1.0 paper

[rgommers]

EDIT: tentative structure and authors:

• [ ] Introduction authors: @stefanv @jarrodmillman
  • [x] History
  • [x] Project goals / scope of library
  • [x] Current status (maturity, users)
  • [x] scikits
  • [x] timeline (example)
• [x] Architecture and implementation choices authors:
  • [x] Submodule organisation authors: @WarrenWeckesser
  • [x] Common infrastructure (build, CI, etc.) this may already be covered in test section below
  • [x] Language choices authors: @tylerjereddy
  • [x] API and ABI evolution
• [x] Recent technical improvements:
  • [x] Data structures:
  • [x] cKDTree authors: @rainwoodman, @tylerjereddy
  • [x] sparse matrices authors: @perimosocordiae
  • [x] LowLevelCallable authors: @tylerjereddy
  • [x] cython_blas/lapack/special authors: @insertinterestingnamehere, @person142
  • [x] optimize (new optimizers, incl. benchmarks) authors: @antonior92, @mdhaber
  • [x] stats distributions authors: @josef-pkt
  • [x] interpolate (polynomial interpolators, PPoly & friends). authors: @ev-br
  • [x] test and benchmark suite authors: @tylerjereddy
• [x] Project organisation and community authors: @rgommers
  • [x] Governance (incl. CoC) authors: @tylerjereddy
  • [x] Roadmap authors: @tylerjereddy
  • [x] Community beyond the SciPy library (ecosystem, scipy.org, etc.) (perhaps covered in history?)
  • [x] Maintainers (i.e., core devs / commit rights) & contributors
  • [ ] may be useful: https://github.com/docathon/watchtower, https://github.com/machine-shop/oss_devkit/blob/master/plot_pr.py (PR over time plot, there may be other better versions out there)
• [ ] Conclusion authors: @tylerjereddy
  • [x] impact now
  • [ ] future development (including key issues: sparse arrays @stefanv, ndimage pixel vs point @stefanv @jni ?, splines, fftpack vs. np.fft and linalg vs. np.linalg, under-maintained submodules)

Original description

We need to decide on the topics/structure/scope of this paper.

My suggested list was:

• Technical topics (relatively new and noteworthy features):
  • Sparse matrices (present & future). Or data structures (include cKDTree, lots of improvements went into that)?
  • LowLevelCallable
  • cython_blas/lapack/special
  • optimize
  • stats distributions
  • benchmark suite
  • Key issues: ndimage pixel vs point, sparse arrays, splines, fftpack vs. np.fft, under-maintained submodules, ...?
• Project/package topics:
  • History (see 1.0 release notes)
  • Language preferences (Python, Cython, C, C++, Fortran) and why. Numba, Pythran? Idea: measure familiarity with these for all core devs
  • Roadmap, governance, CoC, scipy.org and SciPy ecosystem (also, packages depending on SciPy?)
  • Making development easier (CI, runtests, Bento, Docker, etc.)
  • Maintainers, growth of team over time, commit log stats
  • Versioning, evolution of API & ABI

Comment by @pv: I think the harder questions then is the story the paper wants to tell. The challenge here is I think that "Scipy 1.0" consists of bits and pieces, and unlike more field-specific software projects the common theme may be more difficult to state. There's also the previous "Scipy paper", and presumably we want to write a "followup" or "update"?

One possible story could then to be (which also seems what was already suggested) write about what went on in the recent years with the project. In this case the focus would be on the new stuff, and we'd just state what existed before that in the introduction or short background section, and then the body of the paper would proceed on particular things (e.g. as in the list of topics Ralf gave) that we want to talk about.

[rgommers]

I'd prefer someone else to do this, I definitely don't have time before the end of January, maybe longer. IIRC Matt and/or Tyler volunteered?

[mdhaber]

I redlined a draft and have intended to make a PR, but it wouldn't go so far as to unify it into a single voice - yet. I'll work on getting those up when I'm done with the NumFOCUS work.

On Thu, Jan 10, 2019 at 3:55 PM Ralf Gommers notifications@github.com wrote:

I'd prefer someone else to do this, I definitely don't have time before the end of January, maybe longer. IIRC Matt and/or Tyler volunteered?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scipy/scipy-articles/issues/9#issuecomment-453307880, or mute the thread https://github.com/notifications/unsubscribe-auth/AGRCKx5k2XYNin9zrtfvDAzXcpumVHqEks5vB9LwgaJpZM4R4zHo .

[jarrodmillman]

@mdhaber Do you have a rough estimate of when you might be able to make your PR? Do you think you could have it ready by Monday (1/14)?

[mdhaber]

@jarrodmillman No : / But no need to let this hold up anyone else's review.

[tylerjereddy]

See #65 for a check-list of sorts from my full read through today.

[mdhaber]

Reviewed this thread. There are two things agreed upon above that are missing from the current paper:

1. @aarchiba suggested and @rgommers/@endolith agreed that "what can SciPy do?" should be addressed more explicitly
2. There is no "Conclusion" section, and the "Discussion" section doesn't really serve as one. The paper ends with two snippets of Cython code and "Some additional roadmap items are summarized in Table 2." Work on this was discouraged in #123, but since the "Conclusion" section was explicitly proposed by @rgommers twice I thought I should bring it up again.

Action:

1. I'm not sure what to do here. I can support if someone can suggest a concrete plan based on the current version of the paper.
2. I can work on this if others agree that it is still important.

[mdhaber]

(for convenience) The paper is available at: https://427-66140682-gh.circle-artifacts.com/0/papers/artifacts/paper.pdf I'll try to keep this up to date.

[rgommers]

(1) should be stated in only a couple of sentences I think. It should not talk at any individual features or modules, that's not feasible (and there's lots of that). I guess what we need is a few sentences that are an improvement on the first paragraph at https://scipy.org/scipylib/?

[rgommers]

(2): perhaps it doesn't need a separate conclusion. The discussion section is pretty good. What it misses is a strong finish. After the roadmap table, perhaps a 1-2 sentence statement about SciPy having a bright future and ?

[mdhaber]

@rgommers I'll look at that link for 1. For 2, just a strong finish without a separate section is good. You think the Cython code should stay in the discussion?

[rgommers]

You think the Cython code should stay in the discussion?

probably better to remove

[WarrenWeckesser]

You think the Cython code should stay in the discussion?

probably better to remove

Definitely remove it. That comment is about some housekeeping details, and not needed in this paper. (If I'm buying a house, I need to know that the roof doesn't leak, the foundation is strong, the wiring is safe, etc. I don't need to know that there are some cobwebs in the basement that should be cleaned up.)

[WarrenWeckesser]

Here's one possibility for an end to the discussion:

A problem faced by many open source projects is attracting and retaining developers. While it is normal for some individuals to contribute to a project for a while and then move on, too much turn-over can result in the loss of institutional memory, leading to mistakes of the past being repeated, APIs of new code becoming inconsistent with the old code, and a drifting project scope.

We are fortunate that the SciPy project continues to attract enthusiastic and competent new developers while maintaining the involvement of a small but dedicated "old guard". Pauli Virtanen, our BDFL, has been with the project for more than 10 years, and is still actively contributing code. While the person acting as the release manager might change from one release to the next, Ralf Gommers acts as a general manager, keeping the release process moving smoothly and ensuring that pull requests get reviewed and merged where appropriate. Ralf is approaching his tenth year anniversary on the SciPy project and shows no signs of slowing down. There are early contributors, such as Robert Kern and Stéfan van der Walt, who contributed much to SciPy over the last 10 or more years and while not currently writing code for SciPy, still contribute to discussions of bug reports and reviews of new code contributions. There are an additional half dozen or so currently active developers who have been contributing steadily for five or more years. The combination of a committed old guard and a host of new contributors ensures that SciPy will continue to grow while maintaining a high level of quality.

I'm not in love with it, but I'm a slow self-editor. I figure its better to put something up for discussion now instead of putting up a more polished version later.

[tylerjereddy]

I'd say the scope & structure of the paper have been decided at this stage--so, closing.