Pick a name for this open source project

[yuvipanda]

Should be sometching that ties into https://arxiv.org/abs/2310.00865

Tasks

• [ ] Read the paper
• [ ] Come up with 3 names based on the ideas presented in that paper, particularly around frictionless reproducibility and 'competitive challenges'
• [ ] Pick one

[jnywong]

Maybe some permutation of

• django-challenge-website
• django-challenge-competition
• research-challenge-website
• competitive-challenge-website

?

[jnywong]

Also, I can go full authoritarian mode and just pick one to avoid long discussions on this topic 😄 My preference is django-challenge-website.

[yuvipanda]

I do definitely want to tie it to that paper though :)

[yuvipanda]

For context, https://web.stanford.edu/dept/statistics/cgi-bin/donoho/wp-content/uploads/2018/08/wavelab.pdf from the same author helped drive a lot of inspiration for Jupyter itself. I think having a link to donoho here via the name will help a lot in establishing why we are doing this project.

[jnywong]

donoho-challenge-website seems a good fit in this case. I like that we can use the name to earmark this project as an opinionated flavour of a challenge website.

[yuvipanda]

I've updated the issue body with the process I think we should use here. I definitely want us to read the paper first (which I haven't!) before we pick a name. It's a fairly quick read!

[jnywong]

Thanks! Sorry for my ignorance 😆

[yuvipanda]

@jnywong no worries, i also haven't actually read the paper yet fully :D

[yuvipanda]

Posting some useful pieces of information from the paper that are relevant to the naming discussion:

The three initiatives are related but separate; and all three have to come together, and in a particularly strong way, to provide the conditions for the new era. Here they are: [FR-1: Data] datafication of everything, with a culture of research data sharing. One can now find datasets publicly available online on a bewildering variety of topics, from chest x-rays to cosmic microwave background measurements to uber routes to geospatial crop identifications. [FR-2: Re-execution] research code sharing including the ability to exactly re-execute the same complete workflow by different researchers. [FR-3: Challenges] adopting challenge problems as a new paradigm powering scientific research. The paradigm includes: a shared public dataset, a prescribed and quantified task performance metric, a set of enrolled competitors seeking to outperform each other on the task, and a public leaderboard. Thousands of such challenges with millions of entries have now taken place, across many fields.

In particular, FR-3 is relevant to this project:

1. Shared public dataset
2. Prescribed and quantified task performance metric
3. Set of enrolled competitors
4. Public leaderboard!

We got all these, and these were naturally picked as the thing to do, rather than based on reading this paper!

In the past decade, all of these initiatives, in their best implementations, became frictionless open services, essentially offering immediate, permissionless, complete, access to each relevant digital artifact, programmatically from a single line of code. Not all friction is yet gone, but the trend is clear: what frictions still remain will vanish soon enough.

The goal of this project is to reduce some of this friction.

He provides examples:

[FR-1] Research Data repositories: Open Science Collaborative11, Nightingale Open Science 12; [FR-2] Research code re-execution environments like Jupyter 13; and [FR-3] Challenge problem platforms such as CodaLab Competitions 14.

Jupyter gets a nod in here, and https://github.com/codalab/codabench is the open source competition for this particular project. It's already pretty well established, and I didn't know its existence before reading this paper :( It's also a django app...

Then he talks about various combinations of these three things:

The most common leave-one-out setting is surely Reproducible Computational Science (RCS) where we combine [FR-1: Data Sharing] and [FR-2: Code Sharing], without [FR-3]. Here there is a scientific question but no underlying challenge problem being considered; we might simply be doing an exploratory data analysis and reporting what we saw, and giving others access to the data and the analysis scripts. RCS lacks the ability to focus attention of a broad audience on optimizing a performance measure. A less common but very famous and important setting is the Proprietary-Methods Challenge (PMC), with [FR-1: Data] and [FR-3: Challenges] but not widespread code sharing by participant teams, even after the fact. The $1 Million Netflix Challenge (2009), and many large-dollar prize challenges since, are examples.

This explains the need for projects like this one.

Without all three triad legs [FR-1]+[FR-2]+[FR-3], FR is simply blocked.

Less clear is what we might be missing without [FR-3 – Challenges]. We would be missing the task definition which formalized a specific research problem and made it an object of study; the competitive element which attracted our attention in the first place; and the performance measure- ment which crystallized a specific project’s contribution, boiling down an entire research contribution essentially to a single number, which can be reproduced. The quantification of performance – part of practice [FR-3] – makes researchers everywhere interested in reproducing work by others and gives discussion about earlier work clear focus; it enables a community of researchers to care intensely about a single defined performance number, and in discussing how it can be improved.

"Frictionless Research Exchanges"

We see a new institution arising spontaneously; let’s call it a Frictionless Research Exchange (FRX). FRX is an exchange, because participants are constantly bringing something (code, data, re- sults), and taking something (code, data, new ideas), from the exchange; and various globally visible resources - task leaderboards, open review referee reports - broadcast information to the whole com- munity about what works and what doesn’t. Of course, this is a very different type of exchange from those involved in financial markets; it involves intellectual engagement, not money. Finan- cial exchanges produce price discovery. Frictionless Research Exchanges produce community critical review.

I really like the name 'frictionless research exchange' as something that has all 3 legs - open data, open computation, and then open challenges.

Why challenges need to be open too

Challenges alone, without each of [FR-1: Data Sharing] and [FR-2: Code Sharing] and consequent research community engagement, also can’t produce the advantages we are discussing. The typical missing ingredient is [FR-2], reducing us to the Proprietary-Methods Challenge. In such circum- stances, we miss the sharing and cross-fertilization of knowledge, we don’t as a community know what has been done to get a method which bests the competitors, and we don’t get the chance for critical assessment of the methods. It is possible that a ‘method’ is just a set of random tweaks with no rhyme or reason, yet can win a particular competition at a particular moment essentially due to random noise – some competitions are just that close. Competitions don’t, inevitably, produce last- ing results or knowledge. Community review is what ensures that important advances are recognized and propagate.

Why build this?

Section 6 offers many examples of such challenges, but notice that almost all of them had to build their own tooling to run their challenge. Unlike offering FR-2 (via a JupyterHub), running a challenge is harder because you gotta do it from scratch.

How this relates to codabench vs building our own needs to be examined, but not necessarily for naming.

(to be continued)

[yuvipanda]

FR heralds a drop to zero of the human effort required to reproduce a computational result.

I love this sentence.

[yuvipanda]

Ok, with all that, I now propose the following names:

1. frx-challenge-runner (Frictionless Reproducibilty Exchange - Challenge Runner)
2. frx-challenges
3. frx-challenge-hub (to match jupyterhub a bit?)

[yuvipanda]

I personally prefer option 2.

[GeorgianaElena]

For me it's a tie between 2 and 3 :D

[jnywong]

I also prefer 2. Given the consensus between the 3 of us, let's implement the change.

I believe that requires renaming this repo and updating the config files that refer to this repo by name, then we can mark this as closed.

[colliand]

I like 2. Thanks @yuvipanda for this thoughtful and collaborative naming exercise.

I like the way that "frx-" can be used as a prefix. A collection of interoperable components (challenges, hub, data integrations, ...) can be assembled to create the GENOMICS Research Exchange, etc.

[yuvipanda]

I created https://github.com/2i2c-org/unnamed-thingity-thing/issues/98 to handle the actual rename!

Thanks everyone for this process :)