Read "A Vision of Metascience: An Engine of Improvement for the Social Processes of Science" by Michael Nielsen and Kanjun Qiu

I'll be posting some quotes below, expecting to comment on some of them later.

In this essay we sketch a vision in which metascience drives rapid improvement in the social processes of science. This vision requires a strong theoretical discipline of metascience, able to obtain results decisive enough to drive the adoption of new social processes, including processes that may displace incumbents. It also requires a strong ecosystem of metascience entrepreneurs, people working to achieve scalable change in the social processes of science. In some sense, the essay explores what it would mean for humanity to do metascience seriously. And it's about placing that endeavor at the core of science.

Tenure insurance: [..] an instance of two more general patterns: (1) moving risk to parties who can more easily bear it, making the system as a whole less risk averse; and (2) a plausible way to increase people's ambition is to de-risk by improving their fallback options in the event of failure

Open Source Institute: Like a research university, but instead of producing understanding in the form of research papers, it would produce understanding in the form of open source software and open protocols (with a penumbra of concomitant goods, such as prototypes, demos, open data and, yes, papers). Based on the thesis that sometimes important new understanding isn't best expressed in words, but rather in code or protocols. A number of superficially similar programs already operate, but none (as far we know) genuinely change the underlying political economy – the means by which people build their reputation and career – which is the primary point.

Institute for Traveling Scientists: A yacht that sails around the world, boarding and de-boarding scientists in each port. It would be a mobile version of places like the Stanford Center for Advanced Study in the Behavioral Sciences – somewhere stimulating and relaxing for scientists to go on sabbatical, learn a new field, or perhaps write that book they've been meaning to write. Inspired in part by Craig Venter's round-the-world trip for the Ocean Sequencing Project.

This reminds me of The Scholar Ship and Semester at Sea

Long-shot prizes: Purchase insurance premiums against extremely unlikely possibilities that would transform the world. A proof that P ≠ NP. A proof that P = NP. A constructive algorithm solving NP-complete problems quickly. Cold fusion. True faster-than-light travel. A perpetual motion machine. And so on. The more unlikely the outcome, the larger the prize can be, even for a small premium. Cheap, unlikely to succeed, and extraordinarily impactful if it led to solutions to such problems.

Every science funder, every university, and every scientific journal in the world should maintain a [..] anti-portfolio [similar to this one]. Alternately, a collective anti-portfolio could be constructed by a third party willing to tolerate some opprobrium. It wouldn't be popular. But done well it would be invaluable.

This reminds me of The Transformative Nature of Transparency in Research Funding

Interdisciplinary Institute: Most proposals for interdisciplinarity are tepid. Take interdisciplinarity seriously, by setting up an institute which identifies (say) 30 different disciplines, and then hires three people to work at the intersection of every possible pairing of disciplines. That's just 1305 people – a large program, but tiny on the scale of modern science. This would be a deliberately variance-inducing strategy. Most pairings of disciplines would produce little, but there are likely a few where great discoveries would unexpectedly be made through the combination. Those few would pay for all the rest. A less expensive approach would be to sample from randomly chosen pairs of disciplines, with the disciplines potentially coming from a much longer list.

Something to bring up with Ronin and IGDORE.

At-the-Bench Fellowship: [..] This Fellowship would fund senior scientists to spend essentially all their time actually doing science. [..] the thesis is that for some scientists there are increasing returns to focused expertise, not the diminishing returns assumed in the conventional scientist-becomes-manager-of-a-large-group model.

Printing press for funders: An entity endowed with [..] could spin out new funders each year; it could spin out, for instance, a single funder with a [large] endowment, perhaps running a competition to find the operators of the new funder. Or it could spin out a larger number of smaller funders. If each new funder had a radically different thesis, this could significantly increase the structural diversity of science; and perhaps increase the diversity of ambient working environments available to scientists. It would also be possible to set up a similar kind of printing press for research organizations [..] And, perhaps, to set up sunset clauses for the organizations

Excitement quotient for funders: Scientists often apply for grants on the basis of what they believe is fundable, rather than with their best ideas21. One approach to partially address this is for an independent agency to sample people applying to different funders, asking: "how excited were you about this grant application?" They can then publicly rate different funders by comparing excitement scores. This would place pressure on funders to fund work applicants were excited about, and raise questions if it was mostly pro forma.

That reference 21 links to https://future.com/what-we-learned-doing-fast-grants/ , which I read in full. It looks back at an effort to quickly allocate funding for COVID-19-related research as the pandemic was unfolding.

As stated above, the focus of this essay is how the discovery ecosystem improves. Part of the motivation for this focus is the belief that the design space for promising new social processes is vast

Underlying the essay is the assumption that improved social processes can radically transform and improve science. Most scientists we've spoken with agree with at least a weak form of this assumption. For instance, many strongly advocate metascientific principles like: the importance of freedom of inquiry for scientists; or that it strengthens science if outsiders can overturn established theories on the strength of evidence, not their credentials.

That last point is much aligned with Wikimedia's WP:V, though WP:CRED links to a number of essays that explore different shades of the issue, while WP:NOR of course stands in contrast to the above paragraph focused on original research.

It's a humbling experience to talk to the best scientists: what they can do is genuinely astounding. [..] Why not just scale this up? Indeed, sometimes when talking with such people we encounter friendly or skeptical bewilderment. For them and their friends, the current system works well, and they don't see the need for anything different. But perhaps there are very different types of scientist (and scientific work) who could also achieve astounding things in science, perhaps achievements which the current system is unknowingly bottlenecked on, because their personality type doesn't thrive within that system? And perhaps they, and their approach to science, would thrive if there was more structural diversity in the social processes of science? This is a central point to which we shall return.

At first glance, this essay may appear to be an entry in that flourishing genre, what's-wrong-with-science-and-how-to-fix-it? This genre is well represented on social media, in conversation among scientists, and in articles in the scientific and mainstream media. [..] But although there is no shortage of grand hopes and plans, progress is often slow.

Our point of view is different in a crucial way: we are not proposing a single silver bullet. We believe the opportunity is far larger. What we want is a flourishing ecosystem of people with wildly imaginative and insightful ideas for new social processes; and for those ideas to be tested and the best ideas scaled out.

The above comment is becoming unwieldy to edit, so I'll continue in this one, thinking I should perhaps put these thoughts into a separate file.

How do social processes in science change? Is there a general theory of such change? Is it possible to speed up and improve that change? This subject [..] is a fundamental issue in the way human beings make discoveries, is a central problem of metascience, and at the very core of science.

Questions of this kind are one of the main reasons why I find invasion biology and its mathematical modelling so fascinating. In such models, if the species invading a real-world ecosystem are replaced with new ideas entering the research ecosystem, then the dynamics are similar, though much depends of course on the details.

As with the program ideas suggested in the opening section, we're not claiming that any one of these program ideas would revolutionize science. Indeed, we're not even claiming any particular one would work well at all; some might work quite poorly (though we're not sure which!) A healthy discovery system should trial a profusion of ideas, including many which fail; that's what it means to be trying risky things. We do believe it's worth trialling all the ideas above, and many more. Conducting such trials would help answer an immense variety of questions, things like: how much demand is there for discipline switching? What are the resulting flows of scientists between disciplines? What determines those flows? How well do young people perform as Principal Investigators? Are there systematic differences in the directions they explore? And so on, a cornucopia of questions, partial answers, and useful data. In that sense, even "failed" programs would be successful: they will contribute crucial knowledge to our understanding of metascience. And in the event that one of the programs works strikingly well, it can be scaled up. It may even begin to change the culture of science.

Here, I'm reminded of the space of potential foldings of proteins, and how tools like Folding@home, FoldIt or AlphaFold can be used to explore that. Would be nice to see adaptations of such approaches to explore the configuration space of the research ecosystem.

Underlying these heuristics is a view of metascience as an imaginative design practice. It's a view very different from that common in the natural sciences, which are most often about more deeply understanding existing systems, or natural variations thereof52,53. Design, by contrast, is about inventing fundamental new types of object and action, which don't obviously occur in nature. Consider Genovese maritime insurance as an example. It wasn't a change to how ships were built, or sailors trained. Rather, it introduced a radical new interlinked set of abstractions – the insurance premium, counterparties, spreading of risk, insurance payouts. These are beautiful, non-obvious ideas, none of which naturally occur in the world. Rather, they were invented through deep design imagination. And despite being "made up", they transformed humanity's relationship to the world. This is characteristic of design imagination. It must have seemed to the Genovese that financiers were "naturally" reluctant to finance expeditions, given the risks, and that this was a fixed feature of the world. And yet design showed that this was an illusion, which could be radically changed.

We mention this because discussions of metascience often give short shrift to imaginative design. We often meet people who think metascience means studying relatively minor tweaks to existing social processes, for things like peer review, hiring, granting, and so on. Those minor tweaks are genuinely valuable, and can teach us much. But we believe that imaginative design can be at the core of metascience. That means inventing fundamental new primitives for the social processes of science. It means developing tremendous design imagination and insight and new ideas to explore the metascience design space. We believe the most important and powerful social primitives in this design space are yet to be discovered.

We've been developing the idea of metascience as an imaginative design practice. In Part 2 we'll argue that this is one of three major components of metascience. The other two components are: (1) metascience as an entrepreneurial discipline, actually trialling and then scaling out new social processes; and (2) metascience as a research field, aimed at deepening our understanding of the social processes of science, in part as a tool to evaluate their impact on discovery. All three components must work together for metascience to be successful.

Summarizing, and looking ahead, in the vision that will emerge, metascience is not just about the study of science, understanding descriptively what is happening. It has as a fundamental goal interventions to change science. And metascience is not only about incremental interventions. It is also about wildly imaginative design, conjuring new fundamental elements for social processes in science. This is what we mean when we say metascience is an imaginative design discipline. Furthermore, metascience is not just theoretical. That is, it is not just about new understanding (and papers), in the conventional academic mode. It requires building new organizations, new programs, new tools, and new systems. Only by such entrepreneurial building is it possible to test metascientific ideas, and to improve our understanding. That improvement is both valuable in its own right, and also improves what people can build in the future. At the same time, it is not sufficient just to build local systems. Metascience also requires moving from (comparatively) small trials to broader cultural changes in science. That is the main subject of Part 2 of the essay.

scientists are usually "not supposed" to build tools and infrastructure, unless to take data immediately needed in their scientific work94

That footnote then states
- It's striking that the builder of the first telescope is not remembered by most scientists, but Galileo is. The usual view is: Galileo made the scientific discoveries, but the toolbuilder did not. But they did enable discovery. This is an early example of a pattern that persists to this day. It's beyond the scope of this essay to delve deeper, but fascinating to think upon.

Metascience Entrepreneurship Organization: Currently, projects such as the Center for Open Science are funded as bespoke one-offs. It would be better if such efforts could be produced in a scalable way. Metascience entrepreneurship often involves many different activities: theoretical metascience, tool-building, institution-building, community-building, cultural change, partnership-forming, policy work, and so on. This requires multiple people acting in a co-ordinated way.

We've identified several distinct patterns of metascience entrepreneurship. Through the remainder of the essay we'll focus on one in particular, used in the replication crisis. That pattern is perhaps the most challenging, requiring both decisive results in theoretical metascience and effective metascience entrepreneurship to drive change. But first let's make some summary remarks about the other patterns. In general, the decentralized product adoption pattern has limited range of applicability, since many social processes aren't expressed through products. But when it does apply it can be very powerful, and not strongly subject to the earlier inhibiting effects. For that reason we won't discuss it much through the remainder of the essay. The centralized change pattern can work, but has the problem that arbitrary changes can be made on the basis of good optics, political palatability, and so on, without any guarantee of improvement. Of course, no-one will think that's their program! The best way of making this a good pattern is to improve evidentiary standards from theoretical metascience. Our subsequent discussion of theoretical metascience thus applies to this pattern. As we noted earlier social processes that are local and not collectively held can be changed relatively easily, and for that reason we don't discuss them further. Through the remainder of the essay we focus on the theoretical metascience pattern. It applies to many social processes, and it's able to overcome all the inhibiting factors mentioned earlier, including entrenched norms, the shadow of the future, network effects, and so on.

from now we'll use the term decisive result to mean a result sufficiently strong that it would routinely convince someone who was initially hostile or had a vested interest in a different conclusion. This is not a rigorous or precise definition, but it does encode a useful criterion, capturing the notion of a result strong enough to force change, driving the scaling step in the metascience learning loop. To repeat the above in this new language: the key results of the replication crisis were, altogether, a decisive result in this way.

It's notable that the kind of work required to obtain decisive results isn't just business-as-usual.

Much of the challenge in science – and we expect in metascience – is to develop better theories and better instruments to amplify signal amidst the noise. It's a challenge to develop better metascientific instruments, perhaps a metascience microscope or chronoscope125 to help us better and more rapidly understand the importance of scientific work, amplifying currently illegible signals into something meaningful.

Citations have no intrinsic connection to scientific progress at all; they are closer to exhaust from progress, not progress itself.

By contrast to citation analysis, there is already a field that studies the evolution, influence, and importance of ideas in science: the history of science. [..] Why did it matter? Where did the ideas come from? How did they change later thinking? How did they change the course of science, and of human civilization? [..] By contrast, citation analysis seems almost like (a caricature of) behaviorist psychology, studying the external forms of science, but not in any depth the evolution of the intrinsic underlying ideas. It is only by doing the latter that we can properly grasp the changed understanding. That is the remit of history of science, and also (to some extent) of the sociology and philosophy of science.

Now, of course, the history of science is a very active field. But, so far as we know, its approach is not usually used as the basis for program comparisons of the type we are discussing. And it's easy to see why: it's not scalable and superficially "objective" in the way citation analysis appears to be. But we believe it is far more reliable, getting at the actual importance of ideas. It's especially preferable for understanding the importance of individual outlier discoveries. And, as we said above: if you're going to learn to do something well at scale, it makes sense to first learn to do it well locally, even if that approach is not obviously scalable. And so we believe that the techniques of the history of science should take on a central role in making program comparisons, especially in understanding the outliers. It can and should be placed at the foundation of scalable analyses. Citation analysis is also valuable, but we believe it should be secondary.

As a general principle for evaluation, we believe there should be a strong presumption favoring structural diversity in social processes. [..] Problems easily soluble in one environment may be near insoluble in another; and vice versa. Indeed, often we don't a priori know what environment would best enable an attack on an important problem. [..] What we need is a diverse range of very different environments, expressing a wide range of powerful ideas about how to support discovery. In some sense, the range of available environments is a reflection of our collective metascientific intelligence. And monoculture is the enemy of creative work.

It's easy to end up with research monocultures, when it would be far healthier to have a diverse portfolio of approaches co-existing, with the relative scale regulated by some mechanism ensuring the ongoing health of science.

Just to summarize the problems as we see them: there's the portfolio construction problem, and three related subproblems. (1) The unbounded growth problem: finding a healthy regulator for the ultimate scale of a process; (2) The metascience alignment problem: whether the ultimate scale of a process is set by the value to humanity and science, or by fashion and politics; (3) A problem we haven't yet explicitly mentioned, the extinction problem: how to scale down processes which are working poorly, a kind of creative destruction for science. This is something science currently does extremely poorly; the result is long-lived institutions and communities and processes which crowd out new entrants146. It's possible there are no perfect or near-perfect answers to any of these problems. But we believe far better answers are possible than we have today, through imaginative theoretical work and mechanism design. In the meantime, even without solving these global problems of portfolio construction, it seems worth focusing more locally, on identifying new processes that deserve to be amplified.

Daniel-Mietchen / ideas

Read "A Vision of Metascience: An Engine of Improvement for the Social Processes of Science" by Michael Nielsen and Kanjun Qiu #1752