Scenario: Interoperation with Crowdsourced Real-time Fact-checking Services

[AdamSobieski]

Introduction

Towards dialogue systems which can answer questions from users on important and contemporary topics – such as those discussed in the news or in political debates – dialogue systems are discussed which can interoperate with and access crowdsourced real-time fact-checking resources.

Crowdsourced Real-time Fact-checking Resources

Crowdsourced real-time fact-checking resources provide groups of users with the tools, processes, and procedures for collaboratively determining the factuality of statements including the means of engaging in evidence-based argumentation.

Such resources should support non-anonymous users as well as verified users, non-anonymous users who are, for instance, journalists, fact-checkers or experts.

The instantaneous state of and, in some cases, resolution of a crowdsourced real-time fact-checking process can be complex. Multiple mutually-exclusive statements may each be supported by evidence and argumentation.

Crowdsourced real-time fact-checking resources should allow users to opt into receiving notifications as updates occur with respect to ongoing processes. Such resources should provide APIs and should support machine-utilizable output formats. In this way, dialogue systems could also subscribe to events pertaining to information availability and updates.

Which topics require dialogue systems to access fact-checking resources? Dialogue systems could make use of APIs to determine which topics are discussed on fact-checking resources as well as the quantity of recent activity occurring for a given topic.

Centralized, Decentralized and Hybrid Fact-checking Resources

Centralized, decentralized and hybrid approaches to crowdsourced real-time fact-checking resources are described.

An example of a centralized variety of fact-checking resource would be a “Wikifact” resource. In this case, editors login to a centralized Web-based resource and collaborate with one another to provide real-time fact-checking services. Services could utilize the API of a “Wikifact” resource to equip dialogue systems.

Decentralized varieties of fact-checking resources involve contributors making use of separate websites to publish their journalism and fact-checking. Services could spider a collection of websites to provide real-time fact-checking data to dialogue systems.

With respect to hybrid approaches, extensions to popular content management system software could allow journalists and fact-checkers to ping one or more resources as journalism or fact-checking articles were published or edited. Furthermore, such extensions could facilitate the embedding of machine-utilizable data, e.g. via Web schema, into hypertext articles. The instantaneous state of a resource could be transparent to users via one or more centralized Web portals and the storage of the journalism and fact-checking data could be decentralized.

Dialogue System Behaviors

Which behaviors should dialogue systems exhibit when asked about volatile, contentious or dynamic topics, topics of some unfolding discussion on one or more fact-checking resources? Such scenarios might tend to arise as users ask questions about topics recently discussed in the news or discussed in a political debate.

Some options are indicated:

Option 1: I Don’t Know

A dialogue system could indicate that it doesn’t know the answer.

Option 2: I Need More Time to Answer that Question

A dialogue system could indicate that the matter is being reviewed by fact-checkers and could provide a rough estimate of when it should be able to provide a better answer to the question.

A dialogue systems could ask users if they desire notifications with respect to any pertinent updates and could provide such notifications to users via any one of a number of communication channels.

Option 3: Referring Users to Online Fact-checking Resources

A dialogue system could refer users to Web resources, for example to unfolding processes and procedures on fact-checking resources.

Option 4: Explanation of Alternatives and Possibilities

A dialogue system could, in particular for complex scenarios, explain the elements of a set of mutually exclusive alternatives and provide the best supporting evidence and argumentation for each. “Some are indicating X, while others Y.” Interesting questions include how to best generate the natural language in these instances, how to best sort the alternatives and possibilities, how to best specify which parties support which alternatives and possibilities, and so forth.

Option 5: Curation and Automated Decision-making

Computer-aided approaches can be envisioned where teams of personnel review fact-checking resources to make determinations with respect to the outputs of dialogue systems with respect to topics which might be volatile, contentious or dynamic.

Alternatively, dialogue systems could process the evidence and argumentation supporting statements and make automated decisions with respect to their factuality.

[v-kydela]

@AdamSobieski - Can you explain what you want to have done with this issue? Is it a feature request?

[chrimc62]

Thank you for the suggestion, we will add this to our backlog.

[AdamSobieski]

@v-kydela , @chrimc62 , thank you. In my opinion, the scenario is an important one and hopefully one of some interest to the bot developer community. When considering how best to facilitate implementation by bot developers, there could be some new features and services entailed. Please do consider the issue a suggestion or feature request.

[v-kydela]

@chrimc62 - I am assigning you to signify that there is nothing I can do for this issue. I would unassign myself but I have been told that the first commenter must remain assigned for tracking purposes.

[CezaryMarcjan]

We'll track this with #5781, thanks Adam.