RMarland
commented
7 years ago Thanks Kent for sharing this and the various resources - especially appreciate the categorisation of public policy issues/P2 "situations" and the reminder of the TBS risk assessment tool in the planning context. In terms of the evaluation methods - with participants involved in design before and potentially two points of interaction after - think we also need to consider the monitoring/ evaluation during which allows for adjustment of approach - especially where we are talking about iterative engagement over time.
erninghan
kentdaitken
ThomKearney
laurawesley
MaryBethBaker
If this exists elsewhere in the Github ecosystem let me know and I'll just comment there. Section 3 is long and academic but I figured it'd be good for reference, but feel free to skip and weigh in on 4.
Contents
1. Problem
There's lots of guidance for how to evaluate public participation (P2) exercises (including much developed by GC departments), but there's nothing standardized. For good reason, as each P2 process has different goals, stakeholders, and definitions of success, so guidance tends towards this format: "Identify your goals in advance (here are some suggestions), and have a data gathering plan to ascertain whether you've met them (here are some suggestions, e.g., surveys, interviews with a sub-section of the participants, etc.)."
Downside is that it makes it very, very difficult to compare different exercises, methods, or different organizations' capacity for P2, and therefore difficult to systematize learning and improvement.
Any evaluation approach for the GC will have to map to the Principles (see: https://github.com/canada-ca/welcome/wiki/Draft-Guiding-Principles-for-Consultations-and-Public-Engagement), which are not yet set in stone, but I think we can get some discussion going on how we might bridge that gap between defining success in context for each P2 process but assessing against common markers for comparability and quality.
2. Assumptions and premises (i.e,. @kentdaitken's, writing this)
3. Research
References can be looked up here.
Criteria for success (Rowe and Frewer, 2000):(http://www.en.ipea.gov.br/participacao/images/pdfs/participacao/2000%20public%20participation%20methods.pdf):
Acceptance criteria:
Process criteria:
Goals of P2, Shepard (1997):
Literature review (copying from my academic work, so apologies for the, uh, writing):
This last point about incorporating local citizens’ understanding of the area is a recurring element in P2 literature. However, agreement with this principle is not universal; many have concluded or at least considered that complex policies such as environmental and hazard mitigation issues should be the domain of those with specialized expertise. P2 has been shown to push the U.S. Environmental Protection Agency towards preventative decisions (i.e., to make decisions that forego economic benefits disproportionate to assessed risks) (Daley 2007), and citizens tend to assess risks as higher than experts do (Green 1997, Renn 1992). Godshalk, Burby, and Brody (2003, 735) studied the difficulty of explaining subjects with “high complexity and risk” to public audiences. Green (1997, 435) highlights that “the addition of ... public participation rights risks overregulation when public demand for control is high without reducing the possibility of underregulation when public interest is low.”
On the other hand, citing Innes (1990, 1998), Lindblom and Cohen (1979), and Schon (1983), Burby (2003, 34) concludes that “citizens possess ‘ordinary knowledge’” that helps planners understand local conditions. Orosz (2002) and Jonsson (2005) come to a similar conclusion, as does Shepard (1997), citing Susskind and Cruikshank (1987) and Portney (1991) while grounded in the more realist view that the alternative can be appeals, lawsuits, and accordingly, project delays.
It is important to note that these considerations of the benefits of P2 are typically applied across all P2, regardless of political context, jurisdiction, or the P2 format selected. With the exception of Colbourne (2009) (http://www.sciencewise-erc.org.uk/cms/assets/Uploads/Public-Futures15-05-14.pdf), there seems to be comparatively little effort applied to developing a framework for different kinds of public policy issues and whether various P2 activities apply asymmetrically. At most, P2 activities are categorized as (1) technical and/or complex issues and (2) all others. This is particularly true in that the preponderance of P2 case studies focuses on local examples rather than regional or federal ones.
“Type A situations are characterised by low controversy and/or few alternative options due to constraints of time, procedure and resources, or by the existence of a crisis (and need to act immediately).
Type B situations are characterised by a greater number of options, increased uncertainty around the ‘right’ decision and/or the need to make trade-offs and compromises.
Type C situations are characterised by the need to make a decision that will affect many stakeholders (individuals, communities and/or organisations) in a situation with much complexity or uncertainty, a range of (often entrenched) views on the ‘right’ decision and a strong likelihood of conflict and resistance.”
Authors such as Harvey (2009) and Martineau-Delisle and Nadeau (2010)(http://www.cfs.nrcan.gc.ca/pubwarehouse/pdfs/31978.pdf) point to a lack of emphasis in the literature on the participants’ goals and motivations, and what constitutes a success in the eyes of participants. Many authors refer to participants’ goals in realist terms, insofar as they confer legitimacy or efficacy on the goals of those hosting P2 processes; in some cases, the authors also cite Downs’ (1957) rational choice theory about citizen behaviour, to explain why citizens may opt out of participation opportunities. Likewise, Cosmo Howard (Lindquist 2010, p. 15) critiques “citizen-centric” models of evaluation insofar as they can “focus on services that governments had chosen to deliver, the manner in which they had chosen to deliver them, and using criteria and a measurement tool that, while informed by citizen surveys, might not reflect the questions and issues citizens might raise in the context of particular services.”
Rowe and Frewer have covered the evaluation element of P2 research and were setting the research agenda in 2004 for what they believed was an immature field (Rowe and Frewer, 2004). Practitioners have since developed general guides to evaluating the success of P2 (Involve 2011; World Bank 2016), but the difficulty in establishing what constitutes “good policy” given the range of citizens’ views and preferences, and in understanding the extent to which P2 influences policy decisions, makes standardization and comparability elusive.
Lastly, it is also notable that few authors characterize success in terms of economic value or a cost/benefit analysis, but rather, in terms of decontextualized success towards achieving many of the above goals (e.g., public legitimacy, the avoidance of legal challenges, or the incorporation of stakeholder ideas and suggestions) - meaning that defining and comparing “success” in a meta-analysis is difficult.
A couple bonus snippets about participatory budgeting research as it pertains to the P2 goal of "better public policy":
Torgler and Schneider (2009) found that lower tax delinquency rates occur in jurisdictions where citizens have trust in government institutions (tax delinquency being the non-payment of taxes, meaning governments have to spend money enforcing tax law rather than enjoying compliance - this is in contrast to taxpayers enacting strategies for legal tax evasion). Moreover, direct participation—particularly in budgetary decisions—can result in significant increases in tax revenues (Scheider and Baquero 2006).
More strikingly, Scheider and Goldfrank (2002) found that the introduction of participatory budgeting led to more accurate forecasting of expenditures versus budgets as well as higher project completion rates. Other authors found evidence that participatory budgeting increases investment levels or the quality of financial management practices (Zamboni 2007).
4. Resources:
Some of these sources are getting old. There are a couple recent books on the topic that cost $200+ on ebook - will try to get access from one of the universities and distill them.
Evaluating Public Participation Exercises: A Research Agenda: ftp://143.106.76.79/pub/CT001%20SocCiencia/Agosto%2016/Rowe%20and%20Frewer%202004.pdf
Public Participation Methods: A Framework for Evaluation: http://www.en.ipea.gov.br/participacao/images/pdfs/participacao/2000%20public%20participation%20methods.pdf
Assessing the effects of public participation processes from the point of view of participants: significance, achievements, and challenges: http://www.cfs.nrcan.gc.ca/pubwarehouse/pdfs/31978.pdf
World Bank's Evaluating Digital Citizen Engagement A PRACTICAL GUIDE (I find this fits the model of "define your goals and have a data collection plan," and it actually reinforces the "this stuff is complex" conclusion for me): https://openknowledge.worldbank.org/bitstream/handle/10986/23752/deef-book.pdf?sequence=1&isAllowed=y
5. Possibilities
The two most common methods seem to be surveying or interviewing participants after the P2 exercise (this could also be done twice: once right after the interaction and once again after engagement analysis is disseminated or the policy decision is taken). The GC could set standards (with exceptions) for what goes out when, and create a modular template that contains both the process-specific evaluation for program needs and a set of standardized, GC-wide questions that address the principles. All of which would likely be combined with a narrative evaluation from the POV of the convenor. For reference, the principles:
As a side note, before processes, would-be convenors could work through a capacity assessment which would dictate the extent to which they have to engage departmental or central CoEs for guidance (the GC's Project Complexity and Risk Assessment Tool is actually a great example: https://www.tbs-sct.gc.ca/hgw-cgf/oversight-surveillance/itpm-itgp/pm-gp/doc/pcrag-ecrpg/pcrag-ecrpgpr-eng.asp).
I'd tend be an advocate for having a sub-section of the external participants "own" part of the evaluation and get engaged during the design stage, and write their own assessment against principles (with qualitative assessments as well for comparability). Possibly supplemented by surveys and interviews, that would appear either in What We Heard reports or online with the GC self-evaluation on canada.ca/consultingcanadians (https://github.com/canada-ca/welcome/issues/4).
And obviously any of these approaches would need to include data about the context and the methods and channels used for both outreach/promotion and engagement.
Other research, resources, people/orgs that are dominating this concept, possibilities?