Deletion - 502 Reference Transactions (REFERENC) & 502a Reference Transactions Reporting Method (REFERRPT)

Name: Ross Fuqua

State/Affiliation: State Library of Oregon

Description of Change: Delete data elements #502 Reference Transactions and #502a Reference Transactions Reporting Method

Current Definitions: 502 Reference Transactions (REFERENC) Reference Transactions are information consultations in which library staff recommend, interpret, evaluate, and/or use information resources to help others to meet particular information needs. Reference transactions do not include formal instruction or exchanges that provide assistance with locations, schedules, equipment, supplies, or policy statements.

502a Reference Transactions Reporting Method (REFERRPT) Regarding the number of Reference Transactions (data element #502) entered, is this an annual count or an annual estimate based on a typical week or weeks? Select one of the following:

CT—Annual Count
ES—Annual Estimate Based on Typical Week(s)

Justification: Despite two recent attempts to fine-tune the definition for Reference Transactions, this data element continues to weaken public library data practitioners' confidence in the data and is still widely seen as among the most problematic data elements within the Public Library Survey (PLS). The inherent messiness of References Transactions data is largely blamed on how it is interpreted, tracked, and reported locally. But there are also underlying issues of local capacity, statistical validity, and the subjective and evolving nature of reference work itself.

The total number of reference transactions reported across the country has been dwindling - despite the fact (supported with anecdotal and statistical evidence) that library workers are busier than ever. In FY2010, libraries in the US and associated territories reported over 309 million reference transactions. In FY2017, that number had dropped to 240 million; by FY2022, US libraries only reported 128 million reference transactions in the PLS.

Given these trends, we believe that library workers have less time, energy, and training required to parse and track the increasingly diverse informational requests coming their way. In FY2012, 153 libraries either reported zero reference transactions or indicated that they did not track them. By FY2022, this number had grown to 254 (2.7%) libraries nationally. Also in FY2022, 52% of libraries in the US reported that they use estimates to generate the number of reference transactions locally. If over 50% of a surveyed body uses estimates for any given data point, how good is that underlying data? Whether an AE estimates or conducts an actual count of reference transactions, we believe that this data element is of very limited statistical use at the national, or likely even state, level. Libraries may use vastly different estimation methods. Perhaps they select a few weeks a year to do an actual count of reference questions and extrapolate an annual figure based on that. Perhaps they select a single month to do an actual count and multiply it by 12. Perhaps they make guesses based on the number of phone calls, in-person interactions, emails, or chats they receive for their reference staff. Regardless, all these estimation methods result in reported statistics that almost certainly inaccurately reflect the actual reference questions received by the library. Selecting weeks or months means the estimates will vary vastly based on the time of year when those counts are taken, given the cyclic month-to-month usage of libraries. They are also subject to large changes between years if the times chosen to make the estimates are particularly slow or busy. Contacts at service points such as phone and email paint inaccurate pictures, as only a portion of those interactions meet the actual definition of reference transactions, and there is not a good way to estimate that proportion, which likely changes over time anyway.

Even the libraries that conduct actual counts of questions likely are unintentionally reporting inaccurate data. Much like estimation methods, strategies for producing actual counts vary widely. They could be counted on a paper sheet, intranet page, or separate software or a CRM, and they could be reported as they happen, after a person’s desk shift ends, or at the end of the day or even week. Library work is busy and hectic, and it can be hard for library workers to make careful counts of reference questions in such environments. Conversations with library staff who track reference transactions make clear that they routinely forget to record questions or are merely estimating anyway when they report their figures, meaning “actual counts” likely are just estimates themselves. In addition, training may vary among different types of library staff. Dedicated reference or outreach staff may have a good understanding of what constitutes a reference transaction, whereas circulation and operational staff may not have as full understanding even though they also receive questions that could be considered reference.

These deficiencies mean that this data element is statistically unsound. Reference statistics likely can’t even be validly compared among libraries within the same state, based on the different estimation and counting methods, let alone among libraries nationwide. Indeed, due to variations in methodology, time dependency, library staff, and management techniques, this data element may not even be comparable between reporting years at the same AE in many cases.

Moreover, we have concerns about the continued use of the Reference Transactions data element and how it (as a potential measurement of a library or library worker's perceived value) prolongs complicated, defensive, and biased occupational gatekeeping within our profession. Almost half of our nation's public libraries are run entirely by what some would deem as "non-professionals" (4,325 of 9,248, or 47%, of all AEs reported zero ALA/MLS Librarians on staff in the in FY2022 PLS). In FY2022, the median number of reference transactions per capita for libraries without an ALA/MLS librarian was 0.22, whereas the same median number of reference transactions per capita for libraries with an ALA/MLS librarian on staff was 0.30. We feel that while this work continues to get done in all locales, the ways in which we try to quantify it within the PLS unwittingly continues to privilege certain AEs over others. Library workers employed by libraries where an ALA/MLS is required of librarians might be more pre-disposed to demonstrate and quantify the perceived value of that training, thereby painting an inaccurate picture of reference nationwide.

We understand that reference work is a core library function, and library workers understandably want to quantify that work in some way. For that reason, we understand that libraries and even individual states may want to attempt to quantify reference transactions in some way, however flawed. But library reference work is inherently messy and defies quantification. What constitutes a “reference transaction” is a far more subjective question than what constitutes a library program, collection material, or Wi-Fi session. Ultimately, by presenting this data element as a part of the national PLS dataset, we are saying as state data coordinators that we believe it is statistically valid information that can be compared among libraries nationwide. It almost assuredly cannot. For these reasons, we propose to eliminate this as a nationwide data element and leave it up to individual libraries or states on whether and how to track library reference work.

Potential methodological issues: None

I agree with some of Ross's statements above. As someone who worked at a reference desk within the last 5 years, I can attest that within our department, interpretations varied greatly as to what to consider reference, especially when electronic equipment becomes part of the interaction. Using phrasing from the current definition: Reference transactions do not include formal instruction or exchanges that provide assistance with equipment** implies teaching digital literacy tasks when asked to at the reference desk is not reference, yet practicing librarians ask how this is different than providing knowledge from a book or website to complete a task such as write a paper, complete a job application, etc.

I do not think however that we should delete it entirely**** but I would like to propose being open to revising it. Things we might ask ourselves - 1) Will counting the number of reference questions make a difference when advocating for service or staffing support? 2) How could we modify the definition to include using tools like the computer to answer reference questions when we are asked by a patron to teach them something using computer equipment, such as save a file to a flash drive, set up an email account, etc. Is that reference at the reference desk at all? 3) How can we make the definition easier for practicing library staff at the desk to decode and feel more confident answering each year? This one is perhaps most important in my opinion.

Thanks for reading, Jessica Hogan SDC for Hawai'i

I agree with the statement above as well. The responses to this question are essentially meaningless because there's no consistency in how libraries count the number of reference transactions. This feels much like the choice to discontinue asking about database usage.

I also agree with this, it's a flawed number and the method of counting changes in libraries with staff turnover, creating no consistency over the years.

When discussion on the SDC Forum turned to possibly revising the definition of reference, I vigorously nodded my assent as I read along. But to go a step beyond and eliminate it altogether? Ross' well-articulated, multi-prong explanation of why the data element should be retired had me all but dancing in my seat.

All of the arguments provided in his proposal are well-explained/supported, but the single line (well... two lines) that probably hit hardest for me was this:

Ultimately, by presenting this data element as a part of the national PLS dataset, we are saying as state data coordinators that we believe it is statistically valid information that can be compared among libraries nationwide. It almost assuredly cannot.

Just yesterday I had a director ask me if I could provide a precise definition of what reference was, as his library uses the "typical week" method of estimating reference that week is fast approaching. Something like an hour later, long after I moved my reply from the body of the email into its own, gargantuan Word doc trying to provide some guidance and knowing it still wouldn't be the clear-cut "this sort of question counts, this does not" direction he wanted, I concluded with the reminder that this is going to be subjective to a degree and that he do his best to make the calls consistently. I was left feeling somewhat defeated, knowing that's a difficult ask for a library where numerous employees track reference. I have years of email correspondence between myself and directors that I could provide as evidence as to just how differently libraries within a state apply the definitiony. And having spent some time in the Forum archives in an attempt to find the best way to explain certain aspect of the reference definition, it's clear that we are not, as SDCs, providing the same directions across states.

If there's no consistency within libraries/states/nationally and even within the methods of counting (exact, typical week, typical month), what good is the data? Does anyone actually have a high degree of confidence in the data?

I disagree. Why? I believe it is our job to work toward improving the tools and methods we use to collect and interpret this data. If we remove reference transactions now, we risk setting a precedent that any metric that is challenging to quantify can simply be discarded. Instead of giving up on this data point, we should focus on making it better, so it more accurately reflects the meaningful work happening in libraries across the country.

The measurement of reference transactions remains a crucial data element in the PLS because it reflects an enduring and essential function of library services: connecting users with critical information. Despite challenges in data collection and standardization, this metric highlights the value of library reference work, capturing the evolving role of libraries as knowledge centers. Removing this element could diminish the recognition of one of the core services libraries provide, while also obscuring the evidence of libraries’ continued relevance in the digital age.

Reference Transactions Reflect the Core Mission of Libraries: Reference transactions are fundamentally linked to the mission of libraries to provide access to information, support literacy, and enhance educational opportunities. They represent instances where library staff help users navigate an increasingly complex world of information, whether by answering simple factual questions or guiding users through more complex research processes. Removing this metric risks downplaying this vital service, especially as libraries become busier, more multifunctional, and critical to underserved populations.

Demonstrating the Continuity of Reference Services: Even though the number of reference transactions has declined, the fact that libraries across the country still report millions of interactions demonstrates that reference services are far from obsolete. Libraries are continuously adapting, and reference work has evolved, including new formats such as virtual reference, chat services, and digital resources. The current definition of reference transactions captures these changes and demonstrates that reference work is still happening and is valuable. The argument that reference transactions are hard to quantify should not negate their importance.

Reference Data Drives Support and Advocacy: Quantifying reference transactions gives libraries a powerful tool to demonstrate their continued relevance to funders, policymakers, and stakeholders. When libraries can show that they provide essential services to their communities through reference interactions, they can better advocate for resources. Although the data may not always be perfect, eliminating the tracking of reference transactions would strip libraries of a crucial metric that demonstrates their value in serving community information needs.

Imperfections in Data Should Drive Improvement, Not Elimination: While it's true that there are challenges in collecting accurate data on reference transactions, these challenges are not unique to this element. Libraries face similar issues with other data points, including program attendance, digital use, and even circulation figures. The variability in how reference transactions are tracked is not a reason to eliminate the metric but to explore ways to standardize and improve data collection practices. We cannot let perfect be the enemy of good. Removing the data element entirely would mean giving up on understanding a key function of libraries, instead of working to refine and strengthen the data collection process.

Reference Transactions Showcase Evolving Services: The modern library offers more than just books—it provides technology training, job search assistance, literacy support, and more. Many of these services happen through reference transactions, even if they look different than they did in the past. By continuing to track reference transactions, we maintain insight into how libraries serve their communities in ways that go beyond traditional lending. This allows us to document the full scope of what libraries do.

Local Capacity and Training Issues Can Be Addressed: The argument that local capacity issues or staff training hinder the accurate reporting of reference transactions should be seen as an opportunity for improvement rather than a reason for removal. Providing clearer guidance, offering staff training, and supporting standardized data collection practices can mitigate many of these issues. The PLS plays an important role in helping libraries compare themselves against national standards and improve their reporting practices, thus enhancing the credibility and value of reference data.

Reference Services are Vital for Underserved Communities: Many users of public libraries, especially in underserved or rural areas, rely on reference services to meet essential needs like job assistance, legal aid, health information, and more. For many, the library is the only resource they have to access critical information, and reference transactions are often the medium for these life-changing services. Removing this data element would obscure the essential role libraries play in helping these communities navigate complex systems and gain access to information they otherwise might not have.

While there are valid concerns about the data quality for reference transactions, these issues can be addressed through better training, clearer definitions, and improved data collection methods. Eliminating the reference transactions data element from the PLS would diminish the visibility of a core library service and undermine efforts to demonstrate the value and impact of libraries. Instead I believe we should focus on refining the ways we capture and report this data, ensuring that reference services—however they evolve—remain recognized and celebrated as a vital part of library operations.

Thanks for your thoughtful reply Heidi - I've been negative about this category for years, even when I was collecting data for ACRL, but I can appreciate your point of view that it's a vital service. I'm definitely open to improving the definition and giving folks more support on how to answer it accurately.

Hi @heidi-fendrick, Thank you for contributing to this discussion!

I absolutely agree with all of your points here, except for the assertion that better data can be achieved by addressing local and state-level capacity/training issues. Given the expanding complexity of community needs that libraries are trying to address right now, and the resulting staff turnover and overall fatigue that plagues library workers in this moment, I simply do not believe that our field has the human capital it needs to improve data collection or reporting on this particular element, especially when viewed on the national scale.

Another factor in why I've put this proposal forward is that by continuing to measure Reference Transactions in this way, we are attempting a quantify a critical library service function that really demands qualitative measurements and analysis. I fully agree that providing references services (which hopefully fulfill professional and ethical best practices) is a core mission of libraries' work. However, there are better methods to measure whether, or how well, libraries are achieving the task of connecting their communities with informational resources. I believe a sample-based, qualitative approach by another entity would yield way better data with which to drive support, advocacy, and which could potentially illuminate ways in which libraries could serve their communities more equitably. I don't think an annual census, in which 9,000 libraries nationally continue to split definitional hairs over what does or does not constitute a reference transaction, is productive at this stage.

@heidi-fendrick - Your thoughtful defense of keeping the reference question definitely gave pause. Reference has long been such a core service - and core metric - of library service that no longer tracking it can feel almost like sacrilege. But regardless of whether I think reference should be measured, I'm not sure I'm convinced that it can be measured in a meaningful way - at least, not by the PLS.

You write that "[p]roviding clearer guidance, offering staff training, and supporting standardized data collection practices can mitigate many" of the issues surrounding this data element. But how can we provide clearer guidance when we can't agree as an organization as to what reference is? Yes, we all share the same data element definitions, but I don't think it takes too much digging in the SDC archives to show that there is disagreement among the SDCs as to interpreting various questions as to what is and what isn't reference.

Your suggestion that we "explore ways to standardize and improve data collection practices" would seem to be the natural solution to coming to that consensus as a group so that we can train our libraries. But - isn't that what we've been trying to do ever since that question was introduced? How many discussions online and in-person have tried to find a solution, yet there seems to be little to indicate that we've made progress in regards to this element. There's an entire division of the ALA devoted to reference and even then regularly struggle to (re-)define it. You provide so many great examples of what reference work encompasses and how it's changed; is it possible that this already somewhat subjective concept keeps evolving faster than we can hope to come up with a definition that fits our purposes and clear guidelines to share with our libraries? I have so much respect for this group of people and am regularly blown away by the level of discourse that regularly happens surrounding this report, but I'm not so confident that we'll have any more success than any of the other other brilliant people who have tackled this very issue.

I'll also just add that if this proposal passes, there's nothing preventing states from continuing to collect this data if they and their libraries find it valuable. I just feel that we really need to acknowledge that this data element is not comparable at the national level for all of the reasons folks have already said on this thread

@angelakfox

Thank you for your thoughtful feedback on this issue. While I understand the concerns around the accuracy and consistency of the reference transactions data, I’d like to point out that no survey will ever be perfect. The purpose of a survey like the PLS isn’t to achieve flawless data, but to provide a broad understanding of the key activities happening across libraries. Surveys give us valuable information that we can then explore further, rather than perfectly quantifying every interaction. The focus should be on improving the quality of the data we collect, not eliminating important measures because they aren't perfect.

If the PLS isn’t the survey to ask this question on, then what is? As co-chair of PLAs Measurement, Evaluation, and Asessment Group (MEAAG), I can tell you that if we move this question to one of those surveys, we will never get anywhere near the response rate we get with the PLS. The PLS is the most comprehensive and widely responded-to tool we have for measuring library outputs, and it’s the best place to capture this kind of data.

Reference transactions are a fundamental service that libraries provide, and measuring them through the PLS gives us essential insight into how libraries are meeting their communities’ information needs. If we remove this element, we lose an important measure of libraries’ impact. Instead of cutting it, I believe we should work on refining how the data is collected, to better capture the scope of reference services.

@rfuquaSLO Thank you for your thoughtful response and for starting and continuing this important discussion!

I agree that the complexity of community needs and the current strain on library staff make it challenging to improve data collection, especially on a national scale. However, I still believe that working to improve capacity and training at the local and state levels is possible and necessary. While the current environment presents real difficulties, we shouldn’t underestimate the ability of library workers to adapt and innovate—even in the face of fatigue and turnover. I believe that incremental improvements in data collection are still achievable, and completely removing this element would mean losing valuable insight that could inform how we address these challenges in the future.

I also appreciate your point about the qualitative nature of reference services. I agree that a qualitative approach would provide richer data about the effectiveness and equity of reference services. However, qualitative data and quantitative data serve different purposes, and we shouldn’t assume that one should replace the other. The PLS offers a broad, quantitative overview that’s essential for tracking national trends and securing funding and support for libraries. While a qualitative study would certainly deepen our understanding of reference services, it wouldn’t achieve the same broad snapshot that the PLS provides.

And if we were to move reference transactions to another sample-based qualitative survey, I can say from experience, as co-chair of MEAAG, that we won’t get the response rate or reach that the PLS offers. The PLS remains the most comprehensive tool for measuring library outputs, and reference transactions are a critical part of that picture. While the data might not be perfect, it still shows us that reference work is happening, and it provides a valuable metric for demonstrating libraries’ ongoing impact.

Finally, while we continue to "split definitional hairs" about what constitutes a reference transaction, this reflects the broader complexity and evolving nature of reference work. Rather than abandon the effort, I think it’s a sign that we need to refine the process and definitions—not eliminate them. If we stop measuring it altogether, we risk losing sight of a core service that libraries continue to provide, even as it evolves.

@buzzySLO Thank you for bringing up this point! We do this often in Utah when there are changes to the PLS. While it’s true that states could choose to continue collecting reference transaction data on their own, I believe that this would ultimately diminish the value of the data. The strength of the PLS lies in its ability to offer a nationally comparable dataset that provides a broad view of trends across all public libraries. By removing reference transactions from the national survey, we risk creating a fragmented picture where each state is using different methods, definitions, or simply opting out altogether. This would make it even harder to understand the true scope of reference services in public libraries across the country.

If we’re willing to let this data element go because it’s challenging to collect perfectly, then what’s stopping us from abandoning the entire PLS and just letting each state do its own thing? The whole point of the PLS is to give us a national perspective on the important work libraries do. Yes, some data points are hard to measure, but that’s exactly why we should be striving to improve them, not eliminate them. If we start picking apart the survey element by element, we’ll lose the ability to understand trends and make the case for libraries’ value on a broader scale.

@heidi-fendrick You wrote "By removing reference transactions from the national survey, we risk creating a fragmented picture where each state is using different methods, definitions, or simply opting out altogether."

That is already the world that we live in. Not only are different states using different methods, different libraries within the same state are using different methods. As @angelakfox mentioned, states are telling their AEs different things about what constitutes reference questions, even working with the same parameters, so we functionally already operate off different definitions among states. And individual AEs within a state again are interpreting the existing definition differently, too. While I can't speak for other states, we already have AEs that opt out of answering this question for the reasons already articulated by others here. As noted in the proposal, nationally 2.7% of AEs didn't report this statistic in FY2022. I will not be surprised if that figure it higher for FY2023.

I get the desire to want to count reference transactions, as it's a sacred cow of library service. This group and others have tried for decades to define reference service in a way to make it quantifiable. And yes, there are certainly other data elements in the PLS that are weak, but this one is particularly egregious. Keeping this data element in the survey is bad because any trends that are identified by analyzing it are at best suspect. Worse is that those identified trends are flat-out wrong, and we have no way to know that. And the worst result is that having a data element this unreliable throws into question the accuracy of the entire PLS. This proposal isn't talking about other data elements in the PLS; it's talking about this specific data element. It's not just that the reference transactions aren't perfect; it's that they're actively inaccurate, and we know it.

I also want to address your responses to @angelakfox and @rfuquaSLO that we won't get similar response rates if we move this data element to other surveys. I'm sure that you are correct. But I think the real question we need to ask here is whether it is ethical for us to require AEs to spend a bunch of time collecting data that we know is bad. Again, I can't speak for other states, but I am confident saying that next to none of the public libraries in Oregon use reference transactions as a point of comparison with other AEs, whether within Oregon or beyond. Reference statistics are used internally by AEs, if they're used at all. There's no demonstrable benefit to AEs for collecting this data element at the national level. Given the burden of reporting it, I think it's time to admit that in this case, bad data is not better than no data.

@buzzySLO Thank you for your response and for making me think through all of this.

First, the idea that we already live in a world where states and libraries use different methods and definitions isn’t a reason to give up on trying to collect this data—it’s exactly why we need a national framework like the PLS. That was the point I was trying to make - if we start eliminating elements from the survey because of reporting challenges, we risk fragmenting the picture even further. By removing reference transactions from the PLS, we’re essentially abandoning any hope of standardization, not only across states but nationally. Libraries will be left to fend for themselves with no broader benchmarks for comparison.

Additionally, if we only gather and report data that is easy to collect, we leave out so much of the valuable work we do. Reference transactions reflect the crucial, often invisible work being done by library staff and show the importance of direct community interactions. This is one of the key ways we can highlight the ongoing role of libraries in connecting people to information, resources, and services.

Yes, there are challenges in collecting this data, but this is true for other elements in the PLS as well. And, yes, I understand that this proposal is just about this specific data element, thanks. However, if we start cutting elements based on reporting inconsistencies, what’s next? Do we eliminate data on programming or circulation because they, too, are inconsistently defined and tracked? This slippery slope could lead to the complete dismantling of a national survey that has been a critical tool for assessing library service across the country.

In Utah, we haven’t faced the same difficulties in collecting reference transaction data. While I know each state has its own circumstances, it’s clear that the challenges can be mitigated with proper guidance and training. If other states need support in making this easier for their AEs, I’d be happy to share tips and strategies. I have developed simple tools such as standardized Google Forms or other templates to help libraries that struggle to track, or to track more consistently and accurately. These tools are used by libraries large and small, urban and rural. These tools took me very little time and helped streamline the process to make it easier for libraries to report the data without confusion or the extra burden. I have done this for programming and tracking library visits as well.

I think it’s premature to say the data is “bad” when there are ways to improve how it’s collected and interpreted. I also don’t feel that we have exhausted our efforts at cleaning up and standardizing this data element to the best of our ability. The key is to use this data as a tool for advocacy and improvement, rather than as a strict measure of accuracy. Our focus should be on leveraging the data to demonstrate the value of libraries and secure the necessary resources to support their important work. Imperfect data can sometimes provide a general sense of trends or issues that can guide further inquiry or improvement.

As for the ethics of requiring AEs to collect this data, I believe the question should be whether it’s ethical to remove a critical element from the PLS that reflects an essential service libraries provide. We are entrusted with collecting data to paint a comprehensive picture of library services. If we stop trying to measure reference transactions, we risk creating blind spots in our understanding of what libraries are doing on the ground.

You mention that some libraries don’t use reference transactions for comparison, I believe there are still many who do, and having this data at the national level is important for them. This is definitely the case in Utah, but perhaps that is because of the work that has gone into supporting libraries in tracking and reporting this data, and is also reflecting in the effort I have put into working with our directors to help them understand the importance of benchmarking as a tool for improvement.

Finally, I have to ask - if we were to delete this metric which reflects a significant amount of work being done by staff in libraries, what alternative methods or approaches would you propose to measure the impact of the work staff do and the interaction with their communities it represents?

Let’s work together to improve the quality of this data, rather than throwing it out and risking further fragmentation in how libraries report on their services.

IMLS / public-libraries-survey

Deletion - 502 Reference Transactions (REFERENC) & 502a Reference Transactions Reporting Method (REFERRPT) #76