search

GSA-TTS / FAC

GSA's Federal Audit Clearinghouse
Other
20 stars 5 forks source link

FAC - Going concern data being misreported #3433

Open stucka opened 8 months ago

stucka commented 8 months ago

Describe the bug

The "going concern" flag is sometimes being misreported.

I don't know if there is a problem with the data as submitted (someone filling out a form incorrectly) or if something is getting parsed improperly. If the former, perhaps there is value in adding a QA step to show submitting users a list of unusual findings (e.g., when users submit a going concern issue, or material noncompliance, or ...) to verify that's what they intended to do.

@cephillips tells me she's aware of two. I have details on one, provided below.

Steps to reproduce the bug

For a Unity Homes example, visit https://app.fac.gov/dissemination/summary/2023-04-GSAFAC-0000025364

The SF-SAC button allows an Excel-formatted download of the data.

The "general" tab of that data contains a field, is_going_concern_included, that for this report shows as "Yes"

From the original URL, the Single Audit Report button allows a PDF that contains the phrase "going concern" in two spots, both in innocuous uses.

Expected Behavior

Audit data should to the extent possible reflect the accuracy of the audit itself.

Screenshots

No response

System setup

n/a

Additional context

No response

Code of Conduct

jadudm commented 8 months ago

@stucka , thank you.

I may have a question or two, please humor me. We all want the data collection to be excellent.

You're saying:

  1. The SF-SAC data reflects someone having indicated that a going concern existed.
  2. The report text does not reflect this.

Is that the concern?

If it is, part of the challenge is that... it's hard to validate. As you say, we could have more per-field documentation that provides more guidance/examples, but at some level... it feels like something the auditor is supposed to know/do correctly?

I'm not trying to dodge, so much as I'm trying to understand what we can do to improve the collection. Am I understanding your ticket/question correctly?

cephillips commented 8 months ago

Hello Mr. Jadud, it appears this is a persistent issue with the audits now where they are mislabeled as a Yes on the Going Concern issue when the audit does not flag them related to being a going concern. Here is another example.

One City Schools, Inc.https://app.fac.gov/dissemination/report/pdf/2023-06-GSAFAC-0000024938 Madison, WI Accepted date: 2024-02-24 EIN: 471490574 Audit year: 2023 Report ID: 2023-06-GSAFAC-0000024938 Going concern: Yes Result: Unmodified Material weakness: Yes Material non-compliance: No

In the audit itelf (available through the link, the auditors explicitly state there is no going concern issue.

Thanks very much. Cheryl Phillips (I work with Mr. Stucka)

Cheryl Phillips Director, Big Local News Hearst Professional in Residence, Graduate Program in Journalism Stanford University @.**@.> 650-723-2504

[Big Local News logo]

From: Matthew Jadud @.> Date: Tuesday, February 20, 2024 at 12:52 PM To: GSA-TTS/FAC @.> Cc: Cheryl Phillips @.>, Mention @.> Subject: Re: [GSA-TTS/FAC] FAC - Going concern data being misreported (Issue #3433)

@stuckahttps://github.com/stucka , thank you.

I may have a question or two, please humor me. We all want the data collection to be excellent.

You're saying:

  1. The SF-SAC data reflects someone having indicated that a going concern existed.
  2. The report text does not reflect this.

Is that the concern?

If it is, part of the challenge is that... it's hard to validate. As you say, we could have more per-field documentation that provides more guidance/examples, but at some level... it feels like something the auditor is supposed to know/do correctly?

I'm not trying to dodge, so much as I'm trying to understand what we can do to improve the collection. Am I understanding your ticket/question correctly?

— Reply to this email directly, view it on GitHubhttps://github.com/GSA-TTS/FAC/issues/3433#issuecomment-1955067346, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAEFU3RHMZVD3T25QCIJA63YUUEJFAVCNFSM6AAAAABDRRPPZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVGA3DOMZUGY. You are receiving this because you were mentioned.Message ID: @.***>

jadudm commented 8 months ago

Hi @cephillips ; Matt or @jadudm is just fine. :)

For clarity/transparency, this is a Github comment thread, and part of the public record of the FAC repository.

I'd be happy to try and find a time to discuss, if either/both of you have thoughts about how we could help auditors and auditees more easily submit correct audits to the Clearinghouse.

What is difficult is that we have a situation where:

  1. One thing was indicated in the SF-SAC
  2. Another thing was indicated in the report
  3. The auditor and auditee certified everything is correct

The collection process asks that this information be communicated in multiple places (potentially an unfortunate aspect of the collection design at this point in time), but I struggle with how we would reliably (meaning: confidently, 100% of the time) validate that the same information is being conveyed both in the form and in the PDF. It is clear that page 40 of the audit you linked to suggests the answer to going concern is No, but... reliably finding that information in every PDF, and correctly mapping it back to the SF-SAC 100% of the time is a non-trivial task.

We are committed to doing what we can to improve the quality of the data collected in the SF-SAC and SAR, but also are constrained in a number of ways. We cannot, for example, apply a validation to the SAR (the PDF) that is not 100% reliable, if we are going to "gatekeep" on submissions.

We might be able to do some kind of analysis where we attempt (for example) to determine if the going concern field is correctly represented in the SAR. But... what if the PDF is too poorly formed to reliably find this information? Should we reject the audit? Do we... issue a warning? What if that warning is wrong 60% of the time? Should we issue it regardless? (Or, what "correctness" threshold is acceptable for a warning, and how do we ascertain what the correctness is?) Or, perhaps this is something you already have a sense for?

If you have thoughts, tools, or techniques you would like to recommend, we would love to discuss. For example, looking at

https://biglocalnews.org/content/tools/audit-watch.html

if you have open tooling for the PDF analysis that you would be willing to help us understand, we could look at what it would take to incorporate it into the FAC. If you even have a sense for the percentage of audits this is occurring in, that may be of value, and something we can leverage as we attempt to better support auditors/auditees in their submissions. If you reach out to us via the help desk, we can find a time to discuss, if you like.

cephillips commented 8 months ago

It seems like an interesting conversation to have in the long run. I will reach out via the help desk. Cheryl Phillips

From: Matthew Jadud @.> Date: Sunday, February 25, 2024 at 9:42 AM To: GSA-TTS/FAC @.> Cc: Cheryl Phillips @.>, Mention @.> Subject: Re: [GSA-TTS/FAC] FAC - Going concern data being misreported (Issue #3433)

Hi @cephillipshttps://github.com/cephillips ; Matt or @jadudmhttps://github.com/jadudm is just fine. :)

For clarity/transparency, this is a Github comment thread, and part of the public record of the FAC repository.

I'd be happy to try and find a time to discuss, if either/both of you have thoughts about how we could help auditors and auditees more easily submit correct audits to the Clearinghouse.

What is difficult is that we have a situation where:

  1. One thing was indicated in the SF-SAC
  2. Another thing was indicated in the report
  3. The auditor and auditee certified everything is correct

The collection process asks that this information be communicated in multiple places (potentially an unfortunate aspect of the collection design at this point in time), but I struggle with how we would reliably (meaning: confidently, 100% of the time) validate that the same information is being conveyed both in the form and in the PDF. It is clear that page 40 of the audit you linked to suggests the answer to going concern is No, but... reliably finding that information in every PDF, and correctly mapping it back to the SF-SAC 100% of the time is a non-trivial task.

We are committed to doing what we can to improve the quality of the data collected in the SF-SAC and SAR, but also are constrained in a number of ways. We cannot, for example, apply a validation to the SAR (the PDF) that is not 100% reliable, if we are going to "gatekeep" on submissions.

We might be able to do some kind of analysis where we attempt (for example) to determine if the going concern field is correctly represented in the SAR. But... what if the PDF is too poorly formed to reliably find this information? Should we reject the audit? Do we... issue a warning? What if that warning is wrong 60% of the time? Should we issue it regardless? (Or, what "correctness" threshold is acceptable for a warning, and how do we ascertain what the correctness is?) Or, perhaps this is something you already have a sense for?

If you have thoughts, tools, or techniques you would like to recommend, we would love to discuss. For example, looking at

https://biglocalnews.org/content/tools/audit-watch.html

if you have open tooling for the PDF analysis that you would be willing to help us understand, we could look at what it would take to incorporate it into the FAC. If you even have a sense for the percentage of audits this is occurring in, that may be of value, and something we can leverage as we attempt to better support auditors/auditees in their submissions. If you reach out to us via the help desk, we can find a time to discuss, if you like.

— Reply to this email directly, view it on GitHubhttps://github.com/GSA-TTS/FAC/issues/3433#issuecomment-1963010298, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAEFU3SFL3NLLEVT7GC7MS3YVNZYTAVCNFSM6AAAAABDRRPPZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGAYTAMRZHA. You are receiving this because you were mentioned.Message ID: @.***>