Within regulations_document, lots of "NULL" values and "Not Found" in fr_doc_num and agency_id columns

uchicago-capp-30320 / CivicLens

Putting the public back in public commenting

https://civic-lens.org

GNU Affero General Public License v3.0

2 stars 1 forks source link

Within regulations_document, lots of "NULL" values and "Not Found" in fr_doc_num and agency_id columns #96

Closed jchristensonCHI closed 6 months ago

jchristensonCHI commented 6 months ago

Talked to Andrew about this and posted the issue as he asked. What I saw in the PostgreSQL database:

andrewjtdunn commented 6 months ago

Saying this explicitly: issue is that some of the row values appear to not match the column headers. If so, this suggests that something in the data upload process is misaligned

GregoryHo88 commented 6 months ago

1.) lots of "NULL" values and "Not Found" in fr_doc_num and agency_id

@jchristensonCHI, it seems reasonable to expect these discrepancies as not all documents uploaded to regulations.gov are archived in the Federal Register.

2.) Some row values appear not to match column headers.

I will conduct further tests and investigate this issue more thoroughly.

rezarzky commented 6 months ago

Checked on 3 API, not all of them have fr_doc_num

andrewjtdunn commented 6 months ago

discussion as of today:

[ ] build out QA checks on document data (and ideally docket and comment data)
[ ] extract list of document ids in the regulations_document table
[ ] re-pull document data and update in the regulations_document table