manbehindthemadness
commented
3 months ago This can be achieved using python:
-
Using playwright and requests:
- a. Acquire the names of all sitting members of Congress in a
list()object. - b. Perform the desired search(es) by looping through the list of names and commit the financial disclosure information into a
dict()object. - c. Iterate through the resulting dictionary to locate the links to each "full year" report.
- d. Use the
requestsmodule to save each report using a meaningful nomenclature.
- a. Acquire the names of all sitting members of Congress in a
-
Use OCR to extract the scanned text data.
- a. Extract raw text from the PDF documents where able.
- b. Items without raw text can be assumed scanned.
- c. Scanned items processed using Karas or Tesseract (Karas is more accurate but slower).
-
Store the results. (This is going to depend on the level of true automation desired, for fun, I will go all-out)
- a. Using huggingface transformers and langchain's ingestion, vectorize and store all the translated documents into a chroma db.
- b. Bind the resulting vector store into the context of a QA retrieval chain based on a LLama 3, 8b, gguf model (readily available for download here: https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF).
-
Template the desired operation and script the output.
- a. Create a template within the aforementioned QA chain describing the type of hit data that is desired and how to postulate the responses.
- b. Create a series of prompts that request the various portions of the report you wish to create and how they should be constructed.
-
Bring it all together, either on a schedule, or by personal execution we will trigger the following events:
- a. Perform web searches and pdf acquisition (1).
- b. Extract text and image content (2).
- c. Vectorize and store this information (3).
- d. Iterate through the pre-fabricated section prompts to create the top-level reading material.
- e. For each hit and prompt return we receive from the QA chain, sources are included for where the data was found within the context (pdfs). We should collect these and add citations.
- f. Perform proof-reading using a meat-space discriminator agent.
I hope this helps, as it's a merely a method and not a finished solution, I furnish this outside of the scope of the bounty.
-- Delve,
https://github.com/langchain-ai/langchain https://github.com/faustomorales/keras-ocr https://github.com/huggingface/transformers https://github.com/chroma-core/chroma https://github.com/microsoft/playwright
dreslan
geohotstan
jlopp
The Bitcoin Politicians project is woefully out of date due to the hundreds of hours required to manually sift through financial disclosures from 535 members of Congress.
I'm offering a bounty of $1,000 in bitcoin to someone who writes enough code to automate this process. I don't particularly care what tools are used as long as I can reproduce the results and run it on a weekly basis. If you think $1,000 is insufficient for the complexity of this project, feel free to make a bid.
What would automation entail?
This should be straightforward for the members who file digitally, as searching for terms should be as simple as scraping the relevant web page. However, many members of Congress still file on paper and the filings web site will return a PDF, some of which are hundreds of pages long. This will likely be the most challenging part of the project, requiring either a really good OCR library or some kind of LLM that can effectively perform OCR.
In order to consider a submission to be of high enough quality, I'd like to see at least a 95% accuracy rate when it comes to correctly identifying politicians holding the assets for which we're searching.