acheong08 / insider

Financial data scraper for US governmental figures
15 stars 1 forks source link

Automated crawler for Senate #2

Closed acheong08 closed 1 year ago

acheong08 commented 1 year ago

https://efdsearch.senate.gov/search/report/data/

acheong08 commented 1 year ago

Request:

fetch("https://efdsearch.senate.gov/search/report/data/", {
  "headers": {
    "accept": "application/json, text/javascript, */*; q=0.01",
    "accept-language": "en-US,en;q=0.9",
    "content-type": "application/x-www-form-urlencoded; charset=UTF-8",
    "sec-ch-ua": "\"Not_A Brand\";v=\"99\", \"Brave\";v=\"109\", \"Chromium\";v=\"109\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    "sec-gpc": "1",
    "x-csrftoken": "HL0USQFVDenhuLzCQPmoRIOFIwpMqWELYnbzj2mQBne60hLofDLlYpaLRtwylEHt",
    "x-requested-with": "XMLHttpRequest"
  },
  "referrer": "https://efdsearch.senate.gov/search/",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": "draw=5&columns[0][data]=0&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=true&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=1&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=true&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=2&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=true&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=3&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=true&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=4&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=true&columns[4][search][value]=&columns[4][search][regex]=false&order[0][column]=4&order[0][dir]=desc&start=0&length=25&search[value]=&search[regex]=false&report_types=[]&filer_types=[1]&submitted_start_date=01%2F01%2F2012+00%3A00%3A00&submitted_end_date=&candidate_state=&senator_state=&office_id=&first_name=&last_name=",
  "method": "POST",
  "mode": "cors",
  "credentials": "include"
});

Example: Response

{
   "draw":3,
   "recordsTotal":3632,
   "recordsFiltered":3632,
   "data":[
      [
         "Benjamin E",
         "Sasse",
         "Sasse, Ben (Senator)",
         "<a href=\"/search/view/extension-notice/regular/be7c08d6-147c-4482-a82f-aef68cf82a36/\" target=\"_blank\">Termination Report for 2023 Due Date Extension 1</a>",
         "01/20/2023"
      ],
      [
         "JOHN ",
         "BOOZMAN",
         "Senator",
         "<a href=\"/search/view/paper/e22e46da-6779-4ea7-9371-1facf67703e8/\" target=\"_blank\">Annual Report (Amendment)</a>",
         "01/20/2023"
      ],
      [
         "Rick",
         "Scott",
         "Scott, Rick (Senator)",
         "<a href=\"/search/view/ptr/6b92f351-93e8-446a-a4d9-435bd9693d47/\" target=\"_blank\">Periodic Transaction Report for 01/19/2023</a>",
         "01/19/2023"
      ],
      [
         "Roy",
         "Blunt",
         "Blunt, Roy (Senator)",
         "<a href=\"/search/view/ptr/6b2cfa66-ff7e-481c-8fee-01ba0df453ec/\" target=\"_blank\">Periodic Transaction Report for 01/18/2023</a>",
         "01/18/2023"
      ],
      [
         "Angus S",
         "King, Jr.",
         "King, Angus (Senator)",
         "<a href=\"/search/view/ptr/c858ddd7-3c52-4ecb-9d0f-27ae2899d9dd/\" target=\"_blank\">Periodic Transaction Report for 01/18/2023</a>",
         "01/18/2023"
      ],
      [
         "Shelley M",
         "Capito",
         "Capito, Shelley Moore (Senator)",
         "<a href=\"/search/view/ptr/f5c92744-0aa0-4eda-b406-87d9d5d27190/\" target=\"_blank\">Periodic Transaction Report for 01/18/2023</a>",
         "01/17/2023"
      ],
      [
         "Maria",
         "Cantwell",
         "Cantwell, Maria (Senator)",
         "<a href=\"/search/view/ptr/5d41d5a8-eb5b-4092-aae1-99a26f4932f2/\" target=\"_blank\">Periodic Transaction Report for 01/13/2023</a>",
         "01/13/2023"
      ],
      [
         "Thomas H",
         "Tuberville",
         "Tuberville, Tommy (Senator)",
         "<a href=\"/search/view/ptr/6f6c5dfb-d731-4c2c-895b-2c842918bf3e/\" target=\"_blank\">Periodic Transaction Report for 01/13/2023</a>",
         "01/13/2023"
      ],
      [
         "Jacklyn S",
         "Rosen",
         "Rosen, Jacky (Senator)",
         "<a href=\"/search/view/ptr/63e282d7-c362-49de-8c60-4e7389ab2d19/\" target=\"_blank\">Periodic Transaction Report for 01/11/2023</a>",
         "01/11/2023"
      ],
      [
         "Gary C",
         "Peters",
         "Peters, Gary (Senator)",
         "<a href=\"/search/view/ptr/517e976a-8930-4bce-9037-5d93c4cab488/\" target=\"_blank\">Periodic Transaction Report for 01/11/2023</a>",
         "01/10/2023"
      ],
      [
         "Susan M",
         "Collins",
         "Collins, Susan M. (Senator)",
         "<a href=\"/search/view/ptr/44d5e494-6825-487d-9ffd-d2e7725d5cfa/\" target=\"_blank\">Periodic Transaction Report for 01/11/2023</a>",
         "01/10/2023"
      ],
      [
         "Thomas R",
         "Carper",
         "Carper, Thomas R. (Senator)",
         "<a href=\"/search/view/ptr/3091f74a-6d05-49da-b260-8c6866ce7fa5/\" target=\"_blank\">Periodic Transaction Report for 01/05/2023</a>",
         "01/05/2023"
      ],
      [
         "Robert J",
         "Portman",
         "Portman, Rob (Senator)",
         "<a href=\"/search/view/extension-notice/regular/dce0d88a-6109-4719-92da-088316ee5501/\" target=\"_blank\">Termination Report for 2023 Due Date Extension 1</a>",
         "01/04/2023"
      ],
      [
         "Sheldon",
         "Whitehouse",
         "Whitehouse, Sheldon (Senator)",
         "<a href=\"/search/view/ptr/53ddfdc6-dc9a-4945-af41-20de2224ae1c/\" target=\"_blank\">Periodic Transaction Report for 12/30/2022</a>",
         "12/30/2022"
      ],
      [
         "A. Mitchell",
         "McConnell, Jr.",
         "McConnell, A. Mitchell Jr. (Senator)",
         "<a href=\"/search/view/ptr/adcbd2b1-63a0-4c50-b7ef-d093b34336eb/\" target=\"_blank\">Periodic Transaction Report for 12/29/2022</a>",
         "12/29/2022"
      ],
      [
         "Rick",
         "Scott",
         "Scott, Rick (Senator)",
         "<a href=\"/search/view/ptr/a878d31a-cc7f-4402-aea8-e2ad2847ab4c/\" target=\"_blank\">Periodic Transaction Report for 12/23/2022</a>",
         "12/23/2022"
      ],
      [
         "John W",
         "Hickenlooper",
         "Hickenlooper, John (Senator)",
         "<a href=\"/search/view/ptr/0acee755-ed8e-4a93-8785-9946d8c49e03/\" target=\"_blank\">Periodic Transaction Report for 12/22/2022</a>",
         "12/22/2022"
      ],
      [
         "CYNTHIA M",
         "LUMMIS",
         "Senator",
         "<a href=\"/search/view/paper/12ba9d39-339c-4922-8a2a-f7dd52635fae/\" target=\"_blank\">Blind Trust</a>",
         "12/22/2022"
      ],
      [
         "Daniel S",
         "Sullivan",
         "Sullivan, Dan (Senator)",
         "<a href=\"/search/view/ptr/91384996-f2d2-4cb1-b261-43afd6b8c465/\" target=\"_blank\">Periodic Transaction Report for 12/21/2022</a>",
         "12/21/2022"
      ],
      [
         "Patrick J",
         "Toomey",
         "Toomey, Pat (Senator)",
         "<a href=\"/search/view/annual/015c1549-a85e-4a41-9263-5c27a2191fc9/\" target=\"_blank\">Annual Report for CY 2021 (Amendment 1)</a>",
         "12/20/2022"
      ],
      [
         "Susan M",
         "Collins",
         "Collins, Susan M. (Senator)",
         "<a href=\"/search/view/ptr/7471e623-3ee6-4c58-a6ae-64aa50851b03/\" target=\"_blank\">Periodic Transaction Report for 12/16/2022</a>",
         "12/16/2022"
      ],
      [
         "Thomas H",
         "Tuberville",
         "Tuberville, Tommy (Senator)",
         "<a href=\"/search/view/ptr/7da74f60-a4f9-42bb-9663-35bde0f9af53/\" target=\"_blank\">Periodic Transaction Report for 12/14/2022</a>",
         "12/14/2022"
      ],
      [
         "Patrick J",
         "Toomey",
         "Toomey, Pat (Senator)",
         "<a href=\"/search/view/extension-notice/regular/6a5032c6-f81e-4f88-b426-81bebc7b0c38/\" target=\"_blank\">Termination Report for 2023 Due Date Extension 1</a>",
         "12/14/2022"
      ],
      [
         "JOHN ",
         "BOOZMAN",
         "Senator",
         "<a href=\"/search/view/paper/2a393d64-8daa-4b4b-9fe0-82923c3b0544/\" target=\"_blank\">Periodic Transaction Report for 12/14/2022</a>",
         "12/14/2022"
      ],
      [
         "Ron L",
         "Wyden",
         "Wyden, Ron (Senator)",
         "<a href=\"/search/view/ptr/b239e81f-3a2a-4cde-8c96-e82c84266aad/\" target=\"_blank\">Periodic Transaction Report for 12/11/2022</a>",
         "12/11/2022"
      ]
   ],
   "result":"ok"
}
acheong08 commented 1 year ago

We're looking for "Periodic Transaction Report"

acheong08 commented 1 year ago

Got to parse an HTML table pre-rendered

acheong08 commented 1 year ago

The API is currently down due to maintenance. The current code should work once it gets back up.

acheong08 commented 1 year ago

Done:

To do:

acheong08 commented 1 year ago

Done with most things. Storage can come later