Closed acheong08 closed 1 year ago
Request:
fetch("https://efdsearch.senate.gov/search/report/data/", {
"headers": {
"accept": "application/json, text/javascript, */*; q=0.01",
"accept-language": "en-US,en;q=0.9",
"content-type": "application/x-www-form-urlencoded; charset=UTF-8",
"sec-ch-ua": "\"Not_A Brand\";v=\"99\", \"Brave\";v=\"109\", \"Chromium\";v=\"109\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Linux\"",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"sec-gpc": "1",
"x-csrftoken": "HL0USQFVDenhuLzCQPmoRIOFIwpMqWELYnbzj2mQBne60hLofDLlYpaLRtwylEHt",
"x-requested-with": "XMLHttpRequest"
},
"referrer": "https://efdsearch.senate.gov/search/",
"referrerPolicy": "strict-origin-when-cross-origin",
"body": "draw=5&columns[0][data]=0&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=true&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=1&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=true&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=2&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=true&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=3&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=true&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=4&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=true&columns[4][search][value]=&columns[4][search][regex]=false&order[0][column]=4&order[0][dir]=desc&start=0&length=25&search[value]=&search[regex]=false&report_types=[]&filer_types=[1]&submitted_start_date=01%2F01%2F2012+00%3A00%3A00&submitted_end_date=&candidate_state=&senator_state=&office_id=&first_name=&last_name=",
"method": "POST",
"mode": "cors",
"credentials": "include"
});
Example: Response
{
"draw":3,
"recordsTotal":3632,
"recordsFiltered":3632,
"data":[
[
"Benjamin E",
"Sasse",
"Sasse, Ben (Senator)",
"<a href=\"/search/view/extension-notice/regular/be7c08d6-147c-4482-a82f-aef68cf82a36/\" target=\"_blank\">Termination Report for 2023 Due Date Extension 1</a>",
"01/20/2023"
],
[
"JOHN ",
"BOOZMAN",
"Senator",
"<a href=\"/search/view/paper/e22e46da-6779-4ea7-9371-1facf67703e8/\" target=\"_blank\">Annual Report (Amendment)</a>",
"01/20/2023"
],
[
"Rick",
"Scott",
"Scott, Rick (Senator)",
"<a href=\"/search/view/ptr/6b92f351-93e8-446a-a4d9-435bd9693d47/\" target=\"_blank\">Periodic Transaction Report for 01/19/2023</a>",
"01/19/2023"
],
[
"Roy",
"Blunt",
"Blunt, Roy (Senator)",
"<a href=\"/search/view/ptr/6b2cfa66-ff7e-481c-8fee-01ba0df453ec/\" target=\"_blank\">Periodic Transaction Report for 01/18/2023</a>",
"01/18/2023"
],
[
"Angus S",
"King, Jr.",
"King, Angus (Senator)",
"<a href=\"/search/view/ptr/c858ddd7-3c52-4ecb-9d0f-27ae2899d9dd/\" target=\"_blank\">Periodic Transaction Report for 01/18/2023</a>",
"01/18/2023"
],
[
"Shelley M",
"Capito",
"Capito, Shelley Moore (Senator)",
"<a href=\"/search/view/ptr/f5c92744-0aa0-4eda-b406-87d9d5d27190/\" target=\"_blank\">Periodic Transaction Report for 01/18/2023</a>",
"01/17/2023"
],
[
"Maria",
"Cantwell",
"Cantwell, Maria (Senator)",
"<a href=\"/search/view/ptr/5d41d5a8-eb5b-4092-aae1-99a26f4932f2/\" target=\"_blank\">Periodic Transaction Report for 01/13/2023</a>",
"01/13/2023"
],
[
"Thomas H",
"Tuberville",
"Tuberville, Tommy (Senator)",
"<a href=\"/search/view/ptr/6f6c5dfb-d731-4c2c-895b-2c842918bf3e/\" target=\"_blank\">Periodic Transaction Report for 01/13/2023</a>",
"01/13/2023"
],
[
"Jacklyn S",
"Rosen",
"Rosen, Jacky (Senator)",
"<a href=\"/search/view/ptr/63e282d7-c362-49de-8c60-4e7389ab2d19/\" target=\"_blank\">Periodic Transaction Report for 01/11/2023</a>",
"01/11/2023"
],
[
"Gary C",
"Peters",
"Peters, Gary (Senator)",
"<a href=\"/search/view/ptr/517e976a-8930-4bce-9037-5d93c4cab488/\" target=\"_blank\">Periodic Transaction Report for 01/11/2023</a>",
"01/10/2023"
],
[
"Susan M",
"Collins",
"Collins, Susan M. (Senator)",
"<a href=\"/search/view/ptr/44d5e494-6825-487d-9ffd-d2e7725d5cfa/\" target=\"_blank\">Periodic Transaction Report for 01/11/2023</a>",
"01/10/2023"
],
[
"Thomas R",
"Carper",
"Carper, Thomas R. (Senator)",
"<a href=\"/search/view/ptr/3091f74a-6d05-49da-b260-8c6866ce7fa5/\" target=\"_blank\">Periodic Transaction Report for 01/05/2023</a>",
"01/05/2023"
],
[
"Robert J",
"Portman",
"Portman, Rob (Senator)",
"<a href=\"/search/view/extension-notice/regular/dce0d88a-6109-4719-92da-088316ee5501/\" target=\"_blank\">Termination Report for 2023 Due Date Extension 1</a>",
"01/04/2023"
],
[
"Sheldon",
"Whitehouse",
"Whitehouse, Sheldon (Senator)",
"<a href=\"/search/view/ptr/53ddfdc6-dc9a-4945-af41-20de2224ae1c/\" target=\"_blank\">Periodic Transaction Report for 12/30/2022</a>",
"12/30/2022"
],
[
"A. Mitchell",
"McConnell, Jr.",
"McConnell, A. Mitchell Jr. (Senator)",
"<a href=\"/search/view/ptr/adcbd2b1-63a0-4c50-b7ef-d093b34336eb/\" target=\"_blank\">Periodic Transaction Report for 12/29/2022</a>",
"12/29/2022"
],
[
"Rick",
"Scott",
"Scott, Rick (Senator)",
"<a href=\"/search/view/ptr/a878d31a-cc7f-4402-aea8-e2ad2847ab4c/\" target=\"_blank\">Periodic Transaction Report for 12/23/2022</a>",
"12/23/2022"
],
[
"John W",
"Hickenlooper",
"Hickenlooper, John (Senator)",
"<a href=\"/search/view/ptr/0acee755-ed8e-4a93-8785-9946d8c49e03/\" target=\"_blank\">Periodic Transaction Report for 12/22/2022</a>",
"12/22/2022"
],
[
"CYNTHIA M",
"LUMMIS",
"Senator",
"<a href=\"/search/view/paper/12ba9d39-339c-4922-8a2a-f7dd52635fae/\" target=\"_blank\">Blind Trust</a>",
"12/22/2022"
],
[
"Daniel S",
"Sullivan",
"Sullivan, Dan (Senator)",
"<a href=\"/search/view/ptr/91384996-f2d2-4cb1-b261-43afd6b8c465/\" target=\"_blank\">Periodic Transaction Report for 12/21/2022</a>",
"12/21/2022"
],
[
"Patrick J",
"Toomey",
"Toomey, Pat (Senator)",
"<a href=\"/search/view/annual/015c1549-a85e-4a41-9263-5c27a2191fc9/\" target=\"_blank\">Annual Report for CY 2021 (Amendment 1)</a>",
"12/20/2022"
],
[
"Susan M",
"Collins",
"Collins, Susan M. (Senator)",
"<a href=\"/search/view/ptr/7471e623-3ee6-4c58-a6ae-64aa50851b03/\" target=\"_blank\">Periodic Transaction Report for 12/16/2022</a>",
"12/16/2022"
],
[
"Thomas H",
"Tuberville",
"Tuberville, Tommy (Senator)",
"<a href=\"/search/view/ptr/7da74f60-a4f9-42bb-9663-35bde0f9af53/\" target=\"_blank\">Periodic Transaction Report for 12/14/2022</a>",
"12/14/2022"
],
[
"Patrick J",
"Toomey",
"Toomey, Pat (Senator)",
"<a href=\"/search/view/extension-notice/regular/6a5032c6-f81e-4f88-b426-81bebc7b0c38/\" target=\"_blank\">Termination Report for 2023 Due Date Extension 1</a>",
"12/14/2022"
],
[
"JOHN ",
"BOOZMAN",
"Senator",
"<a href=\"/search/view/paper/2a393d64-8daa-4b4b-9fe0-82923c3b0544/\" target=\"_blank\">Periodic Transaction Report for 12/14/2022</a>",
"12/14/2022"
],
[
"Ron L",
"Wyden",
"Wyden, Ron (Senator)",
"<a href=\"/search/view/ptr/b239e81f-3a2a-4cde-8c96-e82c84266aad/\" target=\"_blank\">Periodic Transaction Report for 12/11/2022</a>",
"12/11/2022"
]
],
"result":"ok"
}
We're looking for "Periodic Transaction Report"
Got to parse an HTML table pre-rendered
The API is currently down due to maintenance. The current code should work once it gets back up.
Done:
To do:
Done with most things. Storage can come later
https://efdsearch.senate.gov/search/report/data/