caciviclab / disclosure-backend-static

Campaign finance data powering Open Disclosure California
https://caciviclab.org/odca-jekyll/
15 stars 14 forks source link

Address #284. Need to map data sets to the same Algolia keys. #285

Closed mikeubell closed 3 years ago

mikeubell commented 3 years ago

Add data from the behested donation data. We use the SODA interface to directly pull the data and upload it to the index. Because the field names differ between the behested data and the campaign data we need to map them to common Algolia keys so that the records are ordered by the same criteria regardless of the source. This needs to be coordinated with front end PR 433.

mikeubell commented 3 years ago

On Sep 21, 2021, at 6:12 PM, Tom Dooner @.***> wrote:

@tdooner approved this pull request.

Looks good! No big suggestions, it looks like you've been adding a lot to this file since I've seen it last.

The only big thing is to consider extracting the token to be an environment variable. Is a token necessary to get API results from the data site? Or does it just help with a rate limit?

I think it is just for the rate limit… its been months, Ill check.

In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713516260:

@@ -19,8 +18,9 @@ def contributors_to_committee(name, id, election) return contrib.map do |contributor| { type: :contributor,

  • first_name: contributor['Tran_NamF'],
  • last_name: contributor['Tran_NamL'],
  • c_name: contributor['Tran_NamF'] ? contributor_name would be more explicit, so it isn't confused with "committee name"

I like short names, but I can change.

In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713517087:

@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)

index = client.init_index('election')

+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) +oak_response = oak_client.get("https://data.oaklandca.gov/resource/f4dq-mk8d").body +charity_data = oak_response.map do |donation|

  • {
  • type: :donation,
  • name: donation.official,
  • office_title: donation.office,
  • c_name: donation.payor,
  • location: donation.payor_city,
  • payee: donation.payee,
  • amount: donation.amount.to_i,
  • merge this field with the election_date so sorting is more consistent

  • election_date: donation.payment_date[0,10],
  • description: donation.description,
  • url: if donation.url.nil? a rubyism for you if you want it: you can do:

url: (donation.url["url"].to_s if donation.url), In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713518400:

@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)

index = client.init_index('election')

+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) +oak_response = oak_client.get("https://data.oaklandca.gov/resource/f4dq-mk8d").body +charity_data = oak_response.map do |donation|

  • {
  • type: :donation,
  • name: donation.official,
  • office_title: donation.office,
  • c_name: donation.payor,
  • location: donation.payor_city,
  • payee: donation.payee,
  • amount: donation.amount.to_i,
  • merge this field with the election_date so sorting is more consistent

  • election_date: donation.payment_date[0,10], Yeah, merging this date into election_date probably makes sense for now, given how this field is used for sorting.

In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713518510:

@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)

index = client.init_index('election')

+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) +oak_response = oak_client.get("https://data.oaklandca.gov/resource/f4dq-mk8d").body +charity_data = oak_response.map do |donation|

  • {
  • type: :donation,
  • name: donation.official,
  • office_title: donation.office,
  • c_name: donation.payor, Same gripe about c_name

In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713518771:

puts "total records: #{all_data.length}" -index.replace_all_objects(all_data) + +# Test code so we don't burn all of our allocation on Aloglia +if ENV['ALGOLIASEARCH_SAMPLE_DATA'] Clever!

I was running out of the number of the updates I could do.

In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713520462:

@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)

index = client.init_index('election')

+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) In general we probably shouldn't commit tokens like this to the code, it would be better to pass it in as an environment variable to avoid anyone pretending to be us. This would be configured as part of the built process and on development machines. But, the risk here is pretty low if this token is only used to fetch API resources.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/caciviclab/disclosure-backend-static/pull/285#pullrequestreview-760368068, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTSH6BJHH467VC7QH6WF7LUDEUPRANCNFSM5EMM7XEQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

tdooner commented 3 years ago
Build diff from Commit c5086858ca989f1693875ef301aa0ace9c224f6a: ```diff diff --git a/build/_data/committees/1421001.json b/build/_data/committees/1421001.json index b028463..9350f85 100644 --- a/build/_data/committees/1421001.json +++ b/build/_data/committees/1421001.json @@ -1154,7 +1154,7 @@ "Tran_Date": "2020-07-15", "Tran_NamF": "Jonathan", "Tran_NamL": "Williams", - "Tran_Zip4": "94602" + "Tran_Zip4": "94603" }, { "Filer_ID": "1421001", @@ -1165,7 +1165,7 @@ "Tran_Date": "2020-07-15", "Tran_NamF": "Jonathan", "Tran_NamL": "Williams", - "Tran_Zip4": "94603" + "Tran_Zip4": "94602" }, { "Filer_ID": "1421001", @@ -1456,24 +1456,24 @@ { "Filer_ID": "1421001", "Tran_Emp": "Jordan Custom Builders", - "Tran_Occ": "Operations Manager", + "Tran_Occ": "Owner", "Entity_Cd": "Individual", "Tran_Amt1": 200.0, "Tran_Date": "2020-07-21", "Tran_NamF": "Billy", "Tran_NamL": "Jordan", - "Tran_Zip4": "75154" + "Tran_Zip4": "75204" }, { "Filer_ID": "1421001", "Tran_Emp": "Jordan Custom Builders", - "Tran_Occ": "Owner", + "Tran_Occ": "Operations Manager", "Entity_Cd": "Individual", "Tran_Amt1": 200.0, "Tran_Date": "2020-07-21", "Tran_NamF": "Billy", "Tran_NamL": "Jordan", - "Tran_Zip4": "75204" + "Tran_Zip4": "75154" }, { "Filer_ID": "1421001", diff --git a/build/_data/elections/oakland/2018-06-05.json b/build/_data/elections/oakland/2018-06-05.json index 666352f..e12f489 100644 --- a/build/_data/elections/oakland/2018-06-05.json +++ b/build/_data/elections/oakland/2018-06-05.json @@ -47,7 +47,7 @@ "total_contributions": 15000.0 }, { - "name": "Oakland Athletics Baseball Company", + "name": "Service Employees International Union Local 1021 Issues PAC", "election_name": "oakland-june-2018", "total_contributions": 10000.0 } diff --git a/build/_data/elections/oakland/2020-11-03.json b/build/_data/elections/oakland/2020-11-03.json index 7ca6bd8..26d59fd 100644 --- a/build/_data/elections/oakland/2020-11-03.json +++ b/build/_data/elections/oakland/2020-11-03.json @@ -7,7 +7,7 @@ }, "contributions_by_type": { "Committee": 425323.06, - "Individual": 1708466.14, + "Individual": 1708466.1400000001, "Unitemized": 201316.27999999997, "Self Funding": 51500.59, "Other (includes Businesses)": 325307.44 diff --git a/build/_data/elections/oakland/2022-11-08.json b/build/_data/elections/oakland/2022-11-08.json index 8f1fa12..a2f1319 100644 --- a/build/_data/elections/oakland/2022-11-08.json +++ b/build/_data/elections/oakland/2022-11-08.json @@ -58,7 +58,7 @@ "total_contributions": 3400.0 }, { - "name": "International Brotherhood of Electrical Workers Local 595PAC", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "election_name": "oakland-2022", "total_contributions": 1700.0 }, @@ -76,13 +76,13 @@ "total_contributions": 3400.0 }, { - "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "type": "Office", "election_name": "oakland-2022", "total_contributions": 1700.0 }, { - "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", + "name": "Sprinkler Fitters and Apprentices Local 483 Local Political Action Committee", "type": "Office", "election_name": "oakland-2022", "total_contributions": 1700.0 @@ -99,13 +99,13 @@ "total_spending": 3400.0 }, { - "name": "International Brotherhood of Electrical Workers Local 595PAC", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 }, { - "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund", + "name": "Sprinkler Fitters and Apprentices Local 483 Local Political Action Committee", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 @@ -119,13 +119,13 @@ "total_spending": 3400.0 }, { - "name": "International Brotherhood of Electrical Workers Local 595PAC", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 }, { - "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund", + "name": "Sprinkler Fitters and Apprentices Local 483 Local Political Action Committee", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 diff --git a/build/_data/referendum_supporting/oakland/2020-11-03/ousd-facilities-bond.json b/build/_data/referendum_supporting/oakland/2020-11-03/ousd-facilities-bond.json index 374bb43..6de36e5 100644 --- a/build/_data/referendum_supporting/oakland/2020-11-03/ousd-facilities-bond.json +++ b/build/_data/referendum_supporting/oakland/2020-11-03/ousd-facilities-bond.json @@ -27,7 +27,7 @@ }, { "type": "Individual", - "amount": 81245.5799999999 + "amount": 81245.58 }, { "type": "Other (includes Businesses)", diff --git a/build/_data/stats.json b/build/_data/stats.json index 818a208..3940100 100644 --- a/build/_data/stats.json +++ b/build/_data/stats.json @@ -1,3 +1,3 @@ { - "date_processed": "2021-10-08 00:18:47 -0700" + "date_processed": "2021-10-09 00:20:37 -0700" } diff --git a/build/_data/totals.json b/build/_data/totals.json index fad1ed8..428cd69 100644 --- a/build/_data/totals.json +++ b/build/_data/totals.json @@ -437,7 +437,7 @@ "total_contributions": 15000.0 }, { - "name": "Oakland Athletics Baseball Company", + "name": "Service Employees International Union Local 1021 Issues PAC", "election_name": "oakland-june-2018", "total_contributions": 10000.0 } @@ -713,7 +713,7 @@ }, "contributions_by_type": { "Committee": 425323.06, - "Individual": 1708466.14, + "Individual": 1708466.1400000001, "Unitemized": 201316.27999999997, "Self Funding": 51500.59, "Other (includes Businesses)": 325307.44 @@ -1090,7 +1090,7 @@ "total_contributions": 3400.0 }, { - "name": "International Brotherhood of Electrical Workers Local 595PAC", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "election_name": "oakland-2022", "total_contributions": 1700.0 }, @@ -1108,13 +1108,13 @@ "total_contributions": 3400.0 }, { - "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "type": "Office", "election_name": "oakland-2022", "total_contributions": 1700.0 }, { - "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", + "name": "Sprinkler Fitters and Apprentices Local 483 Local Political Action Committee", "type": "Office", "election_name": "oakland-2022", "total_contributions": 1700.0 @@ -1131,13 +1131,13 @@ "total_spending": 3400.0 }, { - "name": "International Brotherhood of Electrical Workers Local 595PAC", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 }, { - "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund", + "name": "Sprinkler Fitters and Apprentices Local 483 Local Political Action Committee", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 @@ -1151,13 +1151,13 @@ "total_spending": 3400.0 }, { - "name": "International Brotherhood of Electrical Workers Local 595PAC", + "name": "UNITY PAC sponsored by Alameda Labor Council, AFL-CIO", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 }, { - "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund", + "name": "Sprinkler Fitters and Apprentices Local 483 Local Political Action Committee", "type": "Office", "election_name": "oakland-2022", "total_spending": 1700.0 ```