mikeubell
commented
3 years ago On Sep 21, 2021, at 6:12 PM, Tom Dooner @.***> wrote:
@tdooner approved this pull request.
Looks good! No big suggestions, it looks like you've been adding a lot to this file since I've seen it last.
The only big thing is to consider extracting the token to be an environment variable. Is a token necessary to get API results from the data site? Or does it just help with a rate limit?
I think it is just for the rate limit… its been months, Ill check.
In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713516260:
@@ -19,8 +18,9 @@ def contributors_to_committee(name, id, election) return contrib.map do |contributor| { type: :contributor,
- first_name: contributor['Tran_NamF'],
- last_name: contributor['Tran_NamL'],
- c_name: contributor['Tran_NamF'] ? contributor_name would be more explicit, so it isn't confused with "committee name"
I like short names, but I can change.
In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713517087:
@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)
index = client.init_index('election')
+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) +oak_response = oak_client.get("https://data.oaklandca.gov/resource/f4dq-mk8d").body +charity_data = oak_response.map do |donation|
- {
- type: :donation,
- name: donation.official,
- office_title: donation.office,
- c_name: donation.payor,
- location: donation.payor_city,
- payee: donation.payee,
- amount: donation.amount.to_i,
merge this field with the election_date so sorting is more consistent
- election_date: donation.payment_date[0,10],
- description: donation.description,
- url: if donation.url.nil? a rubyism for you if you want it: you can do:
url: (donation.url["url"].to_s if donation.url), In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713518400:
@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)
index = client.init_index('election')
+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) +oak_response = oak_client.get("https://data.oaklandca.gov/resource/f4dq-mk8d").body +charity_data = oak_response.map do |donation|
- {
- type: :donation,
- name: donation.official,
- office_title: donation.office,
- c_name: donation.payor,
- location: donation.payor_city,
- payee: donation.payee,
- amount: donation.amount.to_i,
merge this field with the election_date so sorting is more consistent
- election_date: donation.payment_date[0,10], Yeah, merging this date into election_date probably makes sense for now, given how this field is used for sorting.
In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713518510:
@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)
index = client.init_index('election')
+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) +oak_response = oak_client.get("https://data.oaklandca.gov/resource/f4dq-mk8d").body +charity_data = oak_response.map do |donation|
- {
- type: :donation,
- name: donation.official,
- office_title: donation.office,
- c_name: donation.payor, Same gripe about c_name
In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713518771:
puts "total records: #{all_data.length}" -index.replace_all_objects(all_data) + +# Test code so we don't burn all of our allocation on Aloglia +if ENV['ALGOLIASEARCH_SAMPLE_DATA'] Clever!
I was running out of the number of the updates I could do.
In search_index.rb https://github.com/caciviclab/disclosure-backend-static/pull/285#discussion_r713520462:
@@ -43,6 +43,29 @@ def contributors_to_committee(name, id, election)
index = client.init_index('election')
+oak_client = SODA::Client.new({:domain => "data.oklandca.gov", :app_token => "4FYL4zxMOncsLeANaeDzP455z"}) In general we probably shouldn't commit tokens like this to the code, it would be better to pass it in as an environment variable to avoid anyone pretending to be us. This would be configured as part of the built process and on development machines. But, the risk here is pretty low if this token is only used to fetch API resources.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/caciviclab/disclosure-backend-static/pull/285#pullrequestreview-760368068, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTSH6BJHH467VC7QH6WF7LUDEUPRANCNFSM5EMM7XEQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
tdooner
Add data from the behested donation data. We use the SODA interface to directly pull the data and upload it to the index. Because the field names differ between the behested data and the campaign data we need to map them to common Algolia keys so that the records are ordered by the same criteria regardless of the source. This needs to be coordinated with front end PR 433.