propublica / politwoops_sunlight

Politwoops web front end
Other
44 stars 30 forks source link

Errors importing twitter accounts #15

Closed adieyal closed 10 years ago

adieyal commented 10 years ago

I am having problems setting up Politwoops. I can't populate the database with twitter accounts. After changing my consumer and auth keys I tried to populate using the example csv file. No dice. Since that file has a different format to that suggested in the README, I created my own custom file, same problem.

Below is my output

I see errors when I populate the database with the CSV file. See the output here:

root@4823ef9b0622:/politwoops# bundle exec rake politicians:import_csv CSV=twoopsters.csv 
Faraday::Builder is now Faraday::RackBuilder.
twitter user length 704
rake aborted!
undefined method `add_related_politicians' for nil:NilClass
/politwoops/lib/tasks/politicians.rake:184
/politwoops/lib/tasks/politicians.rake:182:in `each'
/politwoops/lib/tasks/politicians.rake:182
Tasks: TOP => politicians:import_csv
(See full trace by running task with --trace)

Now with my custom file

root@4823ef9b0622:/politwoops# cat satwoops.txt 
,helenzille,,,DA,default

root@4823ef9b0622:/politwoops# bundle exec rake politicians:import_csv CSV=satwoops.txt   
Faraday::Builder is now Faraday::RackBuilder.
rake aborted!
undefined method `downcase' for nil:NilClass
/politwoops/lib/tasks/politicians.rake:108
/politwoops/lib/tasks/politicians.rake:101
Tasks: TOP => politicians:import_csv
(See full trace by running task with --trace)
root@4823ef9b0622:/politwoops# bundle exec rake politicians:import_csv CSV=satwoops.txt

If I now run the website, I see a few populated twitter accounts but clicking on http://172.17.0.3:3000/user/toddakin/ for instance gives me the following error:

undefined method `flat_map' for []:ActiveRecord::Relation

If it is useful, here is a full trace

activerecord (3.2.11) lib/active_record/relation/delegation.rb:45:in `method_missing'
app/models/politician.rb:79:in `get_related_politicians'
app/controllers/politicians_controller.rb:21:in `show'
actionpack (3.2.11) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (3.2.11) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (3.2.11) lib/abstract_controller/base.rb:167:in `process_action'
actionpack (3.2.11) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (3.2.11) lib/abstract_controller/callbacks.rb:18:in `process_action'
activesupport (3.2.11) lib/active_support/callbacks.rb:436:in `_run__885747371__process_action__1067179940__callbacks'
activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `send'
activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `__run_callback'
activesupport (3.2.11) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
activesupport (3.2.11) lib/active_support/callbacks.rb:81:in `send'
activesupport (3.2.11) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.2.11) lib/abstract_controller/callbacks.rb:17:in `process_action'
actionpack (3.2.11) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (3.2.11) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
activesupport (3.2.11) lib/active_support/notifications.rb:123:in `instrument'
activesupport (3.2.11) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (3.2.11) lib/active_support/notifications.rb:123:in `instrument'
actionpack (3.2.11) lib/action_controller/metal/instrumentation.rb:29:in `process_action'
actionpack (3.2.11) lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
activerecord (3.2.11) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (3.2.11) lib/abstract_controller/base.rb:121:in `process'
actionpack (3.2.11) lib/abstract_controller/rendering.rb:45:in `process'
actionpack (3.2.11) lib/action_controller/metal.rb:203:in `dispatch'
actionpack (3.2.11) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
actionpack (3.2.11) lib/action_controller/metal.rb:246:in `action'
actionpack (3.2.11) lib/action_dispatch/routing/route_set.rb:73:in `call'
actionpack (3.2.11) lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
actionpack (3.2.11) lib/action_dispatch/routing/route_set.rb:36:in `call'
journey (1.0.4) lib/journey/router.rb:68:in `call'
journey (1.0.4) lib/journey/router.rb:56:in `each'
journey (1.0.4) lib/journey/router.rb:56:in `call'
actionpack (3.2.11) lib/action_dispatch/routing/route_set.rb:601:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
rack (1.4.3) lib/rack/etag.rb:23:in `call'
rack (1.4.3) lib/rack/conditionalget.rb:25:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/head.rb:14:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/flash.rb:242:in `call'
rack (1.4.3) lib/rack/session/abstract/id.rb:210:in `context'
rack (1.4.3) lib/rack/session/abstract/id.rb:205:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/cookies.rb:341:in `call'
activerecord (3.2.11) lib/active_record/query_cache.rb:64:in `call'
activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/callbacks.rb:28:in `call'
activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `_run__1724966646__call__4__callbacks'
activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `send'
activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `__run_callback'
activesupport (3.2.11) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
activesupport (3.2.11) lib/active_support/callbacks.rb:81:in `send'
activesupport (3.2.11) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.2.11) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/reloader.rb:65:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
railties (3.2.11) lib/rails/rack/logger.rb:32:in `call_app'
railties (3.2.11) lib/rails/rack/logger.rb:16:in `call'
activesupport (3.2.11) lib/active_support/tagged_logging.rb:22:in `tagged'
railties (3.2.11) lib/rails/rack/logger.rb:16:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/request_id.rb:22:in `call'
rack (1.4.3) lib/rack/methodoverride.rb:21:in `call'
rack (1.4.3) lib/rack/runtime.rb:17:in `call'
activesupport (3.2.11) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
rack (1.4.3) lib/rack/lock.rb:15:in `call'
actionpack (3.2.11) lib/action_dispatch/middleware/static.rb:62:in `call'
railties (3.2.11) lib/rails/engine.rb:479:in `call'
railties (3.2.11) lib/rails/application.rb:223:in `call'
rack (1.4.3) lib/rack/content_length.rb:14:in `call'
railties (3.2.11) lib/rails/rack/log_tailer.rb:17:in `call'
rack (1.4.3) lib/rack/handler/webrick.rb:59:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
rack (1.4.3) lib/rack/handler/webrick.rb:13:in `run'
rack (1.4.3) lib/rack/server.rb:268:in `start'
railties (3.2.11) lib/rails/commands/server.rb:70:in `start'
railties (3.2.11) lib/rails/commands.rb:55
railties (3.2.11) lib/rails/commands.rb:50:in `tap'
railties (3.2.11) lib/rails/commands.rb:50
script/rails:6:in `require'
script/rails:6

Any help would be appreciated

dvogel commented 10 years ago

I've updated the README to note that there is now a admin UI for adding users manually. If you would like to continue with the CSV route, I've made a few fixes to the CSV import route and the views that should let you continue without problems. Make sure you pull up to commit faf7b25f3a25be2fbae574acfbf798f1c67a6703.

adieyal commented 10 years ago

Thanks

I'm having another problem using the admin interface. Adding a twitter account I get the following error:

ActiveRecord::StatementInvalid in Admin::PoliticiansController#save_user

Mysql2::Error: Column 'twitter_id' cannot be null: INSERT INTO `politicians` (`account_type_id`, `avatar_content_type`, `avatar_file_name`, `avatar_file_size`, `avatar_updated_at`, `first_name`, `gender`, `last_name`, `middle_name`, `office_id`, `party_id`, `profile_image_url`, `state`, `status`, `suffix`, `twitter_id`, `user_name`) VALUES (NULL, NULL, NULL, NULL, NULL, NULL, 'U', NULL, NULL, NULL, NULL, NULL, NULL, 1, NULL, NULL, 'helenzille')
Rails.root: /politwoops

Application Trace | Framework Trace | Full Trace
app/controllers/admin/politicians_controller.rb:47:in `save_user'
Request

Parameters:

{"twitter_id"=>"",
 "user_name"=>"helenzille",
 "first_name"=>"Helen",
 "middle_name"=>"",
 "last_name"=>"Zille",
 "suffix"=>"",
 "party_id"=>"1",
 "status"=>"1",
 "state"=>"",
 "account_type_id"=>"1",
 "office_id"=>"1",
 "id"=>"0"}
Show session dump

Show env dump

Response

Headers:

None
adieyal commented 10 years ago

Sorry - I forgot to change the api keys

adieyal commented 10 years ago

This error seems to keep on recurring but it seems sporadically

dvogel commented 10 years ago

What's happening is that your browser is failing to look up the numeric ID for the twitter user name you've specified. In commit 26a44edaaad9b027ec97ddb1cab8ae8f7a99da94 I added an explicit error to make this more clear.

I don't have enough info to why the twitter ID could not be found. There's likely an error logged to your browser's developer console.

adieyal commented 10 years ago

Thanks

Just checked my developer's console and there are no errors logged.

Adi

On 10 March 2014 19:41, dvogel notifications@github.com wrote:

What's happening is that your browser is failing to look up the numeric ID for the twitter user name you've specified. In commit 26a44edhttps://github.com/sunlightlabs/politwoops/commit/26a44edaaad9b027ec97ddb1cab8ae8f7a99da94I added an explicit error to make this more clear.

I don't have enough info to why the twitter ID could not be found. There's likely an error logged to your browser's developer console.

Reply to this email directly or view it on GitHubhttps://github.com/sunlightlabs/politwoops/issues/15#issuecomment-37211275 .

Adi Eyal Data Specialist phone: +27 78 014 2469 skype: adieyalcas linkedin: http://za.linkedin.com/pub/dir/Adi/Eyal

dvogel commented 10 years ago

Do you see a network request to twitter after the twitter username field loses focus? For example, if you enter drewpvogel into the username and move the cursor to the next field you should see a request to drewpvogel.json.

adieyal commented 10 years ago

I think what might be happening is that the ajax call is taking a long time to complete. If I save the new user before the call completes then it doesn't receive the id.

After waiting sufficient time, I saved it again and received this error


Twitter::Error::ClientError in Admin::PoliticiansController#save_user

execution expired

Rails.root: /politwoops

Application Trace | Framework Trace | Full Trace

app/models/politician.rb:124:in `reset_avatar'
app/controllers/admin/politicians_controller.rb:83:in `save_user'

Request

Parameters:

{"twitter_id"=>"21011010",
 "user_name"=>"soapsudtycoon",
 "first_name"=>"Adi",
 "middle_name"=>"",
 "last_name"=>"Eyal",
 "suffix"=>"",
 "party_id"=>"1",
 "status"=>"1",
 "state"=>"",
 "account_type_id"=>"0",
 "office_id"=>"0",
 "id"=>"0"}

Show session dump

Show env dump

Response

Headers:

None

On 10 March 2014 23:43, dvogel notifications@github.com wrote:

Do you see a network request to twitter after the twitter username field loses focus? For example, if you enter drewpvogel into the username and move the cursor to the next field you should see a request to drewpvogel.json.

Reply to this email directly or view it on GitHub.

Adi Eyal Data Specialist phone: +27 78 014 2469 skype: adieyalcas linkedin: http://za.linkedin.com/pub/dir/Adi/Eyal

dvogel commented 10 years ago

I've added a UI change to the admin that should avoid the first failure. As of commit 3babf31fb18f20eb0a96c2ecec6814eb8f1181e4, the user form's save button will be disabled until the twitter ID is retrieved.

The execution expired error seems related to your deployment. Either the reset_avatar method timed out while contacting the Twitter API or your web server decided it was taking too long and killed it. Have you set up your twitter API key fields in config/config.yml?

adieyal commented 10 years ago

The execution expired error seems related to your deployment. Either the reset_avatar method timed out while contacting the Twitter API or your web server decided it was taking too long and killed it. Have you set up your twitter API key fields in config/config.yml?

I have set the twitter keys but I'll check again.

Adi

Reply to this email directly or view it on GitHubhttps://github.com/sunlightlabs/politwoops/issues/15#issuecomment-37332573 .

Adi Eyal Data Specialist phone: +27 78 014 2469 skype: adieyalcas linkedin: http://za.linkedin.com/pub/dir/Adi/Eyal