Google not returning results

[bmarsh9]

After installing scumblr, I set up a google developer key and cx. When I run a search, it completes with a success but there are no results. Sidekiq.log does not show anything valuable either, and no errors. Troubleshooting: Version 2.1

rails c

Loading development environment (Rails 4.2.6) [1] pry(main)> SearchProvider::Google.new("scumblr").run NoMethodError: undefined method metadata' for nil:NilClass from /home/monitor/Scumblr/lib/search_providers/provider.rb:43:ininitialize' [2] pry(main)>

Logs from sidekiq.log

2017-11-10T00:20:24.387Z 111771 TID-24pogkmc TaskRunner JID-b82d310c506d57dbbcc8dcfb WARN: Running google 2017-11-10T00:20:24.389Z 111771 TID-24pogj3c TaskWorker JID-b146734f772e3cb59a781ed5 INFO: start 2017-11-10T00:20:24.393Z 111771 TID-24pogkmc TaskRunner JID-b82d310c506d57dbbcc8dcfb WARN: 1 tasks remaining 2017-11-10T00:20:24.393Z 111771 TID-24pogkmc TaskRunner JID-b82d310c506d57dbbcc8dcfb WARN: Task b146734f772e3cb59a781ed5 working 2017-11-10T00:20:24.581Z 111771 TID-24pogj3c TaskWorker JID-b146734f772e3cb59a781ed5 WARN: Response received # 2017-11-10T00:20:24.669Z 111771 TID-24pogj3c TaskWorker JID-b146734f772e3cb59a781ed5 INFO: done: 0.279 sec 2017-11-10T00:20:26.396Z 111771 TID-24pogkmc TaskRunner JID-b82d310c506d57dbbcc8dcfb WARN: 1 tasks remaining 2017-11-10T00:20:26.397Z 111771 TID-24pogkmc TaskRunner JID-b82d310c506d57dbbcc8dcfb WARN: Task b146734f772e3cb59a781ed5 complete 2017-11-10T00:20:28.397Z 111771 TID-24pogkmc TaskRunner JID-b82d310c506d57dbbcc8dcfb INFO: done: 4.017 sec

Additionally, there are no errors when restarting scumblr. When I go to my cx ID custom search page and type in a query in the search bar, I get a result immediately.

Thanks for any future help

[sbehrens]

If you can double check the task id, would you mind running the following from the rails console:

Task.find(<task_id>).perform_task

If you see the same error, could you add the following to line 42 in /home/monitor/Scumblr/lib/search_providers/provider.rb

puts @options[:_self]

And re-run the task from the console, pasting the output of that puts on this ticket? Thanks!

-Scott

[bmarsh9]

Thanks, Here is the output from the console after adding that line:

Loading development environment (Rails 4.2.6) [1] pry(main)> Task.find(5).perform_task Task Load (0.4ms) SELECT "tasks".* FROM "tasks" WHERE "tasks"."id" = $1 LIMIT 1 [["id", 5]] Running Task 5 (0.2ms) BEGIN SQL (0.6ms) INSERT INTO "events" ("action", "source", "details", "eventable_type", "eventable_id", "date", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" [["action", "Error"], ["source", "Task: sample"], ["details", "Unable to run task sample.\n\nError: undefined method []=' for nil:NilClass\n\n[\"/home/monitor/Scumblr/lib/search_providers/provider.rb:42:ininitialize'\", \"/home/monitor/Scumblr/lib/search_providers/google.rb:73:in initialize'\", \"/home/monitor/Scumblr/app/models/task.rb:232:innew'\", \"/home/monitor/Scumblr/app/models/task.rb:232:in perform_task'\", \"(pry):1:inpry'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:355:in eval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:355:inevaluate_ruby'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:323:in handle_line'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:243:inblock (2 levels) in eval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:242:in catch'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:242:inblock in eval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:241:in catch'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:241:ineval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:77:in block in repl'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:67:inloop'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:67:in repl'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:38:inblock in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/input_lock.rb:61:in __with_ownership'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/input_lock.rb:79:inwith_ownership'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:38:in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:15:instart'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_class.rb:169:in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:instart'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:inconsole'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in run_command!'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in<top (required)>'\", \"bin/rails:4:in require'\", \"bin/rails:4:in

'\"]"], ["eventable_type", "Task"], ["eventable_id", 5], ["date", "2017-11-13 17:22:01.500174"], ["created_at", "2017-11-13 17:22:01.500353"], ["updated_at", "2017-11-13 17:22:01.500353"]] (6.7ms) COMMIT undefined method `[]=' for nil:NilClass (0.2ms) BEGIN Task Exists (0.5ms) SELECT 1 AS one FROM "tasks" WHERE ("tasks"."name" = 'sample' AND "tasks"."id" != 5) LIMIT 1 SQL (0.3ms) UPDATE "tasks" SET "metadata" = $1, "updated_at" = $2 WHERE "tasks"."id" = $3 [["metadata", "{\"_last_run\":\"2017-11-13T17:22:01.511+00:00\",\"_start_time\":\"2017-11-13T17:22:01.471+00:00\",\"_last_status\":\"Failed\",\"current_events\":{\"Error\":[110]},\"_last_status_event\":111,\"_last_status_message\":\"Unable to run task sample.\n\nError: undefined me...\",\"_last_successful_run\":\"2017-11-13T17:16:57.614+00:00\"}"], ["updated_at", "2017-11-13 17:22:01.569383"], ["id", 5]] (3.9ms) COMMIT (0.2ms) BEGIN Task Exists (0.3ms) SELECT 1 AS one FROM "tasks" WHERE ("tasks"."name" = 'sample' AND "tasks"."id" != 5) LIMIT 1 SQL (0.4ms) UPDATE "tasks" SET "metadata" = $1, "updated_at" = $2 WHERE "tasks"."id" = $3 [["metadata", "{\"_last_run\":\"2017-11-13T17:22:01.511+00:00\",\"_start_time\":\"2017-11-13T17:22:01.471+00:00\",\"_last_status\":\"Failed\",\"current_events\":{\"Error\":[111]},\"_last_status_event\":111,\"_last_status_message\":\"Unable to run task sample.\n\nError: undefined me...\",\"_last_successful_run\":\"2017-11-13T17:16:57.614+00:00\"}"], ["updated_at", "2017-11-13 17:22:01.599312"], ["id", 5]] (4.4ms) COMMIT No results returned => nil

[sbehrens]

Would you mind adding some newlines? i'm not seeing the puts statement in that stacktrace:

puts "\n\n\n"
puts @options[:_self]
puts "\n\n\n"

Thanks

[bmarsh9]

Sure, here is the output, file looks like this btw: puts "\n\n\n" puts @options[:_self] = options puts "\n\n\n"

Loading development environment (Rails 4.2.6) [1] pry(main)> Task.find(5).perform_task Task Load (0.4ms) SELECT "tasks".* FROM "tasks" WHERE "tasks"."id" = $1 LIMIT 1 [["id", 5]] Running Task 5

(0.2ms) BEGIN SQL (0.6ms) INSERT INTO "events" ("action", "source", "details", "eventable_type", "eventable_id", "date", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" [["action", "Error"], ["source", "Task: sample"], ["details", "Unable to run task sample.\n\nError: undefined method []=' for nil:NilClass\n\n[\"/home/monitor/Scumblr/lib/search_providers/provider.rb:43:ininitialize'\", \"/home/monitor/Scumblr/lib/search_providers/google.rb:73:in initialize'\", \"/home/monitor/Scumblr/app/models/task.rb:232:innew'\", \"/home/monitor/Scumblr/app/models/task.rb:232:in perform_task'\", \"(pry):1:inpry'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:355:in eval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:355:inevaluate_ruby'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:323:in handle_line'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:243:inblock (2 levels) in eval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:242:in catch'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:242:inblock in eval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:241:in catch'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_instance.rb:241:ineval'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:77:in block in repl'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:67:inloop'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:67:in repl'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:38:inblock in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/input_lock.rb:61:in __with_ownership'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/input_lock.rb:79:inwith_ownership'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:38:in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/repl.rb:15:instart'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/pry-0.10.1/lib/pry/pry_class.rb:169:in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:instart'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in start'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:inconsole'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in run_command!'\", \"/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in<top (required)>'\", \"bin/rails:4:in require'\", \"bin/rails:4:in

'\"]"], ["eventable_type", "Task"], ["eventable_id", 5], ["date", "2017-11-13 19:25:25.463264"], ["created_at", "2017-11-13 19:25:25.463414"], ["updated_at", "2017-11-13 19:25:25.463414"]] (11.6ms) COMMIT undefined method `[]=' for nil:NilClass (0.2ms) BEGIN Task Exists (0.6ms) SELECT 1 AS one FROM "tasks" WHERE ("tasks"."name" = 'sample' AND "tasks"."id" != 5) LIMIT 1 SQL (0.4ms) UPDATE "tasks" SET "metadata" = $1, "updated_at" = $2 WHERE "tasks"."id" = $3 [["metadata", "{\"_last_run\":\"2017-11-13T19:25:25.479+00:00\",\"_start_time\":\"2017-11-13T19:25:25.445+00:00\",\"_last_status\":\"Failed\",\"current_events\":{\"Error\":[111]},\"_last_status_event\":171,\"_last_status_message\":\"Unable to run task sample.\n\nError: undefined me...\",\"_last_successful_run\":\"2017-11-13T17:16:57.614+00:00\"}"], ["updated_at", "2017-11-13 19:25:25.513594"], ["id", 5]] (4.7ms) COMMIT (0.2ms) BEGIN Task Exists (0.3ms) SELECT 1 AS one FROM "tasks" WHERE ("tasks"."name" = 'sample' AND "tasks"."id" != 5) LIMIT 1 SQL (0.5ms) UPDATE "tasks" SET "metadata" = $1, "updated_at" = $2 WHERE "tasks"."id" = $3 [["metadata", "{\"_last_run\":\"2017-11-13T19:25:25.479+00:00\",\"_start_time\":\"2017-11-13T19:25:25.445+00:00\",\"_last_status\":\"Failed\",\"current_events\":{\"Error\":[171]},\"_last_status_event\":171,\"_last_status_message\":\"Unable to run task sample.\n\nError: undefined me...\",\"_last_successful_run\":\"2017-11-13T17:16:57.614+00:00\"}"], ["updated_at", "2017-11-13 19:25:25.552419"], ["id", 5]] (3.6ms) COMMIT No results returned => nil

[sbehrens]

I'm still not seeing the output of those puts (I don't see any new lines), can you place them a bit earlier in the file?

Thanks! -Scott

[Jilten]

I'm also experiencing the same issue. However, when I start sidekiq, I get the following warnings

/home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.1/lib/readline.rb:458: warning: already initialized constant Readline::HISTORY /home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.1/lib/readline.rb:486: warning: already initialized constant Readline::FILENAME_COMPLETION_PROC /home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.1/lib/readline.rb:517: warning: already initialized constant Readline::USERNAME_COMPLETION_PROC /home/monitor/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.1/lib/readline.rb:523: warning: already initialized constant Readline::VERSION

[sbehrens]

I'm unfortunately unable to reproduce the bug on my side. A few things could be helpful for troubleshooting.

Can you send a screenshot of your Task config? I'd like to double check your settings. In addition, make sure the 4 task options have green checkmarks in the Task Configuration setting like the screenshot below:

If you do not see the checkmarks, your task isn't configured correctly.

[bmarsh9]

Here is my output for the "New Task" section:

Keys:

The Error for google:

Thanks

[bmarsh9]

And here are the Puts around line 42, and added some more as requested

[bmarsh9]

So I have completely reinstalled everything. Twitter search is working, google search says that it ran and returned with "Complete", but there are no results.

[sk3tch]

@bm1391 would you be willing to share your search term? If not the one you're using could you switch to something that should obviously return results and post that so we can debug on our side?

[bmarsh9]

Sure, I'm using the term "nfl". Thought this one would return lots of results, but nothing shows in the "Results" tab.

[sk3tch]

@bm1391 that certainly should return something. In /lib/search_providers/google.rb could you change line 117 from Rails.logger.warn "Response received #{response}" to Rails.logger.warn "Response received #{response.inspect}"

Then if you tail the logs while it's running you should see a a WARN message that starts "Response received" with a bunch of information. Look through that and see if it gives you a hint. Feel free to paste that log here, but please first look through the log and redact anything sensitive.

[bmarsh9]

Ahh, Thank you sir, It was complaining that the API was created, but not enabled. I don't remember seeing that part in the documentation but I probably missed it. Here is the webpage that it directed me to:

https://console.developers.google.com/apis/library/customsearch.googleapis.com/?project=YOURPROJECTNAME

Results are flowing in now. Thank you for the help and the tool! I'll be sure to document this for my next deployment.