search

googleapis / google-cloud-ruby

Google Cloud Client Library for Ruby
https://googleapis.github.io/google-cloud-ruby/
Apache License 2.0
1.35k stars 543 forks source link

Wrong score from Natural Language Sentiment analysis #2062

Closed mariokam closed 6 years ago

mariokam commented 6 years ago

I am getting very strange error from the ruby client API. This is my code:

require "googleauth"
require "google/cloud/language"

class GoogleLanguageService

  def initialize(params)
    @text = params[:text]
  end

  def analyze_sentiment

    begin
    language_service_client = Google::Cloud::Language.new project: ENV["GOOGLE_CLOUD_PROJECT"],
                                                          keyfile: Rails.root.join('', 'google-cloud-auth.json')

    # Detects the sentiment of the text
    sentiment = language_service_client.sentiment content: @text, type: :PLAIN_TEXT

    # Here the score is different than the score in the public GOOGLE demo page :-) 
    # puts "Text: #{@text}"
    # puts "Score: #{sentiment.score}, #{sentiment.magnitude}"

    return sentiment
    rescue => error
      puts error
    end

  end

  private

  def encoded_access_credentials(username, password)
    Base64.strict_encode64(username+':'+password)
  end

end

and when I want to analyze the sentence: @sentiment = GoogleLanguageService.new({ text: "good luck on monday" }).analyze_sentiment

good luck on monday it returns score: -0.10000000149011612

Good luck on Monday! returns score: 0.0

while on the google demo page both sentences has positive score

quartzmo commented 6 years ago

Hi @mariokam, thanks for opening this issue! I suspect this behavior is determined by the service and is simply pass-through for this library, but I'll take a look at it today.

blowmage commented 6 years ago

I am getting very strange error from the ruby client API.

What error are you getting?

Also, what gem version are you using?

blowmage commented 6 years ago

FWIW, I saw the same results using the Try the API feature on the Natural Language page as I did using the Ruby client.

Good luck on Monday!

language-01 
require "google/cloud/language"

lang = Google::Cloud::Language.new

lang.analyze_sentiment content: "Good luck on Monday!", type: :PLAIN_TEXT
#=> <Google::Cloud::Language::V1::AnalyzeSentimentResponse: document_sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.5, score: 0.5>, language: "en", sentences: [<Google::Cloud::Language::V1::Sentence: text: <Google::Cloud::Language::V1::TextSpan: content: "Good luck on Monday!", begin_offset: -1>, sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.5, score: 0.5>>]>

good luck on monday

language-02 
require "google/cloud/language"

lang = Google::Cloud::Language.new

lang.analyze_sentiment content: "good luck on monday", type: :PLAIN_TEXT
#=> <Google::Cloud::Language::V1::AnalyzeSentimentResponse: document_sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.10000000149011612, score: 0.10000000149011612>, language: "en", sentences: [<Google::Cloud::Language::V1::Sentence: text: <Google::Cloud::Language::V1::TextSpan: content: "good luck on monday", begin_offset: -1>, sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.10000000149011612, score: 0.10000000149011612>>]>
mariokam commented 6 years ago

Dear gents,

as I am saying again, the problem is in the ruby wrapper, and yes I am using probably older version of it, but it cannot be a reason to not work properly. Sending you some logs of my results using this version 0.25 of the wrapper.

/Users/mario/.rvm/rubies/ruby-2.3.0/bin/ruby -e at_exit{sleep(1)};$stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /Users/mario/.rvm/gems/ruby-2.3.0/gems/ruby-debug-ide-0.6.1.beta2/bin/rdebug-ide --disable-int-handler --evaluation-timeout 10 --rubymine-protocol-extensions --port 64258 --host 0.0.0.0 --dispatcher-port 64259 -- /Users/mario/Documents/lovenotes/bin/rails server -b 0.0.0.0 -p 3000 -e development Fast Debugger (ruby-debug-ide 0.6.1.beta2, debase 0.2.2.beta8, file filtering is supported) listens on 0.0.0.0:64258 => Booting Thin => Rails 4.2.7.1 application starting in development on http://0.0.0.0:3000 => Run rails server -h for more startup options => Ctrl-C to shutdown server CONNECTED! Thin web server (v1.7.0 codename Dunder Mifflin) Maximum connections set to 1024 Listening on 0.0.0.0:3000, CTRL+C to stop

Started GET "/love_note" for 127.0.0.1 at 2018-04-23 20:37:25 +0200 ActiveRecord::SchemaMigration Load (13.3ms) SELECT "schema_migrations". FROM "schema_migrations" Processing by LoveNoteController#index as HTML User Load (19.3ms) SELECT "users". FROM "users" WHERE "users"."id" = $1 LIMIT 1 [["id", 4]] BlackList Load (2.0ms) SELECT "black_lists". FROM "black_lists" WHERE "black_lists"."user_id" = $1 [["user_id", 4]] (3.0ms) SELECT DISTINCT "love_notes"."sender_email" FROM "love_notes" WHERE "love_notes"."recipient_email" = $1 AND (sender_email IS NOT NULL) [["recipient_email", "mario@stanford.edu"]] LoveNote Load (0.7ms) SELECT MAX(id) as id, sender_email FROM "love_notes" WHERE 1=0 AND "love_notes"."recipient_email" = $1 GROUP BY "love_notes"."sender_email" [["recipient_email", "mario@stanford.edu"]] (1.0ms) SELECT DISTINCT "love_notes"."recipient_email" FROM "love_notes" WHERE "love_notes"."sender_email" = $1 AND (recipient_email IS NOT NULL) [["sender_email", "mario@stanford.edu"]] LoveNote Load (1.4ms) SELECT MAX(id) as id, recipient_email FROM "love_notes" WHERE "love_notes"."recipient_email" IN ('loveandgsb@gmail.com', 'mario.kam@email.cz', 'mario.kamburov@gmail.com') AND "love_notes"."sender_email" = $1 GROUP BY "love_notes"."recipient_email" [["sender_email", "mario@stanford.edu"]] (1.1ms) SELECT COUNT() FROM "love_notes" WHERE "love_notes"."id" IN (24, 44, 10) LoveNote Load (1.6ms) SELECT "love_notes". FROM "love_notes" WHERE "love_notes"."id" IN (24, 44, 10) ORDER BY created_at DESC (0.9ms) SELECT COUNT() FROM "love_notes" WHERE 1=0 LoveNote Load (1.1ms) SELECT "love_notes". FROM "love_notes" WHERE (recipient_email = 'mario@stanford.edu' ) LoveNote Load (1.3ms) SELECT "love_notes". FROM "love_notes" WHERE (sender_email = 'mario@stanford.edu' ) Redirected to http://localhost:3000/love_notes/stanford_users Completed 302 Found in 809ms (ActiveRecord: 74.9ms)

Started POST "/love_notes/stanford_users" for 127.0.0.1 at 2018-04-23 20:37:54 +0200 Processing by LoveNotes::StanfordUsersController#create as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"hYzhh+TkyUxK0xgFId98fyX4FLuy6i6YfqFyjUHta9RwtZktj4ACbJT/BhQkzlse/vLURKijLDkefCEUs9Kqcw==", "love_note"=>{"note_from"=>"4", "sender_email"=>"mario@stanford.edu", "note_from_email"=>"mario@stanford.edu", "hash_token"=>"782fecbe-4f77-48af-b7e9-a5b61968361b", "recipient_fullname"=>"", "google_contacts_autocomplete"=>"", "recipient_email"=>"mario.kam@email.cz", "stanford_user"=>"mario.kam@email.cz", "text"=>"good luck on monday", "share"=>"0"}} User Load (1.2ms) SELECT "users". FROM "users" WHERE "users"."id" = $1 LIMIT 1 [["id", 4]] BlackList Load (1.0ms) SELECT "black_lists". FROM "black_lists" WHERE "black_lists"."user_id" = $1 [["user_id", 4]] (1.0ms) SELECT DISTINCT "love_notes"."sender_email" FROM "love_notes" WHERE "love_notes"."recipient_email" = $1 AND (sender_email IS NOT NULL) [["recipient_email", "mario@stanford.edu"]] LoveNote Load (1.1ms) SELECT MAX(id) as id, sender_email FROM "love_notes" WHERE 1=0 AND "love_notes"."recipient_email" = $1 GROUP BY "love_notes"."sender_email" [["recipient_email", "mario@stanford.edu"]] (1.3ms) SELECT DISTINCT "love_notes"."recipient_email" FROM "love_notes" WHERE "love_notes"."sender_email" = $1 AND (recipient_email IS NOT NULL) [["sender_email", "mario@stanford.edu"]] LoveNote Load (1.3ms) SELECT MAX(id) as id, recipient_email FROM "love_notes" WHERE "love_notes"."recipient_email" IN ('loveandgsb@gmail.com', 'mario.kam@email.cz', 'mario.kamburov@gmail.com') AND "love_notes"."sender_email" = $1 GROUP BY "love_notes"."recipient_email" [["sender_email", "mario@stanford.edu"]] (1.5ms) SELECT COUNT() FROM "love_notes" WHERE "love_notes"."id" IN (24, 44, 10) LoveNote Load (1.2ms) SELECT "love_notes". FROM "love_notes" WHERE "love_notes"."id" IN (24, 44, 10) ORDER BY created_at DESC (0.9ms) SELECT COUNT() FROM "love_notes" WHERE 1=0 LoveNote Load (1.5ms) SELECT "love_notes". FROM "love_notes" WHERE (recipient_email = 'mario@stanford.edu' ) LoveNote Load (1.2ms) SELECT "love_notes".* FROM "love_notes" WHERE (sender_email = 'mario@stanford.edu' ) (0.7ms) BEGIN

PROJECT_ID: turnkey-aleph-193823 GOOGLE_CLOUD_AUTH_JSON_PATH: /Users/mario/Documents/lovenotes/google-cloud-auth.json

0> language_service_client.inspect => "#<Google::Cloud::Language::Project:0x007f8b8595ab10 @service=Google::Cloud::Language::Service(turnkey-aleph-193823)>"

<Google::Cloud::Language::Project:0x007f8b8595ab10 @service=Google::Cloud::Language::Service(turnkey-aleph-193823)>

0> sentiment => #<Google::Cloud::Language::Annotation::Sentiment:0x007f8b80733cb0 @score=-0.10000000149011612, @magnitude=0.10000000149011612, @sentences=[#<Google::Cloud::Language::Annotation::Sentence:0x007f8b80733d50 @text_span=#<Google::Cloud::Language::Annotation::TextSpan:0x007f8b80733e68 @text="{:content=>\"good luck on monday\", :type=>:PLAIN_TEXT}", @offset=0>, @sentiment=#<Google::Cloud::Language::Annotation::Sentence::Sentiment:0x007f8b80733e40 @score=-0.10000000149011612, @magnitude=0.10000000149011612>>], @language="en">

0> sentiment.score => -0.10000000149011612

using the following versions from my Gemfile.lock google-cloud-core (0.21.1) googleauth (~> 0.5.1) google-cloud-language (0.25.0) google-cloud-core (~> 0.21.0) google-gax (~> 0.8.0) google-gax (0.8.12) google-protobuf (~> 3.2) googleapis-common-protos (~> 1.3.5) googleauth (~> 0.5.1) grpc (~> 1.6.6) rly (~> 0.2.3) google-protobuf (3.5.1.2) googleapis-common-protos (1.3.7) google-protobuf (~> 3.0) googleapis-common-protos-types (~> 1.0) grpc (~> 1.0) googleapis-common-protos-types (1.0.1) google-protobuf (~> 3.0) googleauth (0.5.1) faraday (~> 0.9) jwt (~> 1.4) logging (~> 2.0) memoist (~> 0.12) multi_json (~> 1.11) os (~> 0.9) signet (~> 0.7)

Thanks!

On 23 Apr 2018, at 19:08, Mike Moore notifications@github.com wrote:

FWIW, I saw the same results using the Try the API https://cloud.google.com/natural-language/ feature on the Natural Language page as I did using the Ruby client.

Good luck on Monday!

https://user-images.githubusercontent.com/730/39141883-4084c90a-46e6-11e8-81b1-afde01546420.png require "google/cloud/language"

lang = Google::Cloud::Language.new

lang.analyze_sentiment content: "Good luck on Monday!", type: :PLAIN_TEXT

=> <Google::Cloud::Language::V1::AnalyzeSentimentResponse: document_sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.5, score: 0.5>, language: "en", sentences: [<Google::Cloud::Language::V1::Sentence: text: <Google::Cloud::Language::V1::TextSpan: content: "Good luck on Monday!", begin_offset: -1>, sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.5, score: 0.5>>]>

good luck on monday

https://user-images.githubusercontent.com/730/39141948-778d968e-46e6-11e8-82cc-8c92f2e1a19d.png require "google/cloud/language"

lang = Google::Cloud::Language.new

lang.analyze_sentiment content: "good luck on monday", type: :PLAIN_TEXT

=> <Google::Cloud::Language::V1::AnalyzeSentimentResponse: document_sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.10000000149011612, score: 0.10000000149011612>, language: "en", sentences: [<Google::Cloud::Language::V1::Sentence: text: <Google::Cloud::Language::V1::TextSpan: content: "good luck on monday", begin_offset: -1>, sentiment: <Google::Cloud::Language::V1::Sentiment: magnitude: 0.10000000149011612, score: 0.10000000149011612>>]>

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2062#issuecomment-383650269, or mute the thread https://github.com/notifications/unsubscribe-auth/ABynu_dWRYvS-cHI6HTC7qr2lf0-HELcks5trgqNgaJpZM4Te0th.

quartzmo commented 6 years ago

@mariokam Is it possible for you to update to the current version?

blowmage commented 6 years ago

If you are using an older version then you should use the API provided by that version. When using the 0.25.0 API I still get the sentiment scores that the page returns:

require "google/cloud/language"

lang = Google::Cloud::Language.new

lang.sentiment "Good luck on Monday!", format: :text
#=> #<Google::Cloud::Language::Annotation::Sentiment:0x00007fd37d139d88 @score=0.5, @magnitude=0.5, @sentences=[#<Google::Cloud::Language::Annotation::Sentence:0x00007fd37d139ea0 @text_span=#<Google::Cloud::Language::Annotation::TextSpan:0x00007fd37d13a080 @text="Good luck on Monday!", @offset=0>, @sentiment=#<Google::Cloud::Language::Annotation::Sentence::Sentiment:0x00007fd37d13a008 @score=0.5, @magnitude=0.5>>], @language="en">

lang.sentiment "good luck on monday", format: :text
#=> #<Google::Cloud::Language::Annotation::Sentiment:0x00007fd37d827820 @score=0.10000000149011612, @magnitude=0.10000000149011612, @sentences=[#<Google::Cloud::Language::Annotation::Sentence:0x00007fd37d827898 @text_span=#<Google::Cloud::Language::Annotation::TextSpan:0x00007fd37d8278e8 @text="good luck on monday", @offset=0>, @sentiment=#<Google::Cloud::Language::Annotation::Sentence::Sentiment:0x00007fd37d8278c0 @score=0.10000000149011612, @magnitude=0.10000000149011612>>], @language="en">
irb(main):006:0>

Then biggest difference is that the content value is positional argument and not an optional named argument.

mariokam commented 6 years ago

I was trying to do simple gem install, even tried to specift 0.30, but wasn’t able to download it. My gemfile by default downloads this version.

You don’t need to worry, I have already reimplemented it via curb - easy curl library instead of the ruby wrapper and all works as expected.

On 23 Apr 2018, at 23:14, Chris Smith notifications@github.com wrote:

@mariokam https://github.com/mariokam Is it possible for you to update to the current version?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2062#issuecomment-383725308, or mute the thread https://github.com/notifications/unsubscribe-auth/ABynu7_zux0EMvfOAstxNQUwXGVDawQHks5trkQhgaJpZM4Te0th.

mariokam commented 6 years ago

ok, thank you guys!

good to know that :-) Maybe could be included in the documentation

On 23 Apr 2018, at 23:23, Mike Moore notifications@github.com wrote:

If you are using an older version then you should use the API provided by that version https://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-language/v0.25.0/google/cloud/language/project?method=sentiment-instance. When using the 0.25.0 API I still get the sentiment scores that the page returns:

require "google/cloud/language"

lang = Google::Cloud::Language.new

lang.sentiment "Good luck on Monday!", format: :text

=> #<Google::Cloud::Language::Annotation::Sentiment:0x00007fd37d139d88 @score=0.5, @magnitude=0.5, @sentences=[#<Google::Cloud::Language::Annotation::Sentence:0x00007fd37d139ea0 @text_span=#<Google::Cloud::Language::Annotation::TextSpan:0x00007fd37d13a080 @text="Good luck on Monday!", @offset=0>, @sentiment=#<Google::Cloud::Language::Annotation::Sentence::Sentiment:0x00007fd37d13a008 @score=0.5, @magnitude=0.5>>], @language="en">

lang.sentiment "good luck on monday", format: :text

=> #<Google::Cloud::Language::Annotation::Sentiment:0x00007fd37d827820 @score=0.10000000149011612, @magnitude=0.10000000149011612, @sentences=[#<Google::Cloud::Language::Annotation::Sentence:0x00007fd37d827898 @text_span=#<Google::Cloud::Language::Annotation::TextSpan:0x00007fd37d8278e8 @text="good luck on monday", @offset=0>, @sentiment=#<Google::Cloud::Language::Annotation::Sentence::Sentiment:0x00007fd37d8278c0 @score=0.10000000149011612, @magnitude=0.10000000149011612>>], @language="en">

irb(main):006:0> Then biggest difference is that the content value is positional argument and not an optional named argument.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2062#issuecomment-383727712, or mute the thread https://github.com/notifications/unsubscribe-auth/ABynu1V5gFvtro7ZzrdjkAEDVhgNfH8Zks5trkY0gaJpZM4Te0th.

mariokam commented 6 years ago

Guys, how is this possible?? thanks

On 23 Apr 2018, at 23:46, Chris Smith notifications@github.com wrote:

Closed #2062 https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2062.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2062#event-1590239546, or mute the thread https://github.com/notifications/unsubscribe-auth/ABynu8veszbmTnBfWRk54hpvoqtjTSKmks5trku7gaJpZM4Te0th.

quartzmo commented 6 years ago

I'm sorry, I was mistaken in closing this issue.

Guys, how is this possible??

Can you please provide more detail for this question?

blowmage commented 6 years ago

The API had a breaking change in 0.28.0. If you use a pre-0.28.0 gem version you must use the pre-0.28.0 API, which is documented. @mariokam what specific changes are you asking for?