search

ookamiinc / twitter-analytics-acquirer

0 stars 0 forks source link

11/29の定期実行時のエラーの原因を調査する #30

Open yusuko opened 5 years ago

yusuko commented 5 years ago

https://github.com/ookamiinc/general/issues/4824#issuecomment-442713678 より 11/29の定期実行にてエラーが発生した。 本issueにて原因を調査する 本issueにて対応する。

yusuko commented 5 years ago

log

Nov 29 09:30:07 twitter-data-acquirer app/api:  Starting process with command `bundle exec ruby script.rb` by user scheduler@addons.heroku.com 
Nov 29 09:30:10 twitter-data-acquirer heroku/scheduler.6868:  Starting process with command `bundle exec ruby script.rb` 
Nov 29 09:30:10 twitter-data-acquirer heroku/scheduler.6868:  State changed from starting to up 
Nov 29 09:30:27 twitter-data-acquirer app/scheduler.6868:  Cookie#domain returns dot-less domain name now. Use Cookie#dot_domain if you need "." at the beginning. 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  /app/google_sheet_client.rb:31:in `rescue in write_in_spreadsheet': エラーが起きました！！！ (RuntimeError) 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:   from /app/google_sheet_client.rb:17:in `write_in_spreadsheet' 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:   from script.rb:16:in `block in <main>' 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:   from script.rb:10:in `each' 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:   from script.rb:10:in `<main>' 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  nil!! 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  number_of_loop: 2 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  success! @Playerapp_vb at 1 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  All success! @Playerapp_vb 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  nil!! 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  nil!! 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  nil!! 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  nil!! 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  nil!! 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  number_of_loop: 6 
Nov 29 09:31:34 twitter-data-acquirer app/scheduler.6868:  D, [2018-11-29T00:31:34.497326 #4] DEBUG -- : #<Google::Apis::ServerError: Server error> 
Nov 29 09:31:34 twitter-data-acquirer heroku/scheduler.6868:  Process exited with status 1 
Nov 29 09:31:35 twitter-data-acquirer heroku/scheduler.6868:  State changed from up to complete 
Nov 29 21:46:33 twitter-data-acquirer app/api: Build started by user
Nov 29 21:46:42 twitter-data-acquirer app/api: Deploy 3ecabe0a by user
Nov 29 21:46:42 twitter-data-acquirer app/api: Release v88 created by user Nov 29 21:46:46 twitter-data-acquirer app/api: Build succeeded
yusuko commented 5 years ago 
unhandled response (Mechanize::ResponseCodeError)

２種類あるのか?

yusuko commented 5 years ago 
#<Mechanize:0x00000000036ca798 @agent=#<Mechanize::HTTP::Agent:0x00000000036ca748 @allowed_error_codes=[], @conditional_requests=true, @context=#<Mechanize:0x00000000036ca798 ...>, @content_encoding_hooks=[], @cookie_jar=#<Mechanize::CookieJar:0x00000000036ca6d0 @store=#<HTTP::CookieJar::HashStore:0x00000000036ca5e0 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00000000036ca590>, @logger=nil, @gc_threshold=150, @jar={"twitter.com"=>{"/"=>{"_twitter_sess"=>#<HTTP::Cookie:name="_twitter_sess", value="BAh7CiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCKAXvppmAToMY3NyZl9p%250AZCIlYjU1OTM2NGEzODc2ZjNjNTU0NTJhNzFlMzY4MTZmMWI6B2lkIiUwNzlh%250AZTNiMmU0NjIyZGQzMWU3Y2NmMzFjNjBlYzRlZDoJdXNlcmwrB342AJw%253D--59968939f2a8d025bb99a2610011a8cbd34825f6", domain="twitter.com", for_domain=true, path="/", secure=true, httponly=true, expires=nil, max_age=nil, created_at=2018-10-22 07:47:22 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "personalization_id"=>#<HTTP::Cookie:name="personalization_id", value="v1_pMllvnM6tBBB4zGZRVEJRQ==", domain="twitter.com", for_domain=true, path="/", secure=false, httponly=false, expires=2020-10-21 07:47:21 +0000, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "guest_id"=>#<HTTP::Cookie:name="guest_id", value="v1%3A154019444111871733", domain="twitter.com", for_domain=true, path="/", secure=false, httponly=false, expires=2020-10-21 07:47:21 +0000, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "ads_prefs"=>#<HTTP::Cookie:name="ads_prefs", value="HBIRAAA=", domain="twitter.com", for_domain=true, path="/", secure=false, httponly=false, expires=2028-10-19 07:47:21 +0000, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "kdt"=>#<HTTP::Cookie:name="kdt", value="XSXrbU2ql6SgQu2mSEt9Oqo95jQiqztWcUILOt1G", domain="twitter.com", for_domain=true, path="/", secure=true, httponly=true, expires=2020-04-21 07:47:21 +0000, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "remember_checked_on"=>#<HTTP::Cookie:name="remember_checked_on", value="0", domain="twitter.com", for_domain=true, path="/", secure=false, httponly=false, expires=2028-10-19 07:47:21 +0000, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "twid"=>#<HTTP::Cookie:name="twid", value="u=2617259646", domain="twitter.com", for_domain=true, path="/", secure=true, httponly=false, expires=nil, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "auth_token"=>#<HTTP::Cookie:name="auth_token", value="2e303beaf827caa25ded9c1fe2911c91d2544931", domain="twitter.com", for_domain=true, path="/", secure=true, httponly=true, expires=nil, max_age=nil, created_at=2018-10-22 07:47:21 +0000, accessed_at=2018-12-01 09:17:48 +0000 origin=nil>, "lang"=>#<HTTP::Cookie:name="lang", value="ja", domain="twitter.com", for_domain=false, path="/", secure=false, httponly=false, expires=nil, max_age=nil, created_at=2018-10-22 07:47:22 +0000, accessed_at=2018-10-22 07:47:22 +0000 origin=nil>}}}, @gc_index=10>>, @follow_meta_refresh=false, @follow_meta_refresh_self=false, @gzip_enabled=true, @history=[https://analytics.twitter.com/user/player_twi/tweets/export.json?start_time=1543622400000&end_time=1543655857000&lang=ja, https://analytics.twitter.com/user/player_twi/tweets/bundle?start_time=1543622400000&end_time=1543655857000&lang=ja, https://analytics.twitter.com/user/player_twi/tweets/export.json?start_time=1543622400000&end_time=1543655857000&lang=ja, https://analytics.twitter.com/user/player_twi/tweets/bundle?start_time=1543622400000&end_time=1543655857000&lang=ja, https://analytics.twitter.com/user/player_twi/tweets/export.json?start_time=1543622400000&end_time=1543655857000&lang=ja], @ignore_bad_chunking=false, @keep_alive=true, @max_file_buffer=100000, @open_timeout=nil, @post_connect_hooks=[], @pre_connect_hooks=[], @read_timeout=nil, @redirect_ok=true, @redirection_limit=20, @request_headers={}, @robots=false, @robots_mutex=#<Thread::Mutex:0x00000000036ca400>, @user_agent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030401", @webrobots=nil, @auth_store=#<Mechanize::HTTP::AuthStore:0x00000000036ca3d8 @auth_accounts={}, @default_auth=nil>, @authenticate_parser=#<Mechanize::HTTP::WWWAuthenticateParser:0x00000000036ca338 @scanner=nil>, @authenticate_methods={#<URI::HTTPS https://analytics.twitter.com/>=>{:digest=>[], :iis_digest=>[], :basic=>[]}}, @digest_auth=#<Net::HTTP::DigestAuth:0x00000000036ca298 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00000000036ca270>, @nonce_count=-1>, @digest_challenges={}, @pass=nil, @scheme_handlers={"http"=>#<Proc:0x00000000036ca1d0@/app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:181 (lambda)>, "https"=>#<Proc:0x00000000036ca1d0@/app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:181 (lambda)>, "relative"=>#<Proc:0x00000000036ca1d0@/app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:181 (lambda)>, "file"=>#<Proc:0x00000000036ca1d0@/app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:181 (lambda)>}, @http=#<Net::HTTP::Persistent:0x00000000036ca180 @name="mechanize", @debug_output=nil, @proxy_uri=nil, @no_proxy=[], @headers={}, @override_headers={}, @http_versions={"analytics.twitter.com:443"=>"1.1"}, @keep_alive=300, @open_timeout=nil, @read_timeout=nil, @idle_timeout=5, @max_requests=nil, @socket_options=[[6, 1, 1]], @ssl_generation=1, @pool=#<Net::HTTP::Persistent::Pool:0x00000000036ca018 @size=2500, @timeout=5, @available=#<Net::HTTP::Persistent::TimedStackMulti:0x00000000036c9cd0 @create_block=#<Proc:0x00000000036c9fc8@/app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:525>, @created=1, @que=[], @max=2500, @mutex=#<Thread::Mutex:0x00000000036c9c80>, @resource=#<Thread::ConditionVariable:0x00000000036c9c58>, @shutdown_block=nil, @enqueued=1, @ques={["analytics.twitter.com", 443]=>[#<Net::HTTP::Persistent::Connection:0x00000000031265d0 @http=#<Net::HTTP analytics.twitter.com:443 open=true>, @ssl_generation=1, @last_use=2018-12-01 09:17:48 +0000, @requests=2>]}, @lru={["analytics.twitter.com", 443]=>true}, @key=:"connection_args-28724840">, @key=:"current-28724840", @key_count=:"current-28725140-count">, @certificate=nil, @ca_file=nil, @ca_path=nil, @ciphers=nil, @private_key=nil, @ssl_timeout=nil, @ssl_version=nil, @verify_callback=nil, @verify_depth=nil, @verify_mode=1, @cert_store=nil, @generation=1, @reuse_ssl_sessions=true, @retry_change_requests=false>>, @log=nil, @watch_for_set=nil, @history_added=nil, @pluggable_parser=#<Mechanize::PluggableParser:0x00000000036c9988 @parsers={"text/html"=>Mechanize::Page, "application/xhtml+xml"=>Mechanize::Page, "application/vnd.wap.xhtml+xml"=>Mechanize::Page, "image"=>Mechanize::Image, "text/xml"=>Mechanize::XmlFile, "application/xml"=>Mechanize::XmlFile}, @default=Mechanize::File>, @keep_alive_time=0, @proxy_addr=nil, @proxy_port=nil, @proxy_user=nil, @proxy_pass=nil, @html_parser=Nokogiri::HTML, @default_encoding=nil, @force_default_encoding=false>
/app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:329:in `fetch': 404 => Net::HTTPNotFound for https://analytics.twitter.com/user/player_twi/tweets/bundle?start_time=1543622400000&end_time=1543655857000&lang=ja -- unhandled response (Mechanize::ResponseCodeError)
    from /app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize.rb:464:in `get'
    from /app/twitter_analytics_client.rb:84:in `rescue in block in get_analytics_data'
    from /app/twitter_analytics_client.rb:79:in `block in get_analytics_data'
    from /app/twitter_analytics_client.rb:77:in `each'
    from /app/twitter_analytics_client.rb:77:in `get_analytics_data'
    from /app/twitter_analytics_client.rb:21:in `get_analytics_data_with_cookies'
    from script.rb:12:in `block in <main>'
    from script.rb:9:in `each'
    from script.rb:9:in `<main>'

heroku run bashにて実行。 @agentも出力した。

yusuko commented 5 years ago

heroku run bashにて

bundle exec ruby script.rb

を3回実行した。

1,2回目

3回目

Randomで起きるということだろうか？？？