Open Mauro-Rizzi opened 2 years ago
I let the plugin restart in a loop for a while trying to figure out this issue and suddenly it was able to process a week's worth of data without timing out. It seems my assumption might have not been on point but this still warrants testing in order to find the cause.
@mkreth This is needed for heavier queries. Is there any option to do that? Or, is there an alternative? (3.5m records, will run once a month~)
Added a pull request to support it, even though Salesforce forces the timeout for timeout which is bigger than 2 minutes~
So i'm currently running into the following error:
Error: Net::ReadTimeout Exception: Faraday::TimeoutError Stack: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:181:in
rbuf_fill' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:157:inreaduntil' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:167:in
readline' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:40:inread_status_line' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:29:in
read_new' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1504:inblock in transport_request' org/jruby/RubyKernel.java:1189:in
catch' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1501:intransport_request' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1474:in
request' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1467:inblock in request' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:914:in
start' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465:inrequest' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1223:in
get' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:85:inperform_request' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:43:in
block in call' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:92:inwith_net_http_connection' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/adapter/net_http.rb:38:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/custom_headers.rb:11:incall' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/gzip.rb:14:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-0.14.0/lib/faraday_middleware/response_middleware.rb:31:incall' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/response.rb:8:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-0.14.0/lib/faraday_middleware/response/follow_redirects.rb:87:inperform_with_redirection' redacted/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-0.14.0/lib/faraday_middleware/response/follow_redirects.rb:75:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/instance_url.rb:13:incall' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/authorization.rb:11:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/authentication.rb:16:incall' redacted/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-0.14.0/lib/faraday_middleware/request/encode_json.rb:24:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/multipart.rb:16:incall' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/middleware/mashify.rb:8:in
call' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:143:inbuild_response' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/connection.rb:387:in
run_request' redacted/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/connection.rb:138:inget' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/concerns/verbs.rb:37:in
block in define_verb' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/concerns/verbs.rb:63:inblock in define_api_verb' redacted/vendor/bundle/jruby/2.5.0/gems/restforce-4.3.0/lib/restforce/concerns/api.rb:186:in
query' redacted/vendor/bundle/jruby/2.5.0/gems/logstash-input-salesforce-3.0.7/lib/logstash/inputs/salesforce.rb:97:inrun' redacted/logstash-core/lib/logstash/java_pipeline.rb:405:in
inputworker' redacted/logstash-core/lib/logstash/java_pipeline.rb:396:inblock in start_input'
While trying to ingest a week's worth of data in a single run.
This seems to happen because the plugin is taking too long to read salesforce's response
I might be mistaken, but if that's really the reason I would like a sfdc_timeout setting to be added to the plugin so we can override it and let it run as long as we want, especially considering there's no proper self recurrence in the plugin and also no way to know what was the last thing you queried.