search

logstash-plugins / logstash-input-salesforce

Logstash input for pulling objects from salesforce
Apache License 2.0
16 stars 29 forks source link

Unable to install on Logstash 7.3 or 7.4 Clean OS install AWS AMI Ubuntu 18.04 #21

Open ryanivis opened 5 years ago

ryanivis commented 5 years ago

Installation Aborted, message: Bundler could not find compatible versions for gem "faraday": In snapshot (Gemfile.lock).

Validating logstash-input-salesforce Installing logstash-input-salesforce Plugin version conflict, aborting ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "faraday": In snapshot (Gemfile.lock): faraday (= 0.17.0)

In Gemfile: logstash-core java was resolved to 7.4.1, which depends on elasticsearch (~> 5) java was resolved to 5.0.5, which depends on elasticsearch-transport (= 5.0.5) java was resolved to 5.0.5, which depends on faraday java

logstash-input-salesforce java was resolved to 3.0.5, which depends on
  restforce (~> 2.4.2) java was resolved to 2.4.2, which depends on
    faraday (~> 0.9.0) java

octokit (~> 4) java was resolved to 4.14.0, which depends on
  sawyer (>= 0.5.3, ~> 0.8.0) java was resolved to 0.8.2, which depends on
    faraday (> 0.8, < 2.0) java

Running bundle update will rebuild your snapshot from scratch, using only the gems in your Gemfile, which may resolve the conflict.

Bundler could not find compatible versions for gem "logstash-core": In snapshot (Gemfile.lock): logstash-core (= 7.4.1)

In Gemfile: logstash-core java

logstash-core-plugin-api java was resolved to 2.1.16, which depends on
  logstash-core (= 7.4.1) java

logstash-input-syslog java was resolved to 3.4.1, which depends on
  logstash-filter-grok java was resolved to 4.1.1, which depends on
    logstash-core (>= 5.6.0) java

logstash-input-salesforce java was resolved to 0.1.1, which depends on
  logstash-core (>= 1.4.0, < 2.0.0) java

Running bundle update will rebuild your snapshot from scratch, using only the gems in your Gemfile, which may resolve the conflict.

Bundler could not find compatible versions for gem "logstash-core-plugin-api": In snapshot (Gemfile.lock): logstash-core-plugin-api (= 2.1.16)

In Gemfile: logstash-devutils (~> 1) java was resolved to 1.3.6, which depends on logstash-core-plugin-api (>= 2.0, <= 2.99) java

logstash-input-salesforce java was resolved to 2.0.4, which depends on
  logstash-core-plugin-api (~> 1.0) java

Running bundle update will rebuild your snapshot from scratch, using only the gems in your Gemfile, which may resolve the conflict.

Appears issues in dependencies.

Madere commented 4 years ago

Hi all,

we encounter the same problem on our logstash 7.4.2 Hope this will be solved soon.

Martin.

marekduciuc commented 4 years ago

This also happens on 7.5.1

bharatsantani1410 commented 4 years ago

Also on 7.6.0

oxalampi commented 4 years ago

Hi, we are experiencing the same issue in our project. We follow the advices given in the error logs.

Running bundle update will rebuild your snapshot from scratch, using only the gems in your Gemfile, which may resolve the conflict.

Below are the steps and the results (I did my debugging on windows10):

1) Action: run the command "bundle update --bundler" Result: got an Error : [!] There was an error while loadinglogstash-core-plugin-api.gemspec: The logstash-core-api need to be build on jruby. Bundler cannot continue. 2) Action: installed Jruby and added JRuby\bin directory in my path Result: Success 3) Action: run bundle update --bundler : Result: Error : bundler command does not exist in jruby, need to be installed. 4) Action: installed bundler with the command : : jruby -S gem install bundler Result: Success 5) Action: retry command "bundle update --bundler" Result: Got an error : Could not locate Gemfile 6)Action : Go to the directory C:\Dev\tools\logstash-7.5.0, try to regenerate the gem of logstash-core-plugin-api : gem install logstash-core-plugin-api: Result : Error : 

jar dependencies for logstash-core-5.6.4-java.gemspec . . .
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-parseable POM C:\Dev\tools\jruby-9.2.11.0\lib\ruby\stdlib\jars\gemspec_pom.rb: only whitespace content allowed before start tag and not # (position: START_DOCUMENT seen #... @1:1)  @ line 1, column 1
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project  (C:\Dev\tools\jruby-9.2.11.0\lib\ruby\stdlib\jars\gemspec_pom.rb) has 1 error
[ERROR]     Non-parseable POM C:\Dev\tools\jruby-9.2.11.0\lib\ruby\stdlib\jars\gemspec_pom.rb: only whitespace content allowed before start tag and not # (position: START_DOCUMENT seen #... @1:1)  @ line 1, column 1 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException
ERROR:  While executing gem ... (Errno::ENOENT)
    No such file or directory - C:/Dev/tools/jruby-9.2.11.0/lib/ruby/gems/shared/gems/logstash-core-5.6.4-java/deps.lst

7) Action: removed the first line in the file C:\Dev\tools\jruby-9.2.11.0\lib\ruby\stdlib\jars\gemspec_pom.rb et rerun the command : gem install logstash-core-plugin-api Result : Success

Successfully installed logstash-core-plugin-api-2.1.28-java
Parsing documentation for logstash-core-plugin-api-2.1.28-java
Installing ri documentation for logstash-core-plugin-api-2.1.28-java
Done installing documentation for logstash-core-plugin-api after 0 seconds
1 gem installed

8) Action: retry the command "bundle update --bundler" Result :

lot of logs including the following:

Using logstash-core 7.5.0 (java) from source at `logstash-core`
Using logstash-core-plugin-api 2.1.16 (java) from source at `logstash-core-plugin-api`
Fetching logstash-codec-avro 3.2.3 (java)
Installing logstash-codec-avro 3.2.3 (java)

and at the end: 

Warning: the lockfile is being updated to Bundler 2, after which you will be unable to return to Bundler 1.
Updating files in vendor/cache
  * rake-12.3.3.gem
Permission denied - C:/Dev/tools/logstash-7.5.0/vendor/cache/rake-12.3.3.gem

9)Action: rerun the same command in admin mode : Result : Error, got the same error.

Permission denied - C:/Dev/tools/logstash-7.5.0/vendor/cache/rake-12.3.3.gem

Any hint on how this could be fixed?

drivard commented 4 years ago

Here is how I was able to build the plugin with the logstash version 7.6.1 docker image. I cannot ensure yet that the plugin is a 100% working with the version since I am currently testing with it.

UPDATE I can confirm that we were able to extract data out of salesforce using logstash version 7.6.1

Build the plugin for logstash 7.6.1

First clone the repository locally. Then get in the directory ci/unit.

git clone https://github.com/drivard/logstash-input-salesforce.git
cd logstash-input-salesforce/ci/unit

The docker-compose.yml file references an environment variable named ELASTIC_STACK_VERSION. Export it with the value of the logstash version you want to use. You can check the following url for the version which are availables.

Logstash Versions

Once the version is choosed, please set the environment variable in your shell.

export ELASTIC_STACK_VERSION=7.6.1

Edit the file ci/unit/Dockerfile and change it with the following content. The only difference is that I commented out the line RUN bundle install and I replaced it by RUN gem build ./logstash-input-salesforce.gemspec

ARG ELASTIC_STACK_VERSION
FROM docker.elastic.co/logstash/logstash:$ELASTIC_STACK_VERSION
COPY --chown=logstash:logstash Gemfile /usr/share/plugins/this/Gemfile
COPY --chown=logstash:logstash *.gemspec /usr/share/plugins/this/
RUN cp /usr/share/logstash/logstash-core/versions-gem-copy.yml /usr/share/logstash/versions.yml
ENV PATH="${PATH}:/usr/share/logstash/vendor/jruby/bin"
ENV LOGSTASH_SOURCE=1
ENV JARS_SKIP="true"
RUN gem install bundler -v '< 2'
WORKDIR /usr/share/plugins/this
# RUN bundle install
RUN gem build ./logstash-input-salesforce.gemspec
COPY --chown=logstash:logstash . /usr/share/plugins/this

Now build the docker image with the following command line. The command line could be without the --build if it is the first time you execute it.

docker-compose up -d --build

Once the image is built, use the docker images command line to find the name of the image.

On my laptop the name of the image was unit_logstash, now we need the name of the gem file that was built.

You can find the name of the file by executing the following command or from the output of the build and it should be logstash-input-salesforce-3.0.5.gem.

docker run --rm -it unit_logstash /bin/ls -lh

Now copy the built plugin to your local host

The file will be copied in your current directory. If the build was successfull, it should have also started the logstash container. By doing the command docker ps you will find if a container is running with the name of unit_logstash_1. If this container is running then you can simply copy the file with the following command line:

docker cp unit_logstash_1:/usr/share/plugins/this/logstash-input-salesforce-3.0.5.gem .

Once you have the file on your host. You can simply stop the container by executing the command line docker-compose down.

Distribute and install

You now got the plugin built for logstash version 7.6.1, now you can distribute the file the way you want to your existing logstash installation.

To install the plugin simply execute the command line:

# Logstash 2.3 and higher
bin/logstash-plugin install --no-verify logstash-input-salesforce-3.0.5.gem

# Prior to Logstash 2.3
bin/plugin install --no-verify logstash-input-salesforce-3.0.5.gem

Configure your pipeline and restart logstash.

hariharanlogan commented 4 years ago

Hi Drivard, When I tried your instruction getting the below error, using logstash 7.6.1.

Step 9/12 : RUN gem install bundler -v '< 2' ---> Running in 503f72a18338 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ERROR: Could not find a valid gem 'bundler' (< 2), here is why: Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (initialize: name or service not known) (https://rubygems.org/specs.4.8.gz) ERROR: Service 'logstash' failed to build: The command '/bin/sh -c gem install bundler -v '< 2'' returned a non-zero code: 2

Did manual install of gem install bundler -v '< 2' Fetching: bundler-1.17.3.gem (100%) Successfully installed bundler-1.17.3 1 gem installed

But still issue persisting same error while building at step8 ERROR: Service 'logstash' failed to build: The command '/bin/sh -c gem install bundler -v '< 2'' returned a non-zero code: 2

Did you faced similar issue?

hariharanlogan commented 4 years ago

@drivard ^^

drivard commented 4 years ago

@hariharanlogan I just tried again and I don't get the same errors as you do. Here is my build log:

export ELASTIC_STACK_VERSION=7.6.1
cd logstash-input-salesforce/ci/unit
docker-compose up -d --build
Creating network "unit_default" with the default driver
Building logstash
Step 1/12 : ARG ELASTIC_STACK_VERSION
Step 2/12 : FROM docker.elastic.co/logstash/logstash:$ELASTIC_STACK_VERSION
7.6.1: Pulling from logstash/logstash
c808caf183b6: Pull complete
45a6dd9b68a9: Pull complete
4e4789fc2d59: Pull complete
86b7cc2b2791: Pull complete
d4774147b306: Pull complete
28c693c65176: Pull complete
2cced6b40df3: Pull complete
6ccde4da4cf4: Pull complete
7e27c206e041: Pull complete
4ccf01375919: Pull complete
15ba21b0e67a: Pull complete
Digest: sha256:462ff0cfd3a3ec444e0dccb11dd1ac0a3f52502616b3efaf4ae1aee90f9b7e13
Status: Downloaded newer image for docker.elastic.co/logstash/logstash:7.6.1
 ---> d6d66afe6805
Step 3/12 : COPY --chown=logstash:logstash Gemfile /usr/share/plugins/this/Gemfile
 ---> add22a9b8615
Step 4/12 : COPY --chown=logstash:logstash *.gemspec /usr/share/plugins/this/
 ---> afc2d7f9a811
Step 5/12 : RUN cp /usr/share/logstash/logstash-core/versions-gem-copy.yml /usr/share/logstash/versions.yml
 ---> Running in 0c55e70d2bf1
Removing intermediate container 0c55e70d2bf1
 ---> 3a7ddfe6d8ec
Step 6/12 : ENV PATH="${PATH}:/usr/share/logstash/vendor/jruby/bin"
 ---> Running in 7800138c777b
Removing intermediate container 7800138c777b
 ---> e77787992179
Step 7/12 : ENV LOGSTASH_SOURCE=1
 ---> Running in c715122ae6fa
Removing intermediate container c715122ae6fa
 ---> 28a8b7aaf274
Step 8/12 : ENV JARS_SKIP="true"
 ---> Running in 8db1c0ab9e24
Removing intermediate container 8db1c0ab9e24
 ---> 7f73cc56c5c0
Step 9/12 : RUN gem install bundler -v '< 2'
 ---> Running in ede7a6e46918
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Successfully installed bundler-1.17.3
1 gem installed
Removing intermediate container ede7a6e46918
 ---> 34d702203c8b
Step 10/12 : WORKDIR /usr/share/plugins/this
 ---> Running in b006a038e808
Removing intermediate container b006a038e808
 ---> df163d3544f0
Step 11/12 : RUN gem build ./logstash-input-salesforce.gemspec
 ---> Running in 5d57df5c20c4
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING:  license value 'Apache License (2.0)' is invalid.  Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
Did you mean 'Apache-2.0'?
WARNING:  open-ended dependency on logstash-codec-plain (>= 0) is not recommended
  if logstash-codec-plain is semantically versioned, use:
    add_runtime_dependency 'logstash-codec-plain', '~> 0'
WARNING:  open-ended dependency on logstash-devutils (>= 0, development) is not recommended
  if logstash-devutils is semantically versioned, use:
    add_development_dependency 'logstash-devutils', '~> 0'
WARNING:  open-ended dependency on vcr (>= 0, development) is not recommended
  if vcr is semantically versioned, use:
    add_development_dependency 'vcr', '~> 0'
WARNING:  open-ended dependency on webmock (>= 0, development) is not recommended
  if webmock is semantically versioned, use:
    add_development_dependency 'webmock', '~> 0'
WARNING:  open-ended dependency on json (>= 0, development) is not recommended
  if json is semantically versioned, use:
    add_development_dependency 'json', '~> 0'
WARNING:  pessimistic dependency on public_suffix (~> 1.4.6, development) may be overly strict
  if public_suffix is semantically versioned, use:
    add_development_dependency 'public_suffix', '~> 1.4', '>= 1.4.6'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: logstash-input-salesforce
  Version: 3.0.5
  File: logstash-input-salesforce-3.0.5.gem
Removing intermediate container 5d57df5c20c4
 ---> d0162d5e7850
Step 12/12 : COPY --chown=logstash:logstash . /usr/share/plugins/this
 ---> 20d1ae9d9026
Successfully built 20d1ae9d9026
Successfully tagged unit_logstash:latest
Creating unit_logstash_1 ... done

It built logstash-input-salesforce-3.0.5.gem. Check in the instructions and I have my own fork of this repo git clone https://github.com/drivard/logstash-input-salesforce.git. The current repo has been changed right after I wrote these instruction and they removed the ci/unit folder. Try from from my fork and build it again. Otherwise I built it and you can download the binary file here --> https://github.com/drivard/logstash-input-salesforce/releases/tag/test-3.0.5.

hariharanlogan commented 4 years ago

Thank you so much, @drivard will work on this and keep you updated

hariharanlogan commented 4 years ago

@drivard tried your latest git "git clone https://github.com/drivard/logstash-input-salesforce.git" got same error at step9, so used https://github.com/drivard/logstash-input-salesforce/releases/tag/test-3.0.5. and installed the plugin successfully,

bin/logstash-plugin install --no-verify logstash-input-salesforce-3.0.5.gem OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Installing logstash-input-salesforce Installation successful

Now after updating the salesforce config file and restarting logstash getting the error unable to identify the salesforce plugin, here is the log information: ./logstash -f /data/logstash-7.6.1/config/test.salesforce.conf OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/data/logstash-7.6.1/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Sending Logstash logs to /data/logstash-7.6.1/logs which is now configured via log4j2.properties [2020-04-09T16:28:15,235][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-04-09T16:28:15,389][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.1"} [2020-04-09T16:28:17,200][INFO ][org.reflections.Reflections] Reflections took 41 ms to scan 1 urls, producing 20 keys and 40 values [2020-04-09T16:28:17,331][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/inputs/salesforce>, :path=>"logstash/inputs/salesforce", :type=>"input", :name=>"salesforce"} [2020-04-09T16:28:17,343][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'salesforce'. Are you sure this is correct? Trying to load the salesforce input plugin resulted in this error: no such file to load -- logstash/inputs/salesforce", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:103)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/data/logstash-7.6.1/logstash-core/lib/logstash/java_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/data/logstash-7.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/data/logstash-7.6.1/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:125)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]} warning: thread "Converge PipelineAction::Create

" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main> create at org/logstash/execution/ConvergeResultExt.java:109 add at org/logstash/execution/ConvergeResultExt.java:37 converge_state at /data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:339 [2020-04-09T16:28:17,356][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in create'", "org/logstash/execution/ConvergeResultExt.java:37:inadd'", "/data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:339:in block in converge_state'"]} [2020-04-09T16:28:17,424][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handleJava::JavaLang::IllegalStateExceptionforPipelineAction::Create
>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:increate'", "org/logstash/execution/ConvergeResultExt.java:37:in add'", "/data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:339:inblock in converge_state'"]} [2020-04-09T16:28:17,454][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Not sure if anything I am missing here, tried replacing salesforce name in config file with salesforce-3.0.5.gem for workarround nothing worked, any change required? I used the below basic configuration to check whether request going to salesforce from plugin, once successful, later planning the put in the client id etc, input { salesforce { client_id => 'OAUTH CLIENT ID FROM YOUR SFDC APP' client_secret => 'OAUTH CLIENT SECRET FROM YOUR SFDC APP' username => 'email@example.com' password => 'super-secret' security_token => 'SECURITY TOKEN FOR THIS USER' sfdc_object_name => 'Opportunity' } } filter { } output { stdout {codec => rubydebug} }

hariharanlogan commented 4 years ago

@drivard , please update me

drivard commented 4 years ago

@hariharanlogan It doesn't load the plugins directory as per the error message not finding your input. Look at your command line: ./logstash -f /data/logstash-7.6.1/config/test.salesforce.conf this is not loading any of the logstash main configurations but rather only your specific pipeline. Try something like ./logstash --path.settings /etc/logstash -f /data/logstash-7.6.1/config/test.salesforce.conf where --path.settings /etc/logstash is the path for all your logstash global configurations.

hariharanlogan commented 4 years ago

@drivard Installed logstash7.6.1 in the directory /data all configurations resides in the same folder not in /etc, when i tired with other BEAT plugin from the same location /data/bin/logstash it worked, but only for salesforce getting error out.

hariharanlogan commented 4 years ago

hariharanlogan commented 4 years ago

/data/logstash-7.6.1/bin

hariharanlogan commented 4 years ago

@drivard Still I am getting the error, please suggest.

./logstash --path.settings /data/logstash-7.6.1 -f /data/logstash-7.6.1/config/salesforce.conf OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/data/logstash-7.6.1/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /data/logstash-7.6.1/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-05-01 13:28:09.185 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-05-01 13:28:09.195 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.1"} [INFO ] 2020-05-01 13:28:10.933 [Converge PipelineAction::Create

] Reflections - Reflections took 37 ms to scan 1 urls, producing 20 keys and 40 values [ERROR] 2020-05-01 13:28:11.043 [Converge PipelineAction::Create
] registry - Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/inputs/salesforce>, :path=>"logstash/inputs/salesforce", :type=>"input", :name=>"salesforce"} [ERROR] 2020-05-01 13:28:11.053 [Converge PipelineAction::Create
] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'salesforce'. Are you sure this is correct? Trying to load the salesforce input plugin resulted in this error: no such file to load -- logstash/inputs/salesforce", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:103)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/data/logstash-7.6.1/logstash-core/lib/logstash/java_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/data/logstash-7.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/data/logstash-7.6.1/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "data.logstash_minus_7_dot_6_dot_1.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:125)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]} warning: thread "Converge PipelineAction::Create
" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main> create at org/logstash/execution/ConvergeResultExt.java:109 add at org/logstash/execution/ConvergeResultExt.java:37 converge_state at /data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:339 [ERROR] 2020-05-01 13:28:11.062 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in create'", "org/logstash/execution/ConvergeResultExt.java:37:inadd'", "/data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:339:in block in converge_state'"]} [FATAL] 2020-05-01 13:28:11.074 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handleJava::JavaLang::IllegalStateExceptionforPipelineAction::Create
>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:increate'", "org/logstash/execution/ConvergeResultExt.java:37:in add'", "/data/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:339:inblock in converge_state'"]} [ERROR] 2020-05-01 13:28:11.084 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

drivard commented 4 years ago

Try overriding the path.plugins settings in your configuration file.

https://www.elastic.co/guide/en/logstash/current/dir-layout.html

Or before installing it to a different location try installing it to its common known path.

hariharanlogan commented 4 years ago

@drivard Sure, after installing in common location, installed the salesforce plugin successfully and tried to start, but same error below, please help.

./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/salesforce.conf OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2020-05-03T11:02:41,954][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-05-03T11:02:42,154][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"} [2020-05-03T11:02:45,108][INFO ][org.reflections.Reflections] Reflections took 57 ms to scan 1 urls, producing 20 keys and 40 values [2020-05-03T11:02:45,294][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/inputs/salesforce>, :path=>"logstash/inputs/salesforce", :type=>"input", :name=>"salesforce"} [2020-05-03T11:02:45,311][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'salesforce'. Are you sure this is correct? Trying to load the salesforce input plugin resulted in this error: no such file to load -- logstash/inputs/salesforce", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:103)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:125)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]} warning: thread "Converge PipelineAction::Create

" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main> create at org/logstash/execution/ConvergeResultExt.java:109 add at org/logstash/execution/ConvergeResultExt.java:37 converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:339 [2020-05-03T11:02:45,320][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in create'", "org/logstash/execution/ConvergeResultExt.java:37:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in block in converge_state'"]} [2020-05-03T11:02:45,425][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handleJava::JavaLang::IllegalStateExceptionforPipelineAction::Create
>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:increate'", "org/logstash/execution/ConvergeResultExt.java:37:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:inblock in converge_state'"]} [2020-05-03T11:02:45,437][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

hariharanlogan commented 4 years ago

@drivard anything you see missing, can you share the your thoughts/suggestions