search

embulk / embulk

Embulk: Pluggable Bulk Data Loader.
https://www.embulk.org/
Apache License 2.0
1.74k stars 201 forks source link

Sometimes failed to run(can not copy embedded jar to temp directory) #1148

Open showyou opened 5 years ago

showyou commented 5 years ago

Sometimes embulk failed to run. When I tried the same command at another time, it succeeded. But I have to check everyday.

toru-takahashi commented 5 years ago

I've heard this issue before. At that time, the user said that they faced this issue where Kurbanates on GCP. So, there might be an issue related to Embulk on docker env?

dmikurube commented 5 years ago

Thanks for your follow-up, @toru-takahashi.

Hmm, I've never heard of this, but at least it is happening inside JRuby. https://github.com/jruby/jruby/blob/554b2f1ddc28c8b41279aace76871e9b37afe680/core/src/main/java/org/jruby/util/JRubyClassLoader.java#L98

Some of JRuby's assumption (about temporary directories) may not be satisfied in GCP (and Kubernates), but we're not sure of their assumption actually...

@showyou Is this the entire stacktrace, or did it have much more "causes" of the RuntimeException? We at least need deeper causes of the RuntimeException to try investigating.

showyou commented 5 years ago

Hi @dmikurube(CC: toru-takahashi),

Thanks for your reply. This is entire stack trace. Embulk runs NOT on GKE(kubernestes, Docker), but on GCE.

I tried to run "embulk irb" and "require 'openssl'", but it works.

dmikurube commented 5 years ago

@showyou Thanks for your update. Hmm, but we're not very sure of this. It's inside JRuby, and looks like a kind of mismatch between JRuby and GCP.

Let us keep this ticket open, and we'll update once we get some information.

shase commented 5 years ago

Hi

Similar exceptions sometimes occur in our environment.

Log

 Output:
 2019-07-25 05:06:11.474 +0000: Embulk v0.9.17
 2019-07-25 05:06:12.699 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
 2019-07-25 05:06:21.235 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "/home/ubuntu/foobar/embulk/libs/Gemfile"
 2019-07-25 05:06:21.239 +0000 [INFO] (main): Gem's home and path are being cleared.
 2019-07-25 05:06:34.458 +0000 [INFO] (main): Started Embulk v0.9.17
 2019-07-25 05:06:35.026 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-postgresql (0.10.0)
 RuntimeError:

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

 load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory (LoadError)
               do_require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:348
     block in require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:255
   require_jar_with_block at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:297
              require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:254
              require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:357
                   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:14
                  require at org/jruby/RubyKernel.java:955
                   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/signet.rb:1
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/signet.rb:30
                  require at org/jruby/RubyKernel.java:955
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/compute_engine.rb:1
                  require at org/jruby/RubyKernel.java:955
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/compute_engine.rb:31
                   <main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/application_default.rb:1
                  require at org/jruby/RubyKernel.java:955
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/application_default.rb:30
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth.rb:1
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth.rb:30
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/google-api-client-0.30.2/lib/google/apis/core/base_service.rb:1
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/google-api-client-0.30.2/lib/google/apis/core/base_service.rb:23
                   (root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/google-api-client-0.30.2/generated/google/apis/bigquery_v2/service.rb:1
 org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
 (RuntimeError)
oirik commented 4 years ago

Hello, I have the same problem.

Running embulk from MySQL to BigQuery 206 times and got this error 4 times.

Environment

$ cat /etc/issue
Ubuntu 18.04.4 LTS \n \l

$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

$ embulk -version
embulk 0.9.22

$ grep "embulk" Gemfile.lock
    embulk (0.9.22-java)
    embulk-filter-column (0.7.1)
    embulk-filter-hash (0.5.0)
    embulk-filter-to_json (0.0.5)
    embulk-filter-typecast (0.2.2)
    embulk-input-mysql (0.10.1)
    embulk-output-bigquery (0.6.4)

Exception

LoadError: load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
           require at org/jruby/RubyKernel.java:955
           require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120
            <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri.rb:106
           require at org/jruby/RubyKernel.java:955
           require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120
            <main> at /home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1
           require at org/jruby/RubyKernel.java:955
           require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:133
            (root) at /home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1
            (root) at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:1
  require_and_show at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:81
            search at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:39
            lookup at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:26
            lookup at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin.rb:197
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
        at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
        at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
        at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
        at org.embulk.spi.Exec.newPlugin(Exec.java:67)
        at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
        at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
        at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
        at org.embulk.spi.Exec.doWith(Exec.java:22)
        at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
        at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:290)
        at org.embulk.EmbulkRunner.run(EmbulkRunner.java:154)
        at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
        at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
        at org.embulk.cli.Main.main(Main.java:64)
        Suppressed: org.embulk.plugin.PluginSourceNotMatchException
                at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
                at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
                ... 16 more
        Suppressed: org.embulk.plugin.PluginSourceNotMatchException
                at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:68)
                at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
                ... 16 more
        Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: (LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
                at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
                at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
                ... 16 more
        Caused by: org.jruby.embed.InvokeFailedException: (LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
                at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
                at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
                at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
                at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
                at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
                ... 17 more
        Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
                at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
                at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120)
                at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri.rb:106)
                at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
                at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120)
                at RUBY.<main>(/home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1)
                at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
                at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:133)
                at RUBY.(root)(/home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1)
                at RUBY.(root)(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:1)
                at RUBY.require_and_show(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:81)
                at RUBY.search(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:39)
                at RUBY.lookup(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:26)
                at RUBY.lookup(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin.rb:197)

Error: OutputPlugin 'bigquery' is not found.
(LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
hiroyuki-sato commented 4 years ago

Hello, @oirik

Thank you for your reporting. Do you use GCE? It seems that this issue is GCE specific

oirik commented 4 years ago

@hiroyuki-sato

Hello, thank you for your comment.

Do you use GCE?

No, on-premise and on the actual machine.

oirik commented 4 years ago

Everyone is happening in bigquery output 🤔

hiroyuki-sato commented 4 years ago

Hello, @oirik

Thanks, do you use docker on your environment? Probably, this issue relates to JRuby. embulk-output-bigquery is the only plugin that uses JRuby in your enviroment.

oirik commented 4 years ago

@hiroyuki-sato

do you use docker on your environment?

No, no use of Docker, also no use of VM.

Probably, this issue relates to JRuby. embulk-output-bigquery is the only plugin that uses JRuby in your enviroment.

OK. Many people use embulk for uploading to BigQuery, like me 😸

hiroyuki-sato commented 4 years ago

Hello, @oirik

Could you double-check about your case? This error happened during embulk run correct?

There are many embulk-output-bigquery users. I've never heard that this(original issue)error happened non-GCP users until today. Your error message seems different from other users.

oirik commented 4 years ago

@hiroyuki-sato

This error happened during embulk run correct?

Yes, correct.

The command I've executed was like this: 

# 206 tables
cat tables.txt | while read TABLE_NAME; do embulk run --log log/${TABLE_NAME}.log -c ./diff/${TABLE_NAME}.diff.yml ./any.yml.liquid; done

I've only recently started using embulk, so I can't mention whether the error occurs under certain conditions or frequently.

pyama86 commented 4 years ago

I have happened same error.

stack

2020-04-27 21:31:29.913 +0900: Embulk v0.10.1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
        at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
        at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
        at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
        at org.embulk.spi.Exec.newPlugin(Exec.java:67)
        at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
        at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
        at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
        at org.embulk.spi.Exec.doWith(Exec.java:22)
        at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:243)
        at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
        at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
        at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
        at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
        at org.embulk.cli.Main.main(Main.java:64)
        Suppressed: org.embulk.plugin.PluginSourceNotMatchException
                at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
                at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
                ... 16 more
        Suppressed: org.embulk.plugin.PluginSourceNotMatchException
                at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:67)
                at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
                ... 16 more
        Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
                at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
                at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
                ... 16 more
        Caused by: org.jruby.embed.InvokeFailedException: org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
                at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:322)
                at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
                at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
                at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
                at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
                ... 17 more
        Caused by: org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.jruby.javasupport.proxy.JavaProxyConstructor.newInstanceImpl(JavaProxyConstructor.java:119)
                at org.jruby.javasupport.proxy.JavaProxyConstructor.newInstance(JavaProxyConstructor.java:205)
                at org.jruby.javasupport.Java$JCreateMethod.call(Java.java:618)
                at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
                at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:739)
                at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
                at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
                at org.jruby.java.proxies.ConcreteJavaProxy$InitializeMethod.call(ConcreteJavaProxy.java:48)
                at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:984)
                at org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:69)
                at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
                at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
                at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
                at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:328)
                at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:165)
                at org.jruby.RubyClass.newInstance(RubyClass.java:1001)
                at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
                at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
                at org.jruby.java.proxies.ConcreteJavaProxy$NewMethod.call(ConcreteJavaProxy.java:158)
                at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
                at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
                at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
                at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
                at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
                at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
                at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
                at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
                at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
                at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
                at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:90)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)
                at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)
                at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
                at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
                at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)
                at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
                at org.jruby.RubyClass.finvoke(RubyClass.java:522)
                at org.jruby.RubyBasicObject.send19(RubyBasicObject.java:1684)
                at org.jruby.RubyBasicObject$INVOKER$i$send19.call(RubyBasicObject$INVOKER$i$send19.gen)
                at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
                at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
                at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:432)
                at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
                at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
                at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
                at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174)
                at org.jruby.RubyClass.finvoke(RubyClass.java:899)
                at org.jruby.runtime.Helpers.invoke(Helpers.java:430)
                at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callEachType(EmbedRubyObjectAdapterImpl.java:356)
                at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:309)
                ... 26 more

Error: OutputPlugin 'bigquery' is not found.
org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.

then, I have reailized that segv is occuring.

15270 open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/lib/meta-index", O_RDONLY) = 3
15270 fstat(3, {st_mode=S_IFREG|0644, st_size=1922, ...}) = 0
15270 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f33165ee000
15270 read(3, "% VERSION 2\n% WARNING: this file is auto-generated; do not edit\n% UNSUPPORTED: this file and its format may change and/or\n%   may be removed in a future release\n# charsets.jar\nsun/nio\nsun/awt\n! jce.jar\njavax/crypto\nsun/security\n! jsse.jar\nsun/security\ncom/sun/net/\n! management-agent.jar\n@ resources.jar\ncom/sun/java/util/jar/pack/\nMETA-INF/services/sun.util.spi.XmlPropertiesProvider\nMETA-INF/services/javax.print.PrintServiceLookup\ncom/sun/corba/\nMETA-INF/services/javax.sound.midi.spi.SoundbankReader\nsun/print\nMETA-INF/services/javax.sound.midi.spi.MidiFileReader\nMETA-INF/services/sun.java2d.cmm.CMMServiceProvider\njavax/swing\nMETA-INF/services/javax.sound.sampled.spi.AudioFileReader\nMETA-INF/services/javax.sound.midi.spi.MidiDeviceProvider\nsun/net\nMETA-INF/services/javax.sound.sampled.spi.AudioFileWriter\ncom/sun/imageio/\nMETA-INF/services/sun.java2d.pipe.RenderingEngine\nMETA-INF/mimetypes.default\nMETA-INF/services/javax.sound.midi.spi.MidiFileWriter\nsun/rmi\njavax/sql\nMETA-INF/services/com.sun.tools.internal.ws.wscompile.Plugin\ncom/sun/rowset/\nMETA-INF/services/javax.print.StreamPrintServiceFactory\nMETA-INF/mailcap.default\njava/lang\nsun/text\njavax/xml\nMETA-INF/services/javax.sound.sampled.spi.MixerProvider\ncom/sun/xml/\nMETA-INF/services/com.sun.tools.internal.xjc.Plugin\ncom/sun/java/swing/\ncom/sun/jndi/\ncom/sun/org/\nMETA-INF/services/javax.sound.sampled.spi.FormatConversionProvider\n! rt.jar\ncom/sun/java/util/jar/pack/\njava/\norg/ietf/\ncom/sun/beans/\ncom/sun/tracing/\ncom/sun/java/browser/\ncom/sun/corba/\ncom/sun/media/\ncom/sun/awt/\ncom/sun/management/\nsun/\ncom/sun/jmx/\ncom/sun/demo/\ncom/sun/imageio/\ncom/sun/net/\ncom/sun/rmi/\norg/w3c/\ncom/sun/swing/\ncom/sun/activation/\ncom/sun/nio/\ncom/sun/rowset/\norg/jcp/\ncom/sun/istack/\njdk/\ncom/sun/naming/\norg/xml/\norg/omg/\ncom/sun/security/\ncom/sun/xml/\ncom/sun/java/swing/\ncom/oracle/\ncom/sun/java_cup/\ncom/sun/jndi/\ncom/sun/accessibility/\ncom/sun/org/\njavax/\n", 4096) = 1922
15270 read(3, "", 4096)                 = 0
15270 close(3)                          = 0
15270 munmap(0x7f33165ee000, 4096)      = 0
15270 mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f33005a1000
15270 mmap(0x7f33005a1000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f33005a1000
15270 mmap(NULL, 3932160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f33001e1000
15270 mmap(0x7f33001e1000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f33001e1000
15270 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---

I would like to get a coredump and continue to investigate.

tanakarian commented 4 years ago

Hello. I had struggled with similar problems.

Each problem may have different reason, I think. But at least in my case, It has been solved through export GEM_HOME environment variable.

Environment

My case

My logs are similar to @shase 's one. 

2020-05-14 19:30:03.821 +0000: Embulk v0.9.22
2020-05-14 19:30:07.314 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2020-05-14 19:30:24.451 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "/home/digdag/bundles/Gemfile"
2020-05-14 19:30:24.516 +0000 [INFO] (main): Gem's home and path are being cleared.
2020-05-14 19:30:48.744 +0000 [INFO] (main): Started Embulk v0.9.22
2020-05-14 19:30:49.953 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.10.1)
RuntimeError: 

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
              do_require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:348
    block in require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:255
  require_jar_with_block at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:297
             require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:254
             require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:357
                  <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:14
                 require at org/jruby/RubyKernel.java:955
                  <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30
                 require at org/jruby/RubyKernel.java:955
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1
                 require at org/jruby/RubyKernel.java:955
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31
                  <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1
                 require at org/jruby/RubyKernel.java:955
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23
                  (root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(RuntimeError) 

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
    at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
    at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
    at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
    at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
    at org.embulk.spi.Exec.newPlugin(Exec.java:67)
    at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
    at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
    at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
    at org.embulk.spi.Exec.doWith(Exec.java:22)
    at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
    at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
    at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:290)
    at org.embulk.EmbulkRunner.run(EmbulkRunner.java:154)
    at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
    at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
    at org.embulk.cli.Main.main(Main.java:64)
    Suppressed: org.embulk.plugin.PluginSourceNotMatchException
        at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
        ... 16 more
    Suppressed: org.embulk.plugin.PluginSourceNotMatchException
        at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:68)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
        ... 16 more
    Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: (RuntimeError) 

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
        at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
        ... 16 more
    Caused by: org.jruby.embed.InvokeFailedException: (RuntimeError) 

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
        at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
        at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
        at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
        ... 17 more
    Caused by: org.jruby.exceptions.RaiseException: (RuntimeError) 

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
        at RUBY.do_require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:348)
        at RUBY.block in require_jar(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:255)
        at RUBY.require_jar_with_block(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:297)
        at RUBY.require_jar(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:254)
        at RUBY.require_jar(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:357)
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:14)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1)

Error: OutputPlugin 'bigquery' is not found.
(RuntimeError) 

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)

other than that, logs below were sometime shown.

2020-05-14 23:30:05.157 +0000: Embulk v0.9.22
2020-05-14 23:30:09.556 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2020-05-14 23:30:27.332 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "/home/digdag/bundles/Gemfile"
2020-05-14 23:30:27.333 +0000 [INFO] (main): Gem's home and path are being cleared.
2020-05-14 23:30:52.060 +0000 [INFO] (main): Started Embulk v0.9.22
2020-05-14 23:30:53.841 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.10.1)
LoadError: load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
  require at org/jruby/RubyKernel.java:955
   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:25
  require at org/jruby/RubyKernel.java:955
   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30
  require at org/jruby/RubyKernel.java:955
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1
  require at org/jruby/RubyKernel.java:955
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31
   <main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1
  require at org/jruby/RubyKernel.java:955
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23
   (root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
    at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
    at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
    at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
    at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
    at org.embulk.spi.Exec.newPlugin(Exec.java:67)
    at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
    at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
    at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
    at org.embulk.spi.Exec.doWith(Exec.java:22)
    at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
    at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
    at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:290)
    at org.embulk.EmbulkRunner.run(EmbulkRunner.java:154)
    at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
    at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
    at org.embulk.cli.Main.main(Main.java:64)
    Suppressed: org.embulk.plugin.PluginSourceNotMatchException
        at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
        ... 16 more
    Suppressed: org.embulk.plugin.PluginSourceNotMatchException
        at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:68)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
        ... 16 more
    Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: (LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
        at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
        at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
        ... 16 more
    Caused by: org.jruby.embed.InvokeFailedException: (LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
        at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
        at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
        at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
        ... 17 more
    Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:25)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31)
        at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23)
        at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1)

Error: OutputPlugin 'bigquery' is not found.
(LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory

The problem may be related to msg: Gem's home and path are being cleared. logged at https://github.com/embulk/embulk/blob/v0.9.22/embulk-core/src/main/java/org/embulk/jruby/JRubyInitializer.java#L191 In case of jruby.clearGemPaths(); is called, embulk may use GEM_HOME enviroment variable.

Related comments in JRubyInitializer.java https://github.com/embulk/embulk/blob/v0.9.22/embulk-core/src/main/java/org/embulk/jruby/JRubyInitializer.java#L206

yoruno07 commented 4 years ago

I had same problems. I use embulk & BigQuery plugin in Docker Container. But, maybe I resolved this problem through install plugin jruby-openssl in advance.

In Dockerfile Before

RUN curl -o /usr/local/bin/embulk --create-dirs -L "http://dl.embulk.org/embulk-0.9.22.jar" && \
    chmod +x /usr/local/bin/embulk && \
    /usr/local/bin/embulk gem install embulk-input-mysql -v 0.10.1 && \
    /usr/local/bin/embulk gem install embulk-output-bigquery -v 0.6.4

After

RUN curl -o /usr/local/bin/embulk --create-dirs -L "http://dl.embulk.org/embulk-0.9.22.jar" && \
    chmod +x /usr/local/bin/embulk && \
    /usr/local/bin/embulk gem install embulk-input-mysql -v 0.10.1 && \
    /usr/local/bin/embulk gem install embulk-output-bigquery -v 0.6.4 && \
    /usr/local/bin/embulk gem install jruby-openssl -v 0.10.4 # add

After add jruby plugin, I never occur this problems.

hiroyuki-sato commented 4 years ago

I confirmed @takashiyamazaki that GEM_HOME set to {bundle_dir}/jruby/{version} using ECS task-def environment feature. Twitter Japanese.

progrhyme commented 4 years ago

Hello,

I also came across with this problem and found this issue by Google. And thanks to @yoruno07's comment and @hiroyuki-sato's suggestion, I could fix the problem by installing jruby-openssl and by setting GEM_HOME environment variable, not knowing which was the key factor.

My running environment is ...

(I wonder why jruby-openssl has two versions. These are the result of embulk gem list)

This Docker runs a shell script like this:

#!/bin/bash

# to resolve openjdk java
PATH=/usr/local/openjdk-8/bin:$PATH
export GEM_HOME=/root/.embulk/lib/gems

for table in $TARGET_TABLES; do
  config=config/$table.yml.liquid
  embulk run $config
done

Before the change, when I ran embulk for 20 tables, most of them failed with the error. After I set GEM_HOME and installed jruby-openssl, the error has never happened for now.

WangHaoming commented 2 years ago

I have the same issue, sometimes it fail with "LoadError: load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory" , sometimes the task just sucess. Could any one help me with this issue?

hiroyuki-sato commented 2 years ago

Hello, @WangHaoming. Could you try jruby-openssl v0.10.7? (Not above v0.10.7) I'm not sure your issue is the same to other people. A user resolved the issue using this version.

embulk gem uninstall jruby-openssl
embulk gem install jruby-openssl -v 0.10.7
doctor commented 2 years ago

@hiroyuki-sato I am getting the same error.

embulk gem list

2022-03-07 15:55:37.214 -0800: Embulk v0.9.23

Gem plugin path is: /home/jpillai/.embulk/lib/gems

`uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in require': It seems your ruby installation is missing psych (for YAML output). To eliminate this warning, please install libyaml and reinstall your ruby. LoadError: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory require at org/jruby/RubyKernel.java:955 require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55

at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych_jars.rb:2 require at org/jruby/RubyKernel.java:955 (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55 require at org/jruby/RubyKernel.java:955 (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:5
at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1 require at org/jruby/RubyKernel.java:955 require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55 (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/yaml.rb:6 (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1 require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55 load_yaml at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems.rb:701 load_file at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/config_file.rb:340
at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/config_file.rb:202 `
doctor commented 2 years ago

@hiroyuki-sato this issue is resolved after setting GEM_HOME as suggested by @progrhyme

export GEM_HOME=/home/jpillai/.embulk/lib/gems