Open showyou opened 5 years ago
I've heard this issue before. At that time, the user said that they faced this issue where Kurbanates on GCP. So, there might be an issue related to Embulk on docker env?
Thanks for your follow-up, @toru-takahashi.
Hmm, I've never heard of this, but at least it is happening inside JRuby. https://github.com/jruby/jruby/blob/554b2f1ddc28c8b41279aace76871e9b37afe680/core/src/main/java/org/jruby/util/JRubyClassLoader.java#L98
Some of JRuby's assumption (about temporary directories) may not be satisfied in GCP (and Kubernates), but we're not sure of their assumption actually...
@showyou Is this the entire stacktrace, or did it have much more "causes" of the RuntimeException
? We at least need deeper causes of the RuntimeException
to try investigating.
Hi @dmikurube(CC: toru-takahashi),
Thanks for your reply. This is entire stack trace. Embulk runs NOT on GKE(kubernestes, Docker), but on GCE.
I tried to run "embulk irb" and "require 'openssl'", but it works.
@showyou Thanks for your update. Hmm, but we're not very sure of this. It's inside JRuby, and looks like a kind of mismatch between JRuby and GCP.
Let us keep this ticket open, and we'll update once we get some information.
Hi
Similar exceptions sometimes occur in our environment.
Log
Output:
2019-07-25 05:06:11.474 +0000: Embulk v0.9.17
2019-07-25 05:06:12.699 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2019-07-25 05:06:21.235 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "/home/ubuntu/foobar/embulk/libs/Gemfile"
2019-07-25 05:06:21.239 +0000 [INFO] (main): Gem's home and path are being cleared.
2019-07-25 05:06:34.458 +0000 [INFO] (main): Started Embulk v0.9.17
2019-07-25 05:06:35.026 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-postgresql (0.10.0)
RuntimeError:
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory (LoadError)
do_require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:348
block in require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:255
require_jar_with_block at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:297
require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:254
require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:357
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:14
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/signet.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/signet.rb:30
require at org/jruby/RubyKernel.java:955
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/compute_engine.rb:1
require at org/jruby/RubyKernel.java:955
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/compute_engine.rb:31
<main> at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/application_default.rb:1
require at org/jruby/RubyKernel.java:955
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth/application_default.rb:30
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth.rb:1
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/googleauth-0.8.1/lib/googleauth.rb:30
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/google-api-client-0.30.2/lib/google/apis/core/base_service.rb:1
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/google-api-client-0.30.2/lib/google/apis/core/base_service.rb:23
(root) at /home/ubuntu/foobar/embulk/libs/jruby/2.3.0/gems/google-api-client-0.30.2/generated/google/apis/bigquery_v2/service.rb:1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(RuntimeError)
Hello, I have the same problem.
Running embulk from MySQL to BigQuery 206 times and got this error 4 times.
$ cat /etc/issue
Ubuntu 18.04.4 LTS \n \l
$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
$ embulk -version
embulk 0.9.22
$ grep "embulk" Gemfile.lock
embulk (0.9.22-java)
embulk-filter-column (0.7.1)
embulk-filter-hash (0.5.0)
embulk-filter-to_json (0.0.5)
embulk-filter-typecast (0.2.2)
embulk-input-mysql (0.10.1)
embulk-output-bigquery (0.6.4)
LoadError: load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
require at org/jruby/RubyKernel.java:955
require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri.rb:106
require at org/jruby/RubyKernel.java:955
require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120
<main> at /home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1
require at org/jruby/RubyKernel.java:955
require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:133
(root) at /home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1
(root) at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:1
require_and_show at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:81
search at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:39
lookup at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:26
lookup at uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin.rb:197
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
at org.embulk.spi.Exec.newPlugin(Exec.java:67)
at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:290)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:154)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:68)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: (LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
... 16 more
Caused by: org.jruby.embed.InvokeFailedException: (LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
... 17 more
Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri.rb:106)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:120)
at RUBY.<main>(/home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:133)
at RUBY.(root)(/home/oirik/.embulk/lib/gems/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:1)
at RUBY.(root)(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:1)
at RUBY.require_and_show(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:81)
at RUBY.search(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:39)
at RUBY.lookup(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin_registry.rb:26)
at RUBY.lookup(uri:classloader:/gems/embulk-0.9.22-java/lib/embulk/plugin.rb:197)
Error: OutputPlugin 'bigquery' is not found.
(LoadError) load error: uri/generic -- java.lang.NullPointerException: Inflater has been closed
Hello, @oirik
Thank you for your reporting. Do you use GCE? It seems that this issue is GCE specific
@hiroyuki-sato
Hello, thank you for your comment.
Do you use GCE?
No, on-premise and on the actual machine.
Everyone is happening in bigquery output 🤔
Hello, @oirik
Thanks, do you use docker on your environment? Probably, this issue relates to JRuby. embulk-output-bigquery is the only plugin that uses JRuby in your enviroment.
@hiroyuki-sato
do you use docker on your environment?
No, no use of Docker, also no use of VM.
Probably, this issue relates to JRuby. embulk-output-bigquery is the only plugin that uses JRuby in your enviroment.
OK. Many people use embulk for uploading to BigQuery, like me 😸
Hello, @oirik
Could you double-check about your case?
This error happened during embulk run
correct?
There are many embulk-output-bigquery users. I've never heard that this(original issue)error happened non-GCP users until today. Your error message seems different from other users.
@hiroyuki-sato
This error happened during embulk run correct?
Yes, correct.
The command I've executed was like this:
# 206 tables
cat tables.txt | while read TABLE_NAME; do embulk run --log log/${TABLE_NAME}.log -c ./diff/${TABLE_NAME}.diff.yml ./any.yml.liquid; done
I've only recently started using embulk, so I can't mention whether the error occurs under certain conditions or frequently.
I have happened same error.
stack
2020-04-27 21:31:29.913 +0900: Embulk v0.10.1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
at org.embulk.spi.Exec.newPlugin(Exec.java:67)
at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:243)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:67)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
... 16 more
Caused by: org.jruby.embed.InvokeFailedException: org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:322)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
... 17 more
Caused by: org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.jruby.javasupport.proxy.JavaProxyConstructor.newInstanceImpl(JavaProxyConstructor.java:119)
at org.jruby.javasupport.proxy.JavaProxyConstructor.newInstance(JavaProxyConstructor.java:205)
at org.jruby.javasupport.Java$JCreateMethod.call(Java.java:618)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:739)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
at org.jruby.java.proxies.ConcreteJavaProxy$InitializeMethod.call(ConcreteJavaProxy.java:48)
at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:984)
at org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:69)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:328)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:165)
at org.jruby.RubyClass.newInstance(RubyClass.java:1001)
at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
at org.jruby.java.proxies.ConcreteJavaProxy$NewMethod.call(ConcreteJavaProxy.java:158)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:90)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
at org.jruby.RubyClass.finvoke(RubyClass.java:522)
at org.jruby.RubyBasicObject.send19(RubyBasicObject.java:1684)
at org.jruby.RubyBasicObject$INVOKER$i$send19.call(RubyBasicObject$INVOKER$i$send19.gen)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:432)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:74)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174)
at org.jruby.RubyClass.finvoke(RubyClass.java:899)
at org.jruby.runtime.Helpers.invoke(Helpers.java:430)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callEachType(EmbedRubyObjectAdapterImpl.java:356)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:309)
... 26 more
Error: OutputPlugin 'bigquery' is not found.
org.jruby.proxy.org.embulk.config.ConfigException$Proxy1: Unknown output plugin 'bigquery'. embulk/output/bigquery.rb is not installed. Run 'embulk gem search -rd embulk-output' command to find plugins.
then, I have reailized that segv is occuring.
15270 open("/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/lib/meta-index", O_RDONLY) = 3
15270 fstat(3, {st_mode=S_IFREG|0644, st_size=1922, ...}) = 0
15270 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f33165ee000
15270 read(3, "% VERSION 2\n% WARNING: this file is auto-generated; do not edit\n% UNSUPPORTED: this file and its format may change and/or\n% may be removed in a future release\n# charsets.jar\nsun/nio\nsun/awt\n! jce.jar\njavax/crypto\nsun/security\n! jsse.jar\nsun/security\ncom/sun/net/\n! management-agent.jar\n@ resources.jar\ncom/sun/java/util/jar/pack/\nMETA-INF/services/sun.util.spi.XmlPropertiesProvider\nMETA-INF/services/javax.print.PrintServiceLookup\ncom/sun/corba/\nMETA-INF/services/javax.sound.midi.spi.SoundbankReader\nsun/print\nMETA-INF/services/javax.sound.midi.spi.MidiFileReader\nMETA-INF/services/sun.java2d.cmm.CMMServiceProvider\njavax/swing\nMETA-INF/services/javax.sound.sampled.spi.AudioFileReader\nMETA-INF/services/javax.sound.midi.spi.MidiDeviceProvider\nsun/net\nMETA-INF/services/javax.sound.sampled.spi.AudioFileWriter\ncom/sun/imageio/\nMETA-INF/services/sun.java2d.pipe.RenderingEngine\nMETA-INF/mimetypes.default\nMETA-INF/services/javax.sound.midi.spi.MidiFileWriter\nsun/rmi\njavax/sql\nMETA-INF/services/com.sun.tools.internal.ws.wscompile.Plugin\ncom/sun/rowset/\nMETA-INF/services/javax.print.StreamPrintServiceFactory\nMETA-INF/mailcap.default\njava/lang\nsun/text\njavax/xml\nMETA-INF/services/javax.sound.sampled.spi.MixerProvider\ncom/sun/xml/\nMETA-INF/services/com.sun.tools.internal.xjc.Plugin\ncom/sun/java/swing/\ncom/sun/jndi/\ncom/sun/org/\nMETA-INF/services/javax.sound.sampled.spi.FormatConversionProvider\n! rt.jar\ncom/sun/java/util/jar/pack/\njava/\norg/ietf/\ncom/sun/beans/\ncom/sun/tracing/\ncom/sun/java/browser/\ncom/sun/corba/\ncom/sun/media/\ncom/sun/awt/\ncom/sun/management/\nsun/\ncom/sun/jmx/\ncom/sun/demo/\ncom/sun/imageio/\ncom/sun/net/\ncom/sun/rmi/\norg/w3c/\ncom/sun/swing/\ncom/sun/activation/\ncom/sun/nio/\ncom/sun/rowset/\norg/jcp/\ncom/sun/istack/\njdk/\ncom/sun/naming/\norg/xml/\norg/omg/\ncom/sun/security/\ncom/sun/xml/\ncom/sun/java/swing/\ncom/oracle/\ncom/sun/java_cup/\ncom/sun/jndi/\ncom/sun/accessibility/\ncom/sun/org/\njavax/\n", 4096) = 1922
15270 read(3, "", 4096) = 0
15270 close(3) = 0
15270 munmap(0x7f33165ee000, 4096) = 0
15270 mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f33005a1000
15270 mmap(0x7f33005a1000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f33005a1000
15270 mmap(NULL, 3932160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f33001e1000
15270 mmap(0x7f33001e1000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f33001e1000
15270 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
I would like to get a coredump and continue to investigate.
Hello. I had struggled with similar problems.
Each problem may have different reason, I think. But at least in my case, It has been solved through export GEM_HOME environment variable.
OS Amazon Linux 2 ECS-Optimized (use docker & aws ecs)
JDK openjdk version "1.8.0_242"
Embulk embulk 0.9.22
JRuby jruby 1.7.26 (1.9.3p551) 2018-04-29 fffffff on OpenJDK 64-Bit Server VM 1.8.0_242-b08 +jit
$ embulk mkbundle bundles
~~ edit Gemfile~~
$ cd budles && embulk bundle install
My logs are similar to @shase 's one.
2020-05-14 19:30:03.821 +0000: Embulk v0.9.22
2020-05-14 19:30:07.314 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2020-05-14 19:30:24.451 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "/home/digdag/bundles/Gemfile"
2020-05-14 19:30:24.516 +0000 [INFO] (main): Gem's home and path are being cleared.
2020-05-14 19:30:48.744 +0000 [INFO] (main): Started Embulk v0.9.22
2020-05-14 19:30:49.953 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.10.1)
RuntimeError:
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
do_require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:348
block in require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:255
require_jar_with_block at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:297
require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:254
require_jar at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:357
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:14
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1
require at org/jruby/RubyKernel.java:955
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1
require at org/jruby/RubyKernel.java:955
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30
(root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1
(root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23
(root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(RuntimeError)
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
at org.embulk.spi.Exec.newPlugin(Exec.java:67)
at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:290)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:154)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:68)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: (RuntimeError)
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
... 16 more
Caused by: org.jruby.embed.InvokeFailedException: (RuntimeError)
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
... 17 more
Caused by: org.jruby.exceptions.RaiseException: (RuntimeError)
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
at RUBY.do_require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:348)
at RUBY.block in require_jar(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:255)
at RUBY.require_jar_with_block(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:297)
at RUBY.require_jar(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:254)
at RUBY.require_jar(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:357)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:14)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1)
Error: OutputPlugin 'bigquery' is not found.
(RuntimeError)
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56 -- java.lang.NullPointerException: Inflater has been closed (LoadError)
other than that, logs below were sometime shown.
2020-05-14 23:30:05.157 +0000: Embulk v0.9.22
2020-05-14 23:30:09.556 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2020-05-14 23:30:27.332 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "/home/digdag/bundles/Gemfile"
2020-05-14 23:30:27.333 +0000 [INFO] (main): Gem's home and path are being cleared.
2020-05-14 23:30:52.060 +0000 [INFO] (main): Started Embulk v0.9.22
2020-05-14 23:30:53.841 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.10.1)
LoadError: load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:25
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
require at org/jruby/RubyKernel.java:955
<main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30
require at org/jruby/RubyKernel.java:955
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1
require at org/jruby/RubyKernel.java:955
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31
<main> at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1
require at org/jruby/RubyKernel.java:955
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1
(root) at /home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30
(root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1
(root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23
(root) at /home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1
org.embulk.config.ConfigException: OutputPlugin 'bigquery' is not found.
(LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:75)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:61)
at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:28)
at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:162)
at org.embulk.spi.Exec.newPlugin(Exec.java:67)
at org.embulk.exec.BulkLoader$ProcessPluginSet.<init>(BulkLoader.java:428)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:502)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:290)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:154)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.InjectedPluginSource.newPlugin(InjectedPluginSource.java:53)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException
at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:68)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:49)
... 16 more
Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.jruby.embed.InvokeFailedException: (LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:63)
at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:56)
... 16 more
Caused by: org.jruby.embed.InvokeFailedException: (LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:320)
at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:159)
at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1459)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.embulk.jruby.ScriptingContainerDelegateImpl.callMethod(ScriptingContainerDelegateImpl.java:634)
at org.embulk.jruby.LazyScriptingContainerDelegate.callMethod(LazyScriptingContainerDelegate.java:129)
at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:61)
... 17 more
Caused by: org.jruby.exceptions.RaiseException: (LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jopenssl/load.rb:25)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/openssl.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/signature.rb:4)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt/decode.rb:5)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/jwt-2.2.1/lib/jwt.rb:5)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/signet-0.11.0/lib/signet/oauth_2/client.rb:21)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/signet.rb:30)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/compute_engine.rb:31)
at RUBY.<main>(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:1)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:955)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth/application_default.rb:30)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:1)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/googleauth-0.9.0/lib/googleauth.rb:30)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:1)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/lib/google/apis/core/base_service.rb:23)
at RUBY.(root)(/home/digdag/bundles/jruby/2.3.0/gems/google-api-client-0.32.1/generated/google/apis/bigquery_v2/service.rb:1)
Error: OutputPlugin 'bigquery' is not found.
(LoadError) load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
The problem may be related to msg: Gem's home and path are being cleared.
logged at https://github.com/embulk/embulk/blob/v0.9.22/embulk-core/src/main/java/org/embulk/jruby/JRubyInitializer.java#L191
In case of jruby.clearGemPaths();
is called, embulk may use GEM_HOME enviroment variable.
Related comments in JRubyInitializer.java https://github.com/embulk/embulk/blob/v0.9.22/embulk-core/src/main/java/org/embulk/jruby/JRubyInitializer.java#L206
I had same problems. I use embulk & BigQuery plugin in Docker Container. But, maybe I resolved this problem through install plugin jruby-openssl in advance.
In Dockerfile Before
RUN curl -o /usr/local/bin/embulk --create-dirs -L "http://dl.embulk.org/embulk-0.9.22.jar" && \
chmod +x /usr/local/bin/embulk && \
/usr/local/bin/embulk gem install embulk-input-mysql -v 0.10.1 && \
/usr/local/bin/embulk gem install embulk-output-bigquery -v 0.6.4
After
RUN curl -o /usr/local/bin/embulk --create-dirs -L "http://dl.embulk.org/embulk-0.9.22.jar" && \
chmod +x /usr/local/bin/embulk && \
/usr/local/bin/embulk gem install embulk-input-mysql -v 0.10.1 && \
/usr/local/bin/embulk gem install embulk-output-bigquery -v 0.6.4 && \
/usr/local/bin/embulk gem install jruby-openssl -v 0.10.4 # add
After add jruby plugin, I never occur this problems.
I confirmed @takashiyamazaki that GEM_HOME set to {bundle_dir}/jruby/{version}
using ECS task-def environment feature.
Twitter Japanese.
Hello,
I also came across with this problem and found this issue by Google.
And thanks to @yoruno07's comment and @hiroyuki-sato's suggestion, I could fix the problem by installing jruby-openssl
and by setting GEM_HOME environment variable, not knowing which was the key factor.
My running environment is ...
(I wonder why jruby-openssl
has two versions. These are the result of embulk gem list
)
This Docker runs a shell script like this:
#!/bin/bash
# to resolve openjdk java
PATH=/usr/local/openjdk-8/bin:$PATH
export GEM_HOME=/root/.embulk/lib/gems
for table in $TARGET_TABLES; do
config=config/$table.yml.liquid
embulk run $config
done
Before the change, when I ran embulk for 20 tables, most of them failed with the error. After I set GEM_HOME and installed jruby-openssl, the error has never happened for now.
I have the same issue, sometimes it fail with "LoadError: load error: jopenssl -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory" , sometimes the task just sucess. Could any one help me with this issue?
Hello, @WangHaoming. Could you try jruby-openssl v0.10.7? (Not above v0.10.7) I'm not sure your issue is the same to other people. A user resolved the issue using this version.
embulk gem uninstall jruby-openssl
embulk gem install jruby-openssl -v 0.10.7
@hiroyuki-sato I am getting the same error.
embulk gem list
2022-03-07 15:55:37.214 -0800: Embulk v0.9.23
Gem plugin path is: /home/jpillai/.embulk/lib/gems
`uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in require': It seems your ruby installation is missing psych (for YAML output). To eliminate this warning, please install libyaml and reinstall your ruby. LoadError: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory require at org/jruby/RubyKernel.java:955 require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55
@hiroyuki-sato this issue is resolved after setting GEM_HOME as suggested by @progrhyme
export GEM_HOME=/home/jpillai/.embulk/lib/gems
Sometimes embulk failed to run. When I tried the same command at another time, it succeeded. But I have to check everyday.
OS: CentOS Linux release 7.6.1810 (Core) on GCE
Java: openjdk version "1.8.0_212"
Embulk: 0.9.15
Plugins: embulk (0.9.15 java) embulk-formatter-jsonl (0.1.4) embulk-output-bigquery (0.4.10) embulk-parser-jsonl (0.2.0)
Log: