search

killbill / killbill-plugin-framework-ruby

Framework to write Kill Bill plugins in Ruby
http://killbill.io
8 stars 11 forks source link

Contention in JRuby InvocationMethodFactory.getBlockCallback19 #29

Open pierre opened 9 years ago

pierre commented 9 years ago

Culprit is https://github.com/jruby/jruby/blob/b45a7f080c2b48669fe379f80fc7c750b11e74f2/core/src/main/java/org/jruby/internal/runtime/methods/InvocationMethodFactory.java#L1001 when evaluating https://github.com/rails/rails/blob/ec6c98f7d11e53811fbaa13ed521386f53011630/activerecord/lib/active_record/attribute_methods/read.rb#L53. Threads block on the synchronized object.

kares commented 9 years ago

nice one ... wonder if this is the main cause of the CPU slow-down (for me as well). going to get interesting

kares commented 9 years ago

do you happen to have a full back/stack trace somewhere around ?

pierre commented 9 years ago

Yay for saving YourKit snapshots :-)

At that moment, a dozen or so threads are blocked here:

Plugin-th-172 [BLOCKED]
org.jruby.internal.runtime.methods.InvocationMethodFactory.getBlockCallback19(String, String, int, Object) InvocationMethodFactory.java:989
org.jruby.runtime.Helpers.createBlockCallback19(Object, String, String, int) Helpers.java:154
org.jruby.runtime.Helpers.createCompiledBlockBody19(ThreadContext, Object, String, int, StaticScope, boolean, int, String, int, boolean, String) Helpers.java:244
org.jruby.runtime.Helpers.createCompiledBlockBody19(ThreadContext, Object, StaticScope, String) Helpers.java:236
org.jruby.ast.executable.RuntimeCache.createBlockBody19(Object, ThreadContext, StaticScope, int, String) RuntimeCache.java:452
org.jruby.ast.executable.RuntimeCache.getBlockBody19(Object, ThreadContext, StaticScope, int, String) RuntimeCache.java:86
org.jruby.ast.executable.AbstractScript.getBlockBody190(ThreadContext, StaticScope, String) AbstractScript.java:123
rubyjit.#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594.__file__(#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594, ThreadContext, IRubyObject, Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/activerecord-4.1.9/lib/active_record/attribute_methods/read.rb:87
rubyjit.#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594.__file__(ThreadContext, IRubyObject, Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/activerecord-4.1.9/lib/active_record/attribute_methods/read.rb
org.jruby.internal.runtime.methods.JittedMethod.call(ThreadContext, IRubyObject, RubyModule, String) JittedMethod.java:141
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String) DefaultMethod.java:185
org.jruby.internal.runtime.methods.AliasMethod.call(ThreadContext, IRubyObject, RubyModule, String) AliasMethod.java:56
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject) CachingCallSite.java:134
rubyjit.Killbill::Plugin::ActiveMerchant::ActiveRecord::Response$$to_transaction_info_plugin_6aecfa18f3ceb44c4e4e3def12ee1f5e175a0b181620780594.__file__(Killbill::Plugin::ActiveMerchant::ActiveRecord::Response$$to_transaction_info_plugin_6aecfa18f3ceb44c4e4e3def12ee1f5e175a0b181620780594, ThreadContext, IRubyObject, IRubyObject[], Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/killbill-3.2.2/lib/killbill/helpers/active_merchant/active_record/models/response.rb:123
rubyjit.Killbill::Plugin::ActiveMerchant::ActiveRecord::Response$$to_transaction_info_plugin_6aecfa18f3ceb44c4e4e3def12ee1f5e175a0b181620780594.__file__(ThreadContext, IRubyObject, IRubyObject[], Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/killbill-3.2.2/lib/killbill/helpers/active_merchant/active_record/models/response.rb
org.jruby.ast.executable.AbstractScript.__file__(ThreadContext, IRubyObject, IRubyObject, Block) AbstractScript.java:42
org.jruby.internal.runtime.methods.JittedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject) JittedMethod.java:181
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(IRubyObject, RubyClass, ThreadContext, IRubyObject, IRubyObject) CachingCallSite.java:326
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject) CachingCallSite.java:170
org.jruby.ast.CallOneArgNode.interpret(Ruby, ThreadContext, IRubyObject, Block) CallOneArgNode.java:57
org.jruby.ast.DAsgnNode.interpret(Ruby, ThreadContext, IRubyObject, Block) DAsgnNode.java:110
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(Ruby, ThreadContext, String, int, Node, String, IRubyObject, Block) ASTInterpreter.java:112
org.jruby.runtime.Interpreted19Block.evalBlockBody(ThreadContext, Binding, IRubyObject) Interpreted19Block.java:206
org.jruby.runtime.Interpreted19Block.yield(ThreadContext, IRubyObject, IRubyObject, RubyModule, boolean, Binding, Block$Type, Block) Interpreted19Block.java:194
org.jruby.runtime.Interpreted19Block.yield(ThreadContext, IRubyObject, IRubyObject, RubyModule, boolean, Binding, Block$Type) Interpreted19Block.java:177
org.jruby.runtime.BlockBody.call(ThreadContext, IRubyObject, IRubyObject, Binding, Block$Type) BlockBody.java:131
org.jruby.runtime.Block.call(ThreadContext, IRubyObject, IRubyObject) Block.java:123
org.jruby.RubyEnumerable$EachWithIndex.call(ThreadContext, IRubyObject[], Block) RubyEnumerable.java:933
org.jruby.runtime.CallBlock.yield(ThreadContext, IRubyObject, Binding, Block$Type) CallBlock.java:90
org.jruby.runtime.Block.yield(ThreadContext, IRubyObject) Block.java:142
org.jruby.RubyArray.eachCommon(ThreadContext, Block) RubyArray.java:1606
org.jruby.RubyArray.each(ThreadContext, Block) RubyArray.java:1613
org.jruby.RubyArray$INVOKER$i$0$0$each.call(ThreadContext, IRubyObject, RubyModule, String, Block) RubyArray$INVOKER$i$0$0$each.gen
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) JavaMethod.java:458
org.jruby.RubyClass.finvoke(ThreadContext, IRubyObject, String, IRubyObject[], Block) RubyClass.java:549
org.jruby.runtime.Helpers.invoke(ThreadContext, IRubyObject, String, IRubyObject[], Block) Helpers.java:496
org.jruby.RubyEnumerable.callEach(Ruby, ThreadContext, IRubyObject, IRubyObject[], Arity, BlockCallback) RubyEnumerable.java:114
org.jruby.RubyEnumerable.each_with_indexCommon19(ThreadContext, IRubyObject, Block, IRubyObject[]) RubyEnumerable.java:956
org.jruby.RubyEnumerable.each_with_index19(ThreadContext, IRubyObject, IRubyObject[], Block) RubyEnumerable.java:977
org.jruby.RubyEnumerable$INVOKER$s$0$0$each_with_index19.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) RubyEnumerable$INVOKER$s$0$0$each_with_index19.gen
org.jruby.internal.runtime.methods.DynamicMethod.call(ThreadContext, IRubyObject, RubyModule, String, Block) DynamicMethod.java:202
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(IRubyObject, RubyClass, Block, ThreadContext, IRubyObject) CachingCallSite.java:316
org.jruby.runtime.callsite.CachingCallSite.callBlock(ThreadContext, IRubyObject, IRubyObject, Block) CachingCallSite.java:145
org.jruby.runtime.callsite.CachingCallSite.callIter(ThreadContext, IRubyObject, IRubyObject, Block) CachingCallSite.java:154
org.jruby.ast.CallNoArgBlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) CallNoArgBlockNode.java:64
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject, IRubyObject, IRubyObject) InterpretedMethod.java:268
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject, IRubyObject, IRubyObject) DefaultMethod.java:230
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(IRubyObject, RubyClass, ThreadContext, IRubyObject, IRubyObject, IRubyObject, IRubyObject) CachingCallSite.java:366
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject, IRubyObject, IRubyObject) CachingCallSite.java:238
org.jruby.ast.FCallThreeArgNode.interpret(Ruby, ThreadContext, IRubyObject, Block) FCallThreeArgNode.java:40
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) InterpretedMethod.java:126
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) DefaultMethod.java:173
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject[]) CachingCallSite.java:70
org.jruby.ast.FCallManyArgsNode.interpret(Ruby, ThreadContext, IRubyObject, Block) FCallManyArgsNode.java:60
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) DefaultMethod.java:164
org.jruby.runtime.callsite.SuperCallSite.callBlock(ThreadContext, IRubyObject, IRubyObject, IRubyObject[], Block) SuperCallSite.java:68
org.jruby.runtime.callsite.SuperCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject[], Block) SuperCallSite.java:75
org.jruby.runtime.callsite.SuperCallSite.callVarargs(ThreadContext, IRubyObject, IRubyObject, IRubyObject[], Block) SuperCallSite.java:111
org.jruby.ast.SuperNode.interpret(Ruby, ThreadContext, IRubyObject, Block) SuperNode.java:115
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) InterpretedMethod.java:126
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) DefaultMethod.java:173
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject[]) CachingCallSite.java:70
org.jruby.ast.CallManyArgsNode.interpret(Ruby, ThreadContext, IRubyObject, Block) CallManyArgsNode.java:59
org.jruby.ast.LocalAsgnNode.interpret(Ruby, ThreadContext, IRubyObject, Block) LocalAsgnNode.java:123
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.ast.RescueNode.executeBody(Ruby, ThreadContext, IRubyObject, Block) RescueNode.java:221
org.jruby.ast.RescueNode.interpret(Ruby, ThreadContext, IRubyObject, Block) RescueNode.java:116
org.jruby.ast.EnsureNode.interpret(Ruby, ThreadContext, IRubyObject, Block) EnsureNode.java:96
org.jruby.ast.BeginNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BeginNode.java:83
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) DefaultMethod.java:164
Killbill$$Plugin$$Api$$PaymentPluginApi_1487103343.purchasePayment(UUID, UUID, UUID, UUID, BigDecimal, Currency, Iterable, CallContext) Killbill$$Plugin$$Api$$PaymentPluginApi_1487103343.gen:13
org.killbill.billing.osgi.bundles.jruby.JRubyPaymentPlugin$3.doCall(Ruby) JRubyPaymentPlugin.java:98
org.killbill.billing.osgi.bundles.jruby.JRubyPaymentPlugin$3.doCall(Ruby) JRubyPaymentPlugin.java:95
org.killbill.billing.osgi.bundles.jruby.JRubyPlugin.callWithRuntimeAndChecking(JRubyPlugin$PluginCallback) JRubyPlugin.java:241
org.killbill.billing.osgi.bundles.jruby.JRubyPaymentPlugin.purchasePayment(UUID, UUID, UUID, UUID, BigDecimal, Currency, Iterable, CallContext) JRubyPaymentPlugin.java:95
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:606
org.killbill.billing.osgi.ContextClassLoaderHelper$1$1.execute() ContextClassLoaderHelper.java:83
org.killbill.commons.profiling.Profiling.executeWithProfiling(ProfilingFeature$ProfilingFeatureType, String, Profiling$WithProfilingCallback) Profiling.java:37
org.killbill.billing.osgi.ContextClassLoaderHelper$1.invoke(Object, Method, Object[]) ContextClassLoaderHelper.java:80
com.sun.proxy.$Proxy160.purchasePayment(UUID, UUID, UUID, UUID, BigDecimal, Currency, Iterable, CallContext)
org.killbill.billing.payment.core.sm.PurchaseOperation.doCallSpecificOperationCallback() PurchaseOperation.java:42
org.killbill.billing.payment.core.sm.PaymentOperation.doOperation() PaymentOperation.java:169
org.killbill.billing.payment.core.sm.PaymentOperation.doSimpleOperationCallback() PaymentOperation.java:154
org.killbill.billing.payment.core.sm.PaymentOperation.access$000(PaymentOperation) PaymentOperation.java:50
org.killbill.billing.payment.core.sm.PaymentOperation$2.doOperation() PaymentOperation.java:146
org.killbill.billing.payment.core.sm.PaymentOperation$2.doOperation() PaymentOperation.java:143
org.killbill.billing.payment.core.ProcessorBase$WithAccountLock.processAccountWithLock(GlobalLocker, String, ProcessorBase$WithAccountLockCallback) ProcessorBase.java:219
org.killbill.billing.payment.core.ProcessorBase$CallableWithAccountLock.call() ProcessorBase.java:208
org.killbill.billing.payment.core.ProcessorBase$CallableWithAccountLock.call() ProcessorBase.java:192
java.lang.Thread.run() Thread.java:745

and another handful here:

Plugin-th-154 [BLOCKED]
java.lang.ClassLoader.loadClass(String, boolean) ClassLoader.java:407
java.lang.ClassLoader.loadClass(String) ClassLoader.java:358
rubyjit$#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594$block_0$RUBY$__file__.<init>(Object) rubyjit$#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594$block_0$RUBY$__file__
java.lang.reflect.Constructor.newInstance(Object[]) Constructor.java:526
org.jruby.internal.runtime.methods.InvocationMethodFactory.getBlockCallback19(String, String, int, Object) InvocationMethodFactory.java:1004
org.jruby.runtime.Helpers.createBlockCallback19(Object, String, String, int) Helpers.java:154
org.jruby.runtime.Helpers.createCompiledBlockBody19(ThreadContext, Object, String, int, StaticScope, boolean, int, String, int, boolean, String) Helpers.java:244
org.jruby.runtime.Helpers.createCompiledBlockBody19(ThreadContext, Object, StaticScope, String) Helpers.java:236
org.jruby.ast.executable.RuntimeCache.createBlockBody19(Object, ThreadContext, StaticScope, int, String) RuntimeCache.java:452
org.jruby.ast.executable.RuntimeCache.getBlockBody19(Object, ThreadContext, StaticScope, int, String) RuntimeCache.java:86
org.jruby.ast.executable.AbstractScript.getBlockBody190(ThreadContext, StaticScope, String) AbstractScript.java:123
rubyjit.#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594.__file__(#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594, ThreadContext, IRubyObject, Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/activerecord-4.1.9/lib/active_record/attribute_methods/read.rb:87
rubyjit.#<ActiveRecord::AttributeMethods::GeneratedAttributeMethods:0x6aea8985>$$__temp__166737f527563757c647f5374727565647f5d616473686_8ed3867e39d68c30e5e535f926d961e5b5ceb2cf1620780594.__file__(ThreadContext, IRubyObject, Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/activerecord-4.1.9/lib/active_record/attribute_methods/read.rb
org.jruby.internal.runtime.methods.JittedMethod.call(ThreadContext, IRubyObject, RubyModule, String) JittedMethod.java:141
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String) DefaultMethod.java:185
org.jruby.internal.runtime.methods.AliasMethod.call(ThreadContext, IRubyObject, RubyModule, String) AliasMethod.java:56
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject) CachingCallSite.java:134
rubyjit.Killbill::Plugin::ActiveMerchant::ActiveRecord::Response$$to_transaction_info_plugin_6aecfa18f3ceb44c4e4e3def12ee1f5e175a0b181620780594.__file__(Killbill::Plugin::ActiveMerchant::ActiveRecord::Response$$to_transaction_info_plugin_6aecfa18f3ceb44c4e4e3def12ee1f5e175a0b181620780594, ThreadContext, IRubyObject, IRubyObject[], Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/killbill-3.2.2/lib/killbill/helpers/active_merchant/active_record/models/response.rb:123
rubyjit.Killbill::Plugin::ActiveMerchant::ActiveRecord::Response$$to_transaction_info_plugin_6aecfa18f3ceb44c4e4e3def12ee1f5e175a0b181620780594.__file__(ThreadContext, IRubyObject, IRubyObject[], Block) /var/tmp/bundles/plugins/ruby/killbill-litle/1.10.0/gems/gems/killbill-3.2.2/lib/killbill/helpers/active_merchant/active_record/models/response.rb
org.jruby.ast.executable.AbstractScript.__file__(ThreadContext, IRubyObject, IRubyObject, Block) AbstractScript.java:42
org.jruby.internal.runtime.methods.JittedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject) JittedMethod.java:181
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject) CachingCallSite.java:168
org.jruby.ast.CallOneArgNode.interpret(Ruby, ThreadContext, IRubyObject, Block) CallOneArgNode.java:57
org.jruby.ast.DAsgnNode.interpret(Ruby, ThreadContext, IRubyObject, Block) DAsgnNode.java:110
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(Ruby, ThreadContext, String, int, Node, String, IRubyObject, Block) ASTInterpreter.java:112
org.jruby.runtime.Interpreted19Block.evalBlockBody(ThreadContext, Binding, IRubyObject) Interpreted19Block.java:206
org.jruby.runtime.Interpreted19Block.yield(ThreadContext, IRubyObject, IRubyObject, RubyModule, boolean, Binding, Block$Type, Block) Interpreted19Block.java:194
org.jruby.runtime.Interpreted19Block.yield(ThreadContext, IRubyObject, IRubyObject, RubyModule, boolean, Binding, Block$Type) Interpreted19Block.java:177
org.jruby.runtime.BlockBody.call(ThreadContext, IRubyObject, IRubyObject, Binding, Block$Type) BlockBody.java:131
org.jruby.runtime.Block.call(ThreadContext, IRubyObject, IRubyObject) Block.java:123
org.jruby.RubyEnumerable$EachWithIndex.call(ThreadContext, IRubyObject[], Block) RubyEnumerable.java:933
org.jruby.runtime.CallBlock.yield(ThreadContext, IRubyObject, Binding, Block$Type) CallBlock.java:90
org.jruby.runtime.Block.yield(ThreadContext, IRubyObject) Block.java:142
org.jruby.RubyArray.eachCommon(ThreadContext, Block) RubyArray.java:1606
org.jruby.RubyArray.each(ThreadContext, Block) RubyArray.java:1613
org.jruby.RubyArray$INVOKER$i$0$0$each.call(ThreadContext, IRubyObject, RubyModule, String, Block) RubyArray$INVOKER$i$0$0$each.gen
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) JavaMethod.java:458
org.jruby.RubyClass.finvoke(ThreadContext, IRubyObject, String, IRubyObject[], Block) RubyClass.java:549
org.jruby.runtime.Helpers.invoke(ThreadContext, IRubyObject, String, IRubyObject[], Block) Helpers.java:496
org.jruby.RubyEnumerable.callEach(Ruby, ThreadContext, IRubyObject, IRubyObject[], Arity, BlockCallback) RubyEnumerable.java:114
org.jruby.RubyEnumerable.each_with_indexCommon19(ThreadContext, IRubyObject, Block, IRubyObject[]) RubyEnumerable.java:956
org.jruby.RubyEnumerable.each_with_index19(ThreadContext, IRubyObject, IRubyObject[], Block) RubyEnumerable.java:977
org.jruby.RubyEnumerable$INVOKER$s$0$0$each_with_index19.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) RubyEnumerable$INVOKER$s$0$0$each_with_index19.gen
org.jruby.internal.runtime.methods.DynamicMethod.call(ThreadContext, IRubyObject, RubyModule, String, Block) DynamicMethod.java:202
org.jruby.runtime.callsite.CachingCallSite.callBlock(ThreadContext, IRubyObject, IRubyObject, Block) CachingCallSite.java:143
org.jruby.runtime.callsite.CachingCallSite.callIter(ThreadContext, IRubyObject, IRubyObject, Block) CachingCallSite.java:154
org.jruby.ast.CallNoArgBlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) CallNoArgBlockNode.java:64
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject, IRubyObject, IRubyObject) InterpretedMethod.java:268
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject, IRubyObject, IRubyObject) DefaultMethod.java:230
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject, IRubyObject, IRubyObject) CachingCallSite.java:236
org.jruby.ast.FCallThreeArgNode.interpret(Ruby, ThreadContext, IRubyObject, Block) FCallThreeArgNode.java:40
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) InterpretedMethod.java:126
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) DefaultMethod.java:173
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject[]) CachingCallSite.java:70
org.jruby.ast.FCallManyArgsNode.interpret(Ruby, ThreadContext, IRubyObject, Block) FCallManyArgsNode.java:60
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) DefaultMethod.java:164
org.jruby.runtime.callsite.SuperCallSite.callBlock(ThreadContext, IRubyObject, IRubyObject, IRubyObject[], Block) SuperCallSite.java:68
org.jruby.runtime.callsite.SuperCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject[], Block) SuperCallSite.java:75
org.jruby.runtime.callsite.SuperCallSite.callVarargs(ThreadContext, IRubyObject, IRubyObject, IRubyObject[], Block) SuperCallSite.java:111
org.jruby.ast.SuperNode.interpret(Ruby, ThreadContext, IRubyObject, Block) SuperNode.java:115
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) InterpretedMethod.java:126
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[]) DefaultMethod.java:173
org.jruby.runtime.callsite.CachingCallSite.call(ThreadContext, IRubyObject, IRubyObject, IRubyObject[]) CachingCallSite.java:70
org.jruby.ast.CallManyArgsNode.interpret(Ruby, ThreadContext, IRubyObject, Block) CallManyArgsNode.java:59
org.jruby.ast.LocalAsgnNode.interpret(Ruby, ThreadContext, IRubyObject, Block) LocalAsgnNode.java:123
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.ast.RescueNode.executeBody(Ruby, ThreadContext, IRubyObject, Block) RescueNode.java:221
org.jruby.ast.RescueNode.interpret(Ruby, ThreadContext, IRubyObject, Block) RescueNode.java:116
org.jruby.ast.EnsureNode.interpret(Ruby, ThreadContext, IRubyObject, Block) EnsureNode.java:96
org.jruby.ast.BeginNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BeginNode.java:83
org.jruby.ast.NewlineNode.interpret(Ruby, ThreadContext, IRubyObject, Block) NewlineNode.java:105
org.jruby.ast.BlockNode.interpret(Ruby, ThreadContext, IRubyObject, Block) BlockNode.java:71
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(Ruby, ThreadContext, String, int, RubyModule, Node, String, IRubyObject, Block, boolean) ASTInterpreter.java:74
org.jruby.internal.runtime.methods.InterpretedMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) InterpretedMethod.java:112
org.jruby.internal.runtime.methods.DefaultMethod.call(ThreadContext, IRubyObject, RubyModule, String, IRubyObject[], Block) DefaultMethod.java:164
Killbill$$Plugin$$Api$$PaymentPluginApi_1487103343.purchasePayment(UUID, UUID, UUID, UUID, BigDecimal, Currency, Iterable, CallContext) Killbill$$Plugin$$Api$$PaymentPluginApi_1487103343.gen:13
org.killbill.billing.osgi.bundles.jruby.JRubyPaymentPlugin$3.doCall(Ruby) JRubyPaymentPlugin.java:98
org.killbill.billing.osgi.bundles.jruby.JRubyPaymentPlugin$3.doCall(Ruby) JRubyPaymentPlugin.java:95
org.killbill.billing.osgi.bundles.jruby.JRubyPlugin.callWithRuntimeAndChecking(JRubyPlugin$PluginCallback) JRubyPlugin.java:241
org.killbill.billing.osgi.bundles.jruby.JRubyPaymentPlugin.purchasePayment(UUID, UUID, UUID, UUID, BigDecimal, Currency, Iterable, CallContext) JRubyPaymentPlugin.java:95
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:606
org.killbill.billing.osgi.ContextClassLoaderHelper$1$1.execute() ContextClassLoaderHelper.java:83
org.killbill.commons.profiling.Profiling.executeWithProfiling(ProfilingFeature$ProfilingFeatureType, String, Profiling$WithProfilingCallback) Profiling.java:37
org.killbill.billing.osgi.ContextClassLoaderHelper$1.invoke(Object, Method, Object[]) ContextClassLoaderHelper.java:80
com.sun.proxy.$Proxy160.purchasePayment(UUID, UUID, UUID, UUID, BigDecimal, Currency, Iterable, CallContext)
org.killbill.billing.payment.core.sm.PurchaseOperation.doCallSpecificOperationCallback() PurchaseOperation.java:42
org.killbill.billing.payment.core.sm.PaymentOperation.doOperation() PaymentOperation.java:169
org.killbill.billing.payment.core.sm.PaymentOperation.doSimpleOperationCallback() PaymentOperation.java:154
org.killbill.billing.payment.core.sm.PaymentOperation.access$000(PaymentOperation) PaymentOperation.java:50
org.killbill.billing.payment.core.sm.PaymentOperation$2.doOperation() PaymentOperation.java:146
org.killbill.billing.payment.core.sm.PaymentOperation$2.doOperation() PaymentOperation.java:143
org.killbill.billing.payment.core.ProcessorBase$WithAccountLock.processAccountWithLock(GlobalLocker, String, ProcessorBase$WithAccountLockCallback) ProcessorBase.java:219
org.killbill.billing.payment.core.ProcessorBase$CallableWithAccountLock.call() ProcessorBase.java:208
org.killbill.billing.payment.core.ProcessorBase$CallableWithAccountLock.call() ProcessorBase.java:192
java.lang.Thread.run() Thread.java:745