erlanglab / erlangpl

Tool for developers working with systems running on the Erlang VM (BEAM). It helps with performance analysis.
http://www.erlang.pl/
Apache License 2.0
545 stars 40 forks source link

Crashing under large load #50

Open mdaiter opened 7 years ago

mdaiter commented 7 years ago

My app uses about 1000 processes per machine for controlling and distributing tasks across a cluster. epl_st timeouts when creating the supervisor tree to send to the web GUI. Is there either a way I can specify only certain processes to be tracked, or (better) just extend that timeout and accept a bit of lag?

michalslaski commented 7 years ago

Does the dashboard with metrics render fine, when you connect erlangpl to your app? If so, what throughput is reported?

We got reports from users about epl_st crashing while waiting for a response from which_children/1 - is it the kind of crash you're getting too?

mdaiter commented 7 years ago

The dashboard renders perfectly fine when I connect erlangpl to my app. The throughput is initially reported as N/A.

This is the crash I'm receiving as well. I'll post my crash in a followup report.

NB: I am also using riak_core and poolboy in my project, both of which are incredibly heavy-weight.

mdaiter commented 7 years ago

@michalslaski as promised, here's the log:

Exec: /home/nomoko/Code/erlangpl/rel/erlangpl/erts-7.3.1.2/bin/erlexec -boot /home/nomoko/Code/erlangpl/rel/erlangpl/releases/0.6.1/erlangpl -mode embedded -config /home/nomoko/Code/erlangpl/rel/erlangpl/releases/0.6.1/sys.config -args_file /home/nomoko/Code/erlangpl/rel/erlangpl/releases/0.6.1/vm.args -- console node=foo@192.168.1.26 cookie=erlang
Root: /home/nomoko/Code/erlangpl/rel/erlangpl
Erlang/OTP 18 [erts-7.3.1.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:10] [hipe] [kernel-poll:false]

Visit http://localhost:37575/
Eshell V7.3.1.2  (abort with ^G)
(erlangpl@127.0.0.1)1> ERROR: timed out while collecting data from node

=ERROR REPORT==== 9-May-2017::14:09:41 ===
** Generic server epl_st terminating 
** Last message in was {data,
                           {'foo@192.168.1.26',{1494,331776,600790}},
                           [{process_count,1327},
                            {memory_total,122069840},
                            {spawn,
                                [{<13484.3096.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,579027}},
                                 {<13484.3097.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1096126227998177188652763624537212264741949407232,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,579059}},
                                 {<13484.3098.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,579683}},
                                 {<13484.3099.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1096126227998177188652763624537212264741949407232,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,579703}},
                                 {<13484.3100.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,580205}},
                                 {<13484.3101.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1096126227998177188652763624537212264741949407232,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,580238}},
                                 {<13484.3102.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,580788}},
                                 {<13484.3103.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1118962191081472546749696200048404186924073353216,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,580804}},
                                 {<13484.3104.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,581303}},
                                 {<13484.3105.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1118962191081472546749696200048404186924073353216,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,581326}},
                                 {<13484.3106.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,581853}},
                                 {<13484.3107.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1118962191081472546749696200048404186924073353216,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,581875}},
                                 {<13484.3108.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,582429}},
                                 {<13484.3109.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1118962191081472546749696200048404186924073353216,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,582452}},
                                 {<13484.3110.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,582967}},
                                 {<13484.3111.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1118962191081472546749696200048404186924073353216,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,582997}},
                                 {<13484.3112.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,583525}},
                                 {<13484.3113.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1141798154164767904846628775559596109106197299200,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,583541}},
                                 {<13484.3114.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,584055}},
                                 {<13484.3115.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1141798154164767904846628775559596109106197299200,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,584078}},
                                 {<13484.3116.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,584550}},
                                 {<13484.3117.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1141798154164767904846628775559596109106197299200,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,584566}},
                                 {<13484.3118.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,585062}},
                                 {<13484.3119.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1141798154164767904846628775559596109106197299200,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,585088}},
                                 {<13484.3120.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,585606}},
                                 {<13484.3121.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1141798154164767904846628775559596109106197299200,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,585628}},
                                 {<13484.3122.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,586148}},
                                 {<13484.3123.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1164634117248063262943561351070788031288321245184,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,586166}},
                                 {<13484.3124.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,586643}},
                                 {<13484.3125.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1164634117248063262943561351070788031288321245184,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,586665}},
                                 {<13484.3126.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,587130}},
                                 {<13484.3127.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1164634117248063262943561351070788031288321245184,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,587146}},
                                 {<13484.3128.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,587598}},
                                 {<13484.3129.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1164634117248063262943561351070788031288321245184,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,587616}},
                                 {<13484.3130.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,588059}},
                                 {<13484.3131.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1164634117248063262943561351070788031288321245184,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,588076}},
                                 {<13484.3132.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,588559}},
                                 {<13484.3133.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1187470080331358621040493926581979953470445191168,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,588576}},
                                 {<13484.3134.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,589044}},
                                 {<13484.3135.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1187470080331358621040493926581979953470445191168,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,589069}},
                                 {<13484.3136.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,589515}},
                                 {<13484.3137.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1187470080331358621040493926581979953470445191168,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,589541}},
                                 {<13484.3138.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,589979}},
                                 {<13484.3139.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1187470080331358621040493926581979953470445191168,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,589999}},
                                 {<13484.3140.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,590472}},
                                 {<13484.3141.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1187470080331358621040493926581979953470445191168,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,590492}},
                                 {<13484.3142.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,590975}},
                                 {<13484.3143.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1210306043414653979137426502093171875652569137152,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,590994}},
                                 {<13484.3144.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,591449}},
                                 {<13484.3145.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1210306043414653979137426502093171875652569137152,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,591490}},
                                 {<13484.3146.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,592015}},
                                 {<13484.3147.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1210306043414653979137426502093171875652569137152,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,592045}},
                                 {<13484.3148.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,592556}},
                                 {<13484.3149.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1210306043414653979137426502093171875652569137152,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,592585}},
                                 {<13484.3150.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,593102}},
                                 {<13484.3151.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1210306043414653979137426502093171875652569137152,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,593129}},
                                 {<13484.3152.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,593599}},
                                 {<13484.3153.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1233142006497949337234359077604363797834693083136,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,593621}},
                                 {<13484.3154.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,594077}},
                                 {<13484.3155.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1233142006497949337234359077604363797834693083136,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,594094}},
                                 {<13484.3156.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,594554}},
                                 {<13484.3157.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1233142006497949337234359077604363797834693083136,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,594582}},
                                 {<13484.3158.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,594986}},
                                 {<13484.3159.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1233142006497949337234359077604363797834693083136,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,595009}},
                                 {<13484.3160.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,595473}},
                                 {<13484.3161.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1233142006497949337234359077604363797834693083136,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,595496}},
                                 {<13484.3162.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,595977}},
                                 {<13484.3163.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1255977969581244695331291653115555720016817029120,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,596002}},
                                 {<13484.3164.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,596445}},
                                 {<13484.3165.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1255977969581244695331291653115555720016817029120,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,596472}},
                                 {<13484.3166.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,596941}},
                                 {<13484.3167.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1255977969581244695331291653115555720016817029120,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,596966}},
                                 {<13484.3168.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,597473}},
                                 {<13484.3169.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1255977969581244695331291653115555720016817029120,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,597492}},
                                 {<13484.3170.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,597966}},
                                 {<13484.3171.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1255977969581244695331291653115555720016817029120,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,597992}},
                                 {<13484.3172.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,598473}},
                                 {<13484.3173.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1278813932664540053428224228626747642198940975104,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,598491}},
                                 {<13484.3174.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,598937}},
                                 {<13484.3175.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1278813932664540053428224228626747642198940975104,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,598958}},
                                 {<13484.3176.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,599389}},
                                 {<13484.3177.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1278813932664540053428224228626747642198940975104,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,599415}},
                                 {<13484.3178.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,599859}},
                                 {<13484.3179.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1278813932664540053428224228626747642198940975104,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,599876}},
                                 {<13484.3180.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,600305}},
                                 {<13484.3181.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1278813932664540053428224228626747642198940975104,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,600325}},
                                 {<13484.3182.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,600790}},
                                 {<13484.3183.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1301649895747835411525156804137939564381064921088,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,600806}},
                                 {<13484.3184.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,601328}},
                                 {<13484.3185.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1301649895747835411525156804137939564381064921088,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,601349}},
                                 {<13484.3186.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,601859}},
                                 {<13484.3187.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1301649895747835411525156804137939564381064921088,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,601885}},
                                 {<13484.3188.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,602348}},
                                 {<13484.3189.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1301649895747835411525156804137939564381064921088,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,602370}},
                                 {<13484.3190.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,602813}},
                                 {<13484.3191.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1301649895747835411525156804137939564381064921088,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,602830}},
                                 {<13484.3192.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,603296}},
                                 {<13484.3193.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1324485858831130769622089379649131486563188867072,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,603322}},
                                 {<13484.3194.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,603760}},
                                 {<13484.3195.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1324485858831130769622089379649131486563188867072,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,603780}},
                                 {<13484.3196.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,604211}},
                                 {<13484.3197.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1324485858831130769622089379649131486563188867072,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,604237}},
                                 {<13484.3198.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,604667}},
                                 {<13484.3199.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1324485858831130769622089379649131486563188867072,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,604690}},
                                 {<13484.3200.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,605144}},
                                 {<13484.3201.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1324485858831130769622089379649131486563188867072,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,605168}},
                                 {<13484.3202.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,605645}},
                                 {<13484.3203.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1347321821914426127719021955160323408745312813056,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,605665}},
                                 {<13484.3204.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,606102}},
                                 {<13484.3205.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1347321821914426127719021955160323408745312813056,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,606124}},
                                 {<13484.3206.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,606566}},
                                 {<13484.3207.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1347321821914426127719021955160323408745312813056,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,606591}},
                                 {<13484.3208.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,607040}},
                                 {<13484.3209.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1347321821914426127719021955160323408745312813056,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,607064}},
                                 {<13484.3210.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,607504}},
                                 {<13484.3211.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1347321821914426127719021955160323408745312813056,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,607524}},
                                 {<13484.3212.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,607988}},
                                 {<13484.3213.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1370157784997721485815954530671515330927436759040,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,608003}},
                                 {<13484.3214.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,608502}},
                                 {<13484.3215.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1370157784997721485815954530671515330927436759040,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,608523}},
                                 {<13484.3216.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,609023}},
                                 {<13484.3217.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1370157784997721485815954530671515330927436759040,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,609048}},
                                 {<13484.3218.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,609536}},
                                 {<13484.3219.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1370157784997721485815954530671515330927436759040,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,609563}},
                                 {<13484.3220.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,610033}},
                                 {<13484.3221.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1392993748081016843912887106182707253109560705024,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,610058}},
                                 {<13484.3222.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,610518}},
                                 {<13484.3223.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1370157784997721485815954530671515330927436759040,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,610546}},
                                 {<13484.3224.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,611057}},
                                 {<13484.3225.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1392993748081016843912887106182707253109560705024,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,611073}},
                                 {<13484.3226.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,611560}},
                                 {<13484.3227.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1392993748081016843912887106182707253109560705024,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,611575}},
                                 {<13484.3228.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,612000}},
                                 {<13484.3229.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1392993748081016843912887106182707253109560705024,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,612019}},
                                 {<13484.3230.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,612413}},
                                 {<13484.3231.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1415829711164312202009819681693899175291684651008,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,612443}},
                                 {<13484.3232.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,612914}},
                                 {<13484.3233.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1392993748081016843912887106182707253109560705024,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,612933}},
                                 {<13484.3234.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,613421}},
                                 {<13484.3235.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1415829711164312202009819681693899175291684651008,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,613443}},
                                 {<13484.3236.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,613892}},
                                 {<13484.3237.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1415829711164312202009819681693899175291684651008,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,613916}},
                                 {<13484.3238.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,614342}},
                                 {<13484.3239.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1415829711164312202009819681693899175291684651008,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,614359}},
                                 {<13484.3240.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,614782}},
                                 {<13484.3241.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_features_vnode,
                                         1438665674247607560106752257205091097473808596992,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,614807}},
                                 {<13484.3242.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,615225}},
                                 {<13484.3243.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1415829711164312202009819681693899175291684651008,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,615250}},
                                 {<13484.3244.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,615709}},
                                 {<13484.3245.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_globalsfm_vnode,
                                         1438665674247607560106752257205091097473808596992,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,615726}},
                                 {<13484.3246.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,616128}},
                                 {<13484.3247.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_hypersfm_vnode,
                                         1438665674247607560106752257205091097473808596992,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,616172}},
                                 {<13484.3248.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,616577}},
                                 {<13484.3249.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_io_vnode,
                                         1438665674247607560106752257205091097473808596992,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,616598}},
                                 {<13484.3250.0>,
                                  {erlang,apply,
                                      [#Fun<riak_core_util.6.4247070>,[]]},
                                  {1494,331771,617055}},
                                 {<13484.3251.0>,
                                  {proc_lib,init_p,
                                      [riak_core_vnode_sup,
                                       [riak_core_sup,<13484.2185.0>],
                                       gen,init_it,
                                       [gen_fsm,<13484.2189.0>,<13484.2189.0>,
                                        riak_core_vnode,
                                        [foo_matching_image_collection_vnode,
                                         1438665674247607560106752257205091097473808596992,
                                         0,undefined],
                                        []]]},
                                  {1494,331771,617071}},
                                 {<13484.3252.0>,
                                  {erlang,apply,
                                      [#Fun<exometer_cpu.0.79884867>,[]]},
                                  {1494,331776,486794}}]},
                            {exit,
                                [{<13484.2599.0>,normal,{1494,331771,615571}},
                                 {<13484.2600.0>,normal,{1494,331771,616164}},
                                 {<13484.2601.0>,normal,{1494,331771,616562}},
                                 {<13484.2602.0>,normal,{1494,331771,617028}},
                                 {<13484.2603.0>,normal,{1494,331771,617523}},
                                 {<13484.3017.0>,normal,{1494,331771,761986}},
                                 {<13484.3093.0>,normal,{1494,331771,578944}},
                                 {<13484.3096.0>,normal,{1494,331771,579623}},
                                 {<13484.3098.0>,normal,{1494,331771,580145}},
                                 {<13484.3100.0>,normal,{1494,331771,580723}},
                                 {<13484.3102.0>,normal,{1494,331771,581242}},
                                 {<13484.3104.0>,normal,{1494,331771,581775}},
                                 {<13484.3106.0>,normal,{1494,331771,582358}},
                                 {<13484.3108.0>,normal,{1494,331771,582905}},
                                 {<13484.3110.0>,normal,{1494,331771,583454}},
                                 {<13484.3112.0>,normal,{1494,331771,583983}},
                                 {<13484.3114.0>,normal,{1494,331771,584478}},
                                 {<13484.3116.0>,normal,{1494,331771,584986}},
                                 {<13484.3118.0>,normal,{1494,331771,585535}},
                                 {<13484.3120.0>,normal,{1494,331771,586087}},
                                 {<13484.3122.0>,normal,{1494,331771,586582}},
                                 {<13484.3124.0>,normal,{1494,331771,587066}},
                                 {<13484.3126.0>,normal,{1494,331771,587538}},
                                 {<13484.3128.0>,normal,{1494,331771,587997}},
                                 {<13484.3130.0>,normal,{1494,331771,588498}},
                                 {<13484.3132.0>,normal,{1494,331771,588984}},
                                 {<13484.3134.0>,normal,{1494,331771,589454}},
                                 {<13484.3136.0>,normal,{1494,331771,589918}},
                                 {<13484.3138.0>,normal,{1494,331771,590382}},
                                 {<13484.3140.0>,normal,{1494,331771,590893}},
                                 {<13484.3142.0>,normal,{1494,331771,591373}},
                                 {<13484.3144.0>,normal,{1494,331771,591925}},
                                 {<13484.3146.0>,normal,{1494,331771,592489}},
                                 {<13484.3148.0>,normal,{1494,331771,593021}},
                                 {<13484.3150.0>,normal,{1494,331771,593538}},
                                 {<13484.3152.0>,normal,{1494,331771,594015}},
                                 {<13484.3154.0>,normal,{1494,331771,594492}},
                                 {<13484.3156.0>,normal,{1494,331771,594925}},
                                 {<13484.3158.0>,normal,{1494,331771,595413}},
                                 {<13484.3160.0>,normal,{1494,331771,595914}},
                                 {<13484.3162.0>,normal,{1494,331771,596381}},
                                 {<13484.3164.0>,normal,{1494,331771,596857}},
                                 {<13484.3166.0>,normal,{1494,331771,597406}},
                                 {<13484.3168.0>,normal,{1494,331771,597891}},
                                 {<13484.3170.0>,normal,{1494,331771,598402}},
                                 {<13484.3172.0>,normal,{1494,331771,598865}},
                                 {<13484.3174.0>,normal,{1494,331771,599319}},
                                 {<13484.3176.0>,normal,{1494,331771,599778}},
                                 {<13484.3178.0>,normal,{1494,331771,600235}},
                                 {<13484.3180.0>,normal,{1494,331771,600717}},
                                 {<13484.3182.0>,normal,{1494,331771,601244}},
                                 {<13484.3184.0>,normal,{1494,331771,601757}},
                                 {<13484.3186.0>,normal,{1494,331771,602279}},
                                 {<13484.3188.0>,normal,{1494,331771,602744}},
                                 {<13484.3190.0>,normal,{1494,331771,603212}},
                                 {<13484.3192.0>,normal,{1494,331771,603689}},
                                 {<13484.3194.0>,normal,{1494,331771,604141}},
                                 {<13484.3196.0>,normal,{1494,331771,604598}},
                                 {<13484.3198.0>,normal,{1494,331771,605062}},
                                 {<13484.3200.0>,normal,{1494,331771,605576}},
                                 {<13484.3202.0>,normal,{1494,331771,606033}},
                                 {<13484.3204.0>,normal,{1494,331771,606495}},
                                 {<13484.3206.0>,normal,{1494,331771,606960}},
                                 {<13484.3208.0>,normal,{1494,331771,607434}},
                                 {<13484.3210.0>,normal,{1494,331771,607912}},
                                 {<13484.3212.0>,normal,{1494,331771,608427}},
                                 {<13484.3214.0>,normal,{1494,331771,608961}},
                                 {<13484.3216.0>,normal,{1494,331771,609472}},
                                 {<13484.3218.0>,normal,{1494,331771,609969}},
                                 {<13484.3220.0>,normal,{1494,331771,610438}},
                                 {<13484.3222.0>,normal,{1494,331771,610993}},
                                 {<13484.3224.0>,normal,{1494,331771,611488}},
                                 {<13484.3226.0>,normal,{1494,331771,611923}},
                                 {<13484.3228.0>,normal,{1494,331771,612339}},
                                 {<13484.3230.0>,normal,{1494,331771,612849}},
                                 {<13484.3232.0>,normal,{1494,331771,613350}},
                                 {<13484.3234.0>,normal,{1494,331771,613832}},
                                 {<13484.3236.0>,normal,{1494,331771,614284}},
                                 {<13484.3238.0>,normal,{1494,331771,614723}},
                                 {<13484.3240.0>,normal,{1494,331771,615164}},
                                 {<13484.3242.0>,normal,{1494,331771,615648}},
                                 {<13484.3244.0>,normal,{1494,331771,616022}},
                                 {<13484.3246.0>,normal,{1494,331771,616511}},
                                 {<13484.3248.0>,normal,{1494,331771,616993}},
                                 {<13484.3250.0>,normal,{1494,331771,617483}},
                                 {<13484.3252.0>,
                                  {sample,
                                      [{nprocs,606},
                                       {avg1,8},
                                       {avg5,46},
                                       {avg15,41}]},
                                  {1494,331776,488650}}]},
                            {send,
                                [{{exometer_report,<13484.2158.0>},0,79},
                                 {{global_name_server,<13484.1518.0>},0,1},
                                 {{global_name_server,<13484.3017.0>},0,1},
                                 {{riak_core_handoff_manager,<13484.3097.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3099.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3101.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3103.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3105.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3107.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3109.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3111.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3113.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3115.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3117.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3119.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3121.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3123.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3125.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3127.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3129.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3131.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3133.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3135.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3137.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3139.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3141.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3143.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3145.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3147.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3149.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3151.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3153.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3155.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3157.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3159.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3161.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3163.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3165.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3167.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3169.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3171.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3173.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3175.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3177.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3179.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3181.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3183.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3185.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3187.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3189.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3191.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3193.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3195.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3197.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3199.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3201.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3203.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3205.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3207.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3209.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3211.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3213.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3215.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3217.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3219.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3221.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3223.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3225.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3227.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3229.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3231.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3233.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3235.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3237.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3239.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3241.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3243.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3245.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3247.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3249.0>},
                                  0,1},
                                 {{riak_core_handoff_manager,<13484.3251.0>},
                                  0,1},
                                 {{riak_core_node_watcher_events,
                                      <13484.2213.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3097.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3099.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3101.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3103.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3105.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3107.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3109.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3111.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3113.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3115.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3117.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3119.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3121.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3123.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3125.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3127.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3129.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3131.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3133.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3135.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3137.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3139.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3141.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3143.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3145.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3147.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3149.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3151.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3153.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3155.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3159.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3161.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3163.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3165.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3167.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3169.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3171.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3173.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3175.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3177.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3179.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3181.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3183.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3185.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3187.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3189.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3191.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3193.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3195.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3197.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3199.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3201.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3203.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3205.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3207.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3209.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3211.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3213.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3215.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3217.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3219.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3221.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3223.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3225.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3231.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3233.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3235.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3237.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3239.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3241.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3243.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3247.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3249.0>},
                                  0,1},
                                 {{riak_core_vnode_manager,<13484.3251.0>},
                                  0,1},
                                 {{#Port<13484.4810>,<13484.2172.0>},0,3},
                                 {{<0.243.0>,<13484.1517.0>},0,2},
                                 {{<0.243.0>,<13484.1518.0>},0,1},
                                 {{<13484.1517.0>,<13484.1518.0>},4,0},
                                 {{<13484.1517.0>,<13484.3017.0>},1,0},
                                 {{<13484.1517.0>,
                                   {global_name_server,'erlangpl@127.0.0.1'}},
                                  2,0},
                                 {{<13484.1518.0>,
                                   {global_name_server,'erlangpl@127.0.0.1'}},
                                  2,0},
                                 {{<13484.1518.0>,
                                   {global_name_server,
                                       'foo@192.168.1.26'}},
                                  2,0},
                                 {{<13484.1524.0>,<13484.1526.0>},0,1},
                                 {{<13484.1524.0>,<13484.3013.0>},1,0},
                                 {{<13484.2158.0>,<13484.3093.0>},3,3},
                                 {{<13484.2158.0>,<13484.3096.0>},3,3},
                                 {{<13484.2158.0>,<13484.3098.0>},3,3},
                                 {{<13484.2158.0>,<13484.3100.0>},3,3},
                                 {{<13484.2158.0>,<13484.3102.0>},3,3},
                                 {{<13484.2158.0>,<13484.3104.0>},3,3},
                                 {{<13484.2158.0>,<13484.3106.0>},3,3},
                                 {{<13484.2158.0>,<13484.3108.0>},3,3},
                                 {{<13484.2158.0>,<13484.3110.0>},3,3},
                                 {{<13484.2158.0>,<13484.3112.0>},3,3},
                                 {{<13484.2158.0>,<13484.3114.0>},3,3},
                                 {{<13484.2158.0>,<13484.3116.0>},3,3},
                                 {{<13484.2158.0>,<13484.3118.0>},3,3},
                                 {{<13484.2158.0>,<13484.3120.0>},3,3},
                                 {{<13484.2158.0>,<13484.3122.0>},3,3},
                                 {{<13484.2158.0>,<13484.3124.0>},3,3},
                                 {{<13484.2158.0>,<13484.3126.0>},3,3},
                                 {{<13484.2158.0>,<13484.3128.0>},3,3},
                                 {{<13484.2158.0>,<13484.3130.0>},3,3},
                                 {{<13484.2158.0>,<13484.3132.0>},3,3},
                                 {{<13484.2158.0>,<13484.3134.0>},3,3},
                                 {{<13484.2158.0>,<13484.3136.0>},3,3},
                                 {{<13484.2158.0>,<13484.3138.0>},3,3},
                                 {{<13484.2158.0>,<13484.3140.0>},3,3},
                                 {{<13484.2158.0>,<13484.3142.0>},3,3},
                                 {{<13484.2158.0>,<13484.3144.0>},3,3},
                                 {{<13484.2158.0>,<13484.3146.0>},3,3},
                                 {{<13484.2158.0>,<13484.3148.0>},3,3},
                                 {{<13484.2158.0>,<13484.3150.0>},3,3},
                                 {{<13484.2158.0>,<13484.3152.0>},3,3},
                                 {{<13484.2158.0>,<13484.3154.0>},3,3},
                                 {{<13484.2158.0>,<13484.3156.0>},3,3},
                                 {{<13484.2158.0>,<13484.3158.0>},3,3},
                                 {{<13484.2158.0>,<13484.3160.0>},3,3},
                                 {{<13484.2158.0>,<13484.3162.0>},3,3},
                                 {{<13484.2158.0>,<13484.3164.0>},3,3},
                                 {{<13484.2158.0>,<13484.3166.0>},3,3},
                                 {{<13484.2158.0>,<13484.3168.0>},3,3},
                                 {{<13484.2158.0>,<13484.3170.0>},3,3},
                                 {{<13484.2158.0>,<13484.3172.0>},3,3},
                                 {{<13484.2158.0>,<13484.3174.0>},3,3},
                                 {{<13484.2158.0>,<13484.3176.0>},3,3},
                                 {{<13484.2158.0>,<13484.3178.0>},3,3},
                                 {{<13484.2158.0>,<13484.3180.0>},3,3},
                                 {{<13484.2158.0>,<13484.3182.0>},3,3},
                                 {{<13484.2158.0>,<13484.3184.0>},3,3},
                                 {{<13484.2158.0>,<13484.3186.0>},3,3},
                                 {{<13484.2158.0>,<13484.3188.0>},3,3},
                                 {{<13484.2158.0>,<13484.3190.0>},3,3},
                                 {{<13484.2158.0>,<13484.3192.0>},3,3},
                                 {{<13484.2158.0>,<13484.3194.0>},3,3},
                                 {{<13484.2158.0>,<13484.3196.0>},3,3},
                                 {{<13484.2158.0>,<13484.3198.0>},3,3},
                                 {{<13484.2158.0>,<13484.3200.0>},3,3},
                                 {{<13484.2158.0>,<13484.3202.0>},3,3},
                                 {{<13484.2158.0>,<13484.3204.0>},3,3},
                                 {{<13484.2158.0>,<13484.3206.0>},3,3},
                                 {{<13484.2158.0>,<13484.3208.0>},3,3},
                                 {{<13484.2158.0>,<13484.3210.0>},3,3},
                                 {{<13484.2158.0>,<13484.3212.0>},3,3},
                                 {{<13484.2158.0>,<13484.3214.0>},3,3},
                                 {{<13484.2158.0>,<13484.3216.0>},3,3},
                                 {{<13484.2158.0>,<13484.3218.0>},3,3},
                                 {{<13484.2158.0>,<13484.3220.0>},3,3},
                                 {{<13484.2158.0>,<13484.3222.0>},3,3},
                                 {{<13484.2158.0>,<13484.3224.0>},3,3},
                                 {{<13484.2158.0>,<13484.3226.0>},3,3},
                                 {{<13484.2158.0>,<13484.3228.0>},3,3},
                                 {{<13484.2158.0>,<13484.3230.0>},3,3},
                                 {{<13484.2158.0>,<13484.3232.0>},3,3},
                                 {{<13484.2158.0>,<13484.3234.0>},3,3},
                                 {{<13484.2158.0>,<13484.3236.0>},3,3},
                                 {{<13484.2158.0>,<13484.3238.0>},3,3},
                                 {{<13484.2158.0>,<13484.3240.0>},3,3},
                                 {{<13484.2158.0>,<13484.3242.0>},3,3},
                                 {{<13484.2158.0>,<13484.3244.0>},3,3},
                                 {{<13484.2158.0>,<13484.3246.0>},3,3},
                                 {{<13484.2158.0>,<13484.3248.0>},3,3},
                                 {{<13484.2158.0>,<13484.3250.0>},3,3},
                                 {{<13484.2171.0>,<13484.2172.0>},3,3},
                                 {{<13484.2171.0>,<13484.2246.0>},3,3},
                                 {{<13484.2173.0>,<13484.2174.0>},4,4},
                                 {{<13484.2173.0>,<13484.3252.0>},4,4},
                                 {{<13484.2180.0>,<13484.2181.0>},0,5},
                                 {{<13484.2189.0>,<13484.3093.0>},1,0},
                                 {{<13484.2189.0>,<13484.3096.0>},1,1},
                                 {{<13484.2189.0>,<13484.3097.0>},0,1},
                                 {{<13484.2189.0>,<13484.3098.0>},1,1},
                                 {{<13484.2189.0>,<13484.3099.0>},0,1},
                                 {{<13484.2189.0>,<13484.3100.0>},1,1},
                                 {{<13484.2189.0>,<13484.3101.0>},0,1},
                                 {{<13484.2189.0>,<13484.3102.0>},1,1},
                                 {{<13484.2189.0>,<13484.3103.0>},0,1},
                                 {{<13484.2189.0>,<13484.3104.0>},1,1},
                                 {{<13484.2189.0>,<13484.3105.0>},0,1},
                                 {{<13484.2189.0>,<13484.3106.0>},1,1},
                                 {{<13484.2189.0>,<13484.3107.0>},0,1},
                                 {{<13484.2189.0>,<13484.3108.0>},1,1},
                                 {{<13484.2189.0>,<13484.3109.0>},0,1},
                                 {{<13484.2189.0>,<13484.3110.0>},1,1},
                                 {{<13484.2189.0>,<13484.3111.0>},0,1},
                                 {{<13484.2189.0>,<13484.3112.0>},1,1},
                                 {{<13484.2189.0>,<13484.3113.0>},0,1},
                                 {{<13484.2189.0>,<13484.3114.0>},1,1},
                                 {{<13484.2189.0>,<13484.3115.0>},0,1},
                                 {{<13484.2189.0>,<13484.3116.0>},1,1},
                                 {{<13484.2189.0>,<13484.3117.0>},0,1},
                                 {{<13484.2189.0>,<13484.3118.0>},1,1},
                                 {{<13484.2189.0>,<13484.3119.0>},0,1},
                                 {{<13484.2189.0>,<13484.3120.0>},1,1},
                                 {{<13484.2189.0>,<13484.3121.0>},0,1},
                                 {{<13484.2189.0>,<13484.3122.0>},1,1},
                                 {{<13484.2189.0>,<13484.3123.0>},0,1},
                                 {{<13484.2189.0>,<13484.3124.0>},1,1},
                                 {{<13484.2189.0>,<13484.3125.0>},0,1},
                                 {{<13484.2189.0>,<13484.3126.0>},1,1},
                                 {{<13484.2189.0>,<13484.3127.0>},0,1},
                                 {{<13484.2189.0>,<13484.3128.0>},1,1},
                                 {{<13484.2189.0>,<13484.3129.0>},0,1},
                                 {{<13484.2189.0>,<13484.3130.0>},1,1},
                                 {{<13484.2189.0>,<13484.3131.0>},0,1},
                                 {{<13484.2189.0>,<13484.3132.0>},1,1},
                                 {{<13484.2189.0>,<13484.3133.0>},0,1},
                                 {{<13484.2189.0>,<13484.3134.0>},1,1},
                                 {{<13484.2189.0>,<13484.3135.0>},0,1},
                                 {{<13484.2189.0>,<13484.3136.0>},1,1},
                                 {{<13484.2189.0>,<13484.3137.0>},0,1},
                                 {{<13484.2189.0>,<13484.3138.0>},1,1},
                                 {{<13484.2189.0>,<13484.3139.0>},0,1},
                                 {{<13484.2189.0>,<13484.3140.0>},1,1},
                                 {{<13484.2189.0>,<13484.3141.0>},0,1},
                                 {{<13484.2189.0>,<13484.3142.0>},1,1},
                                 {{<13484.2189.0>,<13484.3143.0>},0,1},
                                 {{<13484.2189.0>,<13484.3144.0>},1,1},
                                 {{<13484.2189.0>,<13484.3145.0>},0,1},
                                 {{<13484.2189.0>,<13484.3146.0>},1,1},
                                 {{<13484.2189.0>,<13484.3147.0>},0,1},
                                 {{<13484.2189.0>,<13484.3148.0>},1,1},
                                 {{<13484.2189.0>,<13484.3149.0>},0,1},
                                 {{<13484.2189.0>,<13484.3150.0>},1,1},
                                 {{<13484.2189.0>,<13484.3151.0>},0,1},
                                 {{<13484.2189.0>,<13484.3152.0>},1,1},
                                 {{<13484.2189.0>,<13484.3153.0>},0,1},
                                 {{<13484.2189.0>,<13484.3154.0>},1,1},
                                 {{<13484.2189.0>,<13484.3155.0>},0,1},
                                 {{<13484.2189.0>,<13484.3156.0>},1,1},
                                 {{<13484.2189.0>,<13484.3157.0>},0,1},
                                 {{<13484.2189.0>,<13484.3158.0>},1,1},
                                 {{<13484.2189.0>,<13484.3159.0>},0,1},
                                 {{<13484.2189.0>,<13484.3160.0>},1,1},
                                 {{<13484.2189.0>,<13484.3161.0>},0,1},
                                 {{<13484.2189.0>,<13484.3162.0>},1,1},
                                 {{<13484.2189.0>,<13484.3163.0>},0,1},
                                 {{<13484.2189.0>,<13484.3164.0>},1,1},
                                 {{<13484.2189.0>,<13484.3165.0>},0,1},
                                 {{<13484.2189.0>,<13484.3166.0>},1,1},
                                 {{<13484.2189.0>,<13484.3167.0>},0,1},
                                 {{<13484.2189.0>,<13484.3168.0>},1,1},
                                 {{<13484.2189.0>,<13484.3169.0>},0,1},
                                 {{<13484.2189.0>,<13484.3170.0>},1,1},
                                 {{<13484.2189.0>,<13484.3171.0>},0,1},
                                 {{<13484.2189.0>,<13484.3172.0>},1,1},
                                 {{<13484.2189.0>,<13484.3173.0>},0,1},
                                 {{<13484.2189.0>,<13484.3174.0>},1,1},
                                 {{<13484.2189.0>,<13484.3175.0>},0,1},
                                 {{<13484.2189.0>,<13484.3176.0>},1,1},
                                 {{<13484.2189.0>,<13484.3177.0>},0,1},
                                 {{<13484.2189.0>,<13484.3178.0>},1,1},
                                 {{<13484.2189.0>,<13484.3179.0>},0,1},
                                 {{<13484.2189.0>,<13484.3180.0>},1,1},
                                 {{<13484.2189.0>,<13484.3181.0>},0,1},
                                 {{<13484.2189.0>,<13484.3182.0>},1,1},
                                 {{<13484.2189.0>,<13484.3183.0>},0,1},
                                 {{<13484.2189.0>,<13484.3184.0>},1,1},
                                 {{<13484.2189.0>,<13484.3185.0>},0,1},
                                 {{<13484.2189.0>,<13484.3186.0>},1,1},
                                 {{<13484.2189.0>,<13484.3187.0>},0,1},
                                 {{<13484.2189.0>,<13484.3188.0>},1,1},
                                 {{<13484.2189.0>,<13484.3189.0>},0,1},
                                 {{<13484.2189.0>,<13484.3190.0>},1,1},
                                 {{<13484.2189.0>,<13484.3191.0>},0,1},
                                 {{<13484.2189.0>,<13484.3192.0>},1,1},
                                 {{<13484.2189.0>,<13484.3193.0>},0,1},
                                 {{<13484.2189.0>,<13484.3194.0>},1,1},
                                 {{<13484.2189.0>,<13484.3195.0>},0,1},
                                 {{<13484.2189.0>,<13484.3196.0>},1,1},
                                 {{<13484.2189.0>,<13484.3197.0>},0,1},
                                 {{<13484.2189.0>,<13484.3198.0>},1,1},
                                 {{<13484.2189.0>,<13484.3199.0>},0,1},
                                 {{<13484.2189.0>,<13484.3200.0>},1,1},
                                 {{<13484.2189.0>,<13484.3201.0>},0,1},
                                 {{<13484.2189.0>,<13484.3202.0>},1,1},
                                 {{<13484.2189.0>,<13484.3203.0>},0,1},
                                 {{<13484.2189.0>,<13484.3204.0>},1,1},
                                 {{<13484.2189.0>,<13484.3205.0>},0,1},
                                 {{<13484.2189.0>,<13484.3206.0>},1,1},
                                 {{<13484.2189.0>,<13484.3207.0>},0,1},
                                 {{<13484.2189.0>,<13484.3208.0>},1,1},
                                 {{<13484.2189.0>,<13484.3209.0>},0,1},
                                 {{<13484.2189.0>,<13484.3210.0>},1,1},
                                 {{<13484.2189.0>,<13484.3211.0>},0,1},
                                 {{<13484.2189.0>,<13484.3212.0>},1,1},
                                 {{<13484.2189.0>,<13484.3213.0>},0,1},
                                 {{<13484.2189.0>,<13484.3214.0>},1,1},
                                 {{<13484.2189.0>,<13484.3215.0>},0,1},
                                 {{<13484.2189.0>,<13484.3216.0>},1,1},
                                 {{<13484.2189.0>,<13484.3217.0>},0,1},
                                 {{<13484.2189.0>,<13484.3218.0>},1,1},
                                 {{<13484.2189.0>,<13484.3219.0>},0,1},
                                 {{<13484.2189.0>,<13484.3220.0>},1,1},
                                 {{<13484.2189.0>,<13484.3221.0>},0,1},
                                 {{<13484.2189.0>,<13484.3222.0>},1,1},
                                 {{<13484.2189.0>,<13484.3223.0>},0,1},
                                 {{<13484.2189.0>,<13484.3224.0>},1,1},
                                 {{<13484.2189.0>,<13484.3225.0>},0,1},
                                 {{<13484.2189.0>,<13484.3226.0>},1,1},
                                 {{<13484.2189.0>,<13484.3227.0>},0,1},
                                 {{<13484.2189.0>,<13484.3228.0>},1,1},
                                 {{<13484.2189.0>,<13484.3229.0>},0,1},
                                 {{<13484.2189.0>,<13484.3230.0>},1,1},
                                 {{<13484.2189.0>,<13484.3231.0>},0,1},
                                 {{<13484.2189.0>,<13484.3232.0>},1,1},
                                 {{<13484.2189.0>,<13484.3233.0>},0,1},
                                 {{<13484.2189.0>,<13484.3234.0>},1,1},
                                 {{<13484.2189.0>,<13484.3235.0>},0,1},
                                 {{<13484.2189.0>,<13484.3236.0>},1,1},
                                 {{<13484.2189.0>,<13484.3237.0>},0,1},
                                 {{<13484.2189.0>,<13484.3238.0>},1,1},
                                 {{<13484.2189.0>,<13484.3239.0>},0,1},
                                 {{<13484.2189.0>,<13484.3240.0>},1,1},
                                 {{<13484.2189.0>,<13484.3241.0>},0,1},
                                 {{<13484.2189.0>,<13484.3242.0>},1,1},
                                 {{<13484.2189.0>,<13484.3243.0>},0,1},
                                 {{<13484.2189.0>,<13484.3244.0>},1,1},
                                 {{<13484.2189.0>,<13484.3245.0>},0,1},
                                 {{<13484.2189.0>,<13484.3246.0>},1,1},
                                 {{<13484.2189.0>,<13484.3247.0>},0,1},
                                 {{<13484.2189.0>,<13484.3248.0>},1,1},
                                 {{<13484.2189.0>,<13484.3249.0>},0,1},
                                 {{<13484.2189.0>,<13484.3250.0>},1,1},
                                 {{<13484.2189.0>,<13484.3251.0>},0,1},
                                 {{<13484.2198.0>,<13484.2246.0>},3,3},
                                 {{<13484.2212.0>,<13484.2246.0>},1,0},
                                 {{<13484.2213.0>,<13484.2599.0>},1,1},
                                 {{<13484.2213.0>,<13484.2600.0>},1,1},
                                 {{<13484.2213.0>,<13484.2601.0>},1,1},
                                 {{<13484.2213.0>,<13484.2602.0>},1,1},
                                 {{<13484.2213.0>,<13484.2603.0>},1,1},
                                 {{<13484.2214.0>,<13484.2599.0>},15,14},
                                 {{<13484.2214.0>,<13484.2600.0>},16,15},
                                 {{<13484.2214.0>,<13484.2601.0>},16,15},
                                 {{<13484.2214.0>,<13484.2602.0>},16,15},
                                 {{<13484.2214.0>,<13484.2603.0>},16,15},
                                 {{<13484.2214.0>,<13484.3093.0>},0,1},
                                 {{<13484.2214.0>,<13484.3096.0>},0,1},
                                 {{<13484.2214.0>,<13484.3098.0>},0,1},
                                 {{<13484.2214.0>,<13484.3100.0>},0,1},
                                 {{<13484.2214.0>,<13484.3102.0>},0,1},
                                 {{<13484.2214.0>,<13484.3104.0>},0,1},
                                 {{<13484.2214.0>,<13484.3106.0>},0,1},
                                 {{<13484.2214.0>,<13484.3108.0>},0,1},
                                 {{<13484.2214.0>,<13484.3110.0>},0,1},
                                 {{<13484.2214.0>,<13484.3112.0>},0,1},
                                 {{<13484.2214.0>,<13484.3114.0>},0,1},
                                 {{<13484.2214.0>,<13484.3116.0>},0,1},
                                 {{<13484.2214.0>,<13484.3118.0>},0,1},
                                 {{<13484.2214.0>,<13484.3120.0>},0,1},
                                 {{<13484.2214.0>,<13484.3122.0>},0,1},
                                 {{<13484.2214.0>,<13484.3124.0>},0,1},
                                 {{<13484.2214.0>,<13484.3126.0>},0,1},
                                 {{<13484.2214.0>,<13484.3128.0>},0,1},
                                 {{<13484.2214.0>,<13484.3130.0>},0,1},
                                 {{<13484.2214.0>,<13484.3132.0>},0,1},
                                 {{<13484.2214.0>,<13484.3134.0>},0,1},
                                 {{<13484.2214.0>,<13484.3136.0>},0,1},
                                 {{<13484.2214.0>,<13484.3138.0>},0,1},
                                 {{<13484.2214.0>,<13484.3140.0>},0,1},
                                 {{<13484.2214.0>,<13484.3142.0>},0,1},
                                 {{<13484.2214.0>,<13484.3144.0>},0,1},
                                 {{<13484.2214.0>,<13484.3146.0>},0,1},
                                 {{<13484.2214.0>,<13484.3148.0>},0,1},
                                 {{<13484.2214.0>,<13484.3150.0>},0,1},
                                 {{<13484.2214.0>,<13484.3152.0>},0,1},
                                 {{<13484.2214.0>,<13484.3154.0>},0,1},
                                 {{<13484.2214.0>,<13484.3156.0>},0,1},
                                 {{<13484.2214.0>,<13484.3158.0>},0,1},
                                 {{<13484.2214.0>,<13484.3160.0>},0,1},
                                 {{<13484.2214.0>,<13484.3162.0>},0,1},
                                 {{<13484.2214.0>,<13484.3164.0>},0,1},
                                 {{<13484.2214.0>,<13484.3166.0>},0,1},
                                 {{<13484.2214.0>,<13484.3168.0>},0,1},
                                 {{<13484.2214.0>,<13484.3170.0>},0,1},
                                 {{<13484.2214.0>,<13484.3172.0>},0,1},
                                 {{<13484.2214.0>,<13484.3174.0>},0,1},
                                 {{<13484.2214.0>,<13484.3176.0>},0,1},
                                 {{<13484.2214.0>,<13484.3178.0>},0,1},
                                 {{<13484.2214.0>,<13484.3180.0>},0,1},
                                 {{<13484.2214.0>,<13484.3182.0>},0,1},
                                 {{<13484.2214.0>,<13484.3184.0>},0,1},
                                 {{<13484.2214.0>,<13484.3186.0>},0,1},
                                 {{<13484.2214.0>,<13484.3188.0>},0,1},
                                 {{<13484.2214.0>,<13484.3190.0>},0,1},
                                 {{<13484.2214.0>,<13484.3192.0>},0,1},
                                 {{<13484.2214.0>,<13484.3194.0>},0,1},
                                 {{<13484.2214.0>,<13484.3196.0>},0,1},
                                 {{<13484.2214.0>,<13484.3198.0>},0,1},
                                 {{<13484.2214.0>,<13484.3200.0>},0,1},
                                 {{<13484.2214.0>,<13484.3202.0>},0,1},
                                 {{<13484.2214.0>,<13484.3204.0>},0,1},
                                 {{<13484.2214.0>,<13484.3206.0>},0,1},
                                 {{<13484.2214.0>,<13484.3208.0>},0,1},
                                 {{<13484.2214.0>,<13484.3210.0>},0,1},
                                 {{<13484.2214.0>,<13484.3212.0>},0,1},
                                 {{<13484.2214.0>,<13484.3214.0>},0,1},
                                 {{<13484.2214.0>,<13484.3216.0>},0,1},
                                 {{<13484.2214.0>,<13484.3218.0>},0,1},
                                 {{<13484.2214.0>,<13484.3220.0>},0,1},
                                 {{<13484.2214.0>,<13484.3222.0>},0,1},
                                 {{<13484.2214.0>,<13484.3224.0>},0,1},
                                 {{<13484.2214.0>,<13484.3226.0>},0,1},
                                 {{<13484.2214.0>,<13484.3228.0>},0,1},
                                 {{<13484.2214.0>,<13484.3230.0>},0,1},
                                 {{<13484.2214.0>,<13484.3232.0>},0,1},
                                 {{<13484.2214.0>,<13484.3234.0>},0,1},
                                 {{<13484.2214.0>,<13484.3236.0>},0,1},
                                 {{<13484.2214.0>,<13484.3238.0>},0,1},
                                 {{<13484.2214.0>,<13484.3240.0>},0,1},
                                 {{<13484.2214.0>,<13484.3242.0>},0,1},
                                 {{<13484.2214.0>,<13484.3244.0>},0,1},
                                 {{<13484.2214.0>,<13484.3246.0>},0,1},
                                 {{<13484.2214.0>,<13484.3248.0>},0,1},
                                 {{<13484.2214.0>,<13484.3250.0>},0,1},
                                 {{<13484.3013.0>,<13484.3094.0>},1,0},
                                 {{<13484.3093.0>,<13484.3095.0>},1,1},
                                 {{<13484.3096.0>,<13484.3097.0>},1,1},
                                 {{<13484.3098.0>,<13484.3099.0>},1,1},
                                 {{<13484.3100.0>,<13484.3101.0>},1,1},
                                 {{<13484.3102.0>,<13484.3103.0>},1,1},
                                 {{<13484.3104.0>,<13484.3105.0>},1,1},
                                 {{<13484.3106.0>,<13484.3107.0>},1,1},
                                 {{<13484.3108.0>,<13484.3109.0>},1,1},
                                 {{<13484.3110.0>,<13484.3111.0>},1,1},
                                 {{<13484.3112.0>,<13484.3113.0>},1,1},
                                 {{<13484.3114.0>,<13484.3115.0>},1,1},
                                 {{<13484.3116.0>,<13484.3117.0>},1,1},
                                 {{<13484.3118.0>,<13484.3119.0>},1,1},
                                 {{<13484.3120.0>,<13484.3121.0>},1,1},
                                 {{<13484.3122.0>,<13484.3123.0>},1,1},
                                 {{<13484.3124.0>,<13484.3125.0>},1,1},
                                 {{<13484.3126.0>,<13484.3127.0>},1,1},
                                 {{<13484.3128.0>,<13484.3129.0>},1,1},
                                 {{<13484.3130.0>,<13484.3131.0>},1,1},
                                 {{<13484.3132.0>,<13484.3133.0>},1,1},
                                 {{<13484.3134.0>,<13484.3135.0>},1,1},
                                 {{<13484.3136.0>,<13484.3137.0>},1,1},
                                 {{<13484.3138.0>,<13484.3139.0>},1,1},
                                 {{<13484.3140.0>,<13484.3141.0>},1,1},
                                 {{<13484.3142.0>,<13484.3143.0>},1,1},
                                 {{<13484.3144.0>,<13484.3145.0>},1,1},
                                 {{<13484.3146.0>,<13484.3147.0>},1,1},
                                 {{<13484.3148.0>,<13484.3149.0>},1,1},
                                 {{<13484.3150.0>,<13484.3151.0>},1,1},
                                 {{<13484.3152.0>,<13484.3153.0>},1,1},
                                 {{<13484.3154.0>,<13484.3155.0>},1,1},
                                 {{<13484.3156.0>,<13484.3157.0>},1,1},
                                 {{<13484.3158.0>,<13484.3159.0>},1,1},
                                 {{<13484.3160.0>,<13484.3161.0>},1,1},
                                 {{<13484.3162.0>,<13484.3163.0>},1,1},
                                 {{<13484.3164.0>,<13484.3165.0>},1,1},
                                 {{<13484.3166.0>,<13484.3167.0>},1,1},
                                 {{<13484.3168.0>,<13484.3169.0>},1,1},
                                 {{<13484.3170.0>,<13484.3171.0>},1,1},
                                 {{<13484.3172.0>,<13484.3173.0>},1,1},
                                 {{<13484.3174.0>,<13484.3175.0>},1,1},
                                 {{<13484.3176.0>,<13484.3177.0>},1,1},
                                 {{<13484.3178.0>,<13484.3179.0>},1,1},
                                 {{<13484.3180.0>,<13484.3181.0>},1,1},
                                 {{<13484.3182.0>,<13484.3183.0>},1,1},
                                 {{<13484.3184.0>,<13484.3185.0>},1,1},
                                 {{<13484.3186.0>,<13484.3187.0>},1,1},
                                 {{<13484.3188.0>,<13484.3189.0>},1,1},
                                 {{<13484.3190.0>,<13484.3191.0>},1,1},
                                 {{<13484.3192.0>,<13484.3193.0>},1,1},
                                 {{<13484.3194.0>,<13484.3195.0>},1,1},
                                 {{<13484.3196.0>,<13484.3197.0>},1,1},
                                 {{<13484.3198.0>,<13484.3199.0>},1,1},
                                 {{<13484.3200.0>,<13484.3201.0>},1,1},
                                 {{<13484.3202.0>,<13484.3203.0>},1,1},
                                 {{<13484.3204.0>,<13484.3205.0>},1,1},
                                 {{<13484.3206.0>,<13484.3207.0>},1,1},
                                 {{<13484.3208.0>,<13484.3209.0>},1,1},
                                 {{<13484.3210.0>,<13484.3211.0>},1,1},
                                 {{<13484.3212.0>,<13484.3213.0>},1,1},
                                 {{<13484.3214.0>,<13484.3215.0>},1,1},
                                 {{<13484.3216.0>,<13484.3217.0>},1,1},
                                 {{<13484.3218.0>,<13484.3219.0>},1,1},
                                 {{<13484.3220.0>,<13484.3221.0>},1,1},
                                 {{<13484.3222.0>,<13484.3223.0>},1,1},
                                 {{<13484.3224.0>,<13484.3225.0>},1,1},
                                 {{<13484.3226.0>,<13484.3227.0>},1,1},
                                 {{<13484.3228.0>,<13484.3229.0>},1,1},
                                 {{<13484.3230.0>,<13484.3231.0>},1,1},
                                 {{<13484.3232.0>,<13484.3233.0>},1,1},
                                 {{<13484.3234.0>,<13484.3235.0>},1,1},
                                 {{<13484.3236.0>,<13484.3237.0>},1,1},
                                 {{<13484.3238.0>,<13484.3239.0>},1,1},
                                 {{<13484.3240.0>,<13484.3241.0>},1,1},
                                 {{<13484.3242.0>,<13484.3243.0>},1,1},
                                 {{<13484.3244.0>,<13484.3245.0>},1,1},
                                 {{<13484.3246.0>,<13484.3247.0>},1,1},
                                 {{<13484.3248.0>,<13484.3249.0>},1,1},
                                 {{<13484.3250.0>,<13484.3251.0>},1,1}]},
                            {send_self,[]},
                            {'receive',
                                [{<13484.1517.0>,8,168},
                                 {<13484.1518.0>,10,90},
                                 {<13484.1524.0>,1,0},
                                 {<13484.1526.0>,1,0},
                                 {<13484.2158.0>,237,15168},
                                 {<13484.2160.0>,79,3950},
                                 {<13484.2171.0>,9,144},
                                 {<13484.2172.0>,48,627},
                                 {<13484.2173.0>,8,60},
                                 {<13484.2174.0>,20,476},
                                 {<13484.2180.0>,5,0},
                                 {<13484.2181.0>,5,0},
                                 {<13484.2189.0>,156,2340},
                                 {<13484.2198.0>,81,1062},
                                 {<13484.2210.0>,5,0},
                                 {<13484.2212.0>,1,8},
                                 {<13484.2213.0>,5,70},
                                 {<13484.2214.0>,227,3400},
                                 {<13484.2227.0>,2,42},
                                 {<13484.2246.0>,46,22808},
                                 {<13484.2599.0>,16,141},
                                 {<13484.2600.0>,17,150},
                                 {<13484.2601.0>,17,150},
                                 {<13484.2602.0>,17,150},
                                 {<13484.2603.0>,17,150},
                                 {<13484.3013.0>,2,6},
                                 {<13484.3017.0>,1,4},
                                 {<13484.3093.0>,5,33},
                                 {<13484.3095.0>,1,10},
                                 {<13484.3096.0>,5,33},
                                 {<13484.3097.0>,3,10},
                                 {<13484.3098.0>,5,33},
                                 {<13484.3099.0>,3,10},
                                 {<13484.3100.0>,5,33},
                                 {<13484.3101.0>,3,10},
                                 {<13484.3102.0>,5,33},
                                 {<13484.3103.0>,3,10},
                                 {<13484.3104.0>,5,33},
                                 {<13484.3105.0>,3,10},
                                 {<13484.3106.0>,5,33},
                                 {<13484.3107.0>,3,10},
                                 {<13484.3108.0>,5,33},
                                 {<13484.3109.0>,3,10},
                                 {<13484.3110.0>,5,33},
                                 {<13484.3111.0>,3,10},
                                 {<13484.3112.0>,5,33},
                                 {<13484.3113.0>,3,10},
                                 {<13484.3114.0>,5,33},
                                 {<13484.3115.0>,3,10},
                                 {<13484.3116.0>,5,33},
                                 {<13484.3117.0>,3,10},
                                 {<13484.3118.0>,5,33},
                                 {<13484.3119.0>,3,10},
                                 {<13484.3120.0>,5,33},
                                 {<13484.3121.0>,3,10},
                                 {<13484.3122.0>,5,33},
                                 {<13484.3123.0>,3,10},
                                 {<13484.3124.0>,5,33},
                                 {<13484.3125.0>,3,10},
                                 {<13484.3126.0>,5,33},
                                 {<13484.3127.0>,3,10},
                                 {<13484.3128.0>,5,33},
                                 {<13484.3129.0>,3,10},
                                 {<13484.3130.0>,5,33},
                                 {<13484.3131.0>,3,10},
                                 {<13484.3132.0>,5,33},
                                 {<13484.3133.0>,3,10},
                                 {<13484.3134.0>,5,33},
                                 {<13484.3135.0>,3,10},
                                 {<13484.3136.0>,5,33},
                                 {<13484.3137.0>,3,10},
                                 {<13484.3138.0>,5,33},
                                 {<13484.3139.0>,3,10},
                                 {<13484.3140.0>,5,33},
                                 {<13484.3141.0>,3,10},
                                 {<13484.3142.0>,5,33},
                                 {<13484.3143.0>,3,10},
                                 {<13484.3144.0>,5,33},
                                 {<13484.3145.0>,3,10},
                                 {<13484.3146.0>,5,33},
                                 {<13484.3147.0>,3,10},
                                 {<13484.3148.0>,5,33},
                                 {<13484.3149.0>,3,10},
                                 {<13484.3150.0>,5,33},
                                 {<13484.3151.0>,3,10},
                                 {<13484.3152.0>,5,33},
                                 {<13484.3153.0>,3,10},
                                 {<13484.3154.0>,5,33},
                                 {<13484.3155.0>,3,10},
                                 {<13484.3156.0>,5,33},
                                 {<13484.3157.0>,2,10},
                                 {<13484.3158.0>,5,33},
                                 {<13484.3159.0>,3,10},
                                 {<13484.3160.0>,5,33},
                                 {<13484.3161.0>,3,10},
                                 {<13484.3162.0>,5,33},
                                 {<13484.3163.0>,3,10},
                                 {<13484.3164.0>,5,33},
                                 {<13484.3165.0>,3,10},
                                 {<13484.3166.0>,5,33},
                                 {<13484.3167.0>,3,10},
                                 {<13484.3168.0>,5,33},
                                 {<13484.3169.0>,3,10},
                                 {<13484.3170.0>,5,33},
                                 {<13484.3171.0>,3,10},
                                 {<13484.3172.0>,5,33},
                                 {<13484.3173.0>,3,10},
                                 {<13484.3174.0>,5,33},
                                 {<13484.3175.0>,3,10},
                                 {<13484.3176.0>,5,33},
                                 {<13484.3177.0>,3,10},
                                 {<13484.3178.0>,5,33},
                                 {<13484.3179.0>,3,10},
                                 {<13484.3180.0>,5,33},
                                 {<13484.3181.0>,3,10},
                                 {<13484.3182.0>,5,33},
                                 {<13484.3183.0>,3,10},
                                 {<13484.3184.0>,5,33},
                                 {<13484.3185.0>,3,10},
                                 {<13484.3186.0>,5,33},
                                 {<13484.3187.0>,3,10},
                                 {<13484.3188.0>,5,33},
                                 {<13484.3189.0>,3,10},
                                 {<13484.3190.0>,5,33},
                                 {<13484.3191.0>,3,10},
                                 {<13484.3192.0>,5,33},
                                 {<13484.3193.0>,3,10},
                                 {<13484.3194.0>,5,33},
                                 {<13484.3195.0>,3,10},
                                 {<13484.3196.0>,5,33},
                                 {<13484.3197.0>,3,10},
                                 {<13484.3198.0>,5,33},
                                 {<13484.3199.0>,3,10},
                                 {<13484.3200.0>,5,33},
                                 {<13484.3201.0>,3,10},
                                 {<13484.3202.0>,5,33},
                                 {<13484.3203.0>,3,10},
                                 {<13484.3204.0>,5,33},
                                 {<13484.3205.0>,3,10},
                                 {<13484.3206.0>,5,33},
                                 {<13484.3207.0>,3,10},
                                 {<13484.3208.0>,5,33},
                                 {<13484.3209.0>,3,10},
                                 {<13484.3210.0>,5,33},
                                 {<13484.3211.0>,3,10},
                                 {<13484.3212.0>,5,33},
                                 {<13484.3213.0>,3,10},
                                 {<13484.3214.0>,5,33},
                                 {<13484.3215.0>,3,10},
                                 {<13484.3216.0>,5,33},
                                 {<13484.3217.0>,3,10},
                                 {<13484.3218.0>,5,33},
                                 {<13484.3219.0>,3,10},
                                 {<13484.3220.0>,5,33},
                                 {<13484.3221.0>,3,10},
                                 {<13484.3222.0>,5,33},
                                 {<13484.3223.0>,3,10},
                                 {<13484.3224.0>,5,33},
                                 {<13484.3225.0>,3,10},
                                 {<13484.3226.0>,5,33},
                                 {<13484.3227.0>,2,10},
                                 {<13484.3228.0>,5,33},
                                 {<13484.3229.0>,2,10},
                                 {<13484.3230.0>,5,33},
                                 {<13484.3231.0>,3,10},
                                 {<13484.3232.0>,5,33},
                                 {<13484.3233.0>,3,10},
                                 {<13484.3234.0>,5,33},
                                 {<13484.3235.0>,3,10},
                                 {<13484.3236.0>,5,33},
                                 {<13484.3237.0>,3,10},
                                 {<13484.3238.0>,5,33},
                                 {<13484.3239.0>,3,10},
                                 {<13484.3240.0>,5,33},
                                 {<13484.3241.0>,3,10},
                                 {<13484.3242.0>,5,33},
                                 {<13484.3243.0>,3,10},
                                 {<13484.3244.0>,5,33},
                                 {<13484.3245.0>,2,10},
                                 {<13484.3246.0>,5,33},
                                 {<13484.3247.0>,3,10},
                                 {<13484.3248.0>,5,33},
                                 {<13484.3249.0>,3,10},
                                 {<13484.3250.0>,5,33},
                                 {<13484.3251.0>,3,10},
                                 {<13484.3252.0>,4,24}]},
                            {trace,[]}]}
** When Server state == {state,[]}
** Reason for termination == 
** {{timeout,{gen_server,call,
                         [epl_tracer,
                          {command,#Fun<supervisor.which_children.1>,
                                   [<13484.2118.0>]}]}},
    [{gen_server,call,2,[{file,"gen_server.erl"},{line,204}]},
     {epl_st,command,2,[{file,"src/epl_st.erl"},{line,129}]},
     {epl_st,generate_sup_tree,1,[{file,"src/epl_st.erl"},{line,104}]},
     {epl_st,'-handle_info/2-fun-0-',2,[{file,"src/epl_st.erl"},{line,76}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1262}]},
     {epl_st,handle_info,2,[{file,"src/epl_st.erl"},{line,75}]},
     {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]}]}

=INFO REPORT==== 9-May-2017::14:09:41 ===
    application: epl_st
    exited: shutdown
    type: permanent
{"Kernel pid terminated",application_controller,"{application_terminated,epl_st,shutdown}"}
Kernel pid terminated (application_controller) ({application_terminated,epl_st,shutdown})
mdaiter commented 7 years ago

Pretty sure we can parallelize most of these list operations (except for foldl). Let's try to make some progress on that end? Think it's timing out due to too list operations stalling.

mdaiter commented 7 years ago

Progress update: increasing the timeout and ?POLL values to 50000 results in successful splash screen and traffic generation. Does not enable sup-tree screen to complete.

mdaiter commented 7 years ago

Alright, so two (non-mutually-exclusive) methods of attack:

  1. Create dynamic timeouts at the risk of delaying output significantly
  2. Parallelize list ops
mdaiter commented 7 years ago

Aha! I think I got to the root of the problem. If a process times out (epl_st.erl, line 75), there seems to be a formatting issue when generating the JSON to be reported back to the main system. Maybe a filtering of these children would fix this problem. I'll try!

michalslaski commented 7 years ago

Thanks for progressing on this issue. If the dashboard renders fine, but reports throughput N/A, it means not a single trace report was collected from the epl_tracer process running on the observed node. This is typically due to a timeout set to 5 seconds, which is not long enough to collect statistics if the system is under heavy load. The suggested optimisations are interesting, but I am not sure we should parallelise list operations. Having just one epl_tracer process spawned on the observed node ensure we will not utilise more than one CPU core for processing trace events. Although we can't avoid creating some overhead, I think we should aim at limiting the impact on the observed node as much as possible. Spawning just one process is a very naive way of limiting the impact, but better than none. I'm sure we can find a way of optimising the processing of long lists. One way would be to use the new tracer backend introduced in Erlang 19, which allows to implement NIF callback functions.

Your crash report confirms that another timeout occurred while querying one of the supervisor processes. This happens if a supervisor process does not comply with the OTP conventions and as a result the which_children/1 function does not get a proper response. This could happen when the 'application' behaviour init/1 callback function does not return a pid of an OTP supervisor. See a similar issue https://github.com/erlanglab/erlangpl/issues/29.

mdaiter commented 7 years ago

@michalslaski I'm pretty sure my applications all return top-level supervisors. If I don't implement handle_info in my classes, however, could this break the system?

OvermindDL1 commented 7 years ago

Identical error when loading this in my large Phoenix application except the Last message could be considered "Bloody Huge!". This is in the latest escript tar.gz download from the website.

OvermindDL1 commented 7 years ago

My application info of :running returns (as was requested in another issue):

iex(my_server@127.0.0.1)1> :application.info()[:running]
[
  sasl: #PID<0.2025.0>,
  iex: #PID<0.2019.0>,
  my_server: #PID<0.1864.0>,
  runtime_tools: #PID<0.1859.0>,
  pid_file: #PID<0.1853.0>,
  gettext: #PID<0.1848.0>,
  mbu: :undefined,
  fs: #PID<0.1841.0>,
  permission_ex: :undefined,
  comeonin: :undefined,
  credo: #PID<0.1829.0>,
  bunt: #PID<0.1826.0>,
  inets: #PID<0.1815.0>,
  cachex: #PID<0.1803.0>,
  eternal: :undefined,
  deppie: #PID<0.1797.0>,
  bodyguard: :undefined,
  ueberauth_identity: :undefined,
  ueberauth: :undefined,
  drab: #PID<0.1786.0>,
  cowboy: #PID<0.1781.0>,
  ranch: #PID<0.1776.0>,
  ssl: #PID<0.1764.0>,
  public_key: :undefined,
  asn1: :undefined,
  cowlib: :undefined,
  phoenix_html: :undefined,
  phoenix: #PID<0.1755.0>,
  phoenix_pubsub: #PID<0.1751.0>,
  poison: :undefined,
  eex: :undefined,
  postgrex: #PID<0.1741.0>,
  db_connection: #PID<0.1734.0>,
  connection: :undefined,
  ecto_interval: :undefined,
  quantum: #PID<0.1729.0>,
  crontab: :undefined,
  exceptional: :undefined,
  phoenix_ecto: :undefined,
  plug: #PID<0.1721.0>,
  crypto: :undefined,
  mime: :undefined,
  ecto: #PID<0.1714.0>,
  logger: #PID<0.1705.0>,
  decimal: :undefined,
  elixir: #PID<0.1698.0>,
  compiler: :undefined,
  poolboy: :undefined,
  stdlib: :undefined,
  kernel: #PID<0.1662.0>
]