search

bytedance / byteps

A high performance and generic framework for distributed DNN training
Other
3.58k stars 486 forks source link

distributed training hang #37

Open tingweiwu opened 4 years ago

tingweiwu commented 4 years ago

I follow the step-by-step-tutorial to run distributed training with mxnet and tensorflow, both hang. I have 3 nodes and on first node I run scheduler and server and second and third node I run worker0 and worker1 cmd and logs as following

scheduler:

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=2 
root@--0002:~# export DMLC_ROLE=scheduler 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching scheduler

server:

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=2 
root@--0002:~# export DMLC_ROLE=server  
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# 
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching server
[[03:13:29] src/customer.cc:03:13:29] 226: Use seperate thread to process pull requests from each worker.
src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[03:13:29[] src/customer.cc:239: Server uses 1 threads to process push requests.
03:13:29] src/customer.cc:137: Server inits Push Thread-0
[03:13:29] src/customer.cc:93: Server inits Pull Thread-1
[03:13:29] src/customer.cc:93: Server inits Pull Thread-0
[03:13:29] src/customer.cc:258: All threads inited, ready to process message 
[03:13:29] src/./zmq_van.h:285: Start ZMQ recv thread

worker0

[root@0001 ~]# docker run --name worker-0 -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
root@0001:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3  
root@0001:~# export DMLC_WORKER_ID=0 
root@0001:~# export DMLC_NUM_WORKER=2 
root@0001:~# export DMLC_ROLE=worker
root@0001:~# export DMLC_NUM_SERVER=1 
root@0001:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@0001:~# export DMLC_PS_ROOT_PORT=1234
root@0001:~# 
root@0001:~# export EVAL_TYPE=benchmark 
root@0001:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32  
BytePS launching worker
[03:14:57] src/customer.cc:363: Do not use thread pool for receiving.
[03:14:57] src/./zmq_van.h:285: Start ZMQ recv thread
[03:15:12] src/./zmq_van.h:285: Start ZMQ recv thread
[03:15:12] src/./zmq_van.h:285: Start ZMQ recv thread

worker1

[root@0002 ~]# docker run --name worker-1 -it --net=host --shm-size=32768m train.registry.docker.com:5000/bytepsimage/worker_mxnet bash
root@0002:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3 
root@0002:~# export DMLC_WORKER_ID=1
root@0002:~# export DMLC_NUM_WORKER=2 
root@0002:~# export DMLC_ROLE=worker 
root@0002:~# export DMLC_NUM_SERVER=1 
root@0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@0002:~# export DMLC_PS_ROOT_PORT=1234
root@0002:~# 
root@0002:~# export EVAL_TYPE=benchmark 
root@0002:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32 
BytePS launching worker
[03:15:17] src/customer.cc:363: Do not use thread pool for receiving.
[03:15:17] src/./zmq_van.h:285: Start ZMQ recv thread
[03:15:17] src/./zmq_van.h:285: Start ZMQ recv thread

In addition after I input ctrl+c to abort the process. and ps -ef |grep python shows that the process still there?

[03:15:17] src/./zmq_van.h:285: Start ZMQ recv thread
^C[[[2019-07-03 03:24:17.2019-07-03 03:24:172019-07-03 03:24:17579377..: 579377579381F: :  FFbyteps/common/communicator.cc  :byteps/common/communicator.cc206byteps/common/communicator.cc:] :206Check failed: (rc) >= (0) Interrupted system call, rank=0206] 
] Check failed: (rc) >= (0) Interrupted system call, rank=2Check failed: (rc) >= (0) Interrupted system call, rank=1

^C^C^C^C^C^C^C^C/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh: line 5:    32 Aborted                 (core dumped) python $path/train_imagenet_byteps.py $@
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/byteps/launcher/launch.py", line 19, in worker
    subprocess.check_call(command, env=my_env, stdout=sys.stdout, stderr=sys.stderr, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh --benchmark 1 --batch-size=32' returned non-zero exit status -2

/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh: line 5:    26 Aborted                 (core dumped) python $path/train_imagenet_byteps.py $@
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/byteps/launcher/launch.py", line 19, in worker
    subprocess.check_call(command, env=my_env, stdout=sys.stdout, stderr=sys.stderr, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh --benchmark 1 --batch-size=32' returned non-zero exit status -2

Traceback (most recent call last):
  File "/usr/local/byteps/launcher/launch.py", line 38, in <module>
    t[i].join() 
  File "/usr/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/usr/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
KeyboardInterrupt
root@bms-bf19-0002:~# /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh: line 5:    28 Aborted                 (core dumped) python $path/train_imagenet_byteps.py $@
^C
root@0002:~# ^C
root@0002:~# ps -ef | grep python
root         31     25 99 03:15 pts/0    00:30:46 python /usr/local/byteps/example/mxnet/train_imagenet_byteps.py --benchmark 1 --batch-size=32
root        768      1  0 03:25 pts/0    00:00:00 grep --color=auto python

and the server and scheduler didn't abort when input ctrl+c

BytePS launching server
[[03:13:29] src/customer.cc:03:13:29] 226: Use seperate thread to process pull requests from each worker.
src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[03:13:29[] src/customer.cc:239: Server uses 1 threads to process push requests.
03:13:29] src/customer.cc:137: Server inits Push Thread-0
[03:13:29] src/customer.cc:93: Server inits Pull Thread-1
[03:13:29] src/customer.cc:93: Server inits Pull Thread-0
[03:13:29] src/customer.cc:258: All threads inited, ready to process message 
[03:13:29] src/./zmq_van.h:285: Start ZMQ recv thread
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
ymjiang commented 4 years ago

You set export DMLC_ROLE=server for your scheduler, please change it to export DMLC_ROLE=scheduler

tingweiwu commented 4 years ago

@ymjiang sorry for pasting incorrectlly.

ymjiang commented 4 years ago

You didn't use nvidia-docker to launch both workers, so I am not sure what will happen... Can you access the GPU in the docker?

tingweiwu commented 4 years ago

@ymjiang I don't use nvidia-docker to launch because I set docker default runtime nvidia on GPU node. and I can access gpu in the docker

here is docker info image

and I have run successfully on single GPU node with docker.

here is log when I run Single Machine Training in the Step-by-Step Tutorial on the same GPU node 

[root@0001 ~]# docker run -it --shm-size=32768m bytepsimage/worker_tensorflow
root@45357267b97a:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3
root@45357267b97a:~# export DMLC_WORKER_ID=0
root@45357267b97a:~# export DMLC_NUM_WORKER=1
root@45357267b97a:~# export DMLC_ROLE=worker
root@45357267b97a:~# export DMLC_NUM_SERVER=1
root@45357267b97a:~# export DMLC_PS_ROOT_URI=10.0.0.1
root@45357267b97a:~# export DMLC_PS_ROOT_PORT=1234
root@45357267b97a:~# export EVAL_TYPE=benchmark
root@45357267b97a:~# 
root@45357267b97a:~# 
root@45357267b97a:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/tensorflow/run_tensorflow_byteps.sh \
>        --model ResNet50 --num-iters 50   
BytePS launching worker
Run synthetic benchmark...
Run synthetic benchmark...
Run synthetic benchmark...
Run synthetic benchmark...
Model: ResNet50
Batch size: 32
Number of GPUs: 4
2019-07-03 06:37:39.838464: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-03 06:37:39.854303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:2d:00.0
totalMemory: 15.75GiB freeMemory: 15.24GiB
2019-07-03 06:37:39.854341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-07-03 06:37:39.854366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-03 06:37:39.854375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-07-03 06:37:39.854382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-07-03 06:37:39.854704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14823 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:2d:00.0, compute capability: 7.0)
2019-07-03 06:37:40.681165: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-03 06:37:40.700947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:39:00.0
totalMemory: 15.75GiB freeMemory: 15.24GiB
2019-07-03 06:37:40.700989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 3
2019-07-03 06:37:40.701009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-03 06:37:40.701018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      3 
2019-07-03 06:37:40.701025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3:   N 
2019-07-03 06:37:40.701340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14823 MB memory) -> physical GPU (device: 3, name: Tesla V100-PCIE-16GB, pci bus id: 0000:39:00.0, compute capability: 7.0)
2019-07-03 06:37:40.728243: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-03 06:37:40.747204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:31:00.0
totalMemory: 15.75GiB freeMemory: 15.24GiB
2019-07-03 06:37:40.747248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 1
2019-07-03 06:37:40.747269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-03 06:37:40.747279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      1 
2019-07-03 06:37:40.747286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   N 
2019-07-03 06:37:40.747622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14823 MB memory) -> physical GPU (device: 1, name: Tesla V100-PCIE-16GB, pci bus id: 0000:31:00.0, compute capability: 7.0)
2019-07-03 06:37:40.809360: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-03 06:37:40.827514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:35:00.0
totalMemory: 15.75GiB freeMemory: 15.24GiB
2019-07-03 06:37:40.827557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 2
2019-07-03 06:37:40.827601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-03 06:37:40.827609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      2 
2019-07-03 06:37:40.827616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2:   N 
2019-07-03 06:37:40.828017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14823 MB memory) -> physical GPU (device: 2, name: Tesla V100-PCIE-16GB, pci bus id: 0000:35:00.0, compute capability: 7.0)
Running warmup...
Running benchmark...
Iter #0: 279.9 img/sec per GPU
Iter #1: 279.1 img/sec per GPU
Iter #2: 279.0 img/sec per GPU
Iter #3: 278.7 img/sec per GPU
ymjiang commented 4 years ago

OK. So can you make sure 10.94.1.77 is the IP of scheduler (and worker/server can ping this IP)? If yes, could you set PS_VERBOSE=2 for all (scheduler/server/worker), run again, and paste the output here? Thank you.

tingweiwu commented 4 years ago

10.94.1.77 is the host ip of scheduler, server and scheduler are on the same node. here is log when set PS_VERBOSE=2

scheduler:

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=2 
root@--0002:~# export DMLC_ROLE=scheduler 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# export PS_VERBOSE=2
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching scheduler
[06:46:25] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[06:46:25] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
[06:46:25] src/van.cc:357[: Bind to role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0
06:46:25] src/./zmq_van.h:285: Start ZMQ recv thread
[06:46:36] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.94.1.77, port=62825, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:01] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.152, port=41120, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.165, port=46880, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:108: assign rank=8 to node role=server, ip=10.94.1.77, port=62825, is_recovery=0
[06:47:17] src/van.cc:108: assign rank=9 to node role=worker, ip=10.94.1.165, port=46880, is_recovery=0
[06:47:17] src/van.cc:108: assign rank=11 to node role=worker, ip=10.94.1.152, port=41120, is_recovery=0
[06:47:17] src/van.cc:446: ? => 9. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=62825, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=46880, is_recovery=0 role=worker, id=11, ip=10.94.1.152, port=41120, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=62825, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=46880, is_recovery=0 role=worker, id=11, ip=10.94.1.152, port=41120, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:446: ? => 8. Meta: request=0, timestamp=2, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=62825, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=46880, is_recovery=0 role=worker, id=11, ip=10.94.1.152, port=41120, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:134: the scheduler is connected to 2 workers and 1 servers
[06:47:17] src/van.cc:471: 11 => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:233: Barrier count for 7 : 1
[06:47:17] src/van.cc:446: ? => 1. Meta: request=1, timestamp=3, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:471: 1 => 1. Meta: request=1, timestamp=3, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:233: Barrier count for 7 : 2
[06:47:17] src/van.cc:471: 9 => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!

server

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=2 
root@--0002:~# export DMLC_ROLE=server  
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# export PS_VERBOSE=2
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching server
[06:46:36] src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[06:46:36] src/customer.cc[06:46:36] src/./zmq_van.h:61:226: Use seperate thread to process pull requests from each worker.
: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[06:46:36] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
[06:46:36] src/customer.cc:239: [Server uses 1 threads to process push requests.
06:46:36] src/customer.cc:137: Server inits Push Thread-0
[06:46:36] src/customer.cc:93: Server inits Pull Thread-1
[06:46:36] src/customer.cc:93: Server inits Pull Thread-0
[06:46:36] src/customer.cc:258: All threads inited, ready to process message 
[06:46:36] src/van.cc:357[: Bind to role=server, ip=10.94.1.77, port=62825, is_recovery=0
06:46:36] src/./zmq_van.h:285: Start ZMQ recv thread
[06:46:36] src/van.cc:446: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.94.1.77, port=62825, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:471: 1 => 2147483647. Meta: request=0, timestamp=2, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=62825, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=46880, is_recovery=0 role=worker, id=11, ip=10.94.1.152, port=41120, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:47:17] src/van.cc:306: S[8] is connected to others
[06:47:17] src/van.cc:446: ? => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!

worker-0

[root@0001 ~]# docker run --name worker-0 -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
root@0001:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3  
root@0001:~# export DMLC_WORKER_ID=0 
root@0001:~# export DMLC_NUM_WORKER=2 
root@0001:~# export DMLC_ROLE=worker
root@0001:~# export DMLC_NUM_SERVER=1 
root@0001:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@0001:~# export DMLC_PS_ROOT_PORT=1234
root@0001:~# export PS_VERBOSE=2
root@0001:~# export EVAL_TYPE=benchmark 
root@0001:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32  
BytePS launching worker
[[06:48:13] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[06:48:13] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
06:48:13] src/customer.cc:363: Do not use thread pool for receiving.
[06:48:13] src/van.cc:357: Bind to role=worker, ip=10.94.1.152, port=41120, is_recovery=0
[06:48:13] src/./zmq_van.h:285: Start ZMQ recv thread
[06:48:13] src/van.cc:446: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.152, port=41120, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:48:29] src/van.cc:471: 1 => 2147483647. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=62825, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=46880, is_recovery=0 role=worker, id=11, ip=10.94.1.152, port=41120, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:48:29] src/./zmq_van.h:285: Start ZMQ recv thread
[06:48:29] src/./zmq_van.h:285: Start ZMQ recv thread
[06:48:29] src/van.cc:306: W[11] is connected to others
[06:48:29] src/van.cc:446: ? => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!

woker-1

[root@0002 ~]# docker run --name worker-1 -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
root@0002:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3 
root@0002:~# export DMLC_WORKER_ID=1
root@0002:~# export DMLC_NUM_WORKER=2 
root@0002:~# export DMLC_ROLE=worker 
root@0002:~# export DMLC_NUM_SERVER=1 
root@0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@0002:~# export DMLC_PS_ROOT_PORT=1234
root@0002:~# export PS_VERBOSE=2
root@0002:~# export EVAL_TYPE=benchmark 
root@0002:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32  
BytePS launching worker
[06:48:34] src/customer.cc:363: Do not use thread pool for receiving.
[06:48:34] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[06:48:34] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
[06:48:34] src/van.cc:357: Bind to role=worker, ip=10.94.1.165, port=46880, is_recovery=0
[06:48:34] src/./zmq_van.h:285: Start ZMQ recv thread
[06:48:34] src/van.cc:446: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.165, port=46880, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:48:34] src/van.cc:471: 1 => 2147483647. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=62825, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=46880, is_recovery=0 role=worker, id=11, ip=10.94.1.152, port=41120, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[06:48:34] src/./zmq_van.h:285: Start ZMQ recv thread
[06:48:34] src/./zmq_van.h:285: Start ZMQ recv thread
[06:48:34] src/van.cc:306: W[9] is connected to others
[06:48:34] src/van.cc:446: ? => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
ymjiang commented 4 years ago

Thank you. Looks like the connection has not been fully established (between worker & server), so it should hang. But I am not clear about why, because we cannot reproduce your case on our testbed.

Could you please do the following and see if it works (or if it doesn't work, show the output). This time we only use one worker, and use BYTEPS_FORCE_DISTRIBUTED=1 for the only worker. I think you can copy-paste the followings.

For the scheduler:

docker run -it --net=host bytepsimage/byteps_server bash
export DMLC_NUM_WORKER=1
export DMLC_ROLE=scheduler 
export DMLC_NUM_SERVER=1 
export DMLC_PS_ROOT_URI=10.94.1.77 
export DMLC_PS_ROOT_PORT=1234  
export PS_VERBOSE=2
python /usr/local/byteps/launcher/launch.py

For the server:

docker run -it --net=host bytepsimage/byteps_server bash
export DMLC_NUM_WORKER=1
export DMLC_ROLE=server 
export DMLC_NUM_SERVER=1 
export DMLC_PS_ROOT_URI=10.94.1.77 
export DMLC_PS_ROOT_PORT=1234  
export PS_VERBOSE=2
python /usr/local/byteps/launcher/launch.py

For the worker:

docker run -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
export NVIDIA_VISIBLE_DEVICES=0,1,2,3 
export DMLC_NUM_WORKER=1
export DMLC_WORKER_ID=0
export DMLC_ROLE=worker 
export DMLC_NUM_SERVER=1 
export DMLC_PS_ROOT_URI=10.94.1.77 
export DMLC_PS_ROOT_PORT=1234  
export PS_VERBOSE=2

export BYTEPS_FORCE_DISTRIBUTED=1

export EVAL_TYPE=benchmark 
python /usr/local/byteps/launcher/launch.py \
        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
        --benchmark 1 --batch-size=32
bobzhuyb commented 4 years ago

In addition to what @ymjiang says, can you try starting worker/server/scheduler on the same machine, and use 127.0.0.1 as DMLC_PS_ROOT_URI?

This will help us check whether this problem is due to the network.

tingweiwu commented 4 years ago

@ymjiang it seems still hang. scheduler:

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=1
root@--0002:~# export DMLC_ROLE=scheduler 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# export PS_VERBOSE=2
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching scheduler
[07:43:09] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[07:43:09] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
[07:43:09] src/van.cc:357: Bind to role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0[
07:43:09] src/./zmq_van.h:285: Start ZMQ recv thread
[07:43:35] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.94.1.77, port=11064, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.152, port=41281, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:108: assign rank=8 to node role=server, ip=10.94.1.77, port=11064, is_recovery=0
[07:43:57] src/van.cc:108: assign rank=9 to node role=worker, ip=10.94.1.152, port=41281, is_recovery=0
[07:43:57] src/van.cc:446: ? => 9. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=11064, is_recovery=0 role=worker, id=9, ip=10.94.1.152, port=41281, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:446: ? => 8. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=11064, is_recovery=0 role=worker, id=9, ip=10.94.1.152, port=41281, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:134: the scheduler is connected to 1 workers and 1 servers
[07:43:57] src/van.cc:446: ? => 1. Meta: request=1, timestamp=2, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:471: 1 => 1. Meta: request=1, timestamp=2, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:233: Barrier count for 7 : 1
[07:43:57] src/van.cc:471: 9 => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:233: Barrier count for 7 : 2

server

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=1
root@--0002:~# export DMLC_ROLE=server 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# export PS_VERBOSE=2
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching server
[07:43:35] src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[07:43:35] src/customer.cc:226: [07:43:35] src/./zmq_van.h:61Use seperate thread to process pull requests from each worker.
: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[07:43:35] src/./zmq_van.h[07:43:35] src/customer.cc:[239:66: Server uses : BYTEPS_ZMQ_NTHREADS set to 41 threads to process push requests.
07:43:35] 
src/customer.cc:137: Server inits Push Thread-0
[07:43:35] src/customer.cc:93: Server inits Pull Thread-0
[07:43:35] src/customer.cc:258: All threads inited, ready to process message 
[07:43:35] src/van.cc:357: Bind to role=server, ip=10.94.1.77, port=11064, is_recovery=0
[07:43:35] src/./zmq_van.h:285: Start ZMQ recv thread
[07:43:35] src/van.cc:446: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.94.1.77, port=11064, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:471: 1 => 2147483647. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=11064, is_recovery=0 role=worker, id=9, ip=10.94.1.152, port=41281, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:43:57] src/van.cc:306: S[8] is connected to others
[07:43:57] src/van.cc:446: ? => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!

worker

[root@0001 ~]# docker run -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
root@0001:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3 
root@0001:~# export DMLC_NUM_WORKER=1
root@0001:~# export DMLC_WORKER_ID=0
root@0001:~# export DMLC_ROLE=worker 
root@0001:~# export DMLC_NUM_SERVER=1 
root@0001:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@0001:~# export DMLC_PS_ROOT_PORT=1234  
root@0001:~# export PS_VERBOSE=2
root@0001:~# 
root@0001:~# export BYTEPS_FORCE_DISTRIBUTED=1
root@0001:~# 
root@0001:~# export EVAL_TYPE=benchmark 
root@0001:~# python /usr/local/byteps/launcher/launch.py \
>         /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>         --benchmark 1 --batch-size=32
BytePS launching worker
[07:45:09] src/customer.cc:363: Do not use thread pool for receiving.[
07:45:09] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[07:45:09] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
[07:45:09] src/van.cc:357: Bind to role=worker, ip=10.94.1.152, port=41281, is_recovery=0
[07:45:09] src/./zmq_van.h:285: Start ZMQ recv thread
[07:45:09] src/van.cc:446: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.152, port=41281, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:45:09] src/van.cc:471: 1 => 2147483647. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=11064, is_recovery=0 role=worker, id=9, ip=10.94.1.152, port=41281, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[07:45:09] src/./zmq_van.h:285: Start ZMQ recv thread
[07:45:09] src/./zmq_van.h:285: Start ZMQ recv thread
[07:45:09] src/van.cc:306: W[9] is connected to others
[07:45:09] src/van.cc:446: ? => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
ymjiang commented 4 years ago

It still cannot establish connections.. then please follow @bobzhuyb 's suggestion and try again.

tingweiwu commented 4 years ago

@ymjiang @bobzhuyb it workes well when worker/server/scheduler on the same machine with 127.0.0.1 as DMLC_PS_ROOT_URI.

let's focus on network.

so I try this:

on first node 10.94.1.77 scheduler

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=1
root@--0002:~# export DMLC_ROLE=scheduler 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching scheduler
[09:09:41] src/./zmq_van.h:285: Start ZMQ recv thread

server

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=1
root@--0002:~# export DMLC_ROLE=server 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching server
[09:09:50] src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[09:09:50] src/customer.cc:226: Use seperate thread to process pull requests from each worker.
[09:09:50[] src/customer.cc:239: Server uses 1 threads to process push requests.
09:09:50] src/customer.cc:137: Server inits Push Thread-0
[09:09:50] src/customer.cc:93: Server inits Pull Thread-0
[09:09:50] src/customer.cc:258: All threads inited, ready to process message 
[09:09:50] src/./zmq_van.h:285: Start ZMQ recv thread

on second node start worker container and test ping 10.94.1.77 and telnet 10.94.1.77 1234 network between scheduler and worker seems ok. but still hang there

[root@-0001 ~]# docker run -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
root@-0001:~# apt-get update && apt-get install -y ip-utils telnet
......
root@-0001:~# ping 10.94.1.77
PING 10.94.1.77 (10.94.1.77) 56(84) bytes of data.
64 bytes from 10.94.1.77: icmp_seq=1 ttl=64 time=1.06 ms
64 bytes from 10.94.1.77: icmp_seq=2 ttl=64 time=0.550 ms
64 bytes from 10.94.1.77: icmp_seq=3 ttl=64 time=0.406 ms
^C
--- 10.94.1.77 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.406/0.673/1.065/0.284 ms
root@-0001:~# telnet 10.94.1.77 1234
Trying 10.94.1.77...
Connected to 10.94.1.77.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.
root@-0001:~# 
root@-0001:~# 
root@-0001:~# 
root@-0001:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3 
root@-0001:~# export DMLC_NUM_WORKER=1
root@-0001:~# export DMLC_WORKER_ID=0
root@-0001:~# export DMLC_ROLE=worker 
root@-0001:~# export DMLC_NUM_SERVER=1 
root@-0001:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@-0001:~# export DMLC_PS_ROOT_PORT=1234  
root@-0001:~# export BYTEPS_FORCE_DISTRIBUTED=1
root@-0001:~# export EVAL_TYPE=benchmark 
root@-0001:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32  
BytePS launching worker
[09:13:45] src/customer.cc:363: Do not use thread pool for receiving.
[09:13:45] src/./zmq_van.h:285: Start ZMQ recv thread
[09:13:45] src/./zmq_van.h:285: Start ZMQ recv thread
[09:13:45] src/./zmq_van.h:285: Start ZMQ recv thread
ymjiang commented 4 years ago

So the problem probably is due to your host network configuration. You can ping through the scheduler node, but the scheduler node may prevent others from connecting to the port... could you check your host configurations?

tingweiwu commented 4 years ago

@ymjiang what do you mean scheduler prevent others from connecting to the port??

I telnet from worker to scheduler. it shows success.

root@-0001:~# telnet 10.94.1.77 1234
Trying 10.94.1.77...
Connected to 10.94.1.77.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.
root@-0001:~#
ymjiang commented 4 years ago

@tingweiwu Can you try to run the basic pslite benchmark? Here are the steps you need to follow:

Install ps-lite

git clone --single-branch --branch byteps https://github.com/bytedance/ps-lite.git
cd ps-lite
make -j

For the scheduler

DMLC_ROLE=scheduler \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT] \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark

For the server

DMLC_ROLE=server \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT]  \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark

For the worker:

DMLC_ROLE=worker \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT] \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark 1024000 100 0

If it succeed, you should be able to see something like this on the worker. 

tests/test_kv_app_benchmark.cc:77: push_byte=4096000, repeat=100, total_time=128.842ms
tests/test_kv_app_benchmark.cc:91: pull_byte=4096000, repeat=100, total_time=353.38ms

If it still hang, then you should really check your network configurations...

tingweiwu commented 4 years ago

@ymjiang ok, I will try later. I wonder what network issue will be if I aready can telnet from worker to scheduler with the SCHEDULER_IP and SCHEDULER_PORT ?

bobzhuyb commented 4 years ago

@tingweiwu From your log, everyone can connect to scheduler. However, the connection between worker and server seems to fail. When starting, the worker and server automatically bind to a port and tell this port to scheduler, then scheduler tells everyone this port. Your problem happens when your worker and server connect to each other.

For example, when your log shows 

[07:43:57] src/van.cc:108: assign rank=8 to node role=server, ip=10.94.1.77, port=11064, is_recovery=0
[07:43:57] src/van.cc:108: assign rank=9 to node role=worker, ip=10.94.1.152, port=41281, is_recovery=0

You should try telnet 10.94.1.77 11064 and telnet 10.94.1.152 41281

tingweiwu commented 4 years ago

@bobzhuyb @ymjiang

here is the log from scheduler

BytePS launching scheduler
[03:23:00] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024
[03:23:00] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4
[03:23:00] src/van.cc:357: Bind to role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0
[03:23:00] src/./zmq_van.h:285: Start ZMQ recv thread
[03:23:09] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.94.1.77, port=6873, is_recovery=0 } }. THIS IS NOT DATA MSG!
[03:24:00] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.94.1.165, port=38566, is_recovery=0 } }. THIS IS NOT DATA MSG!
[03:24:00] src/van.cc:108: assign rank=8 to node role=server, ip=10.94.1.77, port=6873, is_recovery=0
[03:24:00] src/van.cc:108: assign rank=9 to node role=worker, ip=10.94.1.165, port=38566, is_recovery=0
[03:24:00] src/van.cc:446: ? => 9. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=6873, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=38566, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[03:24:00] src/van.cc:446: ? => 8. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ role=server, id=8, ip=10.94.1.77, port=6873, is_recovery=0 role=worker, id=9, ip=10.94.1.165, port=38566, is_recovery=0 role=scheduler, id=1, ip=10.94.1.77, port=1234, is_recovery=0 } }. THIS IS NOT DATA MSG!
[03:24:00] src/van.cc:134: the scheduler is connected to 1 workers and 1 servers
[03:24:00] src/van.cc:446: ? => 1. Meta: request=1, timestamp=2, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[03:24:00] src/van.cc:471: 1 => 1. Meta: request=1, timestamp=2, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG!
[03:24:00] src/van.cc:233: Barrier count for 7 : 1

from worker container I telnet to laucher and server successfully.

root@-0002:~# ping 10.94.1.77
PING 10.94.1.77 (10.94.1.77) 56(84) bytes of data.
64 bytes from 10.94.1.77: icmp_seq=3 ttl=64 time=0.633 ms
64 bytes from 10.94.1.77: icmp_seq=4 ttl=64 time=0.339 ms
64 bytes from 10.94.1.77: icmp_seq=5 ttl=64 time=0.327 ms
^C
--- 10.94.1.77 ping statistics ---
5 packets transmitted, 3 received, 40% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.327/0.433/0.633/0.141 ms

root@-0002:~# telnet 10.94.1.77 1234
Trying 10.94.1.77...
Connected to 10.94.1.77.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.

root@-0002:~# telnet 10.94.1.77 6873
Trying 10.94.1.77...
Connected to 10.94.1.77.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.

from server container I telnet to laucher and worker successfully.

root@--0002:~# ping 10.94.1.77
PING 10.94.1.77 (10.94.1.77) 56(84) bytes of data.
64 bytes from 10.94.1.77: icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from 10.94.1.77: icmp_seq=2 ttl=64 time=0.051 ms
^C
--- 10.94.1.77 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.051/0.056/0.061/0.005 ms

root@--0002:~# telnet 10.94.1.77 1234
Trying 10.94.1.77...
Connected to 10.94.1.77.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.

root@--0002:~# ping 10.94.1.165
PING 10.94.1.165 (10.94.1.165) 56(84) bytes of data.
64 bytes from 10.94.1.165: icmp_seq=1 ttl=64 time=0.977 ms
64 bytes from 10.94.1.165: icmp_seq=2 ttl=64 time=0.939 ms
64 bytes from 10.94.1.165: icmp_seq=3 ttl=64 time=0.420 ms
^C
--- 10.94.1.165 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.420/0.778/0.977/0.256 ms

root@--0002:~# telnet 10.94.1.165 38566
Trying 10.94.1.165...
Connected to 10.94.1.165.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.

from laucher container I telnet to server and worker successfully.

root@--0002:~# ping 10.94.1.77
PING 10.94.1.77 (10.94.1.77) 56(84) bytes of data.
64 bytes from 10.94.1.77: icmp_seq=1 ttl=64 time=0.053 ms
64 bytes from 10.94.1.77: icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 10.94.1.77: icmp_seq=3 ttl=64 time=0.059 ms
^C
--- 10.94.1.77 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.045/0.052/0.059/0.008 ms

root@--0002:~# telnet 10.94.1.77 6873
Trying 10.94.1.77...
Connected to 10.94.1.77.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.

root@--0002:~# ping 10.94.1.165
PING 10.94.1.165 (10.94.1.165) 56(84) bytes of data.
64 bytes from 10.94.1.165: icmp_seq=1 ttl=64 time=0.666 ms
64 bytes from 10.94.1.165: icmp_seq=2 ttl=64 time=0.468 ms
64 bytes from 10.94.1.165: icmp_seq=3 ttl=64 time=0.603 ms
^C
--- 10.94.1.165 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.468/0.579/0.666/0.082 ms

root@--0002:~# telnet 10.94.1.165 38566
Trying 10.94.1.165...
Connected to 10.94.1.165.
Escape character is '^]'.                                                                                                                                                                                                                                                    ^C^CConnection closed by foreign host.

hence. I do not think this is my network configurations issues. but the worker still hang there.

ymjiang commented 4 years ago

@tingweiwu Did you try the pslite benchmark?

@tingweiwu Can you try to run the basic pslite benchmark? Here are the steps you need to follow:

Install ps-lite

git clone --single-branch --branch byteps https://github.com/bytedance/ps-lite.git
cd ps-lite
make -j

For the scheduler

DMLC_ROLE=scheduler \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT] \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark

For the server

DMLC_ROLE=server \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT]  \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark

For the worker:

DMLC_ROLE=worker \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT] \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark 1024000 100 0

If it succeed, you should be able to see something like this on the worker.

tests/test_kv_app_benchmark.cc:77: push_byte=4096000, repeat=100, total_time=128.842ms
tests/test_kv_app_benchmark.cc:91: pull_byte=4096000, repeat=100, total_time=353.38ms

If it still hang, then you should really check your network configurations...

tingweiwu commented 4 years ago

@ymjiang install pslite failed at the make step. should you provider image which contains pslite?

ymjiang commented 4 years ago

@tingweiwu ps-lite is very light-weight and probably does not need an image to run on. Could you paste your Make output here? We may help you debug. Thank you.

tingweiwu commented 4 years ago

@ymjiang 

checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/root/ps-lite/protobuf-3.5.1':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
make: *** [/root/ps-lite/deps/include/google/protobuf/message.h] Error 1
ymjiang commented 4 years ago

Try this:

apt-get update 
apt-get install libprotobuf-dev protobuf-compiler
yangwenhuan commented 4 years ago

I follow the step-by-step-tutorial to run distributed training with mxnet and tensorflow, both hang. I have 3 nodes and on first node I run scheduler and server and second and third node I run worker0 and worker1 cmd and logs as following

scheduler:

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=2 
root@--0002:~# export DMLC_ROLE=scheduler 
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching scheduler

server:

[root@--0002 ~]# docker run -it --net=host bytepsimage/byteps_server bash
root@--0002:~# export DMLC_NUM_WORKER=2 
root@--0002:~# export DMLC_ROLE=server  
root@--0002:~# export DMLC_NUM_SERVER=1 
root@--0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@--0002:~# export DMLC_PS_ROOT_PORT=1234  
root@--0002:~# 
root@--0002:~# python /usr/local/byteps/launcher/launch.py
BytePS launching server
[[03:13:29] src/customer.cc:03:13:29] 226: Use seperate thread to process pull requests from each worker.
src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[03:13:29[] src/customer.cc:239: Server uses 1 threads to process push requests.
03:13:29] src/customer.cc:137: Server inits Push Thread-0
[03:13:29] src/customer.cc:93: Server inits Pull Thread-1
[03:13:29] src/customer.cc:93: Server inits Pull Thread-0
[03:13:29] src/customer.cc:258: All threads inited, ready to process message 
[03:13:29] src/./zmq_van.h:285: Start ZMQ recv thread

worker0

[root@0001 ~]# docker run --name worker-0 -it --net=host --shm-size=32768m bytepsimage/worker_mxnet bash
root@0001:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3  
root@0001:~# export DMLC_WORKER_ID=0 
root@0001:~# export DMLC_NUM_WORKER=2 
root@0001:~# export DMLC_ROLE=worker
root@0001:~# export DMLC_NUM_SERVER=1 
root@0001:~# export DMLC_PS_ROOT_URI=10.94.1.77 
root@0001:~# export DMLC_PS_ROOT_PORT=1234
root@0001:~# 
root@0001:~# export EVAL_TYPE=benchmark 
root@0001:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32  
BytePS launching worker
[03:14:57] src/customer.cc:363: Do not use thread pool for receiving.
[03:14:57] src/./zmq_van.h:285: Start ZMQ recv thread
[03:15:12] src/./zmq_van.h:285: Start ZMQ recv thread
[03:15:12] src/./zmq_van.h:285: Start ZMQ recv thread

worker1

[root@0002 ~]# docker run --name worker-1 -it --net=host --shm-size=32768m train.registry.docker.com:5000/bytepsimage/worker_mxnet bash
root@0002:~# export NVIDIA_VISIBLE_DEVICES=0,1,2,3 
root@0002:~# export DMLC_WORKER_ID=1
root@0002:~# export DMLC_NUM_WORKER=2 
root@0002:~# export DMLC_ROLE=worker 
root@0002:~# export DMLC_NUM_SERVER=1 
root@0002:~# export DMLC_PS_ROOT_URI=10.94.1.77
root@0002:~# export DMLC_PS_ROOT_PORT=1234
root@0002:~# 
root@0002:~# export EVAL_TYPE=benchmark 
root@0002:~# python /usr/local/byteps/launcher/launch.py \
>        /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh \
>        --benchmark 1 --batch-size=32 
BytePS launching worker
[03:15:17] src/customer.cc:363: Do not use thread pool for receiving.
[03:15:17] src/./zmq_van.h:285: Start ZMQ recv thread
[03:15:17] src/./zmq_van.h:285: Start ZMQ recv thread

In addition after I input ctrl+c to abort the process. and ps -ef |grep python shows that the process still there?

[03:15:17] src/./zmq_van.h:285: Start ZMQ recv thread
^C[[[2019-07-03 03:24:17.2019-07-03 03:24:172019-07-03 03:24:17579377..: 579377579381F: :  FFbyteps/common/communicator.cc  :byteps/common/communicator.cc206byteps/common/communicator.cc:] :206Check failed: (rc) >= (0) Interrupted system call, rank=0206] 
] Check failed: (rc) >= (0) Interrupted system call, rank=2Check failed: (rc) >= (0) Interrupted system call, rank=1

^C^C^C^C^C^C^C^C/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh: line 5:    32 Aborted                 (core dumped) python $path/train_imagenet_byteps.py $@
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/byteps/launcher/launch.py", line 19, in worker
    subprocess.check_call(command, env=my_env, stdout=sys.stdout, stderr=sys.stderr, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh --benchmark 1 --batch-size=32' returned non-zero exit status -2

/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh: line 5:    26 Aborted                 (core dumped) python $path/train_imagenet_byteps.py $@
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/byteps/launcher/launch.py", line 19, in worker
    subprocess.check_call(command, env=my_env, stdout=sys.stdout, stderr=sys.stderr, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/usr/local/byteps/example/mxnet/start_mxnet_byteps.sh --benchmark 1 --batch-size=32' returned non-zero exit status -2

Traceback (most recent call last):
  File "/usr/local/byteps/launcher/launch.py", line 38, in <module>
    t[i].join() 
  File "/usr/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/usr/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
KeyboardInterrupt
root@bms-bf19-0002:~# /usr/local/byteps/example/mxnet/start_mxnet_byteps.sh: line 5:    28 Aborted                 (core dumped) python $path/train_imagenet_byteps.py $@
^C
root@0002:~# ^C
root@0002:~# ps -ef | grep python
root         31     25 99 03:15 pts/0    00:30:46 python /usr/local/byteps/example/mxnet/train_imagenet_byteps.py --benchmark 1 --batch-size=32
root        768      1  0 03:25 pts/0    00:00:00 grep --color=auto python

and the server and scheduler didn't abort when input ctrl+c

BytePS launching server
[[03:13:29] src/customer.cc:03:13:29] 226: Use seperate thread to process pull requests from each worker.
src/kvstore/././kvstore_dist_server.h:166: Enable zero copy of pull operations.
[03:13:29[] src/customer.cc:239: Server uses 1 threads to process push requests.
03:13:29] src/customer.cc:137: Server inits Push Thread-0
[03:13:29] src/customer.cc:93: Server inits Pull Thread-1
[03:13:29] src/customer.cc:93: Server inits Pull Thread-0
[03:13:29] src/customer.cc:258: All threads inited, ready to process message 
[03:13:29] src/./zmq_van.h:285: Start ZMQ recv thread
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

@tingweiwu, hi, did you solved the problem? I met the same.

ymjiang commented 4 years ago

@yangwenhuan Did you try this? Just put them on the same machine to see if it works.

In addition to what @ymjiang says, can you try starting worker/server/scheduler on the same machine, and use 127.0.0.1 as DMLC_PS_ROOT_URI?

This will help us check whether this problem is due to the network.

yangwenhuan commented 4 years ago

@ymjiang , not yet, I try it now.

yangwenhuan commented 4 years ago

@ymjiang, it seems communication is successful.

scheduler：

BytePS launching scheduler [10:57:17] src/./zmq_van.h:61: BYTEPS_ZMQ_MAX_SOCKET set to 1024 [10:57:17] src/./zmq_van.h:66: BYTEPS_ZMQ_NTHREADS set to 4 [10:57:17] src/van.cc:357[: Bind to role=scheduler, id=1, ip=127.0.0.1, port=12345, is_recovery=0 10:57:17] src/./zmq_van.h:285: Start ZMQ recv thread [10:57:53] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.0.1.26, port=49581, is_recovery=0 } }. THIS IS NOT DATA MSG! [10:58:54] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.0.1.26, port=49460, is_recovery=0 } }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:471: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.0.1.26, port=49689, is_recovery=0 } }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:108: assign rank=9 to node role=worker, ip=10.0.1.26, port=49460, is_recovery=0 [11:00:12] src/van.cc:108: assign rank=8 to node role=server, ip=10.0.1.26, port=49581, is_recovery=0 [11:00:12] src/van.cc:108: assign rank=11 to node role=worker, ip=10.0.1.26, port=49689, is_recovery=0 [11:00:12] src/van.cc:446: ? => 9. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ role=worker, id=9, ip=10.0.1.26, port=49460, is_recovery=0 role=server, id=8, ip=10.0.1.26, port=49581, is_recovery=0 role=worker, id=11, ip=10.0.1.26, port=49689, is_recovery=0 role=scheduler, id=1, ip=127.0.0.1, port=12345, is_recovery=0 } }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ role=worker, id=9, ip=10.0.1.26, port=49460, is_recovery=0 role=server, id=8, ip=10.0.1.26, port=49581, is_recovery=0 role=worker, id=11, ip=10.0.1.26, port=49689, is_recovery=0 role=scheduler, id=1, ip=127.0.0.1, port=12345, is_recovery=0 } }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:446: ? => 8. Meta: request=0, timestamp=2, control={ cmd=ADD_NODE, node={ role=worker, id=9, ip=10.0.1.26, port=49460, is_recovery=0 role=server, id=8, ip=10.0.1.26, port=49581, is_recovery=0 role=worker, id=11, ip=10.0.1.26, port=49689, is_recovery=0 role=scheduler, id=1, ip=127.0.0.1, port=12345, is_recovery=0 } }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:134: the scheduler is connected to 2 workers and 1 servers [11:00:12] src/van.cc:471: 8 => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:233: Barrier count for 7 : 1 [11:00:12] src/van.cc:471: 9 => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:233: Barrier count for 7 : 2 [11:00:12] src/van.cc:446: ? => 1. Meta: request=1, timestamp=3, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:471: 1 => 1. Meta: request=1, timestamp=3, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:233: Barrier count for 7 : 3 [11:00:12] src/van.cc:471: 11 => 1. Meta: request=1, timestamp=1, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:233: Barrier count for 7 : 4 [11:00:12] src/van.cc:446: ? => 9. Meta: request=0, timestamp=4, control={ cmd=BARRIER, barrier_group=255416528 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:446: ? => 11. Meta: request=0, timestamp=5, control={ cmd=BARRIER, barrier_group=255416528 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:446: ? => 8. Meta: request=0, timestamp=6, control={ cmd=BARRIER, barrier_group=255416528 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:446: ? => 1. Meta: request=0, timestamp=7, control={ cmd=BARRIER, barrier_group=255416528 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:471: 1 => 1. Meta: request=0, timestamp=7, control={ cmd=BARRIER, barrier_group=255416528 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:446: ? => 1. Meta: request=1, timestamp=8, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:471: 1 => 1. Meta: request=1, timestamp=8, control={ cmd=BARRIER, barrier_group=7 }. THIS IS NOT DATA MSG! [11:00:12] src/van.cc:233: Barrier count for 7 : 1

server：

[11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1047, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1047, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1048, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1048, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1048, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1048, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1049, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1049, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1049, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1049, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1050, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1050, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1050, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1050, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1051, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1051, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1051, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1051, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1052, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1052, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1052, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1052, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1053, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1053, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1053, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1053, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1054, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1054, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1054, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1054, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1055, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1055, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1055, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1055, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1056, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1056, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1056, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1056, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1057, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1057, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1057, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1057, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1058, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1058, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1058, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1058, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1059, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1059, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1059, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1059, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1060, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1060, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1060, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1060, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1061, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1061, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1061, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1061, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1062, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1062, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1062, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1062, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1063, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1063, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1063, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1063, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1064, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1064, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1064, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1064, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1065, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1065, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1065, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1065, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1066, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1066, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1066, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1066, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1067, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1067, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1067, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1067, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1068, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1068, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1068, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1068, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1069, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1069, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1069, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1069, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1070, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1070, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1070, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1070, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1071, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1071, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1071, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1071, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1072, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1072, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1072, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1072, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1073, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1073, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1073, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1073, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1074, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1074, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1074, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1074, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1075, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1075, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1075, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1075, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1076, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1076, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1076, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1076, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1077, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1077, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1077, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1077, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1078, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1078, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1078, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1078, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1079, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1079, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1079, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1079, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1080, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1080, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1080, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1080, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1081, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1081, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1081, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1081, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1082, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1082, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1082, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1082, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1083, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1083, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1083, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1083, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1084, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1084, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1084, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1084, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1085, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1085, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1085, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1085, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1086, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1086, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1086, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1086, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1087, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1087, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1087, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1087, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1088, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1088, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1088, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1088, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1089, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1089, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1089, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1089, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1090, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1090, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1090, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1090, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1091, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1091, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1091, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1091, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1092, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1092, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1092, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1092, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1093, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1093, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1093, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1093, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1094, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1094, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1094, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1094, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1095, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2097152 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1095, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2097152 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1095, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1095, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1096, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1096, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1096, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1096, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1097, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1097, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=4096 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1097, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1097, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1098, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1098, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1048576 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1098, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1098, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1099, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1099, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1099, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1099, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1100, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1100, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1100, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1100, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1101, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1101, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2359296 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1101, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1101, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1102, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1102, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1102, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1102, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1103, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1103, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1103, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1103, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1104, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=524288 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1104, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=524288 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1104, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1104, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1105, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1105, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1105, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1105, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1106, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1106, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1106, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1106, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1107, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1107, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1107, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1107, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1108, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1108, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1108, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1108, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1109, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1109, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1109, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1109, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1110, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1110, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1110, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1110, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1111, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1111, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1111, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1111, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1112, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1112, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1112, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1112, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1113, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1113, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1113, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1113, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1114, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1114, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1114, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1114, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1115, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1115, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1115, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1115, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1116, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1116, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1116, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1116, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1117, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1117, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1117, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1117, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1118, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1118, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1118, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1118, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1119, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1119, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1119, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1119, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1120, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1120, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1120, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1120, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1121, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1121, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1121, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1121, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1122, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1122, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1122, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1122, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1123, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1123, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1123, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1123, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1124, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1124, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1124, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1124, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1125, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1125, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1125, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1125, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1126, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1126, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1126, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1126, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1127, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1127, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1127, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1127, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1128, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1128, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1128, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1128, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1129, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1129, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1129, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1129, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1130, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1130, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1130, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1130, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1131, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1131, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1131, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1131, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1132, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1132, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1132, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1132, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1133, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1133, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1133, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1133, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1134, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=524288 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1134, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=524288 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1134, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1134, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1135, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1135, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1135, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1135, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1136, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1136, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=2048 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1136, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1136, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1137, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1137, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=262144 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1137, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1137, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1138, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1138, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1138, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1138, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1139, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1139, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1139, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1139, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1140, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1140, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=589824 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1140, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1140, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1141, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1141, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1141, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1141, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1142, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1142, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=512 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1142, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1142, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1143, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=131072 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1143, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=131072 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1143, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1143, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1144, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1144, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1144, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1144, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1145, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1145, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1145, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1145, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1146, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1146, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1146, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1146, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1147, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1147, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1147, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1147, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1148, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:36] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1148, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:36] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1148, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:36] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1148, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1149, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1149, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1149, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1149, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1150, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1150, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1150, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1150, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1151, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1151, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1151, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1151, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1152, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1152, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1152, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1152, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1153, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1153, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1153, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1153, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1154, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1154, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1154, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1154, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1155, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1155, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1155, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1155, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1156, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1156, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1156, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1156, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1157, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1157, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1157, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1157, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1158, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1158, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1158, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1158, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1159, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1159, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1159, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1159, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1160, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1160, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1160, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1160, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1161, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1161, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1161, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1161, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1162, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1162, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1162, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1162, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1163, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1163, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1163, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1163, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1164, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1164, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1164, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1164, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1165, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1165, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1165, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1165, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1166, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1166, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=1024 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1166, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1166, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1167, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1167, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=65536 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1167, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1167, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=16384 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=16384 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:471: 9 => 8. Meta: request=1, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=37632 data_size=4 [11:00:37] src/van.cc:471: 11 => 8. Meta: request=1, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=37632 data_size=4 [11:00:37] src/van.cc:446: ? => 9. Meta: request=0, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 11. Meta: request=0, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0

worker0：

[11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=16384 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=37632 data_size=4 [11:00:37] src/van.cc:471: 8 => 11. Meta: request=0, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0

worker1：

[11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1168, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1169, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=147456 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1170, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1171, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1172, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=16384 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1173, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1174, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=256 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1175, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0 [11:00:37] src/van.cc:446: ? => 8. Meta: request=1, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER INT32 } Body: data_size=8 data_size=37632 data_size=4 [11:00:37] src/van.cc:471: 8 => 9. Meta: request=0, timestamp=1176, app_id=0, customer_id=0, simple_app=0, push=1, head=0, key=0, data_type={ UINT64 OTHER } Body: data_size=8 data_size=0

tingweiwu commented 4 years ago

@yangwenhuan I haven't solved yet .and I still do not think run successfully on the same node indicates my network configurations incorrect.

@ymjiang you may pay attention to this issue. seems like it is not individual problem I haven't test ps-lite due to I've been busy with other things lately

yangwenhuan commented 4 years ago

@tingweiwu Can you try to run the basic pslite benchmark? Here are the steps you need to follow:

Install ps-lite

git clone --single-branch --branch byteps https://github.com/bytedance/ps-lite.git
cd ps-lite
make -j

For the scheduler

DMLC_ROLE=scheduler \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT] \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark

For the server

DMLC_ROLE=server \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT]  \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark

For the worker:

DMLC_ROLE=worker \
DMLC_NUM_WORKER=1 \
DMLC_NUM_SERVER=1 \
DMLC_PS_ROOT_URI=[YOUR_SCHEDULER_IP] \
DMLC_PS_ROOT_PORT=[YOUR_SCHEDULER_PORT] \
DMLC_INTERFACE=eth0 \
./ps-lite/tests/test_kv_app_benchmark 1024000 100 0

If it succeed, you should be able to see something like this on the worker.

tests/test_kv_app_benchmark.cc:77: push_byte=4096000, repeat=100, total_time=128.842ms
tests/test_kv_app_benchmark.cc:91: pull_byte=4096000, repeat=100, total_time=353.38ms

If it still hang, then you should really check your network configurations...

@ymjiang , the folowing unexpected result means something error with my network ? scheduler:

DMLC_ROLE=scheduler \

DMLC_NUM_WORKER=1 \ DMLC_NUM_SERVER=1 \ DMLC_PS_ROOT_URI=10.0.1.30 \ DMLC_PS_ROOT_PORT=12345 \ DMLC_INTERFACE=eth0 \ ./ps-lite/tests/test_kv_app_benchmark [21:25:17] src/./zmq_van.h:285: Start ZMQ recv thread

server：

DMLC_ROLE=server \ DMLC_NUM_WORKER=1 \ DMLC_NUM_SERVER=1 \ DMLC_PS_ROOT_URI=10.0.1.30 \ DMLC_PS_ROOT_PORT=12345 \ DMLC_NODE_HOST=10.0.1.27 \ DMLC_INTERFACE=eth0 \ ./ps-lite/tests/test_kv_app_benchmark

worker：

DMLC_ROLE=worker \

DMLC_NUM_WORKER=1 \ DMLC_NUM_SERVER=1 \ DMLC_PS_ROOT_URI=10.0.1.30 \ DMLC_PS_ROOT_PORT=12345 \ DMLC_NODE_HOST=10.0.1.26 \ DMLC_INTERFACE=eth0 \ ./ps-lite/tests/test_kv_app_benchmark 1024000 100 0 [21:28:19] src/./zmq_van.h:285: Start ZMQ recv thread

whn09 commented 4 years ago

I think it's your network configuration error. If you are using AWS or other cloud services, you should change the security group inbound rules like this:

Screen Shot 2020-01-20 at 13 56 35

The port you should open can be seen from the log of server, worker0 and worker1.

You can also open a range of port in case of the port change.