During replication server consumes all RAM and crashes

If you suspect this could be a bug, follow the template.

What version of Dgraph are you using? docker dgraph/dgraph:v1.0.5
Have you tried reproducing the issue with latest release? Yes, I tried it on dgraph/dgraph:master
What is the hardware spec (RAM, OS)? I run kubernetes cluster on AWS. Instance is m5.xlarge with 16 GB of RAM and 250 SSD HD

Steps to reproduce the issue (command/config used to run Dgraph).

Start kubernetes cluster with configuration:

# There are 4 public services exposed, users can use:
#       dgraph-zero-public - To load data using Live & Bulk Loaders
#       dgraph-server-public - To connect clients and for HTTP APIs
#       dgraph-ratel-public - For Dgraph UI
#       dgraph-server-x-http-public - Use for debugging & profiling
apiVersion: v1
kind: Service
metadata:
name: dgraph-zero-public
labels:
app: dgraph-zero
spec:
type: LoadBalancer
ports:
- port: 5080
targetPort: 5080
name: zero-grpc
- port: 6080
targetPort: 6080
name: zero-http
selector:
app: dgraph-zero
---
apiVersion: v1
kind: Service
metadata:
name: dgraph-server-public
labels:
app: dgraph-server
spec:
type: LoadBalancer
ports:
- port: 8080
targetPort: 8080
name: server-http
- port: 9080
targetPort: 9080
name: server-grpc
selector:
app: dgraph-server
---
# This service is created in-order to debug & profile a specific server.
# You can create one for each server that you need to profile.
# For a more general HTTP APIs use the above service instead.
apiVersion: v1
kind: Service
metadata:
name: dgraph-ratel-public
labels:
app: dgraph-ratel
spec:
type: NodePort
ports:
- port: 8000
targetPort: 8000
name: ratel-http
selector:
app: dgraph-ratel
---
# This is a headless service which is neccessary for discovery for a dgraph-zero StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
name: dgraph-zero
labels:
app: dgraph-zero
spec:
ports:
- port: 5080
targetPort: 5080
name: zero-grpc
clusterIP: None
selector:
app: dgraph-zero
---
# This is a headless service which is neccessary for discovery for a dgraph-server StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
name: dgraph-server
labels:
app: dgraph-server
spec:
ports:
- port: 7080
targetPort: 7080
name: server-grpc-int
clusterIP: None
selector:
app: dgraph-server
---
# This StatefulSet runs 3 Dgraph Zero.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dgraph-zero
spec:
serviceName: "dgraph-zero"
replicas: 1
selector:
matchLabels:
app: dgraph-zero
template:
metadata:
labels:
app: dgraph-zero
spec:
affinity:
podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
    podAffinityTerm:
      labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - dgraph-zero
      topologyKey: kubernetes.io/hostname
containers:
- name: zero
image: dgraph/dgraph:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5080
  name: zero-grpc
- containerPort: 6080
  name: zero-http
volumeMounts:
- name: datadir
  mountPath: /dgraph
command:
  - bash
  - "-c"
  - |
    set -ex
    [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
    ordinal=${BASH_REMATCH[1]}
    idx=$(($ordinal + 1))
    if [[ $ordinal -eq 0 ]]; then
      dgraph zero --my=$(hostname -f):5080 --idx $idx --replicas 3
    else
      dgraph zero --my=$(hostname -f):5080 --peer dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 --idx $idx --replicas 3
    fi
terminationGracePeriodSeconds: 60
volumes:
- name: datadir
persistentVolumeClaim:
  claimName: datadir
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
  storage: 5Gi
---
# This StatefulSet runs 6 Dgraph Server forming two server groups, 3 servers in each group.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dgraph-server
spec:
serviceName: "dgraph-server"
replicas: 0
selector:
matchLabels:
app: dgraph-server
template:
metadata:
labels:
app: dgraph-server
spec:
affinity:
podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 100
    podAffinityTerm:
      labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - dgraph-server
      topologyKey: kubernetes.io/hostname
containers:
- name: server
image: dgraph/dgraph:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 7080
  name: server-grpc-int
- containerPort: 8080
  name: server-http
- containerPort: 9080
  name: server-grpc
volumeMounts:
- name: datadir
  mountPath: /dgraph
command:
  - bash
  - "-c"
  - |
    set -ex
    dgraph server --my=$(hostname -f):7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
terminationGracePeriodSeconds: 60
volumes:
- name: datadir
persistentVolumeClaim:
  claimName: datadir
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
  storage: 250Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dgraph-ratel
labels:
app: dgraph-ratel
spec:
selector:
matchLabels:
app: dgraph-ratel
template:
metadata:
labels:
app: dgraph-ratel
spec:
containers:
- name: ratel
image: dgraph/dgraph:latest
ports:
- containerPort: 8000
command:
  - dgraph-ratel

When cluster is ready, I start dgraph bulk on separate machine with 32 GB of RAM. Bulk process is connected to k8s zero server and finish with

MAP 01h34m47s rdf_count:465.1M rdf_speed:81.77k/sec edge_count:1.560G edge_speed:274.4k/sec
...
REDUCE 03h11m37s [100.00%] edge_count:1.560G edge_speed:268.7k/sec plist_count:641.3M plist_speed:110.4k/sec

It produce about 111 GB.

I create k8s PVC SSD HDD (250 GB) for dgraph server 0 and copy bulk data on it.
Setup replication count for dgraph server to 1.

Wait for boot on.

kubectl logs -f dgraph-server-0
++ hostname -f
+ dgraph server --my=dgraph-server-0.dgraph-server.default.svc.cluster.local:7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 13:28:24 worker.go:89: Worker listening at address: [::]:7080
2018/06/01 13:28:24 gRPC server started.  Listening on port 9080
2018/06/01 13:28:24 HTTP server started.  Listening on port 8080
2018/06/01 13:28:24 groups.go:78: Current Raft Id: 0
2018/06/01 13:28:25 pool.go:108: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 13:28:25 groups.go:105: Connected to group zero. Assigned group: 1
2018/06/01 13:28:25 draft.go:170: Node ID: 1 with GroupID: 1
2018/06/01 13:28:25 node.go:240: Group 1 found 0 entries
2018/06/01 13:28:25 draft.go:937: New Node for group: 1
2018/06/01 13:28:25 raft.go:567: INFO: 1 became follower at term 0
2018/06/01 13:28:25 raft.go:315: INFO: newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018/06/01 13:28:25 raft.go:567: INFO: 1 became follower at term 1
2018/06/01 13:28:25 node.go:118: Setting conf state to nodes:1 
2018/06/01 13:28:25 raft.go:749: INFO: 1 is starting a new election at term 1
2018/06/01 13:28:25 groups.go:346: Serving tablet for: _predicate_
2018/06/01 13:28:25 raft.go:580: INFO: 1 became candidate at term 2
2018/06/01 13:28:25 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 2
2018/06/01 13:28:25 raft.go:621: INFO: 1 became leader at term 2
2018/06/01 13:28:25 node.go:301: INFO: raft.node: 1 elected leader 1 at term 2
2018/06/01 13:28:25 groups.go:346: Serving tablet for: at
2018/06/01 13:28:25 mutation.go:181: Done schema update predicate:"_predicate_" value_type:STRING list:true 
2018/06/01 13:28:25 groups.go:346: Serving tablet for: in
2018/06/01 13:28:25 groups.go:346: Serving tablet for: count
...
hundred of predicates
...
2018/06/01 13:35:07 groups.go:346: Serving tablet for: user.selected.organization
2018/06/01 13:35:07 groups.go:346: Serving tablet for: user.preferred.organization
2018/06/01 13:35:07 groups.go:346: Serving tablet for: notification.occurrence.date
2018/06/01 13:35:07 groups.go:346: Serving tablet for: dummy

Server is responding and everything looks good.

Replicas count do dgraph server is set on 2. Snapshot is sending:

2018/06/01 15:12:45 raft.go:749: INFO: 1 is starting a new election at term 2
2018/06/01 15:12:45 raft.go:580: INFO: 1 became candidate at term 3
2018/06/01 15:12:45 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 3
2018/06/01 15:12:45 raft.go:621: INFO: 1 became leader at term 3
2018/06/01 15:12:45 node.go:301: INFO: raft.node: 1 elected leader 1 at term 3
2018/06/01 17:45:11 pool.go:158: Echo error from dgraph-server-1.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/06/01 17:45:11 pool.go:108: == CONNECT ==> Setting dgraph-server-1.dgraph-server.default.svc.cluster.local:7080
2018/06/01 17:45:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05s, bytes sent: 22 MB, bytes/sec 4393253
2018/06/01 17:45:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10s, bytes sent: 44 MB, bytes/sec 4449208
2018/06/01 17:45:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15s, bytes sent: 66 MB, bytes/sec 4404129
2018/06/01 17:45:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20s, bytes sent: 88 MB, bytes/sec 4381694
2018/06/01 17:45:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25s, bytes sent: 110 MB, bytes/sec 4391688
2018/06/01 17:45:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30s, bytes sent: 131 MB, bytes/sec 4377680
2018/06/01 17:45:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 35s, bytes sent: 152 MB, bytes/sec 4347668
2018/06/01 17:45:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 40s, bytes sent: 174 MB, bytes/sec 4358675
2018/06/01 17:45:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 45s, bytes sent: 194 MB, bytes/sec 4318528
2018/06/01 17:46:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 50s, bytes sent: 216 MB, bytes/sec 4320323
2018/06/01 17:46:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 55s, bytes sent: 237 MB, bytes/sec 4301221
2018/06/01 17:46:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m00s, bytes sent: 257 MB, bytes/sec 4282766
2018/06/01 17:46:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m05s, bytes sent: 279 MB, bytes/sec 4288907
2018/06/01 17:46:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m10s, bytes sent: 299 MB, bytes/sec 4266223
2018/06/01 17:46:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m15s, bytes sent: 321 MB, bytes/sec 4276578
2018/06/01 17:46:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m20s, bytes sent: 340 MB, bytes/sec 4255449
2018/06/01 17:46:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m25s, bytes sent: 362 MB, bytes/sec 4253578
2018/06/01 17:46:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m30s, bytes sent: 383 MB, bytes/sec 4251331
2018/06/01 17:46:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m35s, bytes sent: 402 MB, bytes/sec 4229044
2018/06/01 17:46:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m40s, bytes sent: 423 MB, bytes/sec 4234240
2018/06/01 17:46:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m45s, bytes sent: 445 MB, bytes/sec 4235367
2018/06/01 17:47:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m50s, bytes sent: 463 MB, bytes/sec 4208357
2018/06/01 17:47:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m55s, bytes sent: 484 MB, bytes/sec 4205258
2018/06/01 17:47:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m00s, bytes sent: 505 MB, bytes/sec 4208297
2018/06/01 17:47:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m05s, bytes sent: 524 MB, bytes/sec 4193215
2018/06/01 17:47:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m10s, bytes sent: 543 MB, bytes/sec 4179465
2018/06/01 17:47:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m15s, bytes sent: 564 MB, bytes/sec 4177842
2018/06/01 17:47:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m20s, bytes sent: 584 MB, bytes/sec 4169674
2018/06/01 17:47:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m25s, bytes sent: 604 MB, bytes/sec 4162970
2018/06/01 17:47:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m30s, bytes sent: 622 MB, bytes/sec 4143588
2018/06/01 17:47:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m35s, bytes sent: 642 MB, bytes/sec 4140368
2018/06/01 17:47:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m40s, bytes sent: 661 MB, bytes/sec 4133150
2018/06/01 17:47:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m45s, bytes sent: 681 MB, bytes/sec 4124900
2018/06/01 17:48:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m50s, bytes sent: 701 MB, bytes/sec 4121609
2018/06/01 17:48:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m55s, bytes sent: 717 MB, bytes/sec 4097400
2018/06/01 17:48:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m00s, bytes sent: 736 MB, bytes/sec 4088239
2018/06/01 17:48:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m05s, bytes sent: 756 MB, bytes/sec 4084180
2018/06/01 17:48:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m10s, bytes sent: 774 MB, bytes/sec 4072266
2018/06/01 17:48:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m15s, bytes sent: 791 MB, bytes/sec 4055813
2018/06/01 17:48:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m19s, bytes sent: 811 MB, bytes/sec 4073739
2018/06/01 17:48:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m25s, bytes sent: 830 MB, bytes/sec 4050333
2018/06/01 17:48:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m30s, bytes sent: 850 MB, bytes/sec 4046088
2018/06/01 17:48:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m35s, bytes sent: 869 MB, bytes/sec 4043575
2018/06/01 17:48:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m40s, bytes sent: 886 MB, bytes/sec 4025607
2018/06/01 17:48:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m45s, bytes sent: 902 MB, bytes/sec 4011002
2018/06/01 17:49:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m50s, bytes sent: 921 MB, bytes/sec 4004382
2018/06/01 17:49:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m55s, bytes sent: 939 MB, bytes/sec 3997608
2018/06/01 17:49:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m00s, bytes sent: 958 MB, bytes/sec 3991401
2018/06/01 17:49:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m05s, bytes sent: 977 MB, bytes/sec 3986740
2018/06/01 17:49:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m10s, bytes sent: 996 MB, bytes/sec 3982052
2018/06/01 17:49:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m15s, bytes sent: 1.0 GB, bytes/sec 3977189
2018/06/01 17:49:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m20s, bytes sent: 1.0 GB, bytes/sec 3973324
2018/06/01 17:49:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m25s, bytes sent: 1.1 GB, bytes/sec 3970729
2018/06/01 17:49:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m30s, bytes sent: 1.1 GB, bytes/sec 3968236
2018/06/01 17:49:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m35s, bytes sent: 1.1 GB, bytes/sec 3965634
2018/06/01 17:49:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m40s, bytes sent: 1.1 GB, bytes/sec 3962986
2018/06/01 17:49:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m45s, bytes sent: 1.1 GB, bytes/sec 3950007
2018/06/01 17:50:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m50s, bytes sent: 1.1 GB, bytes/sec 3949570
2018/06/01 17:50:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m55s, bytes sent: 1.2 GB, bytes/sec 3948760
2018/06/01 17:50:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m00s, bytes sent: 1.2 GB, bytes/sec 3943994
2018/06/01 17:50:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m05s, bytes sent: 1.2 GB, bytes/sec 3938128
2018/06/01 17:50:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m10s, bytes sent: 1.2 GB, bytes/sec 3931558
2018/06/01 17:50:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m15s, bytes sent: 1.2 GB, bytes/sec 3927464
2018/06/01 17:50:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m20s, bytes sent: 1.3 GB, bytes/sec 3924201
2018/06/01 17:50:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m25s, bytes sent: 1.3 GB, bytes/sec 3919021
2018/06/01 17:50:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m30s, bytes sent: 1.3 GB, bytes/sec 3913088
2018/06/01 17:50:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m35s, bytes sent: 1.3 GB, bytes/sec 3908579
2018/06/01 17:50:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m40s, bytes sent: 1.3 GB, bytes/sec 3901949
2018/06/01 17:50:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m45s, bytes sent: 1.3 GB, bytes/sec 3895386
2018/06/01 17:51:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m50s, bytes sent: 1.4 GB, bytes/sec 3890055
2018/06/01 17:51:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m55s, bytes sent: 1.4 GB, bytes/sec 3890257
2018/06/01 17:51:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m00s, bytes sent: 1.4 GB, bytes/sec 3881119
2018/06/01 17:51:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m05s, bytes sent: 1.4 GB, bytes/sec 3883889
2018/06/01 17:51:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m10s, bytes sent: 1.4 GB, bytes/sec 3878649
2018/06/01 17:51:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m15s, bytes sent: 1.4 GB, bytes/sec 3862716
2018/06/01 17:51:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m20s, bytes sent: 1.5 GB, bytes/sec 3857627
2018/06/01 17:51:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m25s, bytes sent: 1.5 GB, bytes/sec 3850813
2018/06/01 17:51:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m30s, bytes sent: 1.5 GB, bytes/sec 3848179
2018/06/01 17:51:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m35s, bytes sent: 1.5 GB, bytes/sec 3852186
2018/06/01 17:51:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m40s, bytes sent: 1.5 GB, bytes/sec 3847207
2018/06/01 17:51:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m45s, bytes sent: 1.6 GB, bytes/sec 3839918
2018/06/01 17:52:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m50s, bytes sent: 1.6 GB, bytes/sec 3830104
2018/06/01 17:52:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m55s, bytes sent: 1.6 GB, bytes/sec 3824781
2018/06/01 17:52:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m00s, bytes sent: 1.6 GB, bytes/sec 3827268
2018/06/01 17:52:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m05s, bytes sent: 1.6 GB, bytes/sec 3823951
2018/06/01 17:52:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m10s, bytes sent: 1.6 GB, bytes/sec 3814258
2018/06/01 17:52:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m15s, bytes sent: 1.7 GB, bytes/sec 3807813
2018/06/01 17:52:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m20s, bytes sent: 1.7 GB, bytes/sec 3796504
2018/06/01 17:52:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m25s, bytes sent: 1.7 GB, bytes/sec 3801479
2018/06/01 17:52:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m30s, bytes sent: 1.7 GB, bytes/sec 3797149
2018/06/01 17:52:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m35s, bytes sent: 1.7 GB, bytes/sec 3784648
2018/06/01 17:52:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m40s, bytes sent: 1.7 GB, bytes/sec 3776229
2018/06/01 17:52:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m45s, bytes sent: 1.8 GB, bytes/sec 3770879
2018/06/01 17:53:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m50s, bytes sent: 1.8 GB, bytes/sec 3775632
2018/06/01 17:53:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m55s, bytes sent: 1.8 GB, bytes/sec 3767350
2018/06/01 17:53:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m00s, bytes sent: 1.8 GB, bytes/sec 3758027
2018/06/01 17:53:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m07s, bytes sent: 1.8 GB, bytes/sec 3744498
2018/06/01 17:53:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m10s, bytes sent: 1.8 GB, bytes/sec 3721572
2018/06/01 17:53:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m15s, bytes sent: 1.9 GB, bytes/sec 3780391
2018/06/01 17:53:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m19s, bytes sent: 1.9 GB, bytes/sec 3861983
2018/06/01 17:53:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m25s, bytes sent: 1.9 GB, bytes/sec 3860727
2018/06/01 17:53:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m30s, bytes sent: 2.0 GB, bytes/sec 3855591
2018/06/01 17:53:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m35s, bytes sent: 2.0 GB, bytes/sec 3848487
2018/06/01 17:53:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m40s, bytes sent: 2.0 GB, bytes/sec 3841572
2018/06/01 17:53:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m45s, bytes sent: 2.0 GB, bytes/sec 3847627
2018/06/01 17:54:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m50s, bytes sent: 2.0 GB, bytes/sec 3841937
2018/06/01 17:54:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m55s, bytes sent: 2.1 GB, bytes/sec 3835490
2018/06/01 17:54:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m00s, bytes sent: 2.1 GB, bytes/sec 3838153
2018/06/01 17:54:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m05s, bytes sent: 2.1 GB, bytes/sec 3836492
2018/06/01 17:54:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m10s, bytes sent: 2.1 GB, bytes/sec 3829414
2018/06/01 17:54:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m15s, bytes sent: 2.1 GB, bytes/sec 3822346
2018/06/01 17:54:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m20s, bytes sent: 2.1 GB, bytes/sec 3828629
2018/06/01 17:54:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m25s, bytes sent: 2.2 GB, bytes/sec 3820716
2018/06/01 17:54:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m30s, bytes sent: 2.2 GB, bytes/sec 3812062
2018/06/01 17:54:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m35s, bytes sent: 2.2 GB, bytes/sec 3817085
2018/06/01 17:54:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m40s, bytes sent: 2.2 GB, bytes/sec 3812566
2018/06/01 17:54:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m45s, bytes sent: 2.2 GB, bytes/sec 3804485
2018/06/01 17:55:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m50s, bytes sent: 2.2 GB, bytes/sec 3803759
2018/06/01 17:55:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m55s, bytes sent: 2.3 GB, bytes/sec 3804736
2018/06/01 17:55:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m59s, bytes sent: 2.3 GB, bytes/sec 3797506
2018/06/01 17:55:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m05s, bytes sent: 2.3 GB, bytes/sec 3783498
2018/06/01 17:55:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m10s, bytes sent: 2.3 GB, bytes/sec 3790308
2018/06/01 17:55:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m15s, bytes sent: 2.3 GB, bytes/sec 3781984
2018/06/01 17:55:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m20s, bytes sent: 2.3 GB, bytes/sec 3774783
2018/06/01 17:55:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m25s, bytes sent: 2.4 GB, bytes/sec 3777479
2018/06/01 17:55:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m30s, bytes sent: 2.4 GB, bytes/sec 3769624
2018/06/01 17:55:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m35s, bytes sent: 2.4 GB, bytes/sec 3767215
2018/06/01 17:55:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m40s, bytes sent: 2.4 GB, bytes/sec 3770315
2018/06/01 17:55:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m45s, bytes sent: 2.4 GB, bytes/sec 3762009
2018/06/01 17:56:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m50s, bytes sent: 2.4 GB, bytes/sec 3762667
2018/06/01 17:56:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m55s, bytes sent: 2.5 GB, bytes/sec 3761090
2018/06/01 17:56:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m00s, bytes sent: 2.5 GB, bytes/sec 3751960
2018/06/01 17:56:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m05s, bytes sent: 2.5 GB, bytes/sec 3757080
2018/06/01 17:56:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m10s, bytes sent: 2.5 GB, bytes/sec 3751188
2018/06/01 17:56:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m15s, bytes sent: 2.5 GB, bytes/sec 3743468
2018/06/01 17:56:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m20s, bytes sent: 2.6 GB, bytes/sec 3750871
2018/06/01 17:56:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m25s, bytes sent: 2.6 GB, bytes/sec 3742799
2018/06/01 17:56:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m30s, bytes sent: 2.6 GB, bytes/sec 3738491
2018/06/01 17:56:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m35s, bytes sent: 2.6 GB, bytes/sec 3738602
2018/06/01 17:56:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m40s, bytes sent: 2.6 GB, bytes/sec 3730983
2018/06/01 17:56:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m45s, bytes sent: 2.6 GB, bytes/sec 3735215
2018/06/01 17:57:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m50s, bytes sent: 2.6 GB, bytes/sec 3730017
2018/06/01 17:57:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m55s, bytes sent: 2.7 GB, bytes/sec 3721870
2018/06/01 17:57:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m00s, bytes sent: 2.7 GB, bytes/sec 3728146
2018/06/01 17:57:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m05s, bytes sent: 2.7 GB, bytes/sec 3720046
2018/06/01 17:57:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m10s, bytes sent: 2.7 GB, bytes/sec 3715153
2018/06/01 17:57:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m15s, bytes sent: 2.7 GB, bytes/sec 3715948
2018/06/01 17:57:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m20s, bytes sent: 2.7 GB, bytes/sec 3707757
2018/06/01 17:57:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m25s, bytes sent: 2.8 GB, bytes/sec 3711457
2018/06/01 17:57:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m30s, bytes sent: 2.8 GB, bytes/sec 3706027
2018/06/01 17:57:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m35s, bytes sent: 2.8 GB, bytes/sec 3699451
2018/06/01 17:57:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m40s, bytes sent: 2.8 GB, bytes/sec 3703611
2018/06/01 17:57:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m45s, bytes sent: 2.8 GB, bytes/sec 3694715
2018/06/01 17:58:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m50s, bytes sent: 2.8 GB, bytes/sec 3689339
2018/06/01 17:58:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m55s, bytes sent: 2.9 GB, bytes/sec 3687580
2018/06/01 17:58:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m00s, bytes sent: 2.9 GB, bytes/sec 3679697
2018/06/01 17:58:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m05s, bytes sent: 2.9 GB, bytes/sec 3685320
2018/06/01 17:58:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m10s, bytes sent: 2.9 GB, bytes/sec 3677117
2018/06/01 17:58:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m15s, bytes sent: 2.9 GB, bytes/sec 3684752
2018/06/01 17:58:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m20s, bytes sent: 2.9 GB, bytes/sec 3676421
2018/06/01 17:58:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m25s, bytes sent: 3.0 GB, bytes/sec 3676760
2018/06/01 17:58:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m30s, bytes sent: 3.0 GB, bytes/sec 3674590
2018/06/01 17:58:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m35s, bytes sent: 3.0 GB, bytes/sec 3668357
2018/06/01 17:58:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m40s, bytes sent: 3.0 GB, bytes/sec 3673005
2018/06/01 17:58:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m45s, bytes sent: 3.0 GB, bytes/sec 3664252
2018/06/01 17:59:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m50s, bytes sent: 3.0 GB, bytes/sec 3670518
2018/06/01 17:59:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m55s, bytes sent: 3.1 GB, bytes/sec 3662421
2018/06/01 17:59:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m00s, bytes sent: 3.1 GB, bytes/sec 3666432
2018/06/01 17:59:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m05s, bytes sent: 3.1 GB, bytes/sec 3660437
2018/06/01 17:59:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m10s, bytes sent: 3.1 GB, bytes/sec 3659825
2018/06/01 17:59:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m15s, bytes sent: 3.1 GB, bytes/sec 3657310
2018/06/01 17:59:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m20s, bytes sent: 3.1 GB, bytes/sec 3652183
2018/06/01 17:59:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m25s, bytes sent: 3.2 GB, bytes/sec 3653679
2018/06/01 17:59:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m30s, bytes sent: 3.2 GB, bytes/sec 3645780
2018/06/01 17:59:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m35s, bytes sent: 3.2 GB, bytes/sec 3651923
2018/06/01 17:59:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m40s, bytes sent: 3.2 GB, bytes/sec 3643520
2018/06/01 17:59:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m45s, bytes sent: 3.2 GB, bytes/sec 3650083
2018/06/01 18:00:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m50s, bytes sent: 3.2 GB, bytes/sec 3641529
2018/06/01 18:00:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m55s, bytes sent: 3.3 GB, bytes/sec 3643243
2018/06/01 18:00:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m00s, bytes sent: 3.3 GB, bytes/sec 3638431
2018/06/01 18:00:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m05s, bytes sent: 3.3 GB, bytes/sec 3636386
2018/06/01 18:00:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m10s, bytes sent: 3.3 GB, bytes/sec 3635220
2018/06/01 18:00:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m15s, bytes sent: 3.3 GB, bytes/sec 3629649
2018/06/01 18:00:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m20s, bytes sent: 3.3 GB, bytes/sec 3632656
2018/06/01 18:00:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m25s, bytes sent: 3.4 GB, bytes/sec 3624341
2018/06/01 18:00:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m30s, bytes sent: 3.4 GB, bytes/sec 3628998
2018/06/01 18:00:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m35s, bytes sent: 3.4 GB, bytes/sec 3619894
2018/06/01 18:00:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m40s, bytes sent: 3.4 GB, bytes/sec 3624584
2018/06/01 18:00:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m45s, bytes sent: 3.4 GB, bytes/sec 3616038
2018/06/01 18:01:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m50s, bytes sent: 3.4 GB, bytes/sec 3621920
2018/06/01 18:01:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m55s, bytes sent: 3.5 GB, bytes/sec 3612809
2018/06/01 18:01:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m00s, bytes sent: 3.5 GB, bytes/sec 3615435
2018/06/01 18:01:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m05s, bytes sent: 3.5 GB, bytes/sec 3609578
2018/06/01 18:01:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m10s, bytes sent: 3.5 GB, bytes/sec 3612228
2018/06/01 18:01:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m15s, bytes sent: 3.5 GB, bytes/sec 3606000
2018/06/01 18:01:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m20s, bytes sent: 3.5 GB, bytes/sec 3598348
2018/06/01 18:01:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m25s, bytes sent: 3.5 GB, bytes/sec 3599619
2018/06/01 18:01:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m30s, bytes sent: 3.6 GB, bytes/sec 3595854
2018/06/01 18:01:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m34s, bytes sent: 3.6 GB, bytes/sec 3599083
2018/06/01 18:01:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m40s, bytes sent: 3.6 GB, bytes/sec 3591672
2018/06/01 18:01:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m45s, bytes sent: 3.6 GB, bytes/sec 3592740
2018/06/01 18:02:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m50s, bytes sent: 3.6 GB, bytes/sec 3587499
2018/06/01 18:02:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m55s, bytes sent: 3.6 GB, bytes/sec 3589162
2018/06/01 18:02:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m00s, bytes sent: 3.7 GB, bytes/sec 3584319
2018/06/01 18:02:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m05s, bytes sent: 3.7 GB, bytes/sec 3579268
2018/06/01 18:02:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m13s, bytes sent: 3.7 GB, bytes/sec 3596038
2018/06/01 18:02:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m15s, bytes sent: 3.8 GB, bytes/sec 3637856
2018/06/01 18:02:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m20s, bytes sent: 3.8 GB, bytes/sec 3631768
2018/06/01 18:02:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m25s, bytes sent: 3.8 GB, bytes/sec 3642069
2018/06/01 18:02:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m30s, bytes sent: 3.8 GB, bytes/sec 3638064
2018/06/01 18:02:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m35s, bytes sent: 3.8 GB, bytes/sec 3640471
2018/06/01 18:02:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m40s, bytes sent: 3.9 GB, bytes/sec 3645572
2018/06/01 18:02:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m45s, bytes sent: 3.9 GB, bytes/sec 3641293
2018/06/01 18:03:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m50s, bytes sent: 3.9 GB, bytes/sec 3650106
2018/06/01 18:03:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m55s, bytes sent: 3.9 GB, bytes/sec 3642771
2018/06/01 18:03:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m00s, bytes sent: 3.9 GB, bytes/sec 3650795
2018/06/01 18:03:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m05s, bytes sent: 4.0 GB, bytes/sec 3651665
2018/06/01 18:03:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m10s, bytes sent: 4.0 GB, bytes/sec 3652878
2018/06/01 18:03:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m15s, bytes sent: 4.0 GB, bytes/sec 3656737
2018/06/01 18:03:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m20s, bytes sent: 4.0 GB, bytes/sec 3650280
2018/06/01 18:03:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m25s, bytes sent: 4.0 GB, bytes/sec 3659106
2018/06/01 18:03:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m30s, bytes sent: 4.1 GB, bytes/sec 3652469
2018/06/01 18:03:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m35s, bytes sent: 4.1 GB, bytes/sec 3660269
2018/06/01 18:03:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m40s, bytes sent: 4.1 GB, bytes/sec 3662390
2018/06/01 18:03:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m45s, bytes sent: 4.1 GB, bytes/sec 3662130
2018/06/01 18:04:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m50s, bytes sent: 4.1 GB, bytes/sec 3670620
2018/06/01 18:04:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m55s, bytes sent: 4.2 GB, bytes/sec 3662772
2018/06/01 18:04:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m00s, bytes sent: 4.2 GB, bytes/sec 3672566
2018/06/01 18:04:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m05s, bytes sent: 4.2 GB, bytes/sec 3672128
2018/06/01 18:04:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m10s, bytes sent: 4.2 GB, bytes/sec 3673396
2018/06/01 18:04:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m15s, bytes sent: 4.3 GB, bytes/sec 3682707
2018/06/01 18:04:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m20s, bytes sent: 4.3 GB, bytes/sec 3674007
2018/06/01 18:04:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m25s, bytes sent: 4.3 GB, bytes/sec 3678505
2018/06/01 18:04:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m30s, bytes sent: 4.3 GB, bytes/sec 3682463
2018/06/01 18:04:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m35s, bytes sent: 4.3 GB, bytes/sec 3679502
2018/06/01 18:04:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m40s, bytes sent: 4.4 GB, bytes/sec 3687799
2018/06/01 18:04:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m45s, bytes sent: 4.4 GB, bytes/sec 3682375
2018/06/01 18:05:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m50s, bytes sent: 4.4 GB, bytes/sec 3688005
2018/06/01 18:05:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m55s, bytes sent: 4.4 GB, bytes/sec 3695026
2018/06/01 18:05:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m00s, bytes sent: 4.4 GB, bytes/sec 3686564
2018/06/01 18:05:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m05s, bytes sent: 4.4 GB, bytes/sec 3691981
2018/06/01 18:05:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m10s, bytes sent: 4.5 GB, bytes/sec 3690294
2018/06/01 18:05:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m15s, bytes sent: 4.5 GB, bytes/sec 3689841
2018/06/01 18:05:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m20s, bytes sent: 4.5 GB, bytes/sec 3694863
2018/06/01 18:05:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m25s, bytes sent: 4.5 GB, bytes/sec 3687340
2018/06/01 18:05:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m30s, bytes sent: 4.5 GB, bytes/sec 3695152
2018/06/01 18:05:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m35s, bytes sent: 4.6 GB, bytes/sec 3699043
2018/06/01 18:05:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m40s, bytes sent: 4.6 GB, bytes/sec 3696077
2018/06/01 18:05:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m45s, bytes sent: 4.6 GB, bytes/sec 3704682
2018/06/01 18:06:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m50s, bytes sent: 4.6 GB, bytes/sec 3697894
2018/06/01 18:06:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m55s, bytes sent: 4.6 GB, bytes/sec 3700931
2018/06/01 18:06:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m00s, bytes sent: 4.7 GB, bytes/sec 3703253
2018/06/01 18:06:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m05s, bytes sent: 4.7 GB, bytes/sec 3695026
2018/06/01 18:06:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m10s, bytes sent: 4.7 GB, bytes/sec 3701953
2018/06/01 18:06:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m15s, bytes sent: 4.7 GB, bytes/sec 3703324
2018/06/01 18:06:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m20s, bytes sent: 4.7 GB, bytes/sec 3702260
2018/06/01 18:06:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m25s, bytes sent: 4.8 GB, bytes/sec 3710631
2018/06/01 18:06:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m30s, bytes sent: 4.8 GB, bytes/sec 3703069
2018/06/01 18:06:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m35s, bytes sent: 4.8 GB, bytes/sec 3709731
2018/06/01 18:06:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m40s, bytes sent: 4.8 GB, bytes/sec 3713884
2018/06/01 18:06:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m45s, bytes sent: 4.8 GB, bytes/sec 3710874
2018/06/01 18:07:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m50s, bytes sent: 4.9 GB, bytes/sec 3714888
2018/06/01 18:07:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m55s, bytes sent: 4.9 GB, bytes/sec 3711779
2018/06/01 18:07:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m00s, bytes sent: 4.9 GB, bytes/sec 3710852
2018/06/01 18:07:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m05s, bytes sent: 4.9 GB, bytes/sec 3718597
2018/06/01 18:07:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m10s, bytes sent: 4.9 GB, bytes/sec 3710743
2018/06/01 18:07:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m15s, bytes sent: 5.0 GB, bytes/sec 3718391
2018/06/01 18:07:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m20s, bytes sent: 5.0 GB, bytes/sec 3720274
2018/06/01 18:07:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m25s, bytes sent: 5.0 GB, bytes/sec 3713716
2018/06/01 18:07:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m30s, bytes sent: 5.0 GB, bytes/sec 3716661
2018/06/01 18:07:57 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m35s, bytes sent: 5.0 GB, bytes/sec 3713399
2018/06/01 18:07:57 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m45s, bytes sent: 5.0 GB, bytes/sec 3686194
2018/06/01 18:07:57 node.go:400: WARN: A tick missed to fire. Node blocks too long!
2018/06/01 18:08:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m50s, bytes sent: 5.0 GB, bytes/sec 3680665
2018/06/01 18:08:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m55s, bytes sent: 5.1 GB, bytes/sec 3686765
2018/06/01 18:08:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m00s, bytes sent: 5.1 GB, bytes/sec 3688270
2018/06/01 18:08:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m05s, bytes sent: 5.1 GB, bytes/sec 3681854
2018/06/01 18:08:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m10s, bytes sent: 5.1 GB, bytes/sec 3687864
2018/06/01 18:08:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m15s, bytes sent: 5.1 GB, bytes/sec 3688813
2018/06/01 18:08:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m20s, bytes sent: 5.2 GB, bytes/sec 3685633
2018/06/01 18:08:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m25s, bytes sent: 5.2 GB, bytes/sec 3692948
2018/06/01 18:08:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m30s, bytes sent: 5.2 GB, bytes/sec 3689993
2018/06/01 18:08:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m35s, bytes sent: 5.2 GB, bytes/sec 3691985
2018/06/01 18:08:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m40s, bytes sent: 5.3 GB, bytes/sec 3697662
2018/06/01 18:08:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m45s, bytes sent: 5.3 GB, bytes/sec 3691618
2018/06/01 18:09:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m50s, bytes sent: 5.3 GB, bytes/sec 3696240
2018/06/01 18:09:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m55s, bytes sent: 5.3 GB, bytes/sec 3703435
2018/06/01 18:09:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m00s, bytes sent: 5.3 GB, bytes/sec 3694984
2018/06/01 18:09:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m05s, bytes sent: 5.3 GB, bytes/sec 3702277
2018/06/01 18:09:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m10s, bytes sent: 5.4 GB, bytes/sec 3707766
2018/06/01 18:09:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m15s, bytes sent: 5.4 GB, bytes/sec 3701348
2018/06/01 18:09:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m20s, bytes sent: 5.4 GB, bytes/sec 3708834
2018/06/01 18:09:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m25s, bytes sent: 5.4 GB, bytes/sec 3710369
2018/06/01 18:09:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m30s, bytes sent: 5.5 GB, bytes/sec 3708364
2018/06/01 18:09:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m35s, bytes sent: 5.5 GB, bytes/sec 3711508
2018/06/01 18:09:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m40s, bytes sent: 5.5 GB, bytes/sec 3709659
2018/06/01 18:09:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m45s, bytes sent: 5.5 GB, bytes/sec 3709562
2018/06/01 18:10:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m50s, bytes sent: 5.5 GB, bytes/sec 3715472
2018/06/01 18:10:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m55s, bytes sent: 5.5 GB, bytes/sec 3709425
2018/06/01 18:10:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m00s, bytes sent: 5.6 GB, bytes/sec 3713845
2018/06/01 18:10:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m05s, bytes sent: 5.6 GB, bytes/sec 3717552
2018/06/01 18:10:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m10s, bytes sent: 5.6 GB, bytes/sec 3709310
2018/06/01 18:10:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m15s, bytes sent: 5.6 GB, bytes/sec 3715326
2018/06/01 18:10:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m20s, bytes sent: 5.7 GB, bytes/sec 3721215
2018/06/01 18:10:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m25s, bytes sent: 5.7 GB, bytes/sec 3713876
2018/06/01 18:10:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m30s, bytes sent: 5.7 GB, bytes/sec 3720092
2018/06/01 18:10:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m35s, bytes sent: 5.7 GB, bytes/sec 3721989
2018/06/01 18:10:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m40s, bytes sent: 5.7 GB, bytes/sec 3718912
2018/06/01 18:10:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m45s, bytes sent: 5.8 GB, bytes/sec 3724028
2018/06/01 18:11:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m50s, bytes sent: 5.8 GB, bytes/sec 3721840
2018/06/01 18:11:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m55s, bytes sent: 5.8 GB, bytes/sec 3720416
2018/06/01 18:11:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m00s, bytes sent: 5.8 GB, bytes/sec 3727138
2018/06/01 18:11:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m05s, bytes sent: 5.8 GB, bytes/sec 3724077
2018/06/01 18:11:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m10s, bytes sent: 5.8 GB, bytes/sec 3724611
2018/06/01 18:11:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m15s, bytes sent: 5.9 GB, bytes/sec 3731143
2018/06/01 18:11:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m20s, bytes sent: 5.9 GB, bytes/sec 3725705
2018/06/01 18:11:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m25s, bytes sent: 5.9 GB, bytes/sec 3729426
2018/06/01 18:11:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m30s, bytes sent: 5.9 GB, bytes/sec 3730807
2018/06/01 18:11:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m35s, bytes sent: 5.9 GB, bytes/sec 3724076
2018/06/01 18:11:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m40s, bytes sent: 6.0 GB, bytes/sec 3726924
2018/06/01 18:11:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m45s, bytes sent: 6.0 GB, bytes/sec 3729892
2018/06/01 18:12:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m50s, bytes sent: 6.0 GB, bytes/sec 3723104
2018/06/01 18:12:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m55s, bytes sent: 6.0 GB, bytes/sec 3727931
2018/06/01 18:12:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m00s, bytes sent: 6.0 GB, bytes/sec 3734131
2018/06/01 18:12:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m05s, bytes sent: 6.1 GB, bytes/sec 3727231
2018/06/01 18:12:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m10s, bytes sent: 6.1 GB, bytes/sec 3731079
2018/06/01 18:12:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m15s, bytes sent: 6.1 GB, bytes/sec 3734923
2018/06/01 18:12:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m20s, bytes sent: 6.1 GB, bytes/sec 3729985
2018/06/01 18:12:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m25s, bytes sent: 6.1 GB, bytes/sec 3731835
2018/06/01 18:12:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m30s, bytes sent: 6.2 GB, bytes/sec 3738031
2018/06/01 18:12:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m35s, bytes sent: 6.2 GB, bytes/sec 3731600
2018/06/01 18:12:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m40s, bytes sent: 6.2 GB, bytes/sec 3731866
2018/06/01 18:12:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m45s, bytes sent: 6.2 GB, bytes/sec 3738084
2018/06/01 18:13:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m50s, bytes sent: 6.2 GB, bytes/sec 3734261
2018/06/01 18:13:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m55s, bytes sent: 6.3 GB, bytes/sec 3735015
2018/06/01 18:13:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m00s, bytes sent: 6.3 GB, bytes/sec 3734983
2018/06/01 18:13:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m05s, bytes sent: 6.3 GB, bytes/sec 3734066
2018/06/01 18:13:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m10s, bytes sent: 6.3 GB, bytes/sec 3731217
2018/06/01 18:13:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m15s, bytes sent: 6.3 GB, bytes/sec 3736654
2018/06/01 18:13:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m20s, bytes sent: 6.4 GB, bytes/sec 3736024
2018/06/01 18:13:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m25s, bytes sent: 6.4 GB, bytes/sec 3733827
2018/06/01 18:13:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m30s, bytes sent: 6.4 GB, bytes/sec 3735442
2018/06/01 18:13:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m35s, bytes sent: 6.4 GB, bytes/sec 3736446
2018/06/01 18:13:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m40s, bytes sent: 6.4 GB, bytes/sec 3730353
2018/06/01 18:13:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m45s, bytes sent: 6.4 GB, bytes/sec 3736661
2018/06/01 18:14:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m50s, bytes sent: 6.5 GB, bytes/sec 3739884
2018/06/01 18:14:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m55s, bytes sent: 6.5 GB, bytes/sec 3733970
2018/06/01 18:14:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m00s, bytes sent: 6.5 GB, bytes/sec 3737049
2018/06/01 18:14:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m05s, bytes sent: 6.5 GB, bytes/sec 3742519
2018/06/01 18:14:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m10s, bytes sent: 6.5 GB, bytes/sec 3734867
2018/06/01 18:14:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m15s, bytes sent: 6.6 GB, bytes/sec 3741173
2018/06/01 18:14:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m20s, bytes sent: 6.6 GB, bytes/sec 3745944
2018/06/01 18:14:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m25s, bytes sent: 6.6 GB, bytes/sec 3738840
2018/06/01 18:14:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m30s, bytes sent: 6.6 GB, bytes/sec 3744760
2018/06/01 18:14:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m35s, bytes sent: 6.7 GB, bytes/sec 3749055
2018/06/01 18:14:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m40s, bytes sent: 6.7 GB, bytes/sec 3743877
2018/06/01 18:14:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m45s, bytes sent: 6.7 GB, bytes/sec 3750013
2018/06/01 18:15:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m50s, bytes sent: 6.7 GB, bytes/sec 3754006
2018/06/01 18:15:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m55s, bytes sent: 6.7 GB, bytes/sec 3746860
2018/06/01 18:15:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m00s, bytes sent: 6.8 GB, bytes/sec 3751817
2018/06/01 18:15:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m05s, bytes sent: 6.8 GB, bytes/sec 3757730
2018/06/01 18:15:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m10s, bytes sent: 6.8 GB, bytes/sec 3749419
2018/06/01 18:15:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m15s, bytes sent: 6.8 GB, bytes/sec 3755456
2018/06/01 18:15:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m20s, bytes sent: 6.8 GB, bytes/sec 3761676
2018/06/01 18:15:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m25s, bytes sent: 6.8 GB, bytes/sec 3753418
2018/06/01 18:15:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m30s, bytes sent: 6.9 GB, bytes/sec 3759134
2018/06/01 18:15:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m35s, bytes sent: 6.9 GB, bytes/sec 3765156
2018/06/01 18:15:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m40s, bytes sent: 6.9 GB, bytes/sec 3758325
2018/06/01 18:15:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m45s, bytes sent: 6.9 GB, bytes/sec 3764308
2018/06/01 18:16:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m49s, bytes sent: 7.0 GB, bytes/sec 3772195
2018/06/01 18:16:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m55s, bytes sent: 7.0 GB, bytes/sec 3762407
2018/06/01 18:16:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 31m00s, bytes sent: 7.0 GB, bytes/sec 3767278
2018/06/01 18:16:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 31m05s, bytes sent: 7.0 GB, bytes/sec 3773062

On second terminal I control memory usage with htop and RAM is constantly consumed and when all memory is used service crashes. After restart 0 bytes of snapshot is send:

++ hostname -f
+ dgraph server --my=dgraph-server-0.dgraph-server.default.svc.cluster.local:7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 18:18:59 gRPC server started.  Listening on port 9080
2018/06/01 18:18:59 HTTP server started.  Listening on port 8080
2018/06/01 18:18:59 worker.go:89: Worker listening at address: [::]:7080
2018/06/01 18:18:59 groups.go:78: Current Raft Id: 1
2018/06/01 18:18:59 pool.go:108: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 18:18:59 groups.go:105: Connected to group zero. Assigned group: 0
2018/06/01 18:18:59 pool.go:108: == CONNECT ==> Setting dgraph-server-1.dgraph-server.default.svc.cluster.local:7080
2018/06/01 18:18:59 draft.go:170: Node ID: 1 with GroupID: 1
2018/06/01 18:18:59 node.go:228: Found hardstate: {Term:3 Vote:1 Commit:7 XXX_unrecognized:[]}
2018/06/01 18:18:59 node.go:240: Group 1 found 7 entries
2018/06/01 18:18:59 draft.go:926: Restarting node for group: 1
2018/06/01 18:18:59 raft.go:567: INFO: 1 became follower at term 3
2018/06/01 18:18:59 raft.go:315: INFO: newRaft 1 [peers: [], term: 3, commit: 7, applied: 0, lastindex: 7, lastterm: 3]
2018/06/01 18:18:59 node.go:118: Setting conf state to nodes:1 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"_predicate_" value_type:STRING list:true 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"element.type.in" value_type:UID directive:REVERSE 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"element.type.in" value_type:UID directive:REVERSE 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"_share_hash_" value_type:STRING directive:INDEX tokenizer:"exact" 
2018/06/01 18:19:03 raft.go:749: INFO: 1 is starting a new election at term 3
2018/06/01 18:19:03 raft.go:580: INFO: 1 became candidate at term 4
2018/06/01 18:19:03 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 4
2018/06/01 18:19:03 raft.go:621: INFO: 1 became leader at term 4
2018/06/01 18:19:03 node.go:301: INFO: raft.node: 1 elected leader 1 at term 4
2018/06/01 18:19:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 44s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 59s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m59s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m00s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m00s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m04s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m24s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m00s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:43 node.go:118: Setting conf state to nodes:1 nodes:2

After that, server is sometimes returns correct results and sometimes return nothing.

Expected behaviour and actual result.

Replication is finished and server returns correct results always

This might be due to using MemoryStorage for Raft WAL. One of the servers is sending Gigabytes of data to another, which is going to be run through Raft based replication + the WAL. So, I reckon that WAL is not being trimmed fast enough, and is causing OOM. See https://github.com/dgraph-io/dgraph/issues/2417.

Can you try one thing for me and confirm if MemoryStorage is in fact the issue. Close to when a lot of RAM is being used, can you try to take a memory profile, take an image of it, and paste it here?

Instructions to do that are here: https://docs.dgraph.io/howto/#retrieving-debug-information

Hi @manishrjain, sorry but I shut down the cluster and start only environment with one node at least for testing. Unfortunately, I really need HA setup for production environment with bulk loaded data. It is impossible now :( because replication is not working with big data set and also bulk loader with shard is not working too.

We should be able to fix this, but a bit more diagnostics would have helped speed things up.

@manishrjain I am wondering whether dgraph server will crash during big query that need lots of computation and memory? In my case, the dgraph server crash without any error logs after the memory meet the lru_mb limitation. BTW, my dgraph version is 1.05.

If it's a big query, then you'd need to allocate a proportionate amount of RAM. Getting killed by OOM is technically outside of Dgraph process' control; all we can do is to reduce the memory footprint required.

In my case, dGraph crash during replication process and no queries are executing.

@selmeci : Can you test your same replicated setup on Dgraph master? The Raft WAL is no longer memory based, and that should help with the OOM issue you were seeing.

of course i can do that

Hi @manishrjain,

i have tried to build dgraph binary from master but now I cannot start zero cluster. I tried it on servers and also on localhost with /etc/hosts set on:

127.0.0.1 localhost
127.0.0.1 zero-1
127.0.0.1 zero-2
127.0.0.1 zero-3

Start commands:

dgraph zero --my=zero-1:5080 --idx 1 --replicas 3 --wal /dgraph/w
dgraph zero --my=zero-2:5081 --idx 2 --replicas 3 --wal /dgraph/w1 --peer zero-1:5080 -o 1
dgraph zero --my=zero-2:5082 --idx 3 --replicas 3 --wal /dgraph/w2 --peer zero-2:5080 -o 2

Logs from zero-1:

Setting up grpc listener at: 0.0.0.0:5080
Setting up http listener at: 0.0.0.0:6080
2018/06/08 15:45:02 node.go:208: Group 0 found 1 entries
2018/06/08 15:45:02 raft.go:567: INFO: 1 became follower at term 0
2018/06/08 15:45:02 raft.go:315: INFO: newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018/06/08 15:45:02 raft.go:567: INFO: 1 became follower at term 1
Running Dgraph zero...
2018/06/08 15:45:02 node.go:115: Setting conf state to nodes:1 
2018/06/08 15:45:06 raft.go:749: INFO: 1 is starting a new election at term 1
2018/06/08 15:45:06 raft.go:580: INFO: 1 became candidate at term 2
2018/06/08 15:45:06 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 2
2018/06/08 15:45:06 raft.go:621: INFO: 1 became leader at term 2
2018/06/08 15:45:06 node.go:301: INFO: raft.node: 1 elected leader 1 at term 2
^CShutting down...
2018/06/08 15:45:57 Stopped taking more http(s) requests. Err: accept tcp [::]:6080: use of closed network connection
2018/06/08 15:45:57 All http(s) requests finished.
2018/06/08 15:45:57 gRpc server stopped : accept tcp [::]:5080: use of closed network connection
All done.
root@ip-172-23-3-224:/dgraph# rm -R w
rm: cannot remove 'w': No such file or directory
root@ip-172-23-3-224:/dgraph# rm -R w
rm: cannot remove 'w': No such file or directory
root@ip-172-23-3-224:/dgraph# dgraph zero --my=zero-1:5080 --idx 1 --replicas 3 --wal /dgraph/w
Setting up grpc listener at: 0.0.0.0:5080
Setting up http listener at: 0.0.0.0:6080
2018/06/08 15:46:05 node.go:208: Group 0 found 1 entries
2018/06/08 15:46:05 raft.go:567: INFO: 1 became follower at term 0
2018/06/08 15:46:05 raft.go:315: INFO: newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018/06/08 15:46:05 raft.go:567: INFO: 1 became follower at term 1
Running Dgraph zero...
2018/06/08 15:46:05 node.go:115: Setting conf state to nodes:1 
2018/06/08 15:46:07 raft.go:749: INFO: 1 is starting a new election at term 1
2018/06/08 15:46:07 raft.go:580: INFO: 1 became candidate at term 2
2018/06/08 15:46:07 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 2
2018/06/08 15:46:07 raft.go:621: INFO: 1 became leader at term 2
2018/06/08 15:46:07 node.go:301: INFO: raft.node: 1 elected leader 1 at term 2
2018/06/08 15:46:38 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:38 pool.go:108: == CONNECT ==> Setting zero-2:5081
2018/06/08 15:46:38 node.go:115: Setting conf state to nodes:1 nodes:2 
2018/06/08 15:46:38 node.go:265: No healthy connection found to node Id: 2, err: Unhealthy connection
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 raft.go:815: INFO: propose conf Type:EntryConfChange Data:"\010\217\315\212\233\356\201\353\205)\020\000\030\002\"\026\t\002\000\000\000\000\000\000\000\032\013zero-2:5081"  ignored since pending unapplied configuration
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 raft.go:815: INFO: propose conf Type:EntryConfChange Data:"\010\302\372\310\271\311\351\247\320\n\020\000\030\002\"\026\t\002\000\000\000\000\000\000\000\032\013zero-2:5081"  ignored since pending unapplied configuration
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 raft.go:815: INFO: propose conf Type:EntryConfChange Data:"\010\225\240\201\330\347\353\200\314\305\001\020\000\030\002\"\026\t\002\000\000\000\000\000\000\000\032\013zero-2:5081"  ignored since pending unapplied configuration
2018/06/08 15:46:39 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:39 raft.go:815: INFO: propose conf Type:EntryConfChange Data:"\010\252\231\355\201\222\265\246\221&\020\000\030\002\"\026\t\002\000\000\000\000\000\000\000\032\013zero-2:5081"  ignored since pending unapplied configuration
2018/06/08 15:46:41 raft.go:793: WARN: 1 stepped down to follower since quorum is not active
2018/06/08 15:46:41 raft.go:567: INFO: 1 became follower at term 2
2018/06/08 15:46:41 node.go:307: INFO: raft.node: 1 lost leader 1 at term 2
2018/06/08 15:46:44 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:44 pool.go:108: == CONNECT ==> Setting zero-2:5082
2018/06/08 15:46:45 raft.go:749: INFO: 1 is starting a new election at term 2
2018/06/08 15:46:45 raft.go:580: INFO: 1 became candidate at term 3
2018/06/08 15:46:45 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 3
2018/06/08 15:46:45 raft.go:651: INFO: 1 [logterm: 2, index: 8] sent MsgVote request to 2 at term 3
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:45 pool.go:158: Echo error from zero-2:5082. Err: rpc error: code = Unavailable desc = transport is closing
2018/06/08 15:46:47 raft.go:749: INFO: 1 is starting a new election at term 3
2018/06/08 15:46:47 raft.go:580: INFO: 1 became candidate at term 4
2018/06/08 15:46:47 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 4
2018/06/08 15:46:47 raft.go:651: INFO: 1 [logterm: 2, index: 8] sent MsgVote request to 2 at term 4
2018/06/08 15:46:48 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
2018/06/08 15:46:49 pool.go:158: Echo error from zero-2:5081. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:5081: connect: connection refused"
^CShutting down...
2018/06/08 15:46:49 Stopped taking more http(s) requests. Err: accept tcp [::]:6080: use of closed network connection
2018/06/08 15:46:49 All http(s) requests finished.
2018/06/08 15:46:49 gRpc server stopped : accept tcp [::]:5080: use of closed network connection
All done.

Logs from zero-2

Setting up grpc listener at: 0.0.0.0:5081
Setting up http listener at: 0.0.0.0:6081
2018/06/08 15:44:25 node.go:208: Group 0 found 1 entries
2018/06/08 15:44:25 pool.go:108: == CONNECT ==> Setting zero-1:5080
2018/06/08 15:44:26 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:27 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:28 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:30 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:31 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:34 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:38 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:46 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:44:46 Max retries exceeded while trying to join cluster: rpc error: code = DeadlineExceeded desc = context deadline exceeded

github.com/dgraph-io/dgraph/x.Fatalf
    /go/src/github.com/dgraph-io/dgraph/x/error.go:100
github.com/dgraph-io/dgraph/dgraph/cmd/zero.(*node).initAndStartNode
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/raft.go:462
github.com/dgraph-io/dgraph/dgraph/cmd/zero.(*state).serveGRPC
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:105
github.com/dgraph-io/dgraph/dgraph/cmd/zero.run
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:180
github.com/dgraph-io/dgraph/dgraph/cmd/zero.init.0.func1
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:59
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).execute
    /go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:702
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).ExecuteC
    /go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:783
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).Execute
    /go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:736
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:42
main.main
    /go/src/github.com/dgraph-io/dgraph/dgraph/main.go:24
runtime.main
    /usr/local/go/src/runtime/proc.go:198
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:2361

Logs from zero-3:

Setting up grpc listener at: 0.0.0.0:5082
Setting up http listener at: 0.0.0.0:6082
2018/06/08 15:46:24 node.go:208: Group 0 found 1 entries
2018/06/08 15:46:24 pool.go:108: == CONNECT ==> Setting zero-2:5080
2018/06/08 15:46:25 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:27 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:28 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:29 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:31 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:34 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:38 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:45 raft.go:458: Error while joining cluster rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018/06/08 15:46:45 Max retries exceeded while trying to join cluster: rpc error: code = DeadlineExceeded desc = context deadline exceeded

github.com/dgraph-io/dgraph/x.Fatalf
    /go/src/github.com/dgraph-io/dgraph/x/error.go:100
github.com/dgraph-io/dgraph/dgraph/cmd/zero.(*node).initAndStartNode
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/raft.go:462
github.com/dgraph-io/dgraph/dgraph/cmd/zero.(*state).serveGRPC
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:105
github.com/dgraph-io/dgraph/dgraph/cmd/zero.run
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:180
github.com/dgraph-io/dgraph/dgraph/cmd/zero.init.0.func1
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:59
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).execute
    /go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:702
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).ExecuteC
    /go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:783
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).Execute
    /go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:736
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
    /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:42
main.main
    /go/src/github.com/dgraph-io/dgraph/dgraph/main.go:24
runtime.main
    /usr/local/go/src/runtime/proc.go:198
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:2361

Can you build docker image for master branch?

Yeah, I could see that issue. Was introduced by my last commit. Fixed it with https://github.com/dgraph-io/dgraph/commit/c717e6de06b46bd78c9fcd67ee50824e28420592.

Try again, please.

Hi @manishrjain, something weird is happening now.

Replication is going for a while and than server1 (origin of data) stop to print logs and server is writing almost same log:

server1

Jun 10 10:52:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:52:59 pool.go:108: == CONNECT ==> Setting dgraph-2.stage.edocu.eu:7080
Jun 10 10:53:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04s, bytes sent: 18 MB, bytes/sec 4391149
Jun 10 10:53:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09s, bytes sent: 32 MB, bytes/sec 3611790
Jun 10 10:53:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14s, bytes sent: 44 MB, bytes/sec 3159741
Jun 10 10:53:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19s, bytes sent: 60 MB, bytes/sec 3145752
Jun 10 10:53:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24s, bytes sent: 72 MB, bytes/sec 3014679
Jun 10 10:53:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29s, bytes sent: 88 MB, bytes/sec 3019926
Jun 10 10:53:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 34s, bytes sent: 102 MB, bytes/sec 2991559
Jun 10 10:53:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 39s, bytes sent: 115 MB, bytes/sec 2957559
Jun 10 10:53:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 44s, bytes sent: 128 MB, bytes/sec 2919657
Jun 10 10:53:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 49s, bytes sent: 144 MB, bytes/sec 2949454
Jun 10 10:53:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 54s, bytes sent: 156 MB, bytes/sec 2896652
Jun 10 10:53:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:53:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 59s, bytes sent: 168 MB, bytes/sec 2843640
Jun 10 10:54:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m04s, bytes sent: 176 MB, bytes/sec 2752555
Jun 10 10:54:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m09s, bytes sent: 186 MB, bytes/sec 2699253
Jun 10 10:54:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m14s, bytes sent: 202 MB, bytes/sec 2727441
Jun 10 10:54:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m19s, bytes sent: 216 MB, bytes/sec 2733734
Jun 10 10:54:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m24s, bytes sent: 230 MB, bytes/sec 2740995
Jun 10 10:54:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m29s, bytes sent: 242 MB, bytes/sec 2723531
Jun 10 10:54:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m35s, bytes sent: 255 MB, bytes/sec 2686897
Jun 10 10:54:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m39s, bytes sent: 268 MB, bytes/sec 2705635
Jun 10 10:54:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m44s, bytes sent: 285 MB, bytes/sec 2743970
Jun 10 10:54:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m49s, bytes sent: 304 MB, bytes/sec 2789822
Jun 10 10:54:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m54s, bytes sent: 315 MB, bytes/sec 2764948
Jun 10 10:54:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:54:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m59s, bytes sent: 326 MB, bytes/sec 2740427
Jun 10 10:55:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m04s, bytes sent: 336 MB, bytes/sec 2706033
Jun 10 10:55:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m09s, bytes sent: 344 MB, bytes/sec 2666176
Jun 10 10:55:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m14s, bytes sent: 353 MB, bytes/sec 2636269
Jun 10 10:55:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m19s, bytes sent: 362 MB, bytes/sec 2602608
Jun 10 10:55:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m24s, bytes sent: 370 MB, bytes/sec 2570495
Jun 10 10:55:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m30s, bytes sent: 381 MB, bytes/sec 2540757
Jun 10 10:55:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m35s, bytes sent: 391 MB, bytes/sec 2523905
Jun 10 10:55:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m39s, bytes sent: 400 MB, bytes/sec 2515308
Jun 10 10:55:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m44s, bytes sent: 408 MB, bytes/sec 2485629
Jun 10 10:55:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m49s, bytes sent: 417 MB, bytes/sec 2466017
Jun 10 10:55:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m55s, bytes sent: 424 MB, bytes/sec 2425396
Jun 10 10:55:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:55:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m00s, bytes sent: 432 MB, bytes/sec 2399539
Jun 10 10:56:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m04s, bytes sent: 440 MB, bytes/sec 2391641
Jun 10 10:56:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m09s, bytes sent: 448 MB, bytes/sec 2370761
Jun 10 10:56:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m14s, bytes sent: 456 MB, bytes/sec 2349124
Jun 10 10:56:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m19s, bytes sent: 464 MB, bytes/sec 2331049
Jun 10 10:56:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m25s, bytes sent: 472 MB, bytes/sec 2301776
Jun 10 10:56:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m30s, bytes sent: 483 MB, bytes/sec 2299914
Jun 10 10:56:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m35s, bytes sent: 493 MB, bytes/sec 2293975
Jun 10 10:56:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m39s, bytes sent: 502 MB, bytes/sec 2292321
Jun 10 10:56:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m44s, bytes sent: 510 MB, bytes/sec 2276648
Jun 10 10:56:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m49s, bytes sent: 517 MB, bytes/sec 2256787
Jun 10 10:56:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m54s, bytes sent: 523 MB, bytes/sec 2235063
Jun 10 10:56:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:56:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m00s, bytes sent: 528 MB, bytes/sec 2200675
Jun 10 10:57:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m04s, bytes sent: 534 MB, bytes/sec 2187581
Jun 10 10:57:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m09s, bytes sent: 540 MB, bytes/sec 2168059
Jun 10 10:57:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m14s, bytes sent: 546 MB, bytes/sec 2148444
Jun 10 10:57:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m19s, bytes sent: 551 MB, bytes/sec 2128416
Jun 10 10:57:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m24s, bytes sent: 559 MB, bytes/sec 2118574
Jun 10 10:57:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m30s, bytes sent: 567 MB, bytes/sec 2101592
Jun 10 10:57:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m34s, bytes sent: 575 MB, bytes/sec 2099169
Jun 10 10:57:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m39s, bytes sent: 583 MB, bytes/sec 2088070
Jun 10 10:57:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m44s, bytes sent: 590 MB, bytes/sec 2076465
Jun 10 10:57:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m49s, bytes sent: 597 MB, bytes/sec 2065542
Jun 10 10:57:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m54s, bytes sent: 605 MB, bytes/sec 2059008
Jun 10 10:57:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:57:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m59s, bytes sent: 613 MB, bytes/sec 2050663
Jun 10 10:58:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m04s, bytes sent: 620 MB, bytes/sec 2039264
Jun 10 10:58:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m09s, bytes sent: 627 MB, bytes/sec 2028946
Jun 10 10:58:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m14s, bytes sent: 636 MB, bytes/sec 2025868
Jun 10 10:58:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m19s, bytes sent: 642 MB, bytes/sec 2013254
Jun 10 10:58:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m25s, bytes sent: 650 MB, bytes/sec 2001616
Jun 10 10:58:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m29s, bytes sent: 658 MB, bytes/sec 1998519
Jun 10 10:58:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m34s, bytes sent: 665 MB, bytes/sec 1991059
Jun 10 10:58:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m40s, bytes sent: 673 MB, bytes/sec 1978334
Jun 10 10:58:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m45s, bytes sent: 680 MB, bytes/sec 1970185
Jun 10 10:58:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m50s, bytes sent: 684 MB, bytes/sec 1954005
Jun 10 10:58:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m54s, bytes sent: 691 MB, bytes/sec 1952100
Jun 10 10:58:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:58:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m59s, bytes sent: 700 MB, bytes/sec 1950253
Jun 10 10:59:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m04s, bytes sent: 706 MB, bytes/sec 1939216
Jun 10 10:59:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m10s, bytes sent: 714 MB, bytes/sec 1929259
Jun 10 10:59:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m14s, bytes sent: 722 MB, bytes/sec 1930415
Jun 10 10:59:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m20s, bytes sent: 728 MB, bytes/sec 1914600
Jun 10 10:59:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m24s, bytes sent: 736 MB, bytes/sec 1915558
Jun 10 10:59:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m29s, bytes sent: 744 MB, bytes/sec 1912320
Jun 10 10:59:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m35s, bytes sent: 750 MB, bytes/sec 1897374
Jun 10 10:59:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m40s, bytes sent: 756 MB, bytes/sec 1890923
Jun 10 10:59:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m44s, bytes sent: 765 MB, bytes/sec 1892853
Jun 10 10:59:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m49s, bytes sent: 772 MB, bytes/sec 1887567
Jun 10 10:59:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m55s, bytes sent: 778 MB, bytes/sec 1874080
Jun 10 10:59:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 10:59:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m00s, bytes sent: 784 MB, bytes/sec 1867040
Jun 10 11:00:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m04s, bytes sent: 792 MB, bytes/sec 1868105
Jun 10 11:00:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m09s, bytes sent: 800 MB, bytes/sec 1864917
Jun 10 11:00:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m14s, bytes sent: 806 MB, bytes/sec 1857722
Jun 10 11:00:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m20s, bytes sent: 811 MB, bytes/sec 1842590
Jun 10 11:00:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m25s, bytes sent: 815 MB, bytes/sec 1832099
Jun 10 11:00:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m29s, bytes sent: 821 MB, bytes/sec 1829224
Jun 10 11:00:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m34s, bytes sent: 829 MB, bytes/sec 1826698
Jun 10 11:00:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m39s, bytes sent: 836 MB, bytes/sec 1820895
Jun 10 11:00:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m44s, bytes sent: 841 MB, bytes/sec 1812885
Jun 10 11:00:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m50s, bytes sent: 847 MB, bytes/sec 1802068
Jun 10 11:00:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m54s, bytes sent: 854 MB, bytes/sec 1802457
Jun 10 11:00:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:00:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m59s, bytes sent: 862 MB, bytes/sec 1799144
Jun 10 11:01:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m04s, bytes sent: 868 MB, bytes/sec 1793603
Jun 10 11:01:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m10s, bytes sent: 874 MB, bytes/sec 1783023
Jun 10 11:01:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m15s, bytes sent: 879 MB, bytes/sec 1776049
Jun 10 11:01:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m20s, bytes sent: 885 MB, bytes/sec 1770098
Jun 10 11:01:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m24s, bytes sent: 892 MB, bytes/sec 1769957
Jun 10 11:01:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m29s, bytes sent: 899 MB, bytes/sec 1767008
Jun 10 11:01:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m35s, bytes sent: 908 MB, bytes/sec 1763306
Jun 10 11:01:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m40s, bytes sent: 915 MB, bytes/sec 1760048
Jun 10 11:01:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m45s, bytes sent: 924 MB, bytes/sec 1759697
Jun 10 11:01:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m49s, bytes sent: 936 MB, bytes/sec 1768535
Jun 10 11:01:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m55s, bytes sent: 942 MB, bytes/sec 1761423
Jun 10 11:01:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:01:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m00s, bytes sent: 949 MB, bytes/sec 1757655
Jun 10 11:02:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m04s, bytes sent: 962 MB, bytes/sec 1767566
Jun 10 11:02:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m09s, bytes sent: 969 MB, bytes/sec 1764860
Jun 10 11:02:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m15s, bytes sent: 975 MB, bytes/sec 1757209
Jun 10 11:02:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m20s, bytes sent: 981 MB, bytes/sec 1752393
Jun 10 11:02:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m24s, bytes sent: 987 MB, bytes/sec 1750465
Jun 10 11:02:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m29s, bytes sent: 994 MB, bytes/sec 1747482
Jun 10 11:02:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m34s, bytes sent: 1.0 GB, bytes/sec 1744676
Jun 10 11:02:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m39s, bytes sent: 1.0 GB, bytes/sec 1745516
Jun 10 11:02:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m45s, bytes sent: 1.0 GB, bytes/sec 1738616
Jun 10 11:02:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m50s, bytes sent: 1.0 GB, bytes/sec 1735179
Jun 10 11:02:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m55s, bytes sent: 1.0 GB, bytes/sec 1732216
Jun 10 11:02:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:02:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m00s, bytes sent: 1.0 GB, bytes/sec 1728993
Jun 10 11:03:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m04s, bytes sent: 1.0 GB, bytes/sec 1732296
Jun 10 11:03:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m09s, bytes sent: 1.1 GB, bytes/sec 1734645
Jun 10 11:03:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m14s, bytes sent: 1.1 GB, bytes/sec 1737234
Jun 10 11:03:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m19s, bytes sent: 1.1 GB, bytes/sec 1739923
Jun 10 11:03:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m25s, bytes sent: 1.1 GB, bytes/sec 1738152
Jun 10 11:03:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m29s, bytes sent: 1.1 GB, bytes/sec 1742094
Jun 10 11:03:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m34s, bytes sent: 1.1 GB, bytes/sec 1743241
Jun 10 11:03:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m39s, bytes sent: 1.1 GB, bytes/sec 1740354
Jun 10 11:03:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m44s, bytes sent: 1.1 GB, bytes/sec 1740596
Jun 10 11:03:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m49s, bytes sent: 1.1 GB, bytes/sec 1740112
Jun 10 11:03:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m54s, bytes sent: 1.1 GB, bytes/sec 1734986
Jun 10 11:03:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:03:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m59s, bytes sent: 1.1 GB, bytes/sec 1735672
Jun 10 11:04:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m05s, bytes sent: 1.2 GB, bytes/sec 1734588
Jun 10 11:04:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m10s, bytes sent: 1.2 GB, bytes/sec 1731542
Jun 10 11:04:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m15s, bytes sent: 1.2 GB, bytes/sec 1734385
Jun 10 11:04:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m20s, bytes sent: 1.2 GB, bytes/sec 1737067
Jun 10 11:04:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m25s, bytes sent: 1.2 GB, bytes/sec 1739823
Jun 10 11:04:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m30s, bytes sent: 1.2 GB, bytes/sec 1741574
Jun 10 11:04:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m34s, bytes sent: 1.2 GB, bytes/sec 1743624
Jun 10 11:04:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m39s, bytes sent: 1.2 GB, bytes/sec 1744043
Jun 10 11:04:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m44s, bytes sent: 1.2 GB, bytes/sec 1745755
Jun 10 11:04:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m50s, bytes sent: 1.2 GB, bytes/sec 1741106
Jun 10 11:04:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m55s, bytes sent: 1.2 GB, bytes/sec 1744716
Jun 10 11:04:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:04:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m00s, bytes sent: 1.3 GB, bytes/sec 1748379
Jun 10 11:05:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m05s, bytes sent: 1.3 GB, bytes/sec 1752372
Jun 10 11:05:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m10s, bytes sent: 1.3 GB, bytes/sec 1752442
Jun 10 11:05:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m14s, bytes sent: 1.3 GB, bytes/sec 1752893
Jun 10 11:05:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m19s, bytes sent: 1.3 GB, bytes/sec 1748375
Jun 10 11:05:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m25s, bytes sent: 1.3 GB, bytes/sec 1746779
Jun 10 11:05:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m30s, bytes sent: 1.3 GB, bytes/sec 1746401
Jun 10 11:05:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m35s, bytes sent: 1.3 GB, bytes/sec 1745931
Jun 10 11:05:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m40s, bytes sent: 1.3 GB, bytes/sec 1747090
Jun 10 11:05:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m45s, bytes sent: 1.3 GB, bytes/sec 1747744
Jun 10 11:05:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m50s, bytes sent: 1.3 GB, bytes/sec 1748582
Jun 10 11:05:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m55s, bytes sent: 1.4 GB, bytes/sec 1750018
Jun 10 11:05:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:05:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m00s, bytes sent: 1.4 GB, bytes/sec 1750521
Jun 10 11:06:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m04s, bytes sent: 1.4 GB, bytes/sec 1753450
Jun 10 11:06:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m09s, bytes sent: 1.4 GB, bytes/sec 1752971
Jun 10 11:06:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m14s, bytes sent: 1.4 GB, bytes/sec 1752497
Jun 10 11:06:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m19s, bytes sent: 1.4 GB, bytes/sec 1753921
Jun 10 11:06:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m24s, bytes sent: 1.4 GB, bytes/sec 1751567
Jun 10 11:06:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m29s, bytes sent: 1.4 GB, bytes/sec 1748935
Jun 10 11:06:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m35s, bytes sent: 1.4 GB, bytes/sec 1746174
Jun 10 11:06:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m40s, bytes sent: 1.4 GB, bytes/sec 1744582
Jun 10 11:06:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m45s, bytes sent: 1.4 GB, bytes/sec 1745272
Jun 10 11:06:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m50s, bytes sent: 1.4 GB, bytes/sec 1745244
Jun 10 11:06:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m55s, bytes sent: 1.5 GB, bytes/sec 1745762
Jun 10 11:06:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:06:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m00s, bytes sent: 1.5 GB, bytes/sec 1745290
Jun 10 11:07:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m05s, bytes sent: 1.5 GB, bytes/sec 1744807
Jun 10 11:07:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m10s, bytes sent: 1.5 GB, bytes/sec 1745799
Jun 10 11:07:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m14s, bytes sent: 1.5 GB, bytes/sec 1745881
Jun 10 11:07:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m19s, bytes sent: 1.5 GB, bytes/sec 1741299
Jun 10 11:07:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m24s, bytes sent: 1.5 GB, bytes/sec 1742897
Jun 10 11:07:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m29s, bytes sent: 1.5 GB, bytes/sec 1744823
Jun 10 11:07:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m34s, bytes sent: 1.5 GB, bytes/sec 1743892
Jun 10 11:07:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m39s, bytes sent: 1.5 GB, bytes/sec 1743403
Jun 10 11:07:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m45s, bytes sent: 1.5 GB, bytes/sec 1741078
Jun 10 11:07:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m49s, bytes sent: 1.5 GB, bytes/sec 1742151
Jun 10 11:07:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m55s, bytes sent: 1.6 GB, bytes/sec 1740731
Jun 10 11:07:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:07:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m00s, bytes sent: 1.6 GB, bytes/sec 1742056
Jun 10 11:08:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m04s, bytes sent: 1.6 GB, bytes/sec 1744940
Jun 10 11:08:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m09s, bytes sent: 1.6 GB, bytes/sec 1747451
Jun 10 11:08:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m14s, bytes sent: 1.6 GB, bytes/sec 1750991
Jun 10 11:08:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m20s, bytes sent: 1.6 GB, bytes/sec 1751836
Jun 10 11:08:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m24s, bytes sent: 1.6 GB, bytes/sec 1752085
Jun 10 11:08:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m29s, bytes sent: 1.6 GB, bytes/sec 1749482
Jun 10 11:08:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m35s, bytes sent: 1.6 GB, bytes/sec 1752000
Jun 10 11:08:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m39s, bytes sent: 1.6 GB, bytes/sec 1754357
Jun 10 11:08:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m45s, bytes sent: 1.7 GB, bytes/sec 1747747
Jun 10 11:08:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m49s, bytes sent: 1.7 GB, bytes/sec 1754041
Jun 10 11:08:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m55s, bytes sent: 1.7 GB, bytes/sec 1755709
Jun 10 11:08:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:08:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m59s, bytes sent: 1.7 GB, bytes/sec 1757133
Jun 10 11:09:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m05s, bytes sent: 1.7 GB, bytes/sec 1753138
Jun 10 11:09:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m09s, bytes sent: 1.7 GB, bytes/sec 1760112
Jun 10 11:09:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m14s, bytes sent: 1.7 GB, bytes/sec 1765296
Jun 10 11:09:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m19s, bytes sent: 1.7 GB, bytes/sec 1763532
Jun 10 11:09:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m24s, bytes sent: 1.7 GB, bytes/sec 1762205
Jun 10 11:09:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m30s, bytes sent: 1.7 GB, bytes/sec 1765109
Jun 10 11:09:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m35s, bytes sent: 1.8 GB, bytes/sec 1770261
Jun 10 11:09:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m39s, bytes sent: 1.8 GB, bytes/sec 1771063
Jun 10 11:09:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m44s, bytes sent: 1.8 GB, bytes/sec 1769324
Jun 10 11:09:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m50s, bytes sent: 1.8 GB, bytes/sec 1768656
Jun 10 11:09:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m54s, bytes sent: 1.8 GB, bytes/sec 1776612
Jun 10 11:09:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:09:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m59s, bytes sent: 1.8 GB, bytes/sec 1774069
Jun 10 11:10:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m04s, bytes sent: 1.8 GB, bytes/sec 1774523
Jun 10 11:10:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m09s, bytes sent: 1.8 GB, bytes/sec 1777209
Jun 10 11:10:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m15s, bytes sent: 1.8 GB, bytes/sec 1770677
Jun 10 11:10:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m20s, bytes sent: 1.8 GB, bytes/sec 1772578
Jun 10 11:10:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m25s, bytes sent: 1.9 GB, bytes/sec 1775435
Jun 10 11:10:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m29s, bytes sent: 1.9 GB, bytes/sec 1776312
Jun 10 11:10:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m34s, bytes sent: 1.9 GB, bytes/sec 1773294
Jun 10 11:10:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m39s, bytes sent: 1.9 GB, bytes/sec 1775649
Jun 10 11:10:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m45s, bytes sent: 1.9 GB, bytes/sec 1776987
Jun 10 11:10:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m49s, bytes sent: 1.9 GB, bytes/sec 1781357
Jun 10 11:10:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m54s, bytes sent: 1.9 GB, bytes/sec 1779876
Jun 10 11:10:59 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:10:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m59s, bytes sent: 1.9 GB, bytes/sec 1775908
Jun 10 11:11:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m04s, bytes sent: 1.9 GB, bytes/sec 1779067
Jun 10 11:11:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m10s, bytes sent: 1.9 GB, bytes/sec 1780688
Jun 10 11:11:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m14s, bytes sent: 2.0 GB, bytes/sec 1782802
Jun 10 11:11:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m19s, bytes sent: 2.0 GB, bytes/sec 1779137
Jun 10 11:11:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m24s, bytes sent: 2.0 GB, bytes/sec 1780091
Jun 10 11:11:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m30s, bytes sent: 2.0 GB, bytes/sec 1781669
Jun 10 11:11:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m34s, bytes sent: 2.0 GB, bytes/sec 1782799
Jun 10 11:11:39 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m39s, bytes sent: 2.0 GB, bytes/sec 1778138
Jun 10 11:11:44 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m44s, bytes sent: 2.0 GB, bytes/sec 1780467
Jun 10 11:11:49 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m50s, bytes sent: 2.0 GB, bytes/sec 1778959
Jun 10 11:11:54 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:11:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m55s, bytes sent: 2.0 GB, bytes/sec 1773277
Jun 10 11:12:00 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:00 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m00s, bytes sent: 2.0 GB, bytes/sec 1777153
Jun 10 11:12:04 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m04s, bytes sent: 2.0 GB, bytes/sec 1780379
Jun 10 11:12:09 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m09s, bytes sent: 2.0 GB, bytes/sec 1775526
Jun 10 11:12:14 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m15s, bytes sent: 2.1 GB, bytes/sec 1776899
Jun 10 11:12:19 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m20s, bytes sent: 2.1 GB, bytes/sec 1780688
Jun 10 11:12:24 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m24s, bytes sent: 2.1 GB, bytes/sec 1780646
Jun 10 11:12:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m30s, bytes sent: 2.1 GB, bytes/sec 1778126

server2:

Jun 10 10:52:59 ip-172-23-1-132 systemd[1]: Started dGraph Server.
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 gRPC server started.  Listening on port 9080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 HTTP server started.  Listening on port 8080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 worker.go:89: Worker listening at address: [::]:7080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 groups.go:77: Current Raft Id: 0
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 pool.go:108: == CONNECT ==> Setting dgraph-1.stage.edocu.eu:5080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 groups.go:104: Connected to group zero. Assigned group: 1
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 pool.go:108: == CONNECT ==> Setting dgraph-1.stage.edocu.eu:7080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 pool.go:108: == CONNECT ==> Setting dgraph-3.stage.edocu.eu:5080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 pool.go:108: == CONNECT ==> Setting dgraph-2.stage.edocu.eu:5080
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 draft.go:153: Node ID: 2 with GroupID: 1
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 node.go:208: Group 1 found 1 entries
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 draft.go:950: Calling IsPeer
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 draft.go:955: Done with IsPeer call
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 draft.go:1002: New Node for group: 1
Jun 10 10:52:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:52:59 draft.go:1007: Retrieving snapshot from peer: 1
Jun 10 10:53:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 14 MB, bytes/sec 2781692
Jun 10 10:53:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10s, bytes written: 29 MB, bytes/sec 2868163
Jun 10 10:53:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15s, bytes written: 41 MB, bytes/sec 2726005
Jun 10 10:53:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20s, bytes written: 56 MB, bytes/sec 2801051
Jun 10 10:53:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25s, bytes written: 69 MB, bytes/sec 2767210
Jun 10 10:53:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30s, bytes written: 84 MB, bytes/sec 2799375
Jun 10 10:53:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35s, bytes written: 97 MB, bytes/sec 2775211
Jun 10 10:53:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40s, bytes written: 112 MB, bytes/sec 2799080
Jun 10 10:53:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 45s, bytes written: 125 MB, bytes/sec 2779276
Jun 10 10:53:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 50s, bytes written: 141 MB, bytes/sec 2815538
Jun 10 10:53:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 55s, bytes written: 153 MB, bytes/sec 2777898
Jun 10 10:53:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:53:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m00s, bytes written: 164 MB, bytes/sec 2730838
Jun 10 10:54:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m05s, bytes written: 172 MB, bytes/sec 2643111
Jun 10 10:54:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m10s, bytes written: 183 MB, bytes/sec 2617409
Jun 10 10:54:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m15s, bytes written: 200 MB, bytes/sec 2670333
Jun 10 10:54:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m20s, bytes written: 215 MB, bytes/sec 2687014
Jun 10 10:54:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m25s, bytes written: 229 MB, bytes/sec 2694358
Jun 10 10:54:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m30s, bytes written: 241 MB, bytes/sec 2681433
Jun 10 10:54:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m35s, bytes written: 255 MB, bytes/sec 2682181
Jun 10 10:54:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m40s, bytes written: 267 MB, bytes/sec 2668691
Jun 10 10:54:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m45s, bytes written: 283 MB, bytes/sec 2695811
Jun 10 10:54:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m50s, bytes written: 300 MB, bytes/sec 2730958
Jun 10 10:54:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m55s, bytes written: 312 MB, bytes/sec 2708439
Jun 10 10:54:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:54:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m00s, bytes written: 322 MB, bytes/sec 2687291
Jun 10 10:55:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m05s, bytes written: 332 MB, bytes/sec 2656759
Jun 10 10:55:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m10s, bytes written: 341 MB, bytes/sec 2620469
Jun 10 10:55:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m15s, bytes written: 349 MB, bytes/sec 2587335
Jun 10 10:55:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m20s, bytes written: 358 MB, bytes/sec 2558603
Jun 10 10:55:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m25s, bytes written: 366 MB, bytes/sec 2521069
Jun 10 10:55:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m30s, bytes written: 378 MB, bytes/sec 2517875
Jun 10 10:55:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m35s, bytes written: 391 MB, bytes/sec 2519982
Jun 10 10:55:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m40s, bytes written: 399 MB, bytes/sec 2492857
Jun 10 10:55:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m45s, bytes written: 407 MB, bytes/sec 2465767
Jun 10 10:55:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m50s, bytes written: 416 MB, bytes/sec 2445970
Jun 10 10:55:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 02m55s, bytes written: 424 MB, bytes/sec 2420737
Jun 10 10:55:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:55:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m00s, bytes written: 431 MB, bytes/sec 2394272
Jun 10 10:56:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m05s, bytes written: 439 MB, bytes/sec 2374906
Jun 10 10:56:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m10s, bytes written: 447 MB, bytes/sec 2352416
Jun 10 10:56:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m15s, bytes written: 455 MB, bytes/sec 2333776
Jun 10 10:56:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m20s, bytes written: 463 MB, bytes/sec 2316145
Jun 10 10:56:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m25s, bytes written: 470 MB, bytes/sec 2293327
Jun 10 10:56:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m30s, bytes written: 481 MB, bytes/sec 2289397
Jun 10 10:56:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m35s, bytes written: 490 MB, bytes/sec 2279151
Jun 10 10:56:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m40s, bytes written: 500 MB, bytes/sec 2273275
Jun 10 10:56:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m45s, bytes written: 509 MB, bytes/sec 2263337
Jun 10 10:56:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m50s, bytes written: 516 MB, bytes/sec 2242370
Jun 10 10:56:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 03m55s, bytes written: 522 MB, bytes/sec 2222111
Jun 10 10:56:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:56:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m00s, bytes written: 527 MB, bytes/sec 2197662
Jun 10 10:57:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m05s, bytes written: 533 MB, bytes/sec 2174292
Jun 10 10:57:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m10s, bytes written: 539 MB, bytes/sec 2155764
Jun 10 10:57:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m15s, bytes written: 545 MB, bytes/sec 2136834
Jun 10 10:57:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m20s, bytes written: 550 MB, bytes/sec 2116625
Jun 10 10:57:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m25s, bytes written: 558 MB, bytes/sec 2105087
Jun 10 10:57:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m30s, bytes written: 566 MB, bytes/sec 2097316
Jun 10 10:57:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m35s, bytes written: 574 MB, bytes/sec 2086882
Jun 10 10:57:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m40s, bytes written: 581 MB, bytes/sec 2076563
Jun 10 10:57:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m45s, bytes written: 589 MB, bytes/sec 2065538
Jun 10 10:57:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m50s, bytes written: 597 MB, bytes/sec 2057400
Jun 10 10:57:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 04m55s, bytes written: 605 MB, bytes/sec 2050965
Jun 10 10:57:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:57:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m00s, bytes written: 612 MB, bytes/sec 2040613
Jun 10 10:58:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m05s, bytes written: 618 MB, bytes/sec 2027848
Jun 10 10:58:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m10s, bytes written: 626 MB, bytes/sec 2019375
Jun 10 10:58:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m15s, bytes written: 634 MB, bytes/sec 2013392
Jun 10 10:58:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m20s, bytes written: 641 MB, bytes/sec 2002919
Jun 10 10:58:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m25s, bytes written: 649 MB, bytes/sec 1998074
Jun 10 10:58:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m30s, bytes written: 656 MB, bytes/sec 1987391
Jun 10 10:58:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m35s, bytes written: 665 MB, bytes/sec 1984490
Jun 10 10:58:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m40s, bytes written: 672 MB, bytes/sec 1975246
Jun 10 10:58:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m45s, bytes written: 680 MB, bytes/sec 1969520
Jun 10 10:58:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m50s, bytes written: 684 MB, bytes/sec 1953368
Jun 10 10:58:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05m55s, bytes written: 690 MB, bytes/sec 1943578
Jun 10 10:58:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:58:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m00s, bytes written: 699 MB, bytes/sec 1941830
Jun 10 10:59:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m05s, bytes written: 706 MB, bytes/sec 1933422
Jun 10 10:59:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m10s, bytes written: 713 MB, bytes/sec 1927133
Jun 10 10:59:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m15s, bytes written: 721 MB, bytes/sec 1923808
Jun 10 10:59:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m20s, bytes written: 727 MB, bytes/sec 1912292
Jun 10 10:59:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m25s, bytes written: 735 MB, bytes/sec 1907965
Jun 10 10:59:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m30s, bytes written: 743 MB, bytes/sec 1904089
Jun 10 10:59:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m35s, bytes written: 749 MB, bytes/sec 1895421
Jun 10 10:59:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m40s, bytes written: 755 MB, bytes/sec 1888485
Jun 10 10:59:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m45s, bytes written: 764 MB, bytes/sec 1885378
Jun 10 10:59:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m50s, bytes written: 772 MB, bytes/sec 1882342
Jun 10 10:59:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 06m55s, bytes written: 777 MB, bytes/sec 1872297
Jun 10 10:59:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 10:59:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m00s, bytes written: 783 MB, bytes/sec 1863763
Jun 10 11:00:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m05s, bytes written: 791 MB, bytes/sec 1860129
Jun 10 11:00:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m10s, bytes written: 798 MB, bytes/sec 1854554
Jun 10 11:00:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m15s, bytes written: 804 MB, bytes/sec 1848654
Jun 10 11:00:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m20s, bytes written: 811 MB, bytes/sec 1842178
Jun 10 11:00:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m25s, bytes written: 815 MB, bytes/sec 1830905
Jun 10 11:00:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m30s, bytes written: 820 MB, bytes/sec 1822778
Jun 10 11:00:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m35s, bytes written: 828 MB, bytes/sec 1819084
Jun 10 11:00:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m40s, bytes written: 836 MB, bytes/sec 1816388
Jun 10 11:00:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m45s, bytes written: 841 MB, bytes/sec 1808532
Jun 10 11:00:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m50s, bytes written: 846 MB, bytes/sec 1800447
Jun 10 11:00:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 07m55s, bytes written: 853 MB, bytes/sec 1795826
Jun 10 11:00:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:00:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m00s, bytes written: 861 MB, bytes/sec 1793326
Jun 10 11:01:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m05s, bytes written: 867 MB, bytes/sec 1788004
Jun 10 11:01:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m10s, bytes written: 874 MB, bytes/sec 1782599
Jun 10 11:01:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m15s, bytes written: 879 MB, bytes/sec 1775185
Jun 10 11:01:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m20s, bytes written: 884 MB, bytes/sec 1767919
Jun 10 11:01:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m25s, bytes written: 891 MB, bytes/sec 1764409
Jun 10 11:01:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m30s, bytes written: 898 MB, bytes/sec 1760532
Jun 10 11:01:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m35s, bytes written: 906 MB, bytes/sec 1758450
Jun 10 11:01:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m40s, bytes written: 914 MB, bytes/sec 1758401
Jun 10 11:01:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m45s, bytes written: 922 MB, bytes/sec 1756790
Jun 10 11:01:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m50s, bytes written: 932 MB, bytes/sec 1757561
Jun 10 11:01:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 08m55s, bytes written: 941 MB, bytes/sec 1759543
Jun 10 11:01:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:01:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m00s, bytes written: 949 MB, bytes/sec 1757357
Jun 10 11:02:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m05s, bytes written: 958 MB, bytes/sec 1757702
Jun 10 11:02:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m10s, bytes written: 966 MB, bytes/sec 1756205
Jun 10 11:02:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m15s, bytes written: 973 MB, bytes/sec 1753503
Jun 10 11:02:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m20s, bytes written: 980 MB, bytes/sec 1749851
Jun 10 11:02:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m25s, bytes written: 987 MB, bytes/sec 1746412
Jun 10 11:02:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m30s, bytes written: 993 MB, bytes/sec 1741964
Jun 10 11:02:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m35s, bytes written: 1000 MB, bytes/sec 1738846
Jun 10 11:02:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m40s, bytes written: 1.0 GB, bytes/sec 1736204
Jun 10 11:02:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m45s, bytes written: 1.0 GB, bytes/sec 1736894
Jun 10 11:02:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m50s, bytes written: 1.0 GB, bytes/sec 1734616
Jun 10 11:02:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 09m55s, bytes written: 1.0 GB, bytes/sec 1730613
Jun 10 11:02:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:02:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m00s, bytes written: 1.0 GB, bytes/sec 1728425
Jun 10 11:03:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m05s, bytes written: 1.0 GB, bytes/sec 1727288
Jun 10 11:03:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m10s, bytes written: 1.1 GB, bytes/sec 1728800
Jun 10 11:03:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m15s, bytes written: 1.1 GB, bytes/sec 1730655
Jun 10 11:03:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m20s, bytes written: 1.1 GB, bytes/sec 1731865
Jun 10 11:03:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m25s, bytes written: 1.1 GB, bytes/sec 1733061
Jun 10 11:03:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m30s, bytes written: 1.1 GB, bytes/sec 1733423
Jun 10 11:03:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m35s, bytes written: 1.1 GB, bytes/sec 1734322
Jun 10 11:03:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m40s, bytes written: 1.1 GB, bytes/sec 1731994
Jun 10 11:03:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m45s, bytes written: 1.1 GB, bytes/sec 1732345
Jun 10 11:03:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m50s, bytes written: 1.1 GB, bytes/sec 1731416
Jun 10 11:03:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10m55s, bytes written: 1.1 GB, bytes/sec 1730579
Jun 10 11:03:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:03:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m00s, bytes written: 1.1 GB, bytes/sec 1731232
Jun 10 11:04:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m05s, bytes written: 1.2 GB, bytes/sec 1733999
Jun 10 11:04:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m10s, bytes written: 1.2 GB, bytes/sec 1730958
Jun 10 11:04:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m15s, bytes written: 1.2 GB, bytes/sec 1732769
Jun 10 11:04:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m20s, bytes written: 1.2 GB, bytes/sec 1736289
Jun 10 11:04:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m25s, bytes written: 1.2 GB, bytes/sec 1736504
Jun 10 11:04:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m30s, bytes written: 1.2 GB, bytes/sec 1736413
Jun 10 11:04:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m35s, bytes written: 1.2 GB, bytes/sec 1736090
Jun 10 11:04:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m40s, bytes written: 1.2 GB, bytes/sec 1736287
Jun 10 11:04:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m45s, bytes written: 1.2 GB, bytes/sec 1738707
Jun 10 11:04:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m50s, bytes written: 1.2 GB, bytes/sec 1739777
Jun 10 11:04:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 11m55s, bytes written: 1.2 GB, bytes/sec 1743743
Jun 10 11:04:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:04:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m00s, bytes written: 1.3 GB, bytes/sec 1746880
Jun 10 11:05:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m05s, bytes written: 1.3 GB, bytes/sec 1748234
Jun 10 11:05:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m10s, bytes written: 1.3 GB, bytes/sec 1747234
Jun 10 11:05:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m15s, bytes written: 1.3 GB, bytes/sec 1745331
Jun 10 11:05:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m20s, bytes written: 1.3 GB, bytes/sec 1745765
Jun 10 11:05:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m25s, bytes written: 1.3 GB, bytes/sec 1746667
Jun 10 11:05:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m30s, bytes written: 1.3 GB, bytes/sec 1746166
Jun 10 11:05:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m35s, bytes written: 1.3 GB, bytes/sec 1745803
Jun 10 11:05:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m40s, bytes written: 1.3 GB, bytes/sec 1745957
Jun 10 11:05:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m45s, bytes written: 1.3 GB, bytes/sec 1746051
Jun 10 11:05:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m50s, bytes written: 1.3 GB, bytes/sec 1746589
Jun 10 11:05:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 12m55s, bytes written: 1.4 GB, bytes/sec 1746660
Jun 10 11:05:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:05:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m00s, bytes written: 1.4 GB, bytes/sec 1746736
Jun 10 11:06:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m05s, bytes written: 1.4 GB, bytes/sec 1746385
Jun 10 11:06:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m10s, bytes written: 1.4 GB, bytes/sec 1746029
Jun 10 11:06:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m15s, bytes written: 1.4 GB, bytes/sec 1746101
Jun 10 11:06:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m20s, bytes written: 1.4 GB, bytes/sec 1745968
Jun 10 11:06:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m25s, bytes written: 1.4 GB, bytes/sec 1745249
Jun 10 11:06:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m30s, bytes written: 1.4 GB, bytes/sec 1744916
Jun 10 11:06:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m35s, bytes written: 1.4 GB, bytes/sec 1744297
Jun 10 11:06:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m40s, bytes written: 1.4 GB, bytes/sec 1744244
Jun 10 11:06:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m45s, bytes written: 1.4 GB, bytes/sec 1745112
Jun 10 11:06:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m50s, bytes written: 1.4 GB, bytes/sec 1744707
Jun 10 11:06:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 13m55s, bytes written: 1.5 GB, bytes/sec 1745562
Jun 10 11:06:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:06:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m00s, bytes written: 1.5 GB, bytes/sec 1744649
Jun 10 11:07:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m05s, bytes written: 1.5 GB, bytes/sec 1741520
Jun 10 11:07:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m10s, bytes written: 1.5 GB, bytes/sec 1742810
Jun 10 11:07:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m15s, bytes written: 1.5 GB, bytes/sec 1741644
Jun 10 11:07:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m20s, bytes written: 1.5 GB, bytes/sec 1738152
Jun 10 11:07:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m25s, bytes written: 1.5 GB, bytes/sec 1737149
Jun 10 11:07:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m30s, bytes written: 1.5 GB, bytes/sec 1738954
Jun 10 11:07:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m35s, bytes written: 1.5 GB, bytes/sec 1739027
Jun 10 11:07:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m40s, bytes written: 1.5 GB, bytes/sec 1739429
Jun 10 11:07:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m45s, bytes written: 1.5 GB, bytes/sec 1739364
Jun 10 11:07:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m50s, bytes written: 1.5 GB, bytes/sec 1739016
Jun 10 11:07:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 14m55s, bytes written: 1.6 GB, bytes/sec 1739845
Jun 10 11:07:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:07:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m00s, bytes written: 1.6 GB, bytes/sec 1741830
Jun 10 11:08:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m05s, bytes written: 1.6 GB, bytes/sec 1742635
Jun 10 11:08:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m10s, bytes written: 1.6 GB, bytes/sec 1744583
Jun 10 11:08:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m15s, bytes written: 1.6 GB, bytes/sec 1748802
Jun 10 11:08:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m20s, bytes written: 1.6 GB, bytes/sec 1747458
Jun 10 11:08:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m25s, bytes written: 1.6 GB, bytes/sec 1746151
Jun 10 11:08:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m30s, bytes written: 1.6 GB, bytes/sec 1746529
Jun 10 11:08:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m35s, bytes written: 1.6 GB, bytes/sec 1748517
Jun 10 11:08:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m40s, bytes written: 1.6 GB, bytes/sec 1748275
Jun 10 11:08:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m45s, bytes written: 1.7 GB, bytes/sec 1747657
Jun 10 11:08:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m50s, bytes written: 1.7 GB, bytes/sec 1750972
Jun 10 11:08:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15m55s, bytes written: 1.7 GB, bytes/sec 1751930
Jun 10 11:08:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:08:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m00s, bytes written: 1.7 GB, bytes/sec 1752010
Jun 10 11:09:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m05s, bytes written: 1.7 GB, bytes/sec 1752727
Jun 10 11:09:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m10s, bytes written: 1.7 GB, bytes/sec 1757746
Jun 10 11:09:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m15s, bytes written: 1.7 GB, bytes/sec 1759098
Jun 10 11:09:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m20s, bytes written: 1.7 GB, bytes/sec 1758149
Jun 10 11:09:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m25s, bytes written: 1.7 GB, bytes/sec 1759722
Jun 10 11:09:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m30s, bytes written: 1.7 GB, bytes/sec 1764603
Jun 10 11:09:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m35s, bytes written: 1.8 GB, bytes/sec 1766440
Jun 10 11:09:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m40s, bytes written: 1.8 GB, bytes/sec 1765451
Jun 10 11:09:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m45s, bytes written: 1.8 GB, bytes/sec 1764351
Jun 10 11:09:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m50s, bytes written: 1.8 GB, bytes/sec 1768075
Jun 10 11:09:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 16m55s, bytes written: 1.8 GB, bytes/sec 1770850
Jun 10 11:09:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:09:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m00s, bytes written: 1.8 GB, bytes/sec 1768394
Jun 10 11:10:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m05s, bytes written: 1.8 GB, bytes/sec 1771197
Jun 10 11:10:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m10s, bytes written: 1.8 GB, bytes/sec 1771192
Jun 10 11:10:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m15s, bytes written: 1.8 GB, bytes/sec 1769945
Jun 10 11:10:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m20s, bytes written: 1.8 GB, bytes/sec 1770129
Jun 10 11:10:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m25s, bytes written: 1.9 GB, bytes/sec 1772226
Jun 10 11:10:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m30s, bytes written: 1.9 GB, bytes/sec 1771188
Jun 10 11:10:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m35s, bytes written: 1.9 GB, bytes/sec 1770277
Jun 10 11:10:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m40s, bytes written: 1.9 GB, bytes/sec 1773706
Jun 10 11:10:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m45s, bytes written: 1.9 GB, bytes/sec 1776210
Jun 10 11:10:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m50s, bytes written: 1.9 GB, bytes/sec 1776978
Jun 10 11:10:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 17m55s, bytes written: 1.9 GB, bytes/sec 1775087
Jun 10 11:10:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:10:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m00s, bytes written: 1.9 GB, bytes/sec 1773871
Jun 10 11:11:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m05s, bytes written: 1.9 GB, bytes/sec 1777294
Jun 10 11:11:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m10s, bytes written: 1.9 GB, bytes/sec 1778842
Jun 10 11:11:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m15s, bytes written: 1.9 GB, bytes/sec 1777879
Jun 10 11:11:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m20s, bytes written: 2.0 GB, bytes/sec 1776345
Jun 10 11:11:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m25s, bytes written: 2.0 GB, bytes/sec 1778340
Jun 10 11:11:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m30s, bytes written: 2.0 GB, bytes/sec 1779053
Jun 10 11:11:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m35s, bytes written: 2.0 GB, bytes/sec 1777634
Jun 10 11:11:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m40s, bytes written: 2.0 GB, bytes/sec 1776057
Jun 10 11:11:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m45s, bytes written: 2.0 GB, bytes/sec 1777706
Jun 10 11:11:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m50s, bytes written: 2.0 GB, bytes/sec 1775127
Jun 10 11:11:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 18m55s, bytes written: 2.0 GB, bytes/sec 1772910
Jun 10 11:11:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:11:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m00s, bytes written: 2.0 GB, bytes/sec 1777092
Jun 10 11:12:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m05s, bytes written: 2.0 GB, bytes/sec 1775756
Jun 10 11:12:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m10s, bytes written: 2.0 GB, bytes/sec 1773286
Jun 10 11:12:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m15s, bytes written: 2.1 GB, bytes/sec 1776710
Jun 10 11:12:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m20s, bytes written: 2.1 GB, bytes/sec 1778523
Jun 10 11:12:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m25s, bytes written: 2.1 GB, bytes/sec 1776154
Jun 10 11:12:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m30s, bytes written: 2.1 GB, bytes/sec 1777234
Jun 10 11:12:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m35s, bytes written: 2.1 GB, bytes/sec 1782995
Jun 10 11:12:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m40s, bytes written: 2.1 GB, bytes/sec 1775440
Jun 10 11:12:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m45s, bytes written: 2.1 GB, bytes/sec 1767948
Jun 10 11:12:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m50s, bytes written: 2.1 GB, bytes/sec 1760520
Jun 10 11:12:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 19m55s, bytes written: 2.1 GB, bytes/sec 1753154
Jun 10 11:12:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:12:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m00s, bytes written: 2.1 GB, bytes/sec 1745849
Jun 10 11:13:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m05s, bytes written: 2.1 GB, bytes/sec 1738605
Jun 10 11:13:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m10s, bytes written: 2.1 GB, bytes/sec 1731420
Jun 10 11:13:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m15s, bytes written: 2.1 GB, bytes/sec 1724295
Jun 10 11:13:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m20s, bytes written: 2.1 GB, bytes/sec 1717228
Jun 10 11:13:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m25s, bytes written: 2.1 GB, bytes/sec 1710219
Jun 10 11:13:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m30s, bytes written: 2.1 GB, bytes/sec 1703267
Jun 10 11:13:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m35s, bytes written: 2.1 GB, bytes/sec 1696371
Jun 10 11:13:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m40s, bytes written: 2.1 GB, bytes/sec 1689531
Jun 10 11:13:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m45s, bytes written: 2.1 GB, bytes/sec 1682746
Jun 10 11:13:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m50s, bytes written: 2.1 GB, bytes/sec 1676015
Jun 10 11:13:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20m55s, bytes written: 2.1 GB, bytes/sec 1669338
Jun 10 11:13:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:13:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m00s, bytes written: 2.1 GB, bytes/sec 1662713
Jun 10 11:14:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m05s, bytes written: 2.1 GB, bytes/sec 1656141
Jun 10 11:14:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m10s, bytes written: 2.1 GB, bytes/sec 1649621
Jun 10 11:14:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m15s, bytes written: 2.1 GB, bytes/sec 1643152
Jun 10 11:14:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m20s, bytes written: 2.1 GB, bytes/sec 1636733
Jun 10 11:14:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m25s, bytes written: 2.1 GB, bytes/sec 1630365
Jun 10 11:14:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m30s, bytes written: 2.1 GB, bytes/sec 1624045
Jun 10 11:14:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m35s, bytes written: 2.1 GB, bytes/sec 1617775
Jun 10 11:14:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m40s, bytes written: 2.1 GB, bytes/sec 1611553
Jun 10 11:14:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m45s, bytes written: 2.1 GB, bytes/sec 1605378
Jun 10 11:14:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m50s, bytes written: 2.1 GB, bytes/sec 1599251
Jun 10 11:14:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 21m55s, bytes written: 2.1 GB, bytes/sec 1593170
Jun 10 11:14:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:14:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m00s, bytes written: 2.1 GB, bytes/sec 1587135
Jun 10 11:15:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m05s, bytes written: 2.1 GB, bytes/sec 1581146
Jun 10 11:15:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m10s, bytes written: 2.1 GB, bytes/sec 1575202
Jun 10 11:15:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m15s, bytes written: 2.1 GB, bytes/sec 1569302
Jun 10 11:15:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m20s, bytes written: 2.1 GB, bytes/sec 1563447
Jun 10 11:15:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m25s, bytes written: 2.1 GB, bytes/sec 1557635
Jun 10 11:15:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m30s, bytes written: 2.1 GB, bytes/sec 1551866
Jun 10 11:15:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m35s, bytes written: 2.1 GB, bytes/sec 1546139
Jun 10 11:15:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m40s, bytes written: 2.1 GB, bytes/sec 1540455
Jun 10 11:15:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m45s, bytes written: 2.1 GB, bytes/sec 1534812
Jun 10 11:15:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m50s, bytes written: 2.1 GB, bytes/sec 1529211
Jun 10 11:15:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 22m55s, bytes written: 2.1 GB, bytes/sec 1523650
Jun 10 11:15:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:15:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m00s, bytes written: 2.1 GB, bytes/sec 1518129
Jun 10 11:16:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m05s, bytes written: 2.1 GB, bytes/sec 1512649
Jun 10 11:16:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m10s, bytes written: 2.1 GB, bytes/sec 1507208
Jun 10 11:16:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m15s, bytes written: 2.1 GB, bytes/sec 1501805
Jun 10 11:16:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m20s, bytes written: 2.1 GB, bytes/sec 1496442
Jun 10 11:16:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m25s, bytes written: 2.1 GB, bytes/sec 1491116
Jun 10 11:16:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m30s, bytes written: 2.1 GB, bytes/sec 1485829
Jun 10 11:16:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m35s, bytes written: 2.1 GB, bytes/sec 1480578
Jun 10 11:16:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m40s, bytes written: 2.1 GB, bytes/sec 1475365
Jun 10 11:16:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m45s, bytes written: 2.1 GB, bytes/sec 1470188
Jun 10 11:16:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m50s, bytes written: 2.1 GB, bytes/sec 1465048
Jun 10 11:16:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 23m55s, bytes written: 2.1 GB, bytes/sec 1459943
Jun 10 11:16:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:16:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m00s, bytes written: 2.1 GB, bytes/sec 1454874
Jun 10 11:17:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m05s, bytes written: 2.1 GB, bytes/sec 1449840
Jun 10 11:17:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m10s, bytes written: 2.1 GB, bytes/sec 1444840
Jun 10 11:17:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m15s, bytes written: 2.1 GB, bytes/sec 1439875
Jun 10 11:17:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m20s, bytes written: 2.1 GB, bytes/sec 1434944
Jun 10 11:17:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m25s, bytes written: 2.1 GB, bytes/sec 1430047
Jun 10 11:17:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m30s, bytes written: 2.1 GB, bytes/sec 1425183
Jun 10 11:17:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m35s, bytes written: 2.1 GB, bytes/sec 1420352
Jun 10 11:17:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m40s, bytes written: 2.1 GB, bytes/sec 1415553
Jun 10 11:17:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m45s, bytes written: 2.1 GB, bytes/sec 1410787
Jun 10 11:17:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m50s, bytes written: 2.1 GB, bytes/sec 1406053
Jun 10 11:17:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 24m55s, bytes written: 2.1 GB, bytes/sec 1401350
Jun 10 11:17:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:17:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m00s, bytes written: 2.1 GB, bytes/sec 1396679
Jun 10 11:18:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m05s, bytes written: 2.1 GB, bytes/sec 1392039
Jun 10 11:18:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m10s, bytes written: 2.1 GB, bytes/sec 1387429
Jun 10 11:18:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m15s, bytes written: 2.1 GB, bytes/sec 1382850
Jun 10 11:18:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m20s, bytes written: 2.1 GB, bytes/sec 1378302
Jun 10 11:18:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m25s, bytes written: 2.1 GB, bytes/sec 1373783
Jun 10 11:18:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m30s, bytes written: 2.1 GB, bytes/sec 1369293
Jun 10 11:18:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m35s, bytes written: 2.1 GB, bytes/sec 1364833
Jun 10 11:18:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m40s, bytes written: 2.1 GB, bytes/sec 1360402
Jun 10 11:18:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m45s, bytes written: 2.1 GB, bytes/sec 1355999
Jun 10 11:18:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m50s, bytes written: 2.1 GB, bytes/sec 1351625
Jun 10 11:18:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25m55s, bytes written: 2.1 GB, bytes/sec 1347279
Jun 10 11:18:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:18:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m00s, bytes written: 2.1 GB, bytes/sec 1342961
Jun 10 11:19:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m05s, bytes written: 2.1 GB, bytes/sec 1338670
Jun 10 11:19:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m10s, bytes written: 2.1 GB, bytes/sec 1334407
Jun 10 11:19:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m15s, bytes written: 2.1 GB, bytes/sec 1330170
Jun 10 11:19:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m20s, bytes written: 2.1 GB, bytes/sec 1325961
Jun 10 11:19:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m25s, bytes written: 2.1 GB, bytes/sec 1321778
Jun 10 11:19:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m30s, bytes written: 2.1 GB, bytes/sec 1317622
Jun 10 11:19:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m35s, bytes written: 2.1 GB, bytes/sec 1313491
Jun 10 11:19:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m40s, bytes written: 2.1 GB, bytes/sec 1309387
Jun 10 11:19:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m45s, bytes written: 2.1 GB, bytes/sec 1305307
Jun 10 11:19:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m50s, bytes written: 2.1 GB, bytes/sec 1301254
Jun 10 11:19:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 26m55s, bytes written: 2.1 GB, bytes/sec 1297225
Jun 10 11:19:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:19:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m00s, bytes written: 2.1 GB, bytes/sec 1293221
Jun 10 11:20:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m05s, bytes written: 2.1 GB, bytes/sec 1289242
Jun 10 11:20:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m10s, bytes written: 2.1 GB, bytes/sec 1285287
Jun 10 11:20:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m15s, bytes written: 2.1 GB, bytes/sec 1281357
Jun 10 11:20:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m20s, bytes written: 2.1 GB, bytes/sec 1277450
Jun 10 11:20:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m25s, bytes written: 2.1 GB, bytes/sec 1273567
Jun 10 11:20:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m30s, bytes written: 2.1 GB, bytes/sec 1269708
Jun 10 11:20:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m35s, bytes written: 2.1 GB, bytes/sec 1265872
Jun 10 11:20:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m40s, bytes written: 2.1 GB, bytes/sec 1262059
Jun 10 11:20:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m45s, bytes written: 2.1 GB, bytes/sec 1258269
Jun 10 11:20:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m50s, bytes written: 2.1 GB, bytes/sec 1254502
Jun 10 11:20:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 27m55s, bytes written: 2.1 GB, bytes/sec 1250757
Jun 10 11:20:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:20:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m00s, bytes written: 2.1 GB, bytes/sec 1247035
Jun 10 11:21:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m05s, bytes written: 2.1 GB, bytes/sec 1243334
Jun 10 11:21:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m10s, bytes written: 2.1 GB, bytes/sec 1239656
Jun 10 11:21:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m15s, bytes written: 2.1 GB, bytes/sec 1235999
Jun 10 11:21:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m20s, bytes written: 2.1 GB, bytes/sec 1232364
Jun 10 11:21:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m25s, bytes written: 2.1 GB, bytes/sec 1228750
Jun 10 11:21:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m30s, bytes written: 2.1 GB, bytes/sec 1225157
Jun 10 11:21:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m35s, bytes written: 2.1 GB, bytes/sec 1221585
Jun 10 11:21:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m40s, bytes written: 2.1 GB, bytes/sec 1218034
Jun 10 11:21:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m45s, bytes written: 2.1 GB, bytes/sec 1214503
Jun 10 11:21:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m50s, bytes written: 2.1 GB, bytes/sec 1210993
Jun 10 11:21:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 28m55s, bytes written: 2.1 GB, bytes/sec 1207503
Jun 10 11:21:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:21:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m00s, bytes written: 2.1 GB, bytes/sec 1204034
Jun 10 11:22:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m05s, bytes written: 2.1 GB, bytes/sec 1200584
Jun 10 11:22:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m10s, bytes written: 2.1 GB, bytes/sec 1197153
Jun 10 11:22:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m15s, bytes written: 2.1 GB, bytes/sec 1193743
Jun 10 11:22:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m20s, bytes written: 2.1 GB, bytes/sec 1190351
Jun 10 11:22:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m25s, bytes written: 2.1 GB, bytes/sec 1186979
Jun 10 11:22:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m30s, bytes written: 2.1 GB, bytes/sec 1183626
Jun 10 11:22:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m35s, bytes written: 2.1 GB, bytes/sec 1180292
Jun 10 11:22:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m40s, bytes written: 2.1 GB, bytes/sec 1176977
Jun 10 11:22:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m45s, bytes written: 2.1 GB, bytes/sec 1173680
Jun 10 11:22:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m50s, bytes written: 2.1 GB, bytes/sec 1170401
Jun 10 11:22:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 29m55s, bytes written: 2.1 GB, bytes/sec 1167141
Jun 10 11:22:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:22:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m00s, bytes written: 2.1 GB, bytes/sec 1163899
Jun 10 11:23:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m05s, bytes written: 2.1 GB, bytes/sec 1160675
Jun 10 11:23:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m10s, bytes written: 2.1 GB, bytes/sec 1157469
Jun 10 11:23:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m15s, bytes written: 2.1 GB, bytes/sec 1154280
Jun 10 11:23:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m20s, bytes written: 2.1 GB, bytes/sec 1151109
Jun 10 11:23:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m25s, bytes written: 2.1 GB, bytes/sec 1147955
Jun 10 11:23:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m30s, bytes written: 2.1 GB, bytes/sec 1144819
Jun 10 11:23:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m35s, bytes written: 2.1 GB, bytes/sec 1141699
Jun 10 11:23:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m40s, bytes written: 2.1 GB, bytes/sec 1138597
Jun 10 11:23:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m45s, bytes written: 2.1 GB, bytes/sec 1135511
Jun 10 11:23:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m50s, bytes written: 2.1 GB, bytes/sec 1132442
Jun 10 11:23:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30m55s, bytes written: 2.1 GB, bytes/sec 1129390
Jun 10 11:23:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:23:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m00s, bytes written: 2.1 GB, bytes/sec 1126354
Jun 10 11:24:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m05s, bytes written: 2.1 GB, bytes/sec 1123334
Jun 10 11:24:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m10s, bytes written: 2.1 GB, bytes/sec 1120331
Jun 10 11:24:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m15s, bytes written: 2.1 GB, bytes/sec 1117343
Jun 10 11:24:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m20s, bytes written: 2.1 GB, bytes/sec 1114371
Jun 10 11:24:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m25s, bytes written: 2.1 GB, bytes/sec 1111416
Jun 10 11:24:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m30s, bytes written: 2.1 GB, bytes/sec 1108475
Jun 10 11:24:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m35s, bytes written: 2.1 GB, bytes/sec 1105551
Jun 10 11:24:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m40s, bytes written: 2.1 GB, bytes/sec 1102641
Jun 10 11:24:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m45s, bytes written: 2.1 GB, bytes/sec 1099747
Jun 10 11:24:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m50s, bytes written: 2.1 GB, bytes/sec 1096868
Jun 10 11:24:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 31m55s, bytes written: 2.1 GB, bytes/sec 1094004
Jun 10 11:24:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:24:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m00s, bytes written: 2.1 GB, bytes/sec 1091155
Jun 10 11:25:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m05s, bytes written: 2.1 GB, bytes/sec 1088321
Jun 10 11:25:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m10s, bytes written: 2.1 GB, bytes/sec 1085502
Jun 10 11:25:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m15s, bytes written: 2.1 GB, bytes/sec 1082697
Jun 10 11:25:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m20s, bytes written: 2.1 GB, bytes/sec 1079906
Jun 10 11:25:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m25s, bytes written: 2.1 GB, bytes/sec 1077130
Jun 10 11:25:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m30s, bytes written: 2.1 GB, bytes/sec 1074368
Jun 10 11:25:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m35s, bytes written: 2.1 GB, bytes/sec 1071621
Jun 10 11:25:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m40s, bytes written: 2.1 GB, bytes/sec 1068887
Jun 10 11:25:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m45s, bytes written: 2.1 GB, bytes/sec 1066167
Jun 10 11:25:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m50s, bytes written: 2.1 GB, bytes/sec 1063461
Jun 10 11:25:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 32m55s, bytes written: 2.1 GB, bytes/sec 1060769
Jun 10 11:25:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:25:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m00s, bytes written: 2.1 GB, bytes/sec 1058090
Jun 10 11:26:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m05s, bytes written: 2.1 GB, bytes/sec 1055425
Jun 10 11:26:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m10s, bytes written: 2.1 GB, bytes/sec 1052773
Jun 10 11:26:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m15s, bytes written: 2.1 GB, bytes/sec 1050134
Jun 10 11:26:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m20s, bytes written: 2.1 GB, bytes/sec 1047509
Jun 10 11:26:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m25s, bytes written: 2.1 GB, bytes/sec 1044897
Jun 10 11:26:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m30s, bytes written: 2.1 GB, bytes/sec 1042298
Jun 10 11:26:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m35s, bytes written: 2.1 GB, bytes/sec 1039711
Jun 10 11:26:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m40s, bytes written: 2.1 GB, bytes/sec 1037138
Jun 10 11:26:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m45s, bytes written: 2.1 GB, bytes/sec 1034577
Jun 10 11:26:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m50s, bytes written: 2.1 GB, bytes/sec 1032029
Jun 10 11:26:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 33m55s, bytes written: 2.1 GB, bytes/sec 1029493
Jun 10 11:26:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:26:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m00s, bytes written: 2.1 GB, bytes/sec 1026970
Jun 10 11:27:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m05s, bytes written: 2.1 GB, bytes/sec 1024459
Jun 10 11:27:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m10s, bytes written: 2.1 GB, bytes/sec 1021960
Jun 10 11:27:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m15s, bytes written: 2.1 GB, bytes/sec 1019474
Jun 10 11:27:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m20s, bytes written: 2.1 GB, bytes/sec 1016999
Jun 10 11:27:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m25s, bytes written: 2.1 GB, bytes/sec 1014537
Jun 10 11:27:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m30s, bytes written: 2.1 GB, bytes/sec 1012086
Jun 10 11:27:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m35s, bytes written: 2.1 GB, bytes/sec 1009647
Jun 10 11:27:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m40s, bytes written: 2.1 GB, bytes/sec 1007220
Jun 10 11:27:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m45s, bytes written: 2.1 GB, bytes/sec 1004805
Jun 10 11:27:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m50s, bytes written: 2.1 GB, bytes/sec 1002401
Jun 10 11:27:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 34m55s, bytes written: 2.1 GB, bytes/sec 1000009
Jun 10 11:27:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:27:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m00s, bytes written: 2.1 GB, bytes/sec 997628
Jun 10 11:28:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m05s, bytes written: 2.1 GB, bytes/sec 995258
Jun 10 11:28:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m10s, bytes written: 2.1 GB, bytes/sec 992900
Jun 10 11:28:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m15s, bytes written: 2.1 GB, bytes/sec 990552
Jun 10 11:28:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m20s, bytes written: 2.1 GB, bytes/sec 988216
Jun 10 11:28:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m25s, bytes written: 2.1 GB, bytes/sec 985891
Jun 10 11:28:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m30s, bytes written: 2.1 GB, bytes/sec 983577
Jun 10 11:28:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m35s, bytes written: 2.1 GB, bytes/sec 981273
Jun 10 11:28:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m40s, bytes written: 2.1 GB, bytes/sec 978980
Jun 10 11:28:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m45s, bytes written: 2.1 GB, bytes/sec 976698
Jun 10 11:28:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m50s, bytes written: 2.1 GB, bytes/sec 974427
Jun 10 11:28:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35m55s, bytes written: 2.1 GB, bytes/sec 972166
Jun 10 11:28:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:28:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m00s, bytes written: 2.1 GB, bytes/sec 969916
Jun 10 11:29:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m05s, bytes written: 2.1 GB, bytes/sec 967676
Jun 10 11:29:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m10s, bytes written: 2.1 GB, bytes/sec 965446
Jun 10 11:29:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m15s, bytes written: 2.1 GB, bytes/sec 963227
Jun 10 11:29:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m20s, bytes written: 2.1 GB, bytes/sec 961018
Jun 10 11:29:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m25s, bytes written: 2.1 GB, bytes/sec 958818
Jun 10 11:29:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m30s, bytes written: 2.1 GB, bytes/sec 956629
Jun 10 11:29:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m35s, bytes written: 2.1 GB, bytes/sec 954450
Jun 10 11:29:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m40s, bytes written: 2.1 GB, bytes/sec 952281
Jun 10 11:29:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m45s, bytes written: 2.1 GB, bytes/sec 950122
Jun 10 11:29:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m50s, bytes written: 2.1 GB, bytes/sec 947972
Jun 10 11:29:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 36m55s, bytes written: 2.1 GB, bytes/sec 945832
Jun 10 11:29:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:29:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m00s, bytes written: 2.1 GB, bytes/sec 943702
Jun 10 11:30:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m05s, bytes written: 2.1 GB, bytes/sec 941581
Jun 10 11:30:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m10s, bytes written: 2.1 GB, bytes/sec 939470
Jun 10 11:30:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m15s, bytes written: 2.1 GB, bytes/sec 937368
Jun 10 11:30:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m20s, bytes written: 2.1 GB, bytes/sec 935276
Jun 10 11:30:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m25s, bytes written: 2.1 GB, bytes/sec 933193
Jun 10 11:30:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m30s, bytes written: 2.1 GB, bytes/sec 931119
Jun 10 11:30:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m35s, bytes written: 2.1 GB, bytes/sec 929055
Jun 10 11:30:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m40s, bytes written: 2.1 GB, bytes/sec 926999
Jun 10 11:30:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m45s, bytes written: 2.1 GB, bytes/sec 924953
Jun 10 11:30:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m50s, bytes written: 2.1 GB, bytes/sec 922915
Jun 10 11:30:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 37m55s, bytes written: 2.1 GB, bytes/sec 920887
Jun 10 11:30:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:30:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m00s, bytes written: 2.1 GB, bytes/sec 918868
Jun 10 11:31:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m05s, bytes written: 2.1 GB, bytes/sec 916857
Jun 10 11:31:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m10s, bytes written: 2.1 GB, bytes/sec 914855
Jun 10 11:31:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m15s, bytes written: 2.1 GB, bytes/sec 912862
Jun 10 11:31:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m20s, bytes written: 2.1 GB, bytes/sec 910877
Jun 10 11:31:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m25s, bytes written: 2.1 GB, bytes/sec 908902
Jun 10 11:31:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m30s, bytes written: 2.1 GB, bytes/sec 906934
Jun 10 11:31:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m35s, bytes written: 2.1 GB, bytes/sec 904975
Jun 10 11:31:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m40s, bytes written: 2.1 GB, bytes/sec 903025
Jun 10 11:31:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m45s, bytes written: 2.1 GB, bytes/sec 901083
Jun 10 11:31:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m50s, bytes written: 2.1 GB, bytes/sec 899149
Jun 10 11:31:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 38m55s, bytes written: 2.1 GB, bytes/sec 897224
Jun 10 11:31:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:31:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m00s, bytes written: 2.1 GB, bytes/sec 895307
Jun 10 11:32:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m05s, bytes written: 2.1 GB, bytes/sec 893398
Jun 10 11:32:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m10s, bytes written: 2.1 GB, bytes/sec 891497
Jun 10 11:32:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m15s, bytes written: 2.1 GB, bytes/sec 889604
Jun 10 11:32:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m20s, bytes written: 2.1 GB, bytes/sec 887720
Jun 10 11:32:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m25s, bytes written: 2.1 GB, bytes/sec 885843
Jun 10 11:32:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m30s, bytes written: 2.1 GB, bytes/sec 883974
Jun 10 11:32:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m35s, bytes written: 2.1 GB, bytes/sec 882113
Jun 10 11:32:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m40s, bytes written: 2.1 GB, bytes/sec 880260
Jun 10 11:32:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m45s, bytes written: 2.1 GB, bytes/sec 878414
Jun 10 11:32:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m50s, bytes written: 2.1 GB, bytes/sec 876577
Jun 10 11:32:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 39m55s, bytes written: 2.1 GB, bytes/sec 874747
Jun 10 11:32:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:32:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m00s, bytes written: 2.1 GB, bytes/sec 872924
Jun 10 11:33:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m05s, bytes written: 2.1 GB, bytes/sec 871109
Jun 10 11:33:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m10s, bytes written: 2.1 GB, bytes/sec 869302
Jun 10 11:33:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m15s, bytes written: 2.1 GB, bytes/sec 867502
Jun 10 11:33:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m20s, bytes written: 2.1 GB, bytes/sec 865710
Jun 10 11:33:24 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m25s, bytes written: 2.1 GB, bytes/sec 863925
Jun 10 11:33:29 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m30s, bytes written: 2.1 GB, bytes/sec 862147
Jun 10 11:33:34 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m35s, bytes written: 2.1 GB, bytes/sec 860377
Jun 10 11:33:39 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m40s, bytes written: 2.1 GB, bytes/sec 858614
Jun 10 11:33:44 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m45s, bytes written: 2.1 GB, bytes/sec 856858
Jun 10 11:33:49 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m50s, bytes written: 2.1 GB, bytes/sec 855109
Jun 10 11:33:54 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40m55s, bytes written: 2.1 GB, bytes/sec 853368
Jun 10 11:33:59 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:33:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 41m00s, bytes written: 2.1 GB, bytes/sec 851633
Jun 10 11:34:04 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:34:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 41m05s, bytes written: 2.1 GB, bytes/sec 849906
Jun 10 11:34:09 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:34:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 41m10s, bytes written: 2.1 GB, bytes/sec 848185
Jun 10 11:34:14 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:34:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 41m15s, bytes written: 2.1 GB, bytes/sec 846472
Jun 10 11:34:19 ip-172-23-1-132 dgraph[9167]: 2018/06/10 11:34:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 41m20s, bytes written: 2.1 GB, bytes/sec 844765

when I start third server I get this errors:

server1

Jun 10 11:12:29 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:12:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m30s, bytes sent: 2.1 GB, bytes/sec 1778126
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:55:34 pool.go:108: == CONNECT ==> Setting dgraph-3.stage.edocu.eu:7080
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: 2018/06/10 11:55:34 Assert failed
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.AssertTrue
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/error.go:55
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.CompareKeys
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/y.go:124
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl.(*Skiplist).findNear
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl/skl.go:210
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl.(*Skiplist).Get
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl/skl.go:376
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).get
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:501
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*Item).yieldItemValue
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/iterator.go:175
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*Item).Value
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/iterator.go:102
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/posting.ReadPostingList
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/posting/mvcc.go:408
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/worker.toKV
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/worker/predicate.go:191
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/worker.(*grpcWorker).PredicateAndSchemaData
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/worker/predicate.go:277
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: github.com/dgraph-io/dgraph/protos/intern._Worker_PredicateAndSchemaData_Handler
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/github.com/dgraph-io/dgraph/protos/intern/internal.pb.go:2652
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: google.golang.org/grpc.(*Server).processStreamingRPC
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/google.golang.org/grpc/server.go:1160
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: google.golang.org/grpc.(*Server).handleStream
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/google.golang.org/grpc/server.go:1253
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: google.golang.org/grpc.(*Server).serveStreams.func1.1
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /go/src/google.golang.org/grpc/server.go:680
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]: runtime.goexit
Jun 10 11:55:34 ip-172-23-3-87 dgraph[9682]:         /usr/local/go/src/runtime/asm_amd64.s:2361
Jun 10 11:55:34 ip-172-23-3-87 systemd[1]: dgraph-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 10 11:55:34 ip-172-23-3-87 systemd[1]: dgraph-server.service: Unit entered failed state.
Jun 10 11:55:34 ip-172-23-3-87 systemd[1]: dgraph-server.service: Failed with result 'exit-code'.
Jun 10 11:55:49 ip-172-23-3-87 systemd[1]: dgraph-server.service: Service hold-off time over, scheduling restart.
Jun 10 11:55:49 ip-172-23-3-87 systemd[1]: Stopped dGraph Server.
Jun 10 11:55:49 ip-172-23-3-87 systemd[1]: Started dGraph Server.
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 groups.go:77: Current Raft Id: 1
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 gRPC server started.  Listening on port 9080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 HTTP server started.  Listening on port 8080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 worker.go:89: Worker listening at address: [::]:7080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 pool.go:108: == CONNECT ==> Setting dgraph-1.stage.edocu.eu:5080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 groups.go:104: Connected to group zero. Assigned group: 0
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 pool.go:108: == CONNECT ==> Setting dgraph-2.stage.edocu.eu:7080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 pool.go:108: == CONNECT ==> Setting dgraph-3.stage.edocu.eu:7080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 pool.go:108: == CONNECT ==> Setting dgraph-2.stage.edocu.eu:5080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 pool.go:108: == CONNECT ==> Setting dgraph-3.stage.edocu.eu:5080
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 draft.go:153: Node ID: 1 with GroupID: 1
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 node.go:199: Found hardstate: {Term:2 Vote:1 Commit:3 XXX_unrecognized:[]}
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 node.go:208: Group 1 found 4 entries
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 draft.go:991: Restarting node for group: 1
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 raft.go:567: INFO: 1 became follower at term 2
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 raft.go:315: INFO: newRaft 1 [peers: [], term: 2, commit: 3, applied: 0, lastindex: 3, lastterm: 2]
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 node.go:115: Setting conf state to nodes:1
Jun 10 11:55:52 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:52 mutation.go:180: Done schema update predicate:"_predicate_" value_type:STRING list:true
Jun 10 11:55:54 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:54 raft.go:749: INFO: 1 is starting a new election at term 2
Jun 10 11:55:54 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:54 raft.go:580: INFO: 1 became candidate at term 3
Jun 10 11:55:54 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:54 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 3
Jun 10 11:55:54 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:54 raft.go:621: INFO: 1 became leader at term 3
Jun 10 11:55:54 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:55:54 node.go:301: INFO: raft.node: 1 elected leader 1 at term 3
Jun 10 11:56:09 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:14 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:14 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:19 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:24 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:29 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:29 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:34 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:34 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:39 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:39 predicate.go:250: Sending SNAPSHOT: Time elapsed: 34s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:44 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:44 predicate.go:250: Sending SNAPSHOT: Time elapsed: 39s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:49 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:49 predicate.go:250: Sending SNAPSHOT: Time elapsed: 44s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:54 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:54 predicate.go:250: Sending SNAPSHOT: Time elapsed: 49s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:56:59 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:56:59 predicate.go:250: Sending SNAPSHOT: Time elapsed: 54s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:57:04 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:57:04 predicate.go:250: Sending SNAPSHOT: Time elapsed: 59s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:57:09 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:57:09 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m04s, bytes sent: 0 B, bytes/sec 0
Jun 10 11:57:12 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:57:12 node.go:115: Setting conf state to nodes:1 nodes:3
Jun 10 11:57:13 ip-172-23-3-87 dgraph[10047]: 2018/06/10 11:57:13 groups.go:494: Got address of a Zero server: dgraph-2.stage.edocu.eu:5080

server3

Jun 10 11:55:33 ip-172-23-2-111 systemd[1]: Started dGraph Server.
Jun 10 11:55:33 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:33 gRPC server started.  Listening on port 9080
Jun 10 11:55:33 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:33 HTTP server started.  Listening on port 8080
Jun 10 11:55:33 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:33 groups.go:77: Current Raft Id: 0
Jun 10 11:55:33 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:33 worker.go:89: Worker listening at address: [::]:7080
Jun 10 11:55:33 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:33 pool.go:108: == CONNECT ==> Setting dgraph-1.stage.edocu.eu:5080
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 groups.go:104: Connected to group zero. Assigned group: 1
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 pool.go:108: == CONNECT ==> Setting dgraph-1.stage.edocu.eu:7080
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 pool.go:108: == CONNECT ==> Setting dgraph-2.stage.edocu.eu:7080
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 pool.go:108: == CONNECT ==> Setting dgraph-2.stage.edocu.eu:5080
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 pool.go:108: == CONNECT ==> Setting dgraph-3.stage.edocu.eu:5080
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 draft.go:153: Node ID: 3 with GroupID: 1
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 node.go:208: Group 1 found 1 entries
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 draft.go:950: Calling IsPeer
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 draft.go:955: Done with IsPeer call
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 draft.go:1002: New Node for group: 1
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 draft.go:1007: Retrieving snapshot from peer: 1
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:34 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = transport is closing
Jun 10 11:55:34 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:35 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:35 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:35 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:36 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:36 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:36 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:37 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:37 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:37 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:38 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:38 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:38 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:39 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:39 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:39 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:40 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:40 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:40 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:41 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:41 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:41 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:42 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:42 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:42 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:43 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:43 draft.go:965: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:43 ip-172-23-2-111 dgraph[9346]: . Retrying...
Jun 10 11:55:44 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:44 pool.go:158: Echo error from dgraph-1.stage.edocu.eu:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:44 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:44 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:45 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:45 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:46 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:46 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:47 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:47 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:48 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:48 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:49 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:49 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:50 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:50 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:51 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:51 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:52 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:52 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:53 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:53 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:54 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:54 pool.go:158: Echo error from dgraph-1.stage.edocu.eu:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.3.87:7080: connect: connection refused"
Jun 10 11:55:54 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:54 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:55 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:55 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:56 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:56 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:57 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:57 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:58 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:58 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:55:59 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:55:59 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:56:00 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:00 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:56:01 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:01 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:56:02 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:02 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:56:03 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:03 draft.go:965: Error while calling fn: Unable to reach leader in group 1. Retrying...
Jun 10 11:56:09 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:14 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:14 predicate.go:42: Getting SNAPSHOT: Time elapsed: 10s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:19 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:19 predicate.go:42: Getting SNAPSHOT: Time elapsed: 15s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:24 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:24 predicate.go:42: Getting SNAPSHOT: Time elapsed: 20s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:29 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:29 predicate.go:42: Getting SNAPSHOT: Time elapsed: 25s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:34 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:34 predicate.go:42: Getting SNAPSHOT: Time elapsed: 30s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:39 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:39 predicate.go:42: Getting SNAPSHOT: Time elapsed: 35s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:44 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:44 predicate.go:42: Getting SNAPSHOT: Time elapsed: 40s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:49 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:49 predicate.go:42: Getting SNAPSHOT: Time elapsed: 45s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:54 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:54 predicate.go:42: Getting SNAPSHOT: Time elapsed: 50s, bytes written: 0 B, bytes/sec 0
Jun 10 11:56:59 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:56:59 predicate.go:42: Getting SNAPSHOT: Time elapsed: 55s, bytes written: 0 B, bytes/sec 0
Jun 10 11:57:04 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:04 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m00s, bytes written: 0 B, bytes/sec 0
Jun 10 11:57:09 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:09 predicate.go:42: Getting SNAPSHOT: Time elapsed: 01m05s, bytes written: 0 B, bytes/sec 0
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 draft.go:1010: Trying to join peers.
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 draft.go:933: Calling JoinCluster via leader: dgraph-1.stage.edocu.eu:7080
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 draft.go:937: Done with JoinCluster call
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 raft.go:567: INFO: 3 became follower at term 0
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 raft.go:315: INFO: newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 raft.go:567: INFO: 3 became follower at term 1
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 groups.go:494: Got address of a Zero server: dgraph-3.stage.edocu.eu:5080
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 raft.go:708: INFO: 3 [term: 1] received a MsgHeartbeat message with higher term from 1 [term: 3]
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 raft.go:567: INFO: 3 became follower at term 3
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 node.go:301: INFO: raft.node: 3 elected leader 1 at term 3
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 node.go:115: Setting conf state to nodes:1
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 node.go:115: Setting conf state to nodes:1 nodes:3
Jun 10 11:57:12 ip-172-23-2-111 dgraph[9346]: 2018/06/10 11:57:12 mutation.go:180: Done schema update predicate:"_predicate_" value_type:STRING list:true

I can't understand why your crash is happening, that seems strange. But, the snapshot moves are now fixed by #2440 .

Can you please retry?

Okay. I could reproduce the crash at my end as well. Happens rarely, but when it happens, it continues to happen. My latest commit, https://github.com/dgraph-io/dgraph/pull/2442 should fix this.

I'm intending to mark this bug as closed. Would like it if you can retry before I close this.

P.S. You won't have to wait for hours to see the data streams happen. They should be very fast now.

Sorry @manishrjain but I cannot verify this fix. I build dgraph from source and now i get:

Jun 14 09:23:48 ip-172-23-3-244 systemd[1]: Started dGraph Server.
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 gRPC server started.  Listening on port 9080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 HTTP server started.  Listening on port 8080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 groups.go:77: Current Raft Id: 1
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 worker.go:89: Worker listening at address: [::]:7080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 pool.go:108: == CONNECT ==> Setting internal-dGraph-zero-stage:5080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 groups.go:104: Connected to group zero. Assigned group: 1
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 pool.go:108: == CONNECT ==> Setting dgraph-1:5080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 pool.go:108: == CONNECT ==> Setting dgraph-2:5080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 pool.go:108: == CONNECT ==> Setting dgraph-3:5080
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 draft.go:153: Node ID: 1 with GroupID: 1
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 node.go:196: Found Snapshot, Metadata: {ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:135 Term:3 XXX_unrecognized:[]}
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 node.go:207: Found hardstate: {Term:3 Vote:1 Commit:135 XXX_unrecognized:[]}
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 node.go:216: Group 1 found 0 entries
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 draft.go:995: Restarting node for group: 1
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: panic: Unable to find raft entry
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: goroutine 2989 [running]:
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft.newLog(0x132a020, 0xc43a31a270, 0x132d2e0, 0xc420336220, 0x0)
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft/log.go:55 +0x1d6
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft.newRaft(0xc43a29e800, 0x808984)
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft/raft.go:270 +0x86
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft.RestartNode(0xc43a29e800, 0x1, 0xc433742700)
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft/node.go:219 +0x43
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode(0xc43a2a0360)
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]:         /go/src/github.com/dgraph-io/dgraph/worker/draft.go:1007 +0x43c
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: github.com/dgraph-io/dgraph/worker.StartRaftNodes(0xc4201ae000, 0x1)
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]:         /go/src/github.com/dgraph-io/dgraph/worker/groups.go:122 +0x7be
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: created by github.com/dgraph-io/dgraph/dgraph/cmd/server.run
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]:         /go/src/github.com/dgraph-io/dgraph/dgraph/cmd/server/run.go:355 +0x8f7
Jun 14 09:23:51 ip-172-23-3-244 systemd[1]: dgraph.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 14 09:23:51 ip-172-23-3-244 systemd[1]: dgraph.service: Unit entered failed state.
Jun 14 09:23:51 ip-172-23-3-244 systemd[1]: dgraph.service: Failed with result 'exit-code'.
Jun 14 09:24:06 ip-172-23-3-244 systemd[1]: dgraph.service: Service hold-off time over, scheduling restart.
Jun 14 09:24:06 ip-172-23-3-244 systemd[1]: Stopped dGraph Server.
Jun 14 09:24:06 ip-172-23-3-244 systemd[1]: Started dGraph Server.
Jun 14 09:24:08 ip-172-23-3-244 systemd[1]: Stopping dGraph Server...
Jun 14 09:24:08 ip-172-23-3-244 systemd[1]: Stopped dGraph Server.

Looks like you're reusing the old WAL. I changed the way it's stored. Can you start with a clean state?

What does it mean?

I started new servers. Build dgraph from master branch. Then I started zero, run dgraph bulk, copied out/0/p dir and than started dgraph server.

I will check it again.

S pozdravom, Roman Selmeci

On Thu, Jun 14, 2018 at 3:49 PM Manish R Jain notifications@github.com wrote:

Looks like you're reusing the old WAL. I changed the way it's stored. Can you start with a clean state?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dgraph-io/dgraph/issues/2424#issuecomment-397302666, or mute the thread https://github.com/notifications/unsubscribe-auth/AANuGyQJCkQhGcRtkOVjqV3CDar_YPD1ks5t8moEgaJpZM4UXJRP .

But, were you using the old directories? Start with no data, no directories.

You were using some old state, because I see these logs:

Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 node.go:196: Found Snapshot, Metadata: {ConfState:{Nodes:[1] XXX_unrecognized:[]} Index:135 Term:3 XXX_unrecognized:[]}
Jun 14 09:23:51 ip-172-23-3-244 dgraph[21466]: 2018/06/14 09:23:51 node.go:207: Found hardstate: {Term:3 Vote:1 Commit:135 XXX_unrecognized:[]}

It found an existing snapshto and hardstate, which is an indication of a previous w directory.

Yes, I did not remove -w dir for dgraph server.

Replication is running. I let it during night and will text at morning if everything is ok.

ok, it crashes with error:

Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]: panic: runtime error: integer divide by zero
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]: goroutine 7736 [running]:
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]: github.com/dgraph-io/dgraph/worker.(*streamLists).streamKVs(0xc4205588d0, 0x1328400, 0xc420558840, 0x125e3b0, 0x10, 0xc43632b560, 0x0, 0x0)
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]:         /go/src/github.com/dgraph-io/dgraph/worker/stream_lists.go:221 +0x755
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]: github.com/dgraph-io/dgraph/worker.(*streamLists).orchestrate.func2(0xc43632b620, 0xc4205588d0, 0x1328400, 0xc420558840, 0x125e3b0, 0x10, 0xc43632b560)
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]:         /go/src/github.com/dgraph-io/dgraph/worker/stream_lists.go:66 +0x6a
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]: created by github.com/dgraph-io/dgraph/worker.(*streamLists).orchestrate
Jun 14 20:53:37 ip-172-23-3-215 dgraph[28703]:         /go/src/github.com/dgraph-io/dgraph/worker/stream_lists.go:65 +0x238

It transfer only 754M and after restart replication is not continue and finish with:

Jun 14 20:57:07 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:07 predicate.go:44: Getting SNAPSHOT: Time elapsed: 03m01s, bytes written: 0 B, 0 B/s
Jun 14 20:57:08 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:08 predicate.go:44: Getting SNAPSHOT: Time elapsed: 03m02s, bytes written: 0 B, 0 B/s
Jun 14 20:57:09 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:09 predicate.go:44: Getting SNAPSHOT: Time elapsed: 03m03s, bytes written: 0 B, 0 B/s
Jun 14 20:57:10 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:10 predicate.go:44: Getting SNAPSHOT: Time elapsed: 03m04s, bytes written: 0 B, 0 B/s
Jun 14 20:57:11 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:11 predicate.go:126: EOF has been reached
Jun 14 20:57:11 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:11 predicate.go:176: Got 4622 keys. DONE.
Jun 14 20:57:11 ip-172-23-1-215 dgraph[15956]: 2018/06/14 20:57:11 draft.go:1017: Trying to join peers.

What a weird bug to hit! The time ticker literally waits for a second, and yet the duration got rounded to zero. Just submitted a fix for that in master. Please retry.

Also, can you include as many logs as possible?

Hi @manishrjain, it crashes agian :/.

You can find logs here:

I run it on AWS server of i3.large type. CPU during replication takes 100% and 14.4 GB of RAM.

I don't think Dgraph crashes on its own. systemd was stopping Dgraph, when the panic happened. We could still improve the handling of the goroutine which calculates table sizes, so that it is exited before calling Close on Badger (which caused the panic here).

In fact on closer inspection, it looks like it was doing its first scan of the store to determine what all is in there. So, Dgraph got killed by systemd very quickly after starting. Logs also show that. Why was it killed immediately after start?

When the RAM usage is high, if you could do a memory profile, that'd help understand what is causing the high RAM usage. I suspect, its Go GC being slow in releasing memory back to the OS, than anything else. When we do the data transfer, it allocates a lot of memory, that is freed quickly. But Go tends to hold onto this memory instead of releasing it back to the OS.

Ok, I will take several memory profiles at Monday.

Dňa pi 15. 6. 2018, 17:11 Manish R Jain notifications@github.com napísal(a):

I don't think Dgraph crashes on its own. systemd was stopping Dgraph, when the panic happened. We could still improve the handling of the goroutine which calculates table sizes, so that it is exited before calling Close on Badger (which caused the panic here).

When the RAM usage is high, if you could do a memory profile, that'd help understand what is causing the high RAM usage. I suspect, its Go GC being slow in releasing memory back to the OS, than anything else. When we do the data transfer, it allocates a lot of memory, that is releases quickly. And Go tends to hold onto this memory instead of releasing it back to the OS.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dgraph-io/dgraph/issues/2424#issuecomment-397652491, or mute the thread https://github.com/notifications/unsubscribe-auth/AANuG2SXjp3AP-SXzpn7HRfNZB6qeftSks5t886KgaJpZM4UXJRP .

Looks like it is because of Value Log being mmapped. I've created a branch, which switches ValueLog to be served out of disk directly: https://github.com/dgraph-io/dgraph/tree/mrjn/low-mem

Can you compile the binary from that branch, and try that? If it seems like it's better -- I could expose an option to set that.

Hmm, I'm little confused.

RAM is constantly low during replication process, but I do not understand logs and why replication is finished when only 2.2 GB of data were transferred (and size of my data are over 11 GB on server 1).

Logs are here.

Also, when I connected third server, the first server crash with error:

Jun 19 14:18:14 ip-172-23-3-215 dgraph[6389]: 2018/06/19 14:18:14 stream_lists.go:231: Sending SNAPSHOT Time elapsed: 14s, bytes sent: 64 MB, speed: 4.6 MB/sec
Jun 19 14:18:15 ip-172-23-3-215 dgraph[6389]: 2018/06/19 14:18:15 stream_lists.go:213: Sent batch of size: 2.4 MB in 102.711705ms.
Jun 19 14:18:15 ip-172-23-3-215 dgraph[6389]: 2018/06/19 14:18:15 stream_lists.go:213: Sent batch of size: 2.4 MB in 170.888909ms.
Jun 19 14:18:15 ip-172-23-3-215 dgraph[6389]: 2018/06/19 14:18:15 stream_lists.go:231: Sending SNAPSHOT Time elapsed: 15s, bytes sent: 69 MB, speed: 4.6 MB/sec
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: 2018/06/19 14:18:16 Assert failed
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.AssertTrue
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/error.go:55
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.CompareKeys
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/y.go:124
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl.(*Skiplist).findNear
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl/skl.go:210
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl.(*Skiplist).Get
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/skl/skl.go:376
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).get
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:501
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*Item).yieldItemValue
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/iterator.go:178
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*Item).Value
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/iterator.go:103
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/posting.ReadPostingList
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/posting/mvcc.go:411
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/worker.(*grpcWorker).PredicateAndSchemaData.func2
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/worker/predicate.go:253
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/worker.(*streamLists).produceKVs.func1
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/worker/stream_lists.go:156
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/worker.(*streamLists).produceKVs
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/worker/stream_lists.go:175
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: github.com/dgraph-io/dgraph/worker.(*streamLists).orchestrate.func1
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /go/src/github.com/dgraph-io/dgraph/worker/stream_lists.go:55
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]: runtime.goexit
Jun 19 14:18:16 ip-172-23-3-215 dgraph[6389]:         /usr/local/go/src/runtime/asm_amd64.s:2361
Jun 19 14:18:16 ip-172-23-3-215 systemd[1]: dgraph.service: Main process exited, code=exited, status=1/FAILURE
Jun 19 14:18:16 ip-172-23-3-215 systemd[1]: dgraph.service: Unit entered failed state.
Jun 19 14:18:16 ip-172-23-3-215 systemd[1]: dgraph.service: Failed with result 'exit-code'.

Hey @selmeci , can you confirm what is the latest commit you're building from?

It's possible that the data on server 1 could be higher, because it contains all the various versions of the keys, while it only transfers out the latest version during a snapshot. Over time, other versions would get GCed. I was working on a debugging tool to help debug things like this. I'll try and push that out in a few days.

git log:

commit 98b48ce06ae777d089c18cdd2e30028094c4f3cd
Merge: 881ea6c3 aaa7d806
Author: Manish R Jain <manish@dgraph.io>
Date:   Mon Jun 18 19:40:57 2018 -0700

    Merge branch 'master' into mrjn/low-mem

I do not know much about Go Lang, but I build it in this steps:

docker run -it --rm --name=dgraph -v /data/delete/:/apps golang bash
go get -u -v -t github.com/dgraph-io/dgraph/...
cd /go/src/github.com/dgraph-io/dgraph
git checkout mrjn/low-mem
cd /go/src/github.com/dgraph-io/dgraph/dgraph
go build
mv src/github.com/dgraph-io/dgraph/dgraph/dgraph /apps/
copy binary into the servers

Are you able to see this crash repeatedly? If so, can you copy over your p and w directories from server 1, upload them somewhere and send me a link?

What do you mean by crash repeatedly? This errors occurs only one on server 1. After the restart, server 1 is connected to the cluster without any other errors. Replication on server 3 but finished almost immediately and do not continue.

Unfortunately I cannot upload this data. I would like to migrate our production data to dGraph and I do not have these problems with test data which are much more smaller.

Ok. I'm not sure how to replicate the crash. I saw it at my end a couple of weeks ago, and pushed a fix. https://github.com/dgraph-io/dgraph/commit/2adb06b675901c7829da13d90e2cd193fee65f6b. If I see it again, I can try and debug.

Anyways, looks like the RAM problem is fixed. I've also made many changes to the way moves are done, due to which they should be as fast as we can write data to disk. I'll add a flag which allows serving value log directly from disk and close this issue.

Ok, thanks for great job.

I'm looking forward to new release.

What flag should than I use?

Dňa ut 19. 6. 2018, 19:35 Manish R Jain notifications@github.com napísal(a):

Ok. I'm not sure how to replicate the crash. I saw it at my end a couple of weeks ago, and pushed a fix. 2adb06b https://github.com/dgraph-io/dgraph/commit/2adb06b675901c7829da13d90e2cd193fee65f6b. If I see it again, I can try and debug.

Anyways, looks like the RAM problem is fixed. I've also made many changes to the way moves are done, due to which they should be as fast as we can write data to disk. I'll add a flag which allows serving value log directly from disk and close this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dgraph-io/dgraph/issues/2424#issuecomment-398482782, or mute the thread https://github.com/notifications/unsubscribe-auth/AANuGyD31NQDGzjYRCyE8NnhNEtqI0pPks5t-TZ_gaJpZM4UXJRP .

Look for flags with --badger. prefixes.

dgraph-io / dgraph

During replication server consumes all RAM and crashes #2424