data losing

[harpy-wings]

What version of Dgraph are you using?

v1.2.2

Have you tried reproducing the issue with the latest release?

No

What is the hardware spec (RAM, OS)?

ubuntu 18.04 - CPU 4Core - 8GB Ram

Dgraph configurations

they registered as service in /etc/systemd/system/dg0.service and ... for example

[Unit]
Description=my main dgraph zero server
After=network.target
StartLimitBurst=5
StartLimitIntervalSec=10

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/local/bin/dgraph zero --config /db/z0/config.json

[Install]
WantedBy=multi-user.target

1 Zero [ { "my": "localhost:5080", "replicas": 3, "wal": "/db/z0/zw" } ] 3 Alphas [ { "my": "localhost:7080", "zero": "localhost:5080", "lru_mb": 2048, "postings": "/db/a0/p", "wal": "/db/a0/w" },{ "my": "localhost:7079", "zero": "localhost:5080", "lru_mb": 2048, "postings": "/db/a1/p", "wal": "/db/a1/w", "port_offset": -1 },{ "my": "localhost:7078", "zero": "localhost:5080", "lru_mb": 2048, "postings": "/db/a2/p", "wal": "/db/a2/w", "port_offset": -2 } ]

Expected behaviour and actual result.

current Dgraph index 0x16993f I have a really edgy and complicated data including 2000 exams and ~50000 Questions and 200000 choices. each node has a ~owner . in other word there is a person who owned all those nodes. there are some other things he owed too. i've got nothing when runing:

{
   data(func:uid(UserUid)) {
         owner @filter(type(Exam)) {
            uid
            name
         }
   }
}

bot i have this one

{
   data(func:uid(UserUid)) {
         owner @filter(uid(ExamUid)) {
            uid
            name
         }
   }
}

and this one

{
   data(func:type(Exam)) @filter(uid(ExamUid)) {
         uid
   }
}

It happened to me twice and i restore from backup each time. i zipped the p&w and also export the database after crashing and zipped it to and here they are the exported data is not complete. i mean don't contain the whole data. so i couldn't restore it. export.zip w.1.zip the p.1.zip

Main Questions

fixing data is not important . the most important thing is that what should i do to prevent this happen again?

can Simultaneous requests cause this?

[MichelDiz]

I think the problem is in the way that you are creating the data. I also encountered some problems that confused me.

For example, in your query, you use the "owner" predicate with the first letter in lower case. However, in your dataset, it has no lower case predicate. And Dgraph is case sensitive. Also, there is no "tuid", maybe it was a typo.

When switching from "owner" to "Owner" I still didn't get data. I decided to appeal to expand using "expand (_all _) {expand (_all _) {expand (_all_)}}" to check the data structure.

What I found is that all data is there, but its type schema does not match what is in the dataset. Or the Type Schema doesn't reflects the structure of your dataset.

Also, you have repeated predicates with different cases. It is considered good practice to adopt a standard. e.g. camelCase or Pascal Case. So you avoid confusion.

Another thing, avoid zipping Dgraph folders. Always use Export for this. If you rely on the files, there may be compatibility issues in some cases unintentionally. It's good to avoid. But if you know what you're doing, okay tho.

Anyway, the problem is in the way you insert the data and how you are designing the structure. If you share these details I may be able to help.

[harpy-wings]

@MichelDiz thanks brother for notations. "tuid" was typing mistake.i updated it. the types defined PascalCase and pridicates are cameCase the structure was fine and working perfect before.

that export file is not correct due to exported after crash. here is one old correct one. but due to avoiding high load to database by programming mistake the types didn't defined completely, so the expand(all) wont work at all. here is one good export. the data are OK. and schema is correct. export.OK.zip

i will prepare the full types definition for you and add it here as soon as possible.

i'm ready to get access to server to you and will be Grateful for your helping and appreciate for that.

[harpy-wings]

some of types are

type Person {
    xid
    name
    fallows
    owner
    buy
    used
    shared
}
type Exam {
    name
    component
}
type Question{
    xid
    name
    choice
}
type ExamSessionGroup {
    xid
    endDate
    startDate
    exam
    examSession
}
type ExamSession {
    xid
    corrected
    result
    answer
    question
    scoring
}

[harpy-wings]

2 Notes : that may cues this. Max Open File was 1024 ; *referring to official documentation it's better to be 65535 . the server configuration is not enogph:

low Memory which cues some daily Out Of Memory Crash *the recommended minimum is 8GB.

is it just it or other thing may inflect it too? there are some bigger projects who are working fine with Dgraph which can be count this as one.

[MichelDiz]

About "Max Open File was 1024" this is standard on the OS. In the documentation, it is recommended that the user increase the limit https://dgraph.io/docs/deploy/#too-many-open-files

Regarding memory, it is good that you do load tests. Imagine scenarios and create a "load test" for these scenarios and see how your instances react. The recommendation in the minimum documentation is for minimum loads. Intense use and giant loads require more resources, so you have to adapt to the demand you need.

The Type seems better now.

Now, after the new dataset. I'm not sure what the issue is. Are you saying that after a crash Dgraph has modified the dataset? In your previous dataset you had an "Owner" predicate and in this new one is "owner". Did the crash change the predicate case? that's weird. In the previous dataset the data was there, but with different predicates.

With the new dataset provided, I got:

{
   data(func: uid(0x5c8e2)) {
    uid
    owner(first:1) @filter(type(Exam)) {
    uid
    name
  }
   }
}

{
  "data": {
    "data": [
      {
        "uid": "0x5c8e2",
        "owner": [
          {
            "uid": "0x509a7",
            "name": "آزمون آنلاین تستی عربی دهم رشته انسانی | درس 8: يا مَنْ فِی الْبِحارِ عَجائِبُهُ"
          }
        ]
      }
    ]
  }
}

So, there's no issue now right?

[MichelDiz]

I have a really edgy and complicated data including 2000 exams and ~50000 Questions and 200000 choices. each node has a ~owner . in other word there is a person who owned all those nodes.

A question. I didn't quite understand that part. Are you saying that a single user holds 252,000 objects? Sometimes the browser may crash with a certain amount of information on the screen. That was what happened?

[harpy-wings]

About "Max Open File was 1024" this is standard on the OS. In the documentation, it is recommended that the user increase the limit https://dgraph.io/docs/deploy/#too-many-open-files

Regarding memory, it is good that you do load tests. Imagine scenarios and create a "load test" for these scenarios and see how your instances react. The recommendation in the minimum documentation is for minimum loads. Intense use and giant loads require more resources, so you have to adapt to the demand you need.

The Type seems better now.

Now, after the new dataset. I'm not sure what the issue is. Are you saying that after a crash Dgraph has modified the dataset? In your previous dataset you had an "Owner" predicate and in this new one is "owner". Did the crash change the predicate case? that's weird. In the previous dataset the data was there, but with different predicates.

With the new dataset provided, I got:

{
   data(func: uid(0x5c8e2)) {
    uid
    owner(first:1) @filter(type(Exam)) {
    uid
    name
  }
   }
}

{
  "data": {
    "data": [
      {
        "uid": "0x5c8e2",
        "owner": [
          {
            "uid": "0x509a7",
            "name": "آزمون آنلاین تستی عربی دهم رشته انسانی | درس 8: يا مَنْ فِی الْبِحارِ عَجائِبُهُ"
          }
        ]
      }
    ]
  }
}

So, there's no issue now right?

yes , the export.OK.zip was exported a month ago it's ok and data are fine. the Owner pridicate was a mistake from old preview versions. you can simply drop it. forget about the Owner . the "owner" is correct and your result is fine . my main question is what was the main reason of this happened? the file attached to the first comment was exported after the crash and it wasn't complete, some predicates disappeared.

[harpy-wings]

I have a really edgy and complicated data including 2000 exams and ~50000 Questions and 200000 choices. each node has a ~owner . in other word there is a person who owned all those nodes.

A question. I didn't quite understand that part. Are you saying that a single user holds 252,000 objects? Sometimes the browser may crash with a certain amount of information on the screen. That was what happened?

the browser is not important , it's something usual that browser can't handle a canvas with too many nodes. the go-client ( dgo ) couldn't get data from this query :

{
   data(func: uid(0x5c8e2)) {
    uid
    owner(first:1) @filter(type(Exam)) {
    uid
    name
  }
   }
}

the point is that this person who owned this 250000 nodes is the admin of website, other users owned things too. other users was fine while the admin lost his owned data.some of this users buied some objects from admin. so they could see that nodes and other part of website working well , but the admin who owned that 250000 nods couldn't see them.

[harpy-wings]

I done some tests with different server configurations . 1RAM - 1CPU

• 1Z - 1A 16RAM - 4CPU
• 1Z - 1A
• 1Z - 3A

test 1 :

• adding Exam with Question Exist in bank; Each test has between one and 48,000 random questions.the average query from database was 17600 questions each exam took about 2.5 second for insertion.

test :

• adding Questions and give its ownership to admin . **5-8 questions per second inserted

the result was interesting and that error occurs again. i thing the reason of it was the Out Of Memory Crash While Mutations. it's like disk sector failure. the data exist but the path of getting them is damaged. we have several crash in each test but none of them didn't damage the data. but data damage in first configuration when the crash happend while Out Of Memory Crash. here is logs. i recorded some video if you are interested in.

View Log

``` ... es:18446744073709551615 expiryTs:1589755638 enabled:true > I0420 06:07:19.498888 25203 node.go:182] Setting conf state to nodes:1 I0420 06:07:19.499218 25203 node.go:182] Setting conf state to nodes:1 nodes:2 I0420 06:07:19.500257 25203 node.go:182] Setting conf state to nodes:1 nodes:2 nodes:3 I0420 06:07:21.540806 25203 log.go:34] raft.node: 1 elected leader 2 at term 17 fatal error: runtime: out of memory runtime stack: runtime.throw(0x189bbf8, 0x16) /usr/local/go/src/runtime/panic.go:774 +0x72 runtime.sysMap(0xc040000000, 0x4000000, 0x24a4298) /usr/local/go/src/runtime/mem_linux.go:169 +0xc5 runtime.(*mheap).sysAlloc(0x248af20, 0x41e000, 0xa280e5, 0x7f8a989ff190) /usr/local/go/src/runtime/malloc.go:701 +0x1cd runtime.(*mheap).grow(0x248af20, 0x20f, 0xffffffff) /usr/local/go/src/runtime/mheap.go:1255 +0xa3 runtime.(*mheap).allocSpanLocked(0x248af20, 0x20f, 0x24a42a8, 0x961805) /usr/local/go/src/runtime/mheap.go:1170 +0x266 runtime.(*mheap).alloc_m(0x248af20, 0x20f, 0x7f894c5f0101, 0x7f894c5fb5c8) /usr/local/go/src/runtime/mheap.go:1022 +0xc2 runtime.(*mheap).alloc.func1() /usr/local/go/src/runtime/mheap.go:1093 +0x4c runtime.(*mheap).alloc(0x248af20, 0x20f, 0x7f8a85010101, 0x7f894c5fb5c8) /usr/local/go/src/runtime/mheap.go:1092 +0x8a runtime.largeAlloc(0x41ca8c, 0x990101, 0x7f894c5fb5c8) /usr/local/go/src/runtime/malloc.go:1138 +0x97 runtime.mallocgc.func1() /usr/local/go/src/runtime/malloc.go:1033 +0x46 runtime.systemstack(0x0) /usr/local/go/src/runtime/asm_amd64.s:370 +0x66 runtime.mstart() /usr/local/go/src/runtime/proc.go:1146 goroutine 693 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0249de758 sp=0xc0249de750 pc=0x9a1a10 runtime.mallocgc(0x41ca8c, 0x1699b60, 0x1, 0x0) /usr/local/go/src/runtime/malloc.go:1032 +0x895 fp=0xc0249de7f8 sp=0xc0249de758 pc=0x94ffa5 runtime.makeslice(0x1699b60, 0x41ca8c, 0x41ca8c, 0x9b0712) /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc0249de828 sp=0xc0249de7f8 pc=0x98a3ec bytes.makeSlice(0x41ca8c, 0x0, 0x0, 0x0) /usr/local/go/src/bytes/buffer.go:229 +0x77 fp=0xc0249de890 sp=0xc0249de828 pc=0xa42e37 bytes.(*Buffer).grow(0xc03f464780, 0xa, 0x1a58c01) /usr/local/go/src/bytes/buffer.go:142 +0x15b fp=0xc0249de8e0 sp=0xc0249de890 pc=0xa4277b bytes.(*Buffer).WriteString(0xc03f464780, 0x173ea8a, 0xa, 0x7f8a98842f00, 0xc03f464780, 0x173ea01) /usr/local/go/src/bytes/buffer.go:184 +0xdd fp=0xc0249de910 sp=0xc0249de8e0 pc=0xa42b9d io.WriteString(0x1a58ca0, 0xc03f464780, 0x173ea8a, 0xa, 0xc03f464780, 0x0, 0x1aa36c0) /usr/local/go/src/io/io.go:291 +0x74 fp=0xc0249de958 sp=0xc0249de910 pc=0xa0dd64 github.com/golang/protobuf/proto.(*textWriter).WriteString(0xc03bf35ce0, 0x173ea8a, 0xa, 0x1, 0x1, 0x5) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:83 +0xb7 fp=0xc0249de9a0 sp=0xc0249de958 pc=0xcd2977 github.com/golang/protobuf/proto.writeName(0xc03bf35ce0, 0xc023ba0e40, 0x185, 0x0) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:163 +0x46 fp=0xc0249de9e0 sp=0xc0249de9a0 pc=0xcd30b6 github.com/golang/protobuf/proto.(*TextMarshaler).writeStruct(0x22a0720, 0xc03bf35ce0, 0x18395c0, 0xc03f18dcc0, 0x199, 0x199, 0x1a58ca0) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:429 +0x7e2 fp=0xc0249dec78 sp=0xc0249de9e0 pc=0xcd4262 github.com/golang/protobuf/proto.(*TextMarshaler).writeAny(0x22a0720, 0xc03bf35ce0, 0x1852f60, 0xc03f3ee7a8, 0x196, 0xc023ba0b40, 0x196, 0x0) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:534 +0x236 fp=0xc0249ded30 sp=0xc0249dec78 pc=0xcd5706 github.com/golang/protobuf/proto.(*TextMarshaler).writeStruct(0x22a0720, 0xc03bf35ce0, 0x1832e40, 0xc03f463cc0, 0x199, 0x199, 0x7f8a989a0520) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:314 +0x149b fp=0xc0249defc8 sp=0xc0249ded30 pc=0xcd4f1b github.com/golang/protobuf/proto.(*TextMarshaler).Marshal(0x22a0720, 0x1a58ca0, 0xc03f464780, 0x1a7c160, 0xc03f463cc0, 0x7f894c851fff, 0x400) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:809 +0x295 fp=0xc0249df068 sp=0xc0249defc8 pc=0xcd7805 github.com/golang/protobuf/proto.(*TextMarshaler).Text(0x22a0720, 0x1a7c160, 0xc03f463cc0, 0x101, 0x7f8a9898d6d0) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:821 +0x6e fp=0xc0249df0b8 sp=0xc0249df068 pc=0xcd7c4e github.com/golang/protobuf/proto.CompactTextString(...) /tmp/go/pkg/mod/github.com/golang/protobuf@v1.3.2/proto/text.go:843 github.com/dgraph-io/dgraph/protos/pb.(*Mutations).String(0xc03f463cc0, 0x1842fe0, 0xc03f463cc0) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:1895 +0x43 fp=0xc0249df0f0 sp=0xc0249df0b8 pc=0xe23f33 fmt.(*pp).handleMethods(0xc03c5861a0, 0x20300f00000076, 0x7f894c851f01) /usr/local/go/src/fmt/print.go:630 +0x302 fp=0xc0249df300 sp=0xc0249df0f0 pc=0xa27b22 fmt.(*pp).printArg(0xc03c5861a0, 0x1842fe0, 0xc03f463cc0, 0x20300f00000076) /usr/local/go/src/fmt/print.go:713 +0x206 fp=0xc0249df398 sp=0xc0249df300 pc=0xa280e6 fmt.(*pp).doPrintf(0xc03c5861a0, 0x189c1e8, 0x17, 0xc0249df6e8, 0x1, 0x1) /usr/local/go/src/fmt/print.go:1030 +0x15b fp=0xc0249df480 sp=0xc0249df398 pc=0xa2baab fmt.Sprintf(0x189c1e8, 0x17, 0xc0249df6e8, 0x1, 0x1, 0xc03c58eae4, 0x10) /usr/local/go/src/fmt/print.go:219 +0x66 fp=0xc0249df4d8 sp=0xc0249df480 pc=0xa24986 go.opencensus.io/trace.(*Span).lazyPrintfInternal(0xc03a141380, 0x0, 0x0, 0x0, 0x189c1e8, 0x17, 0xc0249df6e8, 0x1, 0x1) /tmp/go/pkg/mod/go.opencensus.io@v0.21.0/trace/trace.go:419 +0x99 fp=0xc0249df5a8 sp=0xc0249df4d8 pc=0xe9b089 go.opencensus.io/trace.(*Span).Annotatef(...) /tmp/go/pkg/mod/go.opencensus.io@v0.21.0/trace/trace.go:464 github.com/dgraph-io/dgraph/edgraph.(*Server).doMutate(0x24a1bb0, 0x1a84a60, 0xc03c7620f0, 0xc03bf3b860, 0xc03d51dbc0, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:280 +0xb61 fp=0xc0249df7b8 sp=0xc0249df5a8 pc=0x14a2171 github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery(0x24a1bb0, 0x1a84a60, 0xc03c7620f0, 0xc03c57a070, 0x0, 0xc03d51dbc0, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:755 +0x518 fp=0xc0249df9d0 sp=0xc0249df7b8 pc=0x14a6428 github.com/dgraph-io/dgraph/edgraph.(*Server).Query(0x24a1bb0, 0x1a84a60, 0xc03a10ff50, 0xc03c57a070, 0x24a1bb0, 0xc03a10ff50, 0xc03b6f0a80) /tmp/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:669 +0x52 fp=0xc0249dfa20 sp=0xc0249df9d0 pc=0x14a5df2 github.com/dgraph-io/dgo/v2/protos/api._Dgraph_Query_Handler(0x17ead60, 0x24a1bb0, 0x1a84a60, 0xc03a10ff50, 0xc03bf3eae0, 0x0, 0x1a84a60, 0xc03a10ff50, 0xc03c5f2000, 0xb1c4c) /tmp/go/pkg/mod/github.com/dgraph-io/dgo/v2@v2.1.1-0.20191127085444-c7a02678e8a6/protos/api/api.pb.go:1594 +0x217 fp=0xc0249dfa90 sp=0xc0249dfa20 pc=0xe01187 google.golang.org/grpc.(*Server).processUnaryRPC(0xc00039c420, 0x1a927c0, 0xc000743c80, 0xc03b738600, 0xc0000ebc80, 0x2293618, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:995 +0x460 fp=0xc0249dfe18 sp=0xc0249dfa90 pc=0xde3ca0 google.golang.org/grpc.(*Server).handleStream(0xc00039c420, 0x1a927c0, 0xc000743c80, 0xc03b738600, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1275 +0xd97 fp=0xc0249dff48 sp=0xc0249dfe18 pc=0xde7c07 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0285b1b40, 0xc00039c420, 0x1a927c0, 0xc000743c80, 0xc03b738600) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xbb fp=0xc0249dffb8 sp=0xc0249dff48 pc=0xdf4b6b runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0249dffc0 sp=0xc0249dffb8 pc=0x9a3ae1 created by google.golang.org/grpc.(*Server).serveStreams.func1 /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1 goroutine 1 [semacquire, 9 minutes]: sync.runtime_Semacquire(0xc000873c98) /usr/local/go/src/runtime/sema.go:56 +0x42 sync.(*WaitGroup).Wait(0xc000873c90) /usr/local/go/src/sync/waitgroup.go:130 +0x64 github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer() /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:503 +0x862 github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run() /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:652 +0xda5 github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1(0xc00039e000, 0xc0008e62a0, 0x0, 0x2) /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:90 +0x66 github.com/spf13/cobra.(*Command).execute(0xc00039e000, 0xc0008e6260, 0x2, 0x2, 0xc00039e000, 0xc0008e6260) /tmp/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa github.com/spf13/cobra.(*Command).ExecuteC(0x2296fa0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb github.com/spf13/cobra.(*Command).Execute(...) /tmp/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 github.com/dgraph-io/dgraph/dgraph/cmd.Execute() /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:69 +0x80 main.main() /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:73 +0xb0 goroutine 6 [chan receive]: github.com/golang/glog.(*loggingT).flushDaemon(0x2484c20) /tmp/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x8b created by github.com/golang/glog.init.0 /tmp/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x26f goroutine 10 [select]: go.opencensus.io/stats/view.(*worker).start(0xc0000b0aa0) /tmp/go/pkg/mod/go.opencensus.io@v0.21.0/stats/view/worker.go:154 +0x100 created by go.opencensus.io/stats/view.init.0 /tmp/go/pkg/mod/go.opencensus.io@v0.21.0/stats/view/worker.go:32 +0x57 goroutine 11 [syscall, 9 minutes]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:147 +0x9c os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41 goroutine 12 [chan receive]: github.com/dgraph-io/dgraph/x.init.0.func1(0x1a849e0, 0xc00003a108) /tmp/go/src/github.com/dgraph-io/dgraph/x/metrics.go:212 +0xfd created by github.com/dgraph-io/dgraph/x.init.0 /tmp/go/src/github.com/dgraph-io/dgraph/x/metrics.go:208 +0x93 goroutine 21 [chan receive]: main.main.func1() /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:46 +0x124 created by main.main /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:37 +0xab goroutine 36 [select, 9 minutes]: github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc000325200) /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:96 +0xbe created by github.com/dgraph-io/ristretto.newDefaultPolicy /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:80 +0x129 goroutine 37 [select, 9 minutes]: github.com/dgraph-io/ristretto.(*Cache).processItems(0xc000074660) /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:257 +0xc8 created by github.com/dgraph-io/ristretto.NewCache /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:155 +0x1f3 goroutine 38 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0000322c0, 0xc00000fe00) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:229 +0x2e6 created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:78 +0xa6 goroutine 39 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc000032380, 0xc00000fe00) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:229 +0x2e6 created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:78 +0xa6 goroutine 40 [select]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc000340800, 0xc00000fe60) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:1085 +0x175 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:323 +0xa20 goroutine 796 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8000, 0xc00089c180) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:362 +0x228 created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:343 +0x88 goroutine 797 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8000, 0xc00089c180) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:362 +0x228 created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:343 +0x88 goroutine 798 [chan receive, 9 minutes]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc000340800, 0xc00089c1a0, 0x0, 0x0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:993 +0xc3 github.com/dgraph-io/badger/v2.Open.func4(0xc000010030) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:337 +0x37 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:336 +0x14b7 goroutine 801 [select]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc000340800, 0xc00089c340) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:766 +0x418 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:371 +0xf9c goroutine 800 [select, 9 minutes]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0003409c8) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/value.go:1790 +0x19e created by github.com/dgraph-io/badger/v2.(*valueLog).open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/value.go:1027 +0x22c goroutine 850 [chan receive, 9 minutes]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0003409c8, 0xc00089c360) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/value.go:1693 +0x70 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:375 +0x11c4 goroutine 851 [select, 9 minutes]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc0000eb500, 0xc00089c380) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/publisher.go:67 +0x17f created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:379 +0x107b goroutine 852 [select]: github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc0000eb5f0) /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:96 +0xbe created by github.com/dgraph-io/ristretto.newDefaultPolicy /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:80 +0x129 goroutine 70 [select]: github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0008a20c0) /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:257 +0xc8 created by github.com/dgraph-io/ristretto.NewCache /tmp/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:155 +0x1f3 goroutine 71 [select, 9 minutes]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0000325c0, 0xc0003583a0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:229 +0x2e6 created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:78 +0xa6 goroutine 72 [select, 9 minutes]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0000327c0, 0xc0003583a0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:229 +0x2e6 created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:78 +0xa6 goroutine 52 [select]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc000341000, 0xc0001ee0c0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:1085 +0x175 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:323 +0xa20 goroutine 836 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8070, 0xc0008e6120) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:362 +0x228 created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:343 +0x88 goroutine 837 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8070, 0xc0008e6120) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:362 +0x228 created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/levels.go:343 +0x88 goroutine 838 [chan receive, 9 minutes]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc000341000, 0xc0008e6140, 0x0, 0x0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:993 +0xc3 github.com/dgraph-io/badger/v2.Open.func4(0xc000010270) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:337 +0x37 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:336 +0x14b7 goroutine 841 [select]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc000341000, 0xc0008e62e0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:766 +0x418 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:371 +0xf9c goroutine 840 [select, 9 minutes]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0003411c8) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/value.go:1790 +0x19e created by github.com/dgraph-io/badger/v2.(*valueLog).open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/value.go:1027 +0x22c goroutine 842 [chan receive, 9 minutes]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0003411c8, 0xc0008e6300) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/value.go:1693 +0x70 created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:375 +0x11c4 goroutine 843 [select, 9 minutes]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc00016d200, 0xc0008e6320) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/publisher.go:67 +0x17f created by github.com/dgraph-io/badger/v2.Open /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/db.go:379 +0x107b goroutine 844 [select]: github.com/dgraph-io/dgraph/worker.(*ServerState).runVlogGC(0x2483f60, 0xc000341000) /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:84 +0xfa created by github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:194 +0x740 goroutine 845 [select]: github.com/dgraph-io/dgraph/worker.(*ServerState).runVlogGC(0x2483f60, 0xc000340800) /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:84 +0xfa created by github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:195 +0x76e goroutine 846 [chan receive]: github.com/dgraph-io/dgraph/worker.(*ServerState).fillTimestampRequests(0x2483f60) /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:228 +0x3ba created by github.com/dgraph-io/dgraph/worker.InitServerState /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:59 +0xfb goroutine 847 [select]: github.com/dgraph-io/dgraph/posting.updateMemoryMetrics(0xc0008e63a0) /tmp/go/src/github.com/dgraph-io/dgraph/posting/lists.go:121 +0x16f created by github.com/dgraph-io/dgraph/posting.Init /tmp/go/src/github.com/dgraph-io/dgraph/posting/lists.go:139 +0xd0 goroutine 848 [chan receive]: github.com/dgraph-io/dgraph/worker.(*rateLimiter).bleed(0x2480ca0) /tmp/go/src/github.com/dgraph-io/dgraph/worker/proposal.go:65 +0x100 created by github.com/dgraph-io/dgraph/worker.Init /tmp/go/src/github.com/dgraph-io/dgraph/worker/worker.go:58 +0xeb goroutine 866 [chan receive, 9 minutes]: github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run.func3(0xc0001e6a20) /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:627 +0x4d created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:625 +0xd01 goroutine 868 [IO wait, 9 minutes]: internal/poll.runtime_pollWait(0x7f8a98996d68, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0000b2198, 0x72, 0x0, 0x0, 0x1888dfd) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc0000b2180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8 net.(*netFD).accept(0xc0000b2180, 0x37e11d600, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc0001ee2a0, 0xc000161da8, 0xc000161db0, 0x18) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc0001ee2a0, 0x18f0cc8, 0xc024838000, 0x1a91080, 0xc000130158) /usr/local/go/src/net/tcpsock.go:261 +0x47 google.golang.org/grpc.(*Server).Serve(0xc024838000, 0x1a7f2a0, 0xc0001ee2a0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:586 +0x22e github.com/dgraph-io/dgraph/worker.RunServer(0x94fa01) /tmp/go/src/github.com/dgraph-io/dgraph/worker/worker.go:95 +0x2db created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:417 +0x52 goroutine 869 [IO wait]: internal/poll.runtime_pollWait(0x7f8a98996e38, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0002e6198, 0x72, 0x0, 0x0, 0x1888dfd) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc0002e6180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8 net.(*netFD).accept(0xc0002e6180, 0x37e11d600, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc0008e6480, 0xc024415d28, 0xc024415d30, 0x18) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc0008e6480, 0x18f0cc8, 0xc00039c420, 0x1a91080, 0xc0249ce430) /usr/local/go/src/net/tcpsock.go:261 +0x47 google.golang.org/grpc.(*Server).Serve(0xc00039c420, 0x1a7f2a0, 0xc0008e6480, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:586 +0x22e github.com/dgraph-io/dgraph/dgraph/cmd/alpha.serveGRPC(0x1a7f2a0, 0xc0008e6480, 0x0, 0xc000873c90) /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:388 +0x31c created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:486 +0x68d goroutine 870 [IO wait]: internal/poll.runtime_pollWait(0x7f8a98996f08, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0002e6118, 0x72, 0x0, 0x0, 0x1888dfd) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc0002e6100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8 net.(*netFD).accept(0xc0002e6100, 0xc000065d08, 0xc51704, 0xc0002ec0a0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc0008e6460, 0x5e9d3e38, 0xc000065d08, 0x9cca26) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc0008e6460, 0xc000065d58, 0x18, 0xc000875e00, 0xc50c34) /usr/local/go/src/net/tcpsock.go:261 +0x47 net/http.(*Server).Serve(0xc0002ec000, 0x1a7f2a0, 0xc0008e6460, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2896 +0x280 github.com/dgraph-io/dgraph/dgraph/cmd/alpha.serveHTTP(0x1a7f2a0, 0xc0008e6460, 0x0, 0xc000873c90) /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:406 +0x312 created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:487 +0x6d0 goroutine 871 [chan receive, 9 minutes]: github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer.func1(0xc000873c90, 0x1a7f2a0, 0xc0008e6480, 0x1a7f2a0, 0xc0008e6460) /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:491 +0x7f created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:489 +0x71d goroutine 54 [select, 9 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc024870000) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/balancer_conn_wrappers.go:115 +0x12c created by google.golang.org/grpc.newCCBalancerWrapper /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/balancer_conn_wrappers.go:106 +0x169 goroutine 55 [select]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00035ad70, 0xc000359270, 0x5, 0x5, 0xc024875870, 0x9b036c, 0x18f1b70) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:185 +0xd3 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00035ad70, 0xc000359270, 0x5, 0x5, 0xc024875e98, 0xc023a11ea0, 0xc024875918) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:165 +0x195 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc024828570, 0xc000359270, 0x5, 0x5, 0xc024875940, 0xc024875940, 0xd4c2cc) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:488 +0x55 io.ReadAtLeast(0x1a5a3e0, 0xc024828570, 0xc000359270, 0x5, 0x5, 0x5, 0xc03bf6b300, 0xc024875a28, 0xa4b60a) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0248b0000, 0xc000359270, 0x5, 0x5, 0x1b084bcd, 0xc024875a08, 0x94d4ff) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:472 +0xc7 google.golang.org/grpc.(*parser).recvMsg(0xc000359260, 0x100000000, 0x1, 0xc024875a98, 0x945dc4, 0xc00016b130, 0xc03bf68660, 0x7f8a989ae001) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:508 +0x63 google.golang.org/grpc.recvAndDecompress(0xc000359260, 0xc0248b0000, 0x0, 0x0, 0x100000000, 0xc024875c58, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a0293982, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:639 +0x4d google.golang.org/grpc.recv(0xc000359260, 0x7f8a9895e5f8, 0x24a1bb0, 0xc0248b0000, 0x0, 0x0, 0x1840c80, 0xc03bf5ed00, 0x100000000, 0xc024875c58, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:684 +0x9b google.golang.org/grpc.(*csAttempt).recvMsg(0xc0000b2300, 0x1840c80, 0xc03bf5ed00, 0xc024875c58, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:885 +0xed google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000b2300, 0x20300e, 0x20300e) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:736 +0x46 google.golang.org/grpc.(*clientStream).withRetry(0xc000394120, 0xc024875dc0, 0xc024875d90, 0x0, 0x68) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:590 +0x360 google.golang.org/grpc.(*clientStream).RecvMsg(0xc000394120, 0x1840c80, 0xc03bf5ed00, 0x5e9d3e4b, 0x1b085acc) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:735 +0x103 github.com/dgraph-io/dgraph/protos/pb.(*raftHeartbeatClient).Recv(0xc0003a83b0, 0x83ad4411d8, 0x24842c0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4850 +0x62 github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat(0xc000096270, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:227 +0x1e5 github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc000096270) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:250 +0xce created by github.com/dgraph-io/dgraph/conn.newPool /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:179 +0x402 goroutine 872 [chan receive, 9 minutes]: google.golang.org/grpc.(*addrConn).resetTransport(0xc0000bc780) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/clientconn.go:1077 +0x6ac created by google.golang.org/grpc.(*addrConn).connect /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/clientconn.go:743 +0x120 goroutine 876 [runnable]: internal/poll.runtime_pollWait(0x7f8a98996c98, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0002e6418, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0002e6400, 0xc024894000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc0002e6400, 0xc024894000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0001300b0, 0xc024894000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc0001e6ea0, 0xc0248ac038, 0x9, 0x9, 0x24859e0, 0x7f8a989af0c8, 0x0) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc0001e6ea0, 0xc0248ac038, 0x9, 0x9, 0x9, 0xd1dd25, 0xc03b975f4c, 0xc024cf2e00) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc0248ac038, 0x9, 0x9, 0x1a58c40, 0xc0001e6ea0, 0x0, 0xbf9f6d7200000000, 0x83be05252a, 0x24842c0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc0248ac000, 0xc03b975f40, 0xc03b975f40, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000188380) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:1249 +0x196 created by google.golang.org/grpc/internal/transport.newHTTP2Client /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:289 +0xd75 goroutine 877 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00089e5f0, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001e6f60, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000188380) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:333 +0x7b created by google.golang.org/grpc/internal/transport.newHTTP2Client /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:331 +0xf12 goroutine 56 [select, 9 minutes]: google.golang.org/grpc.newClientStream.func5(0xc000876700, 0xc000394120, 0x1a84a60, 0xc0248900f0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:319 +0xd7 created by google.golang.org/grpc.newClientStream /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:318 +0x9a5 goroutine 57 [select, 9 minutes]: github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat.func1(0x1a849a0, 0xc023ab5ac0, 0xc000096270, 0xc00016b0c0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:218 +0xc7 created by github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:217 +0x1c3 goroutine 41 [select, 9 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc024974300) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/balancer_conn_wrappers.go:115 +0x12c created by google.golang.org/grpc.newCCBalancerWrapper /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/balancer_conn_wrappers.go:106 +0x169 goroutine 42 [select]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0249ea000, 0xc0249d01b0, 0x5, 0x5, 0xc0249e3870, 0x9b036c, 0x18f1b70) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:185 +0xd3 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0249ea000, 0xc0249d01b0, 0x5, 0x5, 0xc0249e3e98, 0xc0249ec000, 0xc0249e3918) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:165 +0x195 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0249cc090, 0xc0249d01b0, 0x5, 0x5, 0xc0249e3940, 0xc0249e3940, 0xd4c2cc) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:488 +0x55 io.ReadAtLeast(0x1a5a3e0, 0xc0249cc090, 0xc0249d01b0, 0x5, 0x5, 0x5, 0xc03c550080, 0xc0249e3a28, 0xa4b60a) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0249e8000, 0xc0249d01b0, 0x5, 0x5, 0x1b9a5e61, 0xc0249e3a08, 0x94d4ff) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:472 +0xc7 google.golang.org/grpc.(*parser).recvMsg(0xc0249d01a0, 0x100000000, 0x1, 0xc0249e3a98, 0x945dc4, 0xc00006b8d0, 0xc03bef7f50, 0x7f8a989afe01) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:508 +0x63 google.golang.org/grpc.recvAndDecompress(0xc0249d01a0, 0xc0249e8000, 0x0, 0x0, 0x100000000, 0xc0249e3c58, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a0bb4b72, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:639 +0x4d google.golang.org/grpc.recv(0xc0249d01a0, 0x7f8a9895e5f8, 0x24a1bb0, 0xc0249e8000, 0x0, 0x0, 0x1840c80, 0xc03bf15f80, 0x100000000, 0xc0249e3c58, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:684 +0x9b google.golang.org/grpc.(*csAttempt).recvMsg(0xc0002e6700, 0x1840c80, 0xc03bf15f80, 0xc0249e3c58, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:885 +0xed google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0002e6700, 0x20300e, 0x20300e) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:736 +0x46 google.golang.org/grpc.(*clientStream).withRetry(0xc0249ac120, 0xc0249e3dc0, 0xc0249e3d90, 0x0, 0x68) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:590 +0x360 google.golang.org/grpc.(*clientStream).RecvMsg(0xc0249ac120, 0x1840c80, 0xc03bf15f80, 0x5e9d3e4b, 0x1b9a747c) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:735 +0x103 github.com/dgraph-io/dgraph/protos/pb.(*raftHeartbeatClient).Recv(0xc0249d4040, 0x83add62b7a, 0x24842c0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4850 +0x62 github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat(0xc0249785b0, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:227 +0x1e5 github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc0249785b0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:250 +0xce created by github.com/dgraph-io/dgraph/conn.newPool /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:179 +0x402 goroutine 878 [chan receive, 9 minutes]: google.golang.org/grpc.(*addrConn).resetTransport(0xc0000bcf00) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/clientconn.go:1077 +0x6ac created by google.golang.org/grpc.(*addrConn).connect /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/clientconn.go:743 +0x120 goroutine 853 [runnable]: internal/poll.runtime_pollWait(0x7f8a98996bc8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0002e6618, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0002e6600, 0xc024990000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc0002e6600, 0xc024990000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0001180a8, 0xc024990000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc00039b2c0, 0xc0002ec118, 0x9, 0x9, 0xc000508000, 0x7f8a989ae040, 0x0) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc00039b2c0, 0xc0002ec118, 0x9, 0x9, 0x9, 0xd1dd25, 0xc03bf6ca2c, 0xc0001f0e00) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc0002ec118, 0x9, 0x9, 0x1a58c40, 0xc00039b2c0, 0x0, 0xbf9f6d7200000000, 0x83add6958a, 0x24842c0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002ec0e0, 0xc03bf6ca20, 0xc03bf6ca20, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0249aa000) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:1249 +0x196 created by google.golang.org/grpc/internal/transport.newHTTP2Client /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:289 +0xd75 goroutine 884 [select]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00089ec30, 0xc0008e6a90, 0x5, 0x5, 0x30, 0x48, 0x18f1b70) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:185 +0xd3 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00089ec30, 0xc0008e6a90, 0x5, 0x5, 0x0, 0xc024832360, 0xc024bcd978) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:165 +0x195 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0248ae930, 0xc0008e6a90, 0x5, 0x5, 0x0, 0xc024bcd9a0, 0xd4c2cc) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:488 +0x55 io.ReadAtLeast(0x1a5a3e0, 0xc0248ae930, 0xc0008e6a90, 0x5, 0x5, 0x5, 0xc03be098c0, 0xc024bcda88, 0xa4b60a) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00019df00, 0xc0008e6a90, 0x5, 0x5, 0xc01d47beca, 0xc024bcda68, 0x94d4ff) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:472 +0xc7 google.golang.org/grpc.(*parser).recvMsg(0xc0008e6a80, 0x100000000, 0x1, 0xc024bcdaf8, 0x945dc4, 0xc00006b9a0, 0xc03bdfb2e0, 0x7f8a9899fe01) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:508 +0x63 google.golang.org/grpc.recvAndDecompress(0xc0008e6a80, 0xc00019df00, 0x0, 0x0, 0x100000000, 0xc024bcdcb8, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a268ab69, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:639 +0x4d google.golang.org/grpc.recv(0xc0008e6a80, 0x7f8a9895e5f8, 0x24a1bb0, 0xc00019df00, 0x0, 0x0, 0x1849c60, 0xc03be10800, 0x100000000, 0xc024bcdcb8, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:684 +0x9b google.golang.org/grpc.(*csAttempt).recvMsg(0xc0002e6800, 0x1849c60, 0xc03be10800, 0xc024bcdcb8, 0xc024bcdd10, 0x9a05e0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:885 +0xed google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0002e6800, 0x20300e, 0x20300e) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:736 +0x46 google.golang.org/grpc.(*clientStream).withRetry(0xc0249ac360, 0xc024bcde20, 0xc024bcddf0, 0x1, 0x68) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:590 +0x360 google.golang.org/grpc.(*clientStream).RecvMsg(0xc0249ac360, 0x1849c60, 0xc03be10800, 0xc024bcdedc, 0x2) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:735 +0x103 github.com/dgraph-io/dgraph/protos/pb.(*zeroStreamMembershipClient).Recv(0xc00006ba20, 0xc024bcded8, 0x2, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:5116 +0x62 github.com/dgraph-io/dgraph/worker.(*groupi).receiveMembershipUpdates.func1(0xc000096270, 0x1a8e4a0, 0xc00006ba20, 0x1a849a0, 0xc024824500, 0xc00006b930, 0xc0001e7740) /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:775 +0xf8 created by github.com/dgraph-io/dgraph/worker.(*groupi).receiveMembershipUpdates /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:771 +0x665 goroutine 854 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000b0a00, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00039b380, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0249aa000) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:333 +0x7b created by google.golang.org/grpc/internal/transport.newHTTP2Client /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:331 +0xf12 goroutine 43 [select, 9 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0249744c0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/balancer_conn_wrappers.go:115 +0x12c created by google.golang.org/grpc.newCCBalancerWrapper /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/balancer_conn_wrappers.go:106 +0x169 goroutine 44 [select]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0249c2280, 0xc0249c42d0, 0x5, 0x5, 0xc0249e1870, 0x9b036c, 0x18f1b70) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:185 +0xd3 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0249c2280, 0xc0249c42d0, 0x5, 0x5, 0xc0249e1e98, 0xc0249b6080, 0xc0249e1918) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:165 +0x195 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0249bc360, 0xc0249c42d0, 0x5, 0x5, 0xc0249e1940, 0xc0249e1940, 0xd4c2cc) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:488 +0x55 io.ReadAtLeast(0x1a5a3e0, 0xc0249bc360, 0xc0249c42d0, 0x5, 0x5, 0x5, 0xc03bf6b340, 0xc0249e1a28, 0xa4b60a) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc024a1a100, 0xc0249c42d0, 0x5, 0x5, 0x1ba07018, 0xc0249e1a08, 0x94d4ff) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:472 +0xc7 google.golang.org/grpc.(*parser).recvMsg(0xc0249c42c0, 0x100000000, 0x1, 0xc0249e1a98, 0x945dc4, 0xc00016b210, 0xc03bf68670, 0x7f8a989ae001) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:508 +0x63 google.golang.org/grpc.recvAndDecompress(0xc0249c42c0, 0xc024a1a100, 0x0, 0x0, 0x100000000, 0xc0249e1c58, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a0c15fa2, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:639 +0x4d google.golang.org/grpc.recv(0xc0249c42c0, 0x7f8a9895e5f8, 0x24a1bb0, 0xc024a1a100, 0x0, 0x0, 0x1840c80, 0xc03bf5ed80, 0x100000000, 0xc0249e1c58, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:684 +0x9b google.golang.org/grpc.(*csAttempt).recvMsg(0xc0000b2480, 0x1840c80, 0xc03bf5ed80, 0xc0249e1c58, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:885 +0xed google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000b2480, 0x20300e, 0x20300e) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:736 +0x46 google.golang.org/grpc.(*clientStream).withRetry(0xc000394360, 0xc0249e1dc0, 0xc0249e1d90, 0x0, 0x68) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:590 +0x360 google.golang.org/grpc.(*clientStream).RecvMsg(0xc000394360, 0x1840c80, 0xc03bf5ed80, 0x5e9d3e4b, 0x1ba0790e) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:735 +0x103 github.com/dgraph-io/dgraph/protos/pb.(*raftHeartbeatClient).Recv(0xc0249c0140, 0x83addc3015, 0x24842c0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4850 +0x62 github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat(0xc024978750, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:227 +0x1e5 github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc024978750) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:250 +0xce created by github.com/dgraph-io/dgraph/conn.newPool /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:179 +0x402 goroutine 74 [chan receive, 9 minutes]: google.golang.org/grpc.(*addrConn).resetTransport(0xc00026a500) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/clientconn.go:1077 +0x6ac created by google.golang.org/grpc.(*addrConn).connect /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/clientconn.go:743 +0x120 goroutine 90 [select, 9 minutes]: google.golang.org/grpc.newClientStream.func5(0xc00085a700, 0xc0249ac120, 0x1a84a60, 0xc0248ae4b0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:319 +0xd7 created by google.golang.org/grpc.newClientStream /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:318 +0x9a5 goroutine 91 [select, 9 minutes]: github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat.func1(0x1a849a0, 0xc024824400, 0xc0249785b0, 0xc00006b870) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:218 +0xc7 created by github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:217 +0x1c3 goroutine 883 [select, 9 minutes]: google.golang.org/grpc.newClientStream.func5(0xc000876700, 0xc0249ac360, 0x1a84a60, 0xc0248ae840) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:319 +0xd7 created by google.golang.org/grpc.newClientStream /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:318 +0x9a5 goroutine 77 [runnable]: internal/poll.runtime_pollWait(0x7f8a98996af8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc024a1c118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc024a1c100, 0xc024a24000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc024a1c100, 0xc024a24000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc024a1e008, 0xc024a24000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc0249ca240, 0xc024a3c038, 0x9, 0x9, 0x20, 0x18, 0x17667a0) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc0249ca240, 0xc024a3c038, 0x9, 0x9, 0x9, 0x100000000d1dd25, 0x0, 0x9) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc024a3c038, 0x9, 0x9, 0x1a58c40, 0xc0249ca240, 0x0, 0xc000000000, 0x950298, 0xc0249c21a0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc024a3c000, 0xc03ba87180, 0xc03ba87180, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc024a42000) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:1249 +0x196 created by google.golang.org/grpc/internal/transport.newHTTP2Client /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:289 +0xd75 goroutine 78 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0249c2190, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0249ca3c0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc024a42000) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:333 +0x7b created by google.golang.org/grpc/internal/transport.newHTTP2Client /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_client.go:331 +0xf12 goroutine 79 [select, 9 minutes]: google.golang.org/grpc.newClientStream.func5(0xc00085aa80, 0xc000394360, 0x1a84a60, 0xc0248905d0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:319 +0xd7 created by google.golang.org/grpc.newClientStream /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:318 +0x9a5 goroutine 80 [select, 9 minutes]: github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat.func1(0x1a849a0, 0xc023ab5c40, 0xc024978750, 0xc00016b1b0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:218 +0xc7 created by github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:217 +0x1c3 goroutine 45 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc024984430, 0x0) /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:229 +0x2e6 created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /tmp/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-0.20191220102048-ab4352b00a17/y/watermark.go:78 +0xa6 goroutine 1114 [select]: github.com/dgraph-io/dgraph/worker.(*groupi).processOracleDeltaStream(0x2484640) /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:976 +0x19a created by github.com/dgraph-io/dgraph/worker.StartRaftNodes /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:157 +0x77a goroutine 1112 [select]: github.com/dgraph-io/dgraph/worker.(*groupi).sendMembershipUpdates(0x2484640) /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:714 +0x1df created by github.com/dgraph-io/dgraph/worker.StartRaftNodes /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:155 +0x732 goroutine 1110 [select]: github.com/dgraph-io/dgraph/worker.(*node).checkpointAndClose(0xc02498e420, 0xc0249b8c00) /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:707 +0x11d created by github.com/dgraph-io/dgraph/worker.(*node).Run /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:790 +0x11e goroutine 1109 [runnable]: github.com/dgraph-io/dgraph/worker.(*node).Run(0xc02498e420) /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:802 +0xcc5 created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1474 +0x50a goroutine 1108 [chan receive]: github.com/dgraph-io/dgraph/conn.(*Node).BatchAndSendMessages(0xc024984360) /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:347 +0x5c1 created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1473 +0x4e5 goroutine 1107 [select]: github.com/dgraph-io/dgraph/worker.(*node).processApplyCh(0xc02498e420) /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:516 +0x218 created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1472 +0x4bd goroutine 1113 [select]: github.com/dgraph-io/dgraph/worker.(*groupi).receiveMembershipUpdates(0x2484640) /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:802 +0x200 created by github.com/dgraph-io/dgraph/worker.StartRaftNodes /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:156 +0x756 goroutine 1106 [select, 5 minutes]: github.com/dgraph-io/dgraph/worker.(*node).processRollups(0xc02498e420) /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:430 +0x1b5 created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1471 +0x498 goroutine 1101 [select]: github.com/dgraph-io/dgraph/conn.(*Node).doSendMessage(0xc024984360, 0x2, 0xc024fe81e0, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:468 +0x4b1 github.com/dgraph-io/dgraph/conn.(*Node).streamMessages(0xc024984360, 0x2, 0xc024aeabe0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:413 +0x1b2 created by github.com/dgraph-io/dgraph/conn.(*Node).BatchAndSendMessages /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:386 +0x51a goroutine 890 [runnable]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001f9e50, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc024e2e4e0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00078be00) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:283 +0xcb created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:280 +0xfbe goroutine 892 [runnable]: internal/poll.runtime_pollWait(0x7f8a98996958, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc024972d18, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc024972d00, 0xc024f8c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc024972d00, 0xc024f8c000, 0x8000, 0x8000, 0x4, 0xc024b46d50, 0xc00077c600) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc02482a318, 0xc024f8c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc024b117a0, 0xc024f84118, 0x9, 0x9, 0xc024b46d00, 0xd488e2, 0xc00078bf01) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc024b117a0, 0xc024f84118, 0x9, 0x9, 0x9, 0xc03b94b710, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc024f84118, 0x9, 0x9, 0x1a58c40, 0xc024b117a0, 0x0, 0x0, 0x5924fa4000, 0x5900000001) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc024f840e0, 0xc024fa4000, 0xc024fa4000, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00078be00, 0xc024f47b60, 0x18f0d20) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:440 +0x9e google.golang.org/grpc.(*Server).serveStreams(0xc024838000, 0x1a927c0, 0xc00078be00) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:706 +0xf3 google.golang.org/grpc.(*Server).handleRawConn.func1(0xc024838000, 0x1a927c0, 0xc00078be00) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:668 +0x3f created by google.golang.org/grpc.(*Server).handleRawConn /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:667 +0x562 goroutine 894 [select]: github.com/dgraph-io/dgraph/conn.(*RaftServer).Heartbeat(0x2482960, 0xc024824e80, 0x1a8e2c0, 0xc024f41260, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:298 +0x37f github.com/dgraph-io/dgraph/protos/pb._Raft_Heartbeat_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024d32e40, 0xc0000b7ef8, 0xc024dd0200) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4942 +0x109 google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00078bc80, 0xc024dd0200, 0xc00016d9e0, 0x228f2e0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1199 +0xb2e google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00078bc80, 0xc024dd0200, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1279 +0xd30 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc024f49430, 0xc024838000, 0x1a927c0, 0xc00078bc80, 0xc024dd0200) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1 goroutine 1096 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc024f8a3c0, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc024b11860, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00081ad80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:283 +0xcb created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:280 +0xfbe goroutine 81 [select]: go.etcd.io/etcd/raft.(*node).run(0xc0249ca780, 0xc024b40000) /tmp/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/node.go:342 +0x4c1 created by go.etcd.io/etcd/raft.RestartNode /tmp/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/node.go:246 +0x31b goroutine 887 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001f9d60, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc024e2e1e0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00078bc80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:283 +0xcb created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:280 +0xfbe goroutine 6353 [IO wait]: internal/poll.runtime_pollWait(0x7f8a989967b8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc023e77718, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc023e77700, 0xc023bf4000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc023e77700, 0xc023bf4000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0249ce430, 0xc023bf4000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc025ed4000, 0xc0248ac3b8, 0x9, 0x9, 0x9cc4aa, 0x8000000000000000, 0x83a91da048) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc025ed4000, 0xc0248ac3b8, 0x9, 0x9, 0x9, 0xd1dd25, 0xc023bccf18, 0xc024fd0e38) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc0248ac3b8, 0x9, 0x9, 0x1a58c40, 0xc025ed4000, 0x0, 0x0, 0x7070e0910100402, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc0248ac380, 0xc03bf6c980, 0xc03bf6c980, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000743c80, 0xc02701a6f0, 0x18f0d20) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:440 +0x9e google.golang.org/grpc.(*Server).serveStreams(0xc00039c420, 0x1a927c0, 0xc000743c80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:706 +0xf3 google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00039c420, 0x1a927c0, 0xc000743c80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:668 +0x3f created by google.golang.org/grpc.(*Server).handleRawConn /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:667 +0x562 goroutine 6351 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc023bcceb0, 0x1, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:395 +0x122 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc025eb2240, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/controlbuf.go:513 +0x1e3 google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000743c80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:283 +0xcb created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:280 +0xfbe goroutine 1128 [select, 9 minutes]: google.golang.org/grpc.newClientStream.func5(0xc00085aa80, 0xc02518c000, 0x1a84a60, 0xc025170870) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:319 +0xd7 created by google.golang.org/grpc.newClientStream /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:318 +0x9a5 goroutine 889 [IO wait]: internal/poll.runtime_pollWait(0x7f8a98996a28, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc024972c98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc024972c80, 0xc024f6c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc024972c80, 0xc024f6c000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc02482a310, 0xc024f6c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc024b11740, 0xc024f84038, 0x9, 0x9, 0xc024b45d10, 0x950298, 0x20) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc024b11740, 0xc024f84038, 0x9, 0x9, 0x9, 0xc024b45dc8, 0xc024b45d70, 0xc024b45e38) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc024f84038, 0x9, 0x9, 0x1a58c40, 0xc024b11740, 0x0, 0x8300000000, 0xc03ba86f60, 0x20) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc024f84000, 0xc03ba86f60, 0xc03ba86f60, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00078bc80, 0xc024f47a40, 0x18f0d20) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:440 +0x9e google.golang.org/grpc.(*Server).serveStreams(0xc024838000, 0x1a927c0, 0xc00078bc80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:706 +0xf3 google.golang.org/grpc.(*Server).handleRawConn.func1(0xc024838000, 0x1a927c0, 0xc00078bc80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:668 +0x3f created by google.golang.org/grpc.(*Server).handleRawConn /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:667 +0x562 goroutine 891 [select, 9 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00078be00) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:943 +0x218 created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:289 +0xfe3 goroutine 888 [select, 9 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00078bc80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:943 +0x218 created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:289 +0xfe3 goroutine 6352 [select]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000743c80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:943 +0x218 created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:289 +0xfe3 goroutine 1099 [select]: github.com/dgraph-io/dgraph/conn.(*RaftServer).Heartbeat(0x2482960, 0xc024975000, 0x1a8e2c0, 0xc024aeaa60, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:298 +0x37f github.com/dgraph-io/dgraph/protos/pb._Raft_Heartbeat_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024a77380, 0xc0000b7ef8, 0xc024ab6200) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4942 +0x109 google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00081ad80, 0xc024ab6200, 0xc00016d9e0, 0x228f2e0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1199 +0xb2e google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00081ad80, 0xc024ab6200, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1279 +0xd30 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00084edc0, 0xc024838000, 0x1a927c0, 0xc00081ad80, 0xc024ab6200) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1 goroutine 1097 [select, 9 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00081ad80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:943 +0x218 created by google.golang.org/grpc/internal/transport.newHTTP2Server /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:289 +0xfe3 goroutine 1098 [IO wait]: internal/poll.runtime_pollWait(0x7f8a98996888, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0002e7118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0002e7100, 0xc024fb0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc0002e7100, 0xc024fb0000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000130158, 0xc024fb0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 bufio.(*Reader).Read(0xc024e2e960, 0xc0248ac118, 0x9, 0x9, 0xc024cecd10, 0x950298, 0x20) /usr/local/go/src/bufio/bufio.go:226 +0x26a io.ReadAtLeast(0x1a58c40, 0xc024e2e960, 0xc0248ac118, 0x9, 0x9, 0x9, 0xc024cecdc8, 0xc024cecd70, 0xc024cece38) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 golang.org/x/net/http2.readFrameHeader(0xc0248ac118, 0x9, 0x9, 0x1a58c40, 0xc024e2e960, 0x0, 0x8300000000, 0xc03bf6ca80, 0x20) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:237 +0x87 golang.org/x/net/http2.(*Framer).ReadFrame(0xc0248ac0e0, 0xc03bf6ca80, 0xc03bf6ca80, 0x0, 0x0) /tmp/go/pkg/mod/golang.org/x/net@v0.0.0-20191209160850-c0dbc17a3553/http2/frame.go:492 +0xa1 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00081ad80, 0xc024fa42d0, 0x18f0d20) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/http2_server.go:440 +0x9e google.golang.org/grpc.(*Server).serveStreams(0xc024838000, 0x1a927c0, 0xc00081ad80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:706 +0xf3 google.golang.org/grpc.(*Server).handleRawConn.func1(0xc024838000, 0x1a927c0, 0xc00081ad80) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:668 +0x3f created by google.golang.org/grpc.(*Server).handleRawConn /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:667 +0x562 goroutine 1100 [runnable]: github.com/dgraph-io/dgraph/conn.(*RaftServer).Heartbeat(0x2482960, 0xc0249751c0, 0x1a8e2c0, 0xc024aeab90, 0x0, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:298 +0x37f github.com/dgraph-io/dgraph/protos/pb._Raft_Heartbeat_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024a77440, 0xc0000b7ef8, 0xc024ab6300) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4942 +0x109 google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024ab6300, 0xc00016d9e0, 0x228f2e0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1199 +0xb2e google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024ab6300, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1279 +0xd30 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc024f49450, 0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024ab6300) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1 goroutine 110 [runnable]: google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc024df8d20, 0xc024f6a770, 0x5, 0x5, 0xb700000000af8957, 0x0, 0x18f1b70) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:173 +0xc7 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc024df8d20, 0xc024f6a770, 0x5, 0x5, 0xc024ff9940, 0xc024f5e640, 0xc024ff9630) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:167 +0x21a google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc024de3f20, 0xc024f6a770, 0x5, 0x5, 0xc024ff9658, 0xc024ff9658, 0xd4d65c) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:488 +0x55 io.ReadAtLeast(0x1a5a3e0, 0xc024de3f20, 0xc024f6a770, 0x5, 0x5, 0x5, 0x11, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:329 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc024f54300, 0xc024f6a770, 0x5, 0x5, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/internal/transport/transport.go:472 +0xc7 google.golang.org/grpc.(*parser).recvMsg(0xc024f6a760, 0x100000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:508 +0x63 google.golang.org/grpc.recvAndDecompress(0xc024f6a760, 0xc024f54300, 0x0, 0x0, 0x100000000, 0xc024ff99b8, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0xc024ff9a98, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:639 +0x4d google.golang.org/grpc.recv(0xc024f6a760, 0x7f8a9895e5f8, 0x24a1bb0, 0xc024f54300, 0x0, 0x0, 0x18289c0, 0xc03ebd61e0, 0x100000000, 0xc024ff99b8, ...) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/rpc_util.go:684 +0x9b google.golang.org/grpc.(*serverStream).RecvMsg(0xc024f560c0, 0x18289c0, 0xc03ebd61e0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/stream.go:1474 +0x170 github.com/dgraph-io/dgraph/protos/pb.(*raftRaftMessageServer).Recv(0xc024f62800, 0x21, 0x30, 0x0) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4978 +0x62 github.com/dgraph-io/dgraph/conn.(*RaftServer).RaftMessage(0x2482960, 0x1a90d80, 0xc024f62800, 0x1a84c20, 0x2482960) /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:249 +0x206 github.com/dgraph-io/dgraph/protos/pb._Raft_RaftMessage_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024f560c0, 0xc0000b7ef8, 0xc024f54300) /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4959 +0xad google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024f54300, 0xc00016d9e0, 0x228f300, 0x0, 0x0, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1199 +0xb2e google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024f54300, 0x0) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1279 +0xd30 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc024f49450, 0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024f54300) /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 /tmp/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1 ```

[harpy-wings]

after another test the server crashed again. how would you like to check it out? here is some screenshots

16G Ram , 4 CPU

1Zero - 3Alpha

"uid" filtering is fine while other filtering like type filtering and equality filtering "eq" are failed.

it sounds like a bug.

dgraph alpha logs

View Log

``` Apr 20 16:55:28 bahamaamooz dgraph[28766]: I0420 16:55:28.877248 28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 104.361534ms. Apr 20 16:55:28 bahamaamooz dgraph[28766]: I0420 16:55:28.935575 28766 log.go:34] Rolling up Created batch of size: 1.4 MB in 55.47142ms. Apr 20 16:55:28 bahamaamooz dgraph[28766]: I0420 16:55:28.993690 28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 57.210653ms. Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.140797 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 49.984726ms. Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.650206 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 9.4 MB, speed: 4.7 MB/sec Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.812894 28766 log.go:34] Rolling up Created batch of size: 4.1 MB in 91.447791ms. Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.812934 28766 log.go:34] Rolling up Sent 55240 keys Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.915312 28766 draft.go:1102] Rolled up 55214 keys. Done Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.915786 28766 draft.go:445] List rollup at Ts 213334: OK. Apr 20 16:55:33 bahamaamooz dgraph[28766]: W0420 16:55:33.125191 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 270ms. Breakdown: [{sync 270ms} {disk 0s} {proposals 0s} {ad Apr 20 16:55:35 bahamaamooz dgraph[28766]: W0420 16:55:35.843292 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 475ms. Breakdown: [{sync 475ms} {disk 0s} {proposals 0s} {ad Apr 20 16:55:51 bahamaamooz dgraph[28766]: W0420 16:55:51.615249 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 245ms. Breakdown: [{sync 244ms} {disk 0s} {proposals 0s} {ad Apr 20 16:55:52 bahamaamooz dgraph[28766]: W0420 16:55:52.006268 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 298ms. Breakdown: [{sync 298ms} {disk 0s} {proposals 0s} {ad Apr 20 16:55:52 bahamaamooz dgraph[28766]: W0420 16:55:52.309106 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 231ms. Breakdown: [{sync 231ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:07 bahamaamooz dgraph[28766]: W0420 16:56:07.653744 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 549ms. Breakdown: [{sync 548ms} {disk 1ms} {proposals 0s} {a Apr 20 16:56:10 bahamaamooz dgraph[28766]: W0420 16:56:10.324320 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 253ms. Breakdown: [{sync 253ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:24 bahamaamooz dgraph[28766]: W0420 16:56:24.807072 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 276ms. Breakdown: [{sync 276ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:25 bahamaamooz dgraph[28766]: W0420 16:56:25.352025 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 247ms. Breakdown: [{sync 247ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:25 bahamaamooz dgraph[28766]: W0420 16:56:25.944496 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 281ms. Breakdown: [{sync 281ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:56 bahamaamooz dgraph[28766]: W0420 16:56:56.206866 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 274ms. Breakdown: [{sync 274ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:56 bahamaamooz dgraph[28766]: W0420 16:56:56.489422 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad Apr 20 16:56:56 bahamaamooz dgraph[28766]: W0420 16:56:56.813798 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 220ms. Breakdown: [{sync 220ms} {disk 0s} {proposals 0s} {ad Apr 20 16:57:43 bahamaamooz dgraph[28766]: W0420 16:57:43.675913 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 286ms. Breakdown: [{sync 286ms} {disk 0s} {proposals 0s} {ad Apr 20 16:58:27 bahamaamooz dgraph[28766]: I0420 16:58:27.759193 28766 draft.go:403] Creating snapshot at index: 217037. ReadTs: 225250. Apr 20 16:58:34 bahamaamooz dgraph[28766]: W0420 16:58:34.350002 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.302s. Breakdown: [{sync 1.301s} {disk 0s} {proposals 0s} { Apr 20 16:58:36 bahamaamooz dgraph[28766]: W0420 16:58:36.277534 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 765ms. Breakdown: [{sync 765ms} {disk 0s} {proposals 0s} {ad Apr 20 16:58:37 bahamaamooz dgraph[28766]: W0420 16:58:37.840466 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 627ms. Breakdown: [{disk 627ms} {proposals 0s} {advance 0s}] Apr 20 16:59:22 bahamaamooz dgraph[28766]: I0420 16:59:22.675589 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 16:59:22 bahamaamooz dgraph[28766]: I0420 16:59:22.676434 28766 log.go:34] Running for level: 0 Apr 20 16:59:23 bahamaamooz dgraph[28766]: I0420 16:59:23.109463 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 432.889606ms Apr 20 16:59:23 bahamaamooz dgraph[28766]: I0420 16:59:23.109515 28766 log.go:34] Compaction for level: 0 DONE Apr 20 16:59:42 bahamaamooz dgraph[28766]: W0420 16:59:42.626777 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 202ms. Breakdown: [{sync 202ms} {disk 0s} {proposals 0s} {ad Apr 20 17:00:04 bahamaamooz dgraph[28766]: W0420 17:00:04.614443 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 339ms. Breakdown: [{sync 338ms} {disk 1ms} {proposals 0s} {a Apr 20 17:00:06 bahamaamooz dgraph[28766]: W0420 17:00:06.000347 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 313ms. Breakdown: [{sync 313ms} {disk 0s} {proposals 0s} {ad Apr 20 17:00:07 bahamaamooz dgraph[28766]: W0420 17:00:07.558387 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 273ms. Breakdown: [{sync 272ms} {disk 1ms} {proposals 0s} {a Apr 20 17:00:27 bahamaamooz dgraph[28766]: I0420 17:00:27.912439 28766 log.go:34] Rolling up Created batch of size: 110 kB in 27.74666ms. Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.312017 28766 log.go:34] Rolling up Created batch of size: 526 kB in 40.101007ms. Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.395887 28766 log.go:34] Rolling up Created batch of size: 298 kB in 54.284858ms. Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.657942 28766 log.go:34] Rolling up Created batch of size: 867 kB in 46.594905ms. Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.658346 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.8 MB, speed: 1.8 MB/sec Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.724868 28766 log.go:34] Rolling up Created batch of size: 906 kB in 66.014821ms. Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.817357 28766 log.go:34] Rolling up Created batch of size: 1.0 MB in 91.663612ms. Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.835354 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 17.724056ms. Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.433018 28766 log.go:34] Rolling up Created batch of size: 3.9 MB in 31.095653ms. Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.433771 28766 log.go:34] Rolling up Sent 32830 keys Apr 20 17:00:29 bahamaamooz dgraph[28766]: W0420 17:00:29.705151 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 255ms. Breakdown: [{sync 255ms} {disk 0s} {proposals 0s} {ad Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.705425 28766 draft.go:1102] Rolled up 32804 keys. Done Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.705448 28766 draft.go:445] List rollup at Ts 225250: OK. Apr 20 17:01:26 bahamaamooz dgraph[28766]: W0420 17:01:26.562592 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 211ms. Breakdown: [{sync 211ms} {disk 0s} {proposals 0s} {ad Apr 20 17:02:27 bahamaamooz dgraph[28766]: I0420 17:02:27.782971 28766 draft.go:403] Creating snapshot at index: 228547. ReadTs: 236646. Apr 20 17:02:30 bahamaamooz dgraph[28766]: W0420 17:02:30.878437 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 215ms. Breakdown: [{sync 182ms} {disk 33ms} {proposals 0s} { Apr 20 17:03:53 bahamaamooz dgraph[28766]: W0420 17:03:53.478850 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 556ms. Breakdown: [{sync 555ms} {proposals 1ms} {disk 0s} {a Apr 20 17:03:53 bahamaamooz dgraph[28766]: W0420 17:03:53.817084 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 313ms. Breakdown: [{sync 312ms} {disk 0s} {proposals 0s} {ad Apr 20 17:04:54 bahamaamooz dgraph[28766]: W0420 17:04:54.923402 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 524ms. Breakdown: [{sync 523ms} {disk 1ms} {proposals 0s} {a Apr 20 17:04:56 bahamaamooz dgraph[28766]: W0420 17:04:56.594668 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 208ms. Breakdown: [{sync 207ms} {disk 0s} {proposals 0s} {ad Apr 20 17:05:27 bahamaamooz dgraph[28766]: I0420 17:05:27.990930 28766 log.go:34] Rolling up Created batch of size: 105 kB in 47.529521ms. Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.368632 28766 log.go:34] Rolling up Created batch of size: 530 kB in 41.622235ms. Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.535526 28766 log.go:34] Rolling up Created batch of size: 285 kB in 68.817359ms. Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.602977 28766 log.go:34] Rolling up Created batch of size: 854 kB in 66.942751ms. Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.650452 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.8 MB, speed: 1.8 MB/sec Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.766191 28766 log.go:34] Rolling up Created batch of size: 892 kB in 29.443225ms. Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.860600 28766 log.go:34] Rolling up Created batch of size: 992 kB in 40.126009ms. Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.974255 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 8.313996ms. Apr 20 17:05:29 bahamaamooz dgraph[28766]: I0420 17:05:29.620639 28766 log.go:34] Rolling up Created batch of size: 4.0 MB in 26.099804ms. Apr 20 17:05:29 bahamaamooz dgraph[28766]: I0420 17:05:29.621298 28766 log.go:34] Rolling up Sent 31392 keys Apr 20 17:05:31 bahamaamooz dgraph[28766]: I0420 17:05:31.501380 28766 draft.go:1102] Rolled up 31366 keys. Done Apr 20 17:05:31 bahamaamooz dgraph[28766]: I0420 17:05:31.501442 28766 draft.go:445] List rollup at Ts 236646: OK. Apr 20 17:06:27 bahamaamooz dgraph[28766]: I0420 17:06:27.770930 28766 draft.go:403] Creating snapshot at index: 240831. ReadTs: 248818. Apr 20 17:06:35 bahamaamooz dgraph[28766]: W0420 17:06:35.599135 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 456ms. Breakdown: [{sync 455ms} {disk 0s} {proposals 0s} {ad Apr 20 17:06:35 bahamaamooz dgraph[28766]: W0420 17:06:35.886562 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 249ms. Breakdown: [{sync 248ms} {disk 0s} {proposals 0s} {ad Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.109557 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.109635 28766 log.go:34] Running for level: 0 Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.792852 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 683.189212ms Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.792899 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:08:10 bahamaamooz dgraph[28766]: W0420 17:08:10.256750 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 202ms. Breakdown: [{sync 200ms} {disk 2ms} {proposals 0s} {a Apr 20 17:08:10 bahamaamooz dgraph[28766]: W0420 17:08:10.974192 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 227ms. Breakdown: [{sync 226ms} {disk 1ms} {proposals 0s} {a Apr 20 17:08:28 bahamaamooz dgraph[28766]: W0420 17:08:28.993525 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 209ms. Breakdown: [{sync 209ms} {disk 0s} {proposals 0s} {ad Apr 20 17:09:28 bahamaamooz dgraph[28766]: W0420 17:09:28.260749 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 248ms. Breakdown: [{sync 247ms} {disk 1ms} {proposals 0s} {a Apr 20 17:10:27 bahamaamooz dgraph[28766]: I0420 17:10:27.799116 28766 draft.go:403] Creating snapshot at index: 252248. ReadTs: 260140. Apr 20 17:10:27 bahamaamooz dgraph[28766]: I0420 17:10:27.860104 28766 log.go:34] Rolling up Created batch of size: 113 kB in 39.875248ms. Apr 20 17:10:28 bahamaamooz dgraph[28766]: I0420 17:10:28.677161 28766 log.go:34] Rolling up Created batch of size: 561 kB in 37.342572ms. Apr 20 17:10:28 bahamaamooz dgraph[28766]: I0420 17:10:28.677518 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 674 kB, speed: 674 kB/sec Apr 20 17:10:28 bahamaamooz dgraph[28766]: I0420 17:10:28.902838 28766 log.go:34] Rolling up Created batch of size: 304 kB in 53.451758ms. Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.082979 28766 log.go:34] Rolling up Created batch of size: 929 kB in 70.922104ms. Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.281349 28766 log.go:34] Rolling up Created batch of size: 889 kB in 66.522093ms. Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.376724 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 25.575976ms. Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.432646 28766 log.go:34] Rolling up Created batch of size: 312 kB in 40.603µs. Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.503263 28766 log.go:34] Rolling up Created batch of size: 1.0 MB in 57.82951ms. Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.650458 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 8.5 MB, speed: 4.3 MB/sec Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.621580 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 52.862591ms. Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.621645 28766 log.go:34] Rolling up Sent 33540 keys Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.696088 28766 draft.go:1102] Rolled up 33514 keys. Done Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.696132 28766 draft.go:445] List rollup at Ts 248818: OK. Apr 20 17:11:23 bahamaamooz dgraph[28766]: W0420 17:11:23.435541 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 318ms. Breakdown: [{sync 318ms} {disk 0s} {proposals 0s} {ad Apr 20 17:12:03 bahamaamooz dgraph[28766]: W0420 17:12:03.934698 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 225ms. Breakdown: [{sync 224ms} {disk 1ms} {proposals 0s} {a Apr 20 17:12:41 bahamaamooz dgraph[28766]: W0420 17:12:41.040913 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 248ms. Breakdown: [{sync 248ms} {disk 0s} {proposals 0s} {ad Apr 20 17:14:27 bahamaamooz dgraph[28766]: I0420 17:14:27.794983 28766 draft.go:403] Creating snapshot at index: 263009. ReadTs: 270790. Apr 20 17:15:03 bahamaamooz dgraph[28766]: W0420 17:15:03.652858 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 785ms. Breakdown: [{sync 784ms} {disk 1ms} {proposals 0s} {a Apr 20 17:15:25 bahamaamooz dgraph[28766]: W0420 17:15:25.453327 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 328ms. Breakdown: [{sync 328ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:26 bahamaamooz dgraph[28766]: W0420 17:15:26.038692 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 268ms. Breakdown: [{sync 267ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:26 bahamaamooz dgraph[28766]: W0420 17:15:26.553953 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 207ms. Breakdown: [{sync 207ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:27 bahamaamooz dgraph[28766]: W0420 17:15:27.168443 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 326ms. Breakdown: [{sync 326ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:27 bahamaamooz dgraph[28766]: I0420 17:15:27.895821 28766 log.go:34] Rolling up Created batch of size: 203 kB in 54.381181ms. Apr 20 17:15:28 bahamaamooz dgraph[28766]: I0420 17:15:28.480769 28766 log.go:34] Rolling up Created batch of size: 790 kB in 88.479115ms. Apr 20 17:15:28 bahamaamooz dgraph[28766]: I0420 17:15:28.654812 28766 log.go:34] Rolling up Created batch of size: 549 kB in 105.288689ms. Apr 20 17:15:28 bahamaamooz dgraph[28766]: I0420 17:15:28.654853 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.5 MB, speed: 1.5 MB/sec Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.070099 28766 log.go:34] Rolling up Created batch of size: 1.3 MB in 136.848507ms. Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.248852 28766 log.go:34] Rolling up Created batch of size: 6.0 MB in 177.10802ms. Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.314059 28766 log.go:34] Rolling up Created batch of size: 2.0 MB in 63.865349ms. Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.650261 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 11 MB, speed: 5.4 MB/sec Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.769228 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 68.021461ms. Apr 20 17:15:30 bahamaamooz dgraph[28766]: I0420 17:15:30.135327 28766 log.go:34] Rolling up Created batch of size: 594 kB in 42.739573ms. Apr 20 17:15:30 bahamaamooz dgraph[28766]: I0420 17:15:30.136304 28766 log.go:34] Rolling up Sent 59654 keys Apr 20 17:15:33 bahamaamooz dgraph[28766]: I0420 17:15:33.831545 28766 draft.go:1102] Rolled up 59628 keys. Done Apr 20 17:15:33 bahamaamooz dgraph[28766]: I0420 17:15:33.831595 28766 draft.go:445] List rollup at Ts 270790: OK. Apr 20 17:15:35 bahamaamooz dgraph[28766]: I0420 17:15:35.109541 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:15:35 bahamaamooz dgraph[28766]: I0420 17:15:35.109615 28766 log.go:34] Running for level: 0 Apr 20 17:15:38 bahamaamooz dgraph[28766]: W0420 17:15:38.357770 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 590ms. Breakdown: [{sync 562ms} {disk 28ms} {proposals 0s} { Apr 20 17:15:38 bahamaamooz dgraph[28766]: I0420 17:15:38.483341 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 3.373697571s Apr 20 17:15:38 bahamaamooz dgraph[28766]: I0420 17:15:38.483443 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:15:40 bahamaamooz dgraph[28766]: W0420 17:15:40.722243 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.59s. Breakdown: [{sync 1.59s} {disk 0s} {proposals 0s} {ad Apr 20 17:15:44 bahamaamooz dgraph[28766]: W0420 17:15:44.948660 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 3.626s. Breakdown: [{sync 3.625s} {disk 0s} {proposals 0s} { Apr 20 17:15:49 bahamaamooz dgraph[28766]: W0420 17:15:49.905504 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 247ms. Breakdown: [{disk 228ms} {sync 18ms} {proposals 0s} { Apr 20 17:15:50 bahamaamooz dgraph[28766]: W0420 17:15:50.817021 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 325ms. Breakdown: [{sync 325ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:51 bahamaamooz dgraph[28766]: W0420 17:15:51.192417 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 219ms. Breakdown: [{sync 219ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:52 bahamaamooz dgraph[28766]: W0420 17:15:52.904968 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad Apr 20 17:15:57 bahamaamooz dgraph[28766]: W0420 17:15:57.296240 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 247ms. Breakdown: [{sync 247ms} {disk 0s} {proposals 0s} {ad Apr 20 17:16:00 bahamaamooz dgraph[28766]: W0420 17:16:00.381517 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad Apr 20 17:16:00 bahamaamooz dgraph[28766]: W0420 17:16:00.674039 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 262ms. Breakdown: [{sync 261ms} {disk 0s} {proposals 0s} {ad Apr 20 17:16:02 bahamaamooz dgraph[28766]: W0420 17:16:02.064656 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.241s. Breakdown: [{sync 1.24s} {disk 1ms} {proposals 0s} { Apr 20 17:16:08 bahamaamooz dgraph[28766]: W0420 17:16:08.220373 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 220ms. Breakdown: [{sync 219ms} {disk 1ms} {proposals 0s} {a Apr 20 17:18:10 bahamaamooz dgraph[28766]: W0420 17:18:10.242891 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 214ms. Breakdown: [{sync 214ms} {disk 0s} {proposals 0s} {ad Apr 20 17:20:02 bahamaamooz dgraph[28766]: W0420 17:20:02.578438 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 301ms. Breakdown: [{sync 300ms} {advance 1ms} {disk 0s} {pro Apr 20 17:20:09 bahamaamooz dgraph[28766]: W0420 17:20:09.110351 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 252ms. Breakdown: [{sync 252ms} {disk 0s} {proposals 0s} {ad Apr 20 17:20:27 bahamaamooz dgraph[28766]: I0420 17:20:27.757012 28766 draft.go:403] Creating snapshot at index: 274610. ReadTs: 282290. Apr 20 17:20:41 bahamaamooz dgraph[28766]: W0420 17:20:41.352767 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 226ms. Breakdown: [{sync 222ms} {disk 4ms} {proposals 0s} {a Apr 20 17:20:42 bahamaamooz dgraph[28766]: W0420 17:20:42.003781 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 271ms. Breakdown: [{sync 271ms} {disk 0s} {proposals 0s} {ad Apr 20 17:21:25 bahamaamooz dgraph[28766]: W0420 17:21:25.071562 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 203ms. Breakdown: [{sync 202ms} {disk 1ms} {proposals 0s} {a Apr 20 17:24:27 bahamaamooz dgraph[28766]: I0420 17:24:27.784905 28766 draft.go:403] Creating snapshot at index: 284970. ReadTs: 292556. Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.109584 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.109754 28766 log.go:34] Running for level: 0 Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.860445 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 750.639156ms Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.860814 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:25:28 bahamaamooz dgraph[28766]: I0420 17:25:28.196467 28766 log.go:34] Rolling up Created batch of size: 201 kB in 38.274057ms. Apr 20 17:25:28 bahamaamooz dgraph[28766]: I0420 17:25:28.708029 28766 log.go:34] Rolling up Created batch of size: 812 kB in 63.863736ms. Apr 20 17:25:28 bahamaamooz dgraph[28766]: I0420 17:25:28.708083 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.0 MB, speed: 1.0 MB/sec Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.001872 28766 log.go:34] Rolling up Created batch of size: 544 kB in 193.812676ms. Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.239683 28766 log.go:34] Rolling up Created batch of size: 1.3 MB in 136.271068ms. Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.367924 28766 log.go:34] Rolling up Created batch of size: 1.3 MB in 121.678506ms. Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.597316 28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 24.135486ms. Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.707355 28766 log.go:34] Rolling up Created batch of size: 1.5 MB in 107.384506ms. Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.707466 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 10 MB, speed: 5.1 MB/sec Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.826113 28766 log.go:34] Rolling up Created batch of size: 1.1 MB in 36.396µs. Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.243835 28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 147.521131ms. Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.641936 28766 log.go:34] Rolling up Created batch of size: 623 kB in 15.062856ms. Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.641980 28766 log.go:34] Rolling up Sent 59096 keys Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.659444 28766 draft.go:1102] Rolled up 59070 keys. Done Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.659493 28766 draft.go:445] List rollup at Ts 292556: OK. Apr 20 17:25:35 bahamaamooz dgraph[28766]: W0420 17:25:35.597698 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 409ms. Breakdown: [{sync 409ms} {disk 0s} {proposals 0s} {ad Apr 20 17:27:27 bahamaamooz dgraph[28766]: I0420 17:27:27.768234 28766 draft.go:403] Creating snapshot at index: 295286. ReadTs: 302778. Apr 20 17:29:27 bahamaamooz dgraph[28766]: W0420 17:29:27.781013 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 377ms. Breakdown: [{sync 376ms} {disk 0s} {proposals 0s} {ad Apr 20 17:29:45 bahamaamooz dgraph[28766]: W0420 17:29:45.131899 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 452ms. Breakdown: [{sync 452ms} {disk 0s} {proposals 0s} {ad Apr 20 17:30:10 bahamaamooz dgraph[28766]: W0420 17:30:10.155710 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 239ms. Breakdown: [{sync 239ms} {disk 0s} {proposals 0s} {ad Apr 20 17:30:27 bahamaamooz dgraph[28766]: I0420 17:30:27.956105 28766 log.go:34] Rolling up Created batch of size: 95 kB in 15.298369ms. Apr 20 17:30:28 bahamaamooz dgraph[28766]: I0420 17:30:28.581262 28766 log.go:34] Rolling up Created batch of size: 589 kB in 68.546857ms. Apr 20 17:30:28 bahamaamooz dgraph[28766]: I0420 17:30:28.650365 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 683 kB, speed: 683 kB/sec Apr 20 17:30:28 bahamaamooz dgraph[28766]: I0420 17:30:28.972709 28766 log.go:34] Rolling up Created batch of size: 256 kB in 75.623184ms. Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.218469 28766 log.go:34] Rolling up Created batch of size: 847 kB in 115.691039ms. Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.404551 28766 log.go:34] Rolling up Created batch of size: 970 kB in 107.183733ms. Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.468659 28766 log.go:34] Rolling up Created batch of size: 882 kB in 63.274622ms. Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.576570 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 35.345788ms. Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.650639 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 7.9 MB, speed: 4.0 MB/sec Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.691596 28766 log.go:34] Rolling up Created batch of size: 1.1 MB in 52.47µs. Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.244371 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 41.508643ms. Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.543926 28766 log.go:34] Rolling up Created batch of size: 549 kB in 10.845674ms. Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.543964 28766 log.go:34] Rolling up Sent 28201 keys Apr 20 17:30:30 bahamaamooz dgraph[28766]: W0420 17:30:30.657917 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 410ms. Breakdown: [{sync 410ms} {disk 0s} {proposals 0s} {ad Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.863283 28766 draft.go:1102] Rolled up 28173 keys. Done Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.863333 28766 draft.go:445] List rollup at Ts 302778: OK. Apr 20 17:30:35 bahamaamooz dgraph[28766]: W0420 17:30:35.857710 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 377ms. Breakdown: [{sync 376ms} {disk 0s} {proposals 0s} {ad Apr 20 17:30:38 bahamaamooz dgraph[28766]: W0420 17:30:38.296406 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 224ms. Breakdown: [{sync 223ms} {disk 0s} {proposals 0s} {ad Apr 20 17:30:41 bahamaamooz dgraph[28766]: W0420 17:30:41.191843 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 364ms. Breakdown: [{sync 364ms} {disk 0s} {proposals 0s} {ad Apr 20 17:31:27 bahamaamooz dgraph[28766]: I0420 17:31:27.787463 28766 draft.go:403] Creating snapshot at index: 307075. ReadTs: 314474. Apr 20 17:31:54 bahamaamooz dgraph[28766]: I0420 17:31:54.427110 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:31:54 bahamaamooz dgraph[28766]: I0420 17:31:54.427261 28766 log.go:34] Running for level: 0 Apr 20 17:32:03 bahamaamooz dgraph[28766]: I0420 17:32:03.547464 28766 log.go:34] LOG Compact 0->1, del 102 tables, add 1 tables, took 9.120163784s Apr 20 17:32:03 bahamaamooz dgraph[28766]: I0420 17:32:03.548096 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:32:05 bahamaamooz dgraph[28766]: I0420 17:32:05.675812 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:32:05 bahamaamooz dgraph[28766]: I0420 17:32:05.676772 28766 log.go:34] Running for level: 0 Apr 20 17:32:06 bahamaamooz dgraph[28766]: I0420 17:32:06.413693 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 736.283404ms Apr 20 17:32:06 bahamaamooz dgraph[28766]: I0420 17:32:06.413737 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:32:09 bahamaamooz dgraph[28766]: W0420 17:32:09.094520 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad Apr 20 17:33:29 bahamaamooz dgraph[28766]: W0420 17:33:29.744632 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 321ms. Breakdown: [{sync 320ms} {disk 1ms} {proposals 0s} {a Apr 20 17:35:27 bahamaamooz dgraph[28766]: I0420 17:35:27.785205 28766 draft.go:403] Creating snapshot at index: 318389. ReadTs: 325674. Apr 20 17:35:28 bahamaamooz dgraph[28766]: I0420 17:35:28.652565 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 17:35:29 bahamaamooz dgraph[28766]: I0420 17:35:29.651975 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 0 B, speed: 0 B/sec Apr 20 17:35:30 bahamaamooz dgraph[28766]: I0420 17:35:30.062443 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 109.279337ms. Apr 20 17:35:30 bahamaamooz dgraph[28766]: I0420 17:35:30.652010 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.4 MB, speed: 1.5 MB/sec Apr 20 17:35:31 bahamaamooz dgraph[28766]: I0420 17:35:31.661646 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 112.126478ms. Apr 20 17:35:31 bahamaamooz dgraph[28766]: I0420 17:35:31.661723 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.6 MB, speed: 2.1 MB/sec Apr 20 17:35:32 bahamaamooz dgraph[28766]: I0420 17:35:32.011756 28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 7.012549ms. Apr 20 17:35:32 bahamaamooz dgraph[28766]: I0420 17:35:32.651930 28766 log.go:34] Rolling up Time elapsed: 05s, bytes sent: 13 MB, speed: 2.6 MB/sec Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.417130 28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 55.733975ms. Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.417172 28766 log.go:34] Rolling up Sent 32242 keys Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.456689 28766 draft.go:1102] Rolled up 32214 keys. Done Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.457124 28766 draft.go:445] List rollup at Ts 314474: OK. Apr 20 17:38:29 bahamaamooz dgraph[28766]: W0420 17:38:29.268407 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 224ms. Breakdown: [{sync 222ms} {disk 1ms} {proposals 0s} {a Apr 20 17:38:36 bahamaamooz dgraph[28766]: W0420 17:38:36.095798 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 217ms. Breakdown: [{sync 216ms} {disk 1ms} {proposals 0s} {a Apr 20 17:39:04 bahamaamooz dgraph[28766]: W0420 17:39:04.533391 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 381ms. Breakdown: [{sync 380ms} {disk 0s} {proposals 0s} {ad Apr 20 17:39:16 bahamaamooz dgraph[28766]: W0420 17:39:16.848751 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 217ms. Breakdown: [{sync 217ms} {disk 0s} {proposals 0s} {ad Apr 20 17:39:21 bahamaamooz dgraph[28766]: W0420 17:39:21.441346 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 467ms. Breakdown: [{sync 466ms} {disk 0s} {proposals 0s} {ad Apr 20 17:39:31 bahamaamooz dgraph[28766]: W0420 17:39:31.890610 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 215ms. Breakdown: [{sync 214ms} {disk 1ms} {proposals 0s} {a Apr 20 17:40:27 bahamaamooz dgraph[28766]: I0420 17:40:27.809335 28766 draft.go:403] Creating snapshot at index: 330313. ReadTs: 337496. Apr 20 17:40:28 bahamaamooz dgraph[28766]: I0420 17:40:28.650412 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 17:40:29 bahamaamooz dgraph[28766]: I0420 17:40:29.318832 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 159.928634ms. Apr 20 17:40:29 bahamaamooz dgraph[28766]: I0420 17:40:29.650277 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.4 MB, speed: 2.2 MB/sec Apr 20 17:40:30 bahamaamooz dgraph[28766]: I0420 17:40:30.605924 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 124.241092ms. Apr 20 17:40:30 bahamaamooz dgraph[28766]: I0420 17:40:30.651682 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 8.6 MB, speed: 2.9 MB/sec Apr 20 17:40:31 bahamaamooz dgraph[28766]: I0420 17:40:31.057692 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 8.31469ms. Apr 20 17:40:31 bahamaamooz dgraph[28766]: I0420 17:40:31.650249 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.2 MB/sec Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.181384 28766 log.go:34] Rolling up Created batch of size: 1.5 MB in 58.13928ms. Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.181595 28766 log.go:34] Rolling up Sent 30868 keys Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.222282 28766 draft.go:1102] Rolled up 30838 keys. Done Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.222766 28766 draft.go:445] List rollup at Ts 325674: OK. Apr 20 17:41:07 bahamaamooz dgraph[28766]: I0420 17:41:07.675484 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:41:07 bahamaamooz dgraph[28766]: I0420 17:41:07.676361 28766 log.go:34] Running for level: 0 Apr 20 17:41:08 bahamaamooz dgraph[28766]: I0420 17:41:08.783031 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.106512061s Apr 20 17:41:08 bahamaamooz dgraph[28766]: I0420 17:41:08.783081 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:41:42 bahamaamooz dgraph[28766]: W0420 17:41:42.419940 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 764ms. Breakdown: [{sync 763ms} {disk 1ms} {proposals 0s} {a Apr 20 17:43:27 bahamaamooz dgraph[28766]: I0420 17:43:27.767561 28766 draft.go:403] Creating snapshot at index: 341286. ReadTs: 348376. Apr 20 17:45:28 bahamaamooz dgraph[28766]: I0420 17:45:28.650439 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 17:45:29 bahamaamooz dgraph[28766]: I0420 17:45:29.045354 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 282.454936ms. Apr 20 17:45:29 bahamaamooz dgraph[28766]: I0420 17:45:29.650479 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.3 MB, speed: 2.1 MB/sec Apr 20 17:45:30 bahamaamooz dgraph[28766]: I0420 17:45:30.081884 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 140.864727ms. Apr 20 17:45:30 bahamaamooz dgraph[28766]: I0420 17:45:30.637112 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 72.212754ms. Apr 20 17:45:30 bahamaamooz dgraph[28766]: I0420 17:45:30.650702 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 13 MB, speed: 4.3 MB/sec Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.675493 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 120.445796ms. Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.675547 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 17 MB, speed: 4.2 MB/sec Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.683566 28766 log.go:34] Rolling up Created batch of size: 309 kB in 7.832957ms. Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.683602 28766 log.go:34] Rolling up Sent 61620 keys Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.805878 28766 draft.go:1102] Rolled up 61588 keys. Done Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.806360 28766 draft.go:445] List rollup at Ts 348376: OK. Apr 20 17:47:27 bahamaamooz dgraph[28766]: I0420 17:47:27.774453 28766 draft.go:403] Creating snapshot at index: 352494. ReadTs: 359462. Apr 20 17:48:30 bahamaamooz dgraph[28766]: I0420 17:48:30.682441 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:48:30 bahamaamooz dgraph[28766]: I0420 17:48:30.683396 28766 log.go:34] Running for level: 0 Apr 20 17:48:31 bahamaamooz dgraph[28766]: I0420 17:48:31.347651 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 664.033182ms Apr 20 17:48:31 bahamaamooz dgraph[28766]: I0420 17:48:31.347765 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:49:55 bahamaamooz dgraph[28766]: W0420 17:49:55.426192 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 432ms. Breakdown: [{sync 432ms} {disk 1ms} {proposals 0s} {a Apr 20 17:50:28 bahamaamooz dgraph[28766]: I0420 17:50:28.650430 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 17:50:29 bahamaamooz dgraph[28766]: I0420 17:50:29.437822 28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 184.569795ms. Apr 20 17:50:29 bahamaamooz dgraph[28766]: I0420 17:50:29.650874 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.5 MB, speed: 2.3 MB/sec Apr 20 17:50:30 bahamaamooz dgraph[28766]: I0420 17:50:30.650455 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.5 MB, speed: 1.5 MB/sec Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.019603 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 137.740531ms. Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.354189 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 2.44292ms. Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.650456 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.986989 28766 log.go:34] Rolling up Created batch of size: 1.1 MB in 53.488444ms. Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.987063 28766 log.go:34] Rolling up Sent 30565 keys Apr 20 17:50:32 bahamaamooz dgraph[28766]: I0420 17:50:32.000591 28766 draft.go:1102] Rolled up 30531 keys. Done Apr 20 17:50:32 bahamaamooz dgraph[28766]: I0420 17:50:32.000699 28766 draft.go:445] List rollup at Ts 359462: OK. Apr 20 17:51:10 bahamaamooz dgraph[28766]: W0420 17:51:10.574433 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 214ms. Breakdown: [{sync 213ms} {disk 0s} {proposals 0s} {ad Apr 20 17:51:16 bahamaamooz dgraph[28766]: W0420 17:51:16.699467 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 303ms. Breakdown: [{sync 303ms} {disk 0s} {proposals 0s} {ad Apr 20 17:51:18 bahamaamooz dgraph[28766]: W0420 17:51:18.550492 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 335ms. Breakdown: [{sync 335ms} {disk 0s} {proposals 0s} {ad Apr 20 17:51:27 bahamaamooz dgraph[28766]: I0420 17:51:27.758368 28766 draft.go:403] Creating snapshot at index: 364846. ReadTs: 371710. Apr 20 17:51:37 bahamaamooz dgraph[28766]: W0420 17:51:37.734522 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 554ms. Breakdown: [{sync 554ms} {disk 0s} {proposals 0s} {ad Apr 20 17:51:38 bahamaamooz dgraph[28766]: W0420 17:51:38.728562 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 345ms. Breakdown: [{sync 345ms} {disk 0s} {proposals 0s} {ad Apr 20 17:51:52 bahamaamooz dgraph[28766]: W0420 17:51:52.350879 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 614ms. Breakdown: [{sync 613ms} {disk 1ms} {proposals 0s} {a Apr 20 17:51:53 bahamaamooz dgraph[28766]: W0420 17:51:53.284614 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 604ms. Breakdown: [{sync 604ms} {disk 0s} {proposals 0s} {ad Apr 20 17:52:56 bahamaamooz dgraph[28766]: W0420 17:52:56.980791 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 210ms. Breakdown: [{sync 210ms} {disk 0s} {proposals 0s} {ad Apr 20 17:53:11 bahamaamooz dgraph[28766]: W0420 17:53:11.656884 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 217ms. Breakdown: [{sync 216ms} {disk 0s} {proposals 0s} {ad Apr 20 17:54:50 bahamaamooz dgraph[28766]: W0420 17:54:50.632289 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 240ms. Breakdown: [{sync 240ms} {disk 1ms} {proposals 0s} {a Apr 20 17:54:57 bahamaamooz dgraph[28766]: W0420 17:54:57.595442 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 375ms. Breakdown: [{sync 375ms} {disk 0s} {proposals 0s} {ad Apr 20 17:55:27 bahamaamooz dgraph[28766]: I0420 17:55:27.863959 28766 draft.go:403] Creating snapshot at index: 375428. ReadTs: 382208. Apr 20 17:55:28 bahamaamooz dgraph[28766]: I0420 17:55:28.650372 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 17:55:29 bahamaamooz dgraph[28766]: I0420 17:55:29.415184 28766 log.go:34] Rolling up Created batch of size: 4.7 MB in 147.820275ms. Apr 20 17:55:29 bahamaamooz dgraph[28766]: I0420 17:55:29.651032 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.7 MB, speed: 2.3 MB/sec Apr 20 17:55:30 bahamaamooz dgraph[28766]: I0420 17:55:30.596777 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 108.848499ms. Apr 20 17:55:30 bahamaamooz dgraph[28766]: I0420 17:55:30.650416 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 8.9 MB, speed: 3.0 MB/sec Apr 20 17:55:30 bahamaamooz dgraph[28766]: I0420 17:55:30.954348 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 848.458µs. Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.650611 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.745629 28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 45.890597ms. Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.746086 28766 log.go:34] Rolling up Sent 33740 keys Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.907607 28766 draft.go:1102] Rolled up 33706 keys. Done Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.907656 28766 draft.go:445] List rollup at Ts 371710: OK. Apr 20 17:55:37 bahamaamooz dgraph[28766]: W0420 17:55:37.476764 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 225ms. Breakdown: [{sync 225ms} {disk 0s} {proposals 0s} {ad Apr 20 17:56:27 bahamaamooz dgraph[28766]: I0420 17:56:27.109518 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 17:56:27 bahamaamooz dgraph[28766]: I0420 17:56:27.109590 28766 log.go:34] Running for level: 0 Apr 20 17:56:28 bahamaamooz dgraph[28766]: I0420 17:56:28.020378 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 910.757046ms Apr 20 17:56:28 bahamaamooz dgraph[28766]: I0420 17:56:28.020428 28766 log.go:34] Compaction for level: 0 DONE Apr 20 17:59:27 bahamaamooz dgraph[28766]: I0420 17:59:27.776412 28766 draft.go:403] Creating snapshot at index: 389306. ReadTs: 395974. Apr 20 18:00:28 bahamaamooz dgraph[28766]: I0420 18:00:28.650469 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:00:29 bahamaamooz dgraph[28766]: I0420 18:00:29.612647 28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 223.948853ms. Apr 20 18:00:29 bahamaamooz dgraph[28766]: I0420 18:00:29.650368 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.5 MB, speed: 2.3 MB/sec Apr 20 18:00:30 bahamaamooz dgraph[28766]: I0420 18:00:30.650483 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.5 MB, speed: 1.5 MB/sec Apr 20 18:00:30 bahamaamooz dgraph[28766]: I0420 18:00:30.968780 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 219.050328ms. Apr 20 18:00:31 bahamaamooz dgraph[28766]: I0420 18:00:31.477868 28766 log.go:34] Rolling up Created batch of size: 4.6 MB in 37.609314ms. Apr 20 18:00:31 bahamaamooz dgraph[28766]: I0420 18:00:31.650406 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 14 MB, speed: 3.4 MB/sec Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.459630 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 128.021876ms. Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.499964 28766 log.go:34] Rolling up Created batch of size: 329 kB in 14.666922ms. Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.500007 28766 log.go:34] Rolling up Sent 65774 keys Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.556360 28766 draft.go:1102] Rolled up 65740 keys. Done Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.556940 28766 draft.go:445] List rollup at Ts 395974: OK. Apr 20 18:02:01 bahamaamooz dgraph[28766]: W0420 18:02:01.454977 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 374ms. Breakdown: [{sync 374ms} {disk 0s} {proposals 0s} {ad Apr 20 18:02:25 bahamaamooz dgraph[28766]: W0420 18:02:25.396453 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 204ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad Apr 20 18:03:02 bahamaamooz dgraph[28766]: W0420 18:03:02.288886 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 338ms. Breakdown: [{sync 337ms} {disk 0s} {proposals 0s} {ad Apr 20 18:03:03 bahamaamooz dgraph[28766]: W0420 18:03:03.001275 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 554ms. Breakdown: [{sync 553ms} {disk 1ms} {proposals 0s} {a Apr 20 18:03:07 bahamaamooz dgraph[28766]: W0420 18:03:07.618005 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 327ms. Breakdown: [{sync 327ms} {disk 0s} {proposals 0s} {ad Apr 20 18:03:27 bahamaamooz dgraph[28766]: I0420 18:03:27.727312 28766 draft.go:403] Creating snapshot at index: 399401. ReadTs: 405990. Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.109524 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.109614 28766 log.go:34] Running for level: 0 Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.794071 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 684.427226ms Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.794126 28766 log.go:34] Compaction for level: 0 DONE Apr 20 18:05:28 bahamaamooz dgraph[28766]: I0420 18:05:28.653014 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:05:29 bahamaamooz dgraph[28766]: I0420 18:05:29.419442 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 122.518883ms. Apr 20 18:05:29 bahamaamooz dgraph[28766]: I0420 18:05:29.656640 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.3 MB, speed: 2.1 MB/sec Apr 20 18:05:30 bahamaamooz dgraph[28766]: I0420 18:05:30.653035 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.3 MB, speed: 1.4 MB/sec Apr 20 18:05:31 bahamaamooz dgraph[28766]: I0420 18:05:31.177333 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 109.623607ms. Apr 20 18:05:31 bahamaamooz dgraph[28766]: I0420 18:05:31.624712 28766 log.go:34] Rolling up Created batch of size: 4.6 MB in 9.0166ms. Apr 20 18:05:31 bahamaamooz dgraph[28766]: I0420 18:05:31.653053 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.563344 28766 log.go:34] Rolling up Created batch of size: 1.6 MB in 52.749284ms. Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.564131 28766 log.go:34] Rolling up Sent 27605 keys Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.583824 28766 draft.go:1102] Rolled up 27571 keys. Done Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.583906 28766 draft.go:445] List rollup at Ts 405990: OK. Apr 20 18:06:27 bahamaamooz dgraph[28766]: I0420 18:06:27.792530 28766 draft.go:403] Creating snapshot at index: 410167. ReadTs: 416672. Apr 20 18:07:51 bahamaamooz dgraph[28766]: W0420 18:07:51.573044 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 209ms. Breakdown: [{sync 208ms} {disk 0s} {proposals 0s} {ad Apr 20 18:09:14 bahamaamooz dgraph[28766]: W0420 18:09:14.727818 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 371ms. Breakdown: [{sync 371ms} {disk 0s} {proposals 0s} {ad Apr 20 18:09:44 bahamaamooz dgraph[28766]: W0420 18:09:44.347542 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 393ms. Breakdown: [{sync 393ms} {disk 0s} {proposals 0s} {ad Apr 20 18:10:09 bahamaamooz dgraph[28766]: I0420 18:10:09.675581 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 18:10:09 bahamaamooz dgraph[28766]: I0420 18:10:09.676369 28766 log.go:34] Running for level: 0 Apr 20 18:10:10 bahamaamooz dgraph[28766]: I0420 18:10:10.693406 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.016823623s Apr 20 18:10:10 bahamaamooz dgraph[28766]: I0420 18:10:10.693586 28766 log.go:34] Compaction for level: 0 DONE Apr 20 18:10:27 bahamaamooz dgraph[28766]: I0420 18:10:27.815062 28766 draft.go:403] Creating snapshot at index: 424051. ReadTs: 430432. Apr 20 18:10:28 bahamaamooz dgraph[28766]: I0420 18:10:28.650160 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:10:29 bahamaamooz dgraph[28766]: I0420 18:10:29.546656 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 116.356416ms. Apr 20 18:10:29 bahamaamooz dgraph[28766]: I0420 18:10:29.650221 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.4 MB, speed: 2.2 MB/sec Apr 20 18:10:30 bahamaamooz dgraph[28766]: I0420 18:10:30.650186 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.4 MB, speed: 1.5 MB/sec Apr 20 18:10:31 bahamaamooz dgraph[28766]: I0420 18:10:31.237963 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 97.218639ms. Apr 20 18:10:31 bahamaamooz dgraph[28766]: I0420 18:10:31.511487 28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 4.664048ms. Apr 20 18:10:31 bahamaamooz dgraph[28766]: I0420 18:10:31.650164 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.467521 28766 log.go:34] Rolling up Created batch of size: 2.2 MB in 62.428891ms. Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.467597 28766 log.go:34] Rolling up Sent 29462 keys Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.709805 28766 draft.go:1102] Rolled up 29422 keys. Done Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.709890 28766 draft.go:445] List rollup at Ts 416672: OK. Apr 20 18:10:33 bahamaamooz dgraph[28766]: W0420 18:10:33.868649 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.527s. Breakdown: [{disk 1.527s} {advance 1ms} {proposals 0 Apr 20 18:10:36 bahamaamooz dgraph[28766]: I0420 18:10:36.053683 28766 log.go:34] 1 [logterm: 2, index: 424185, vote: 1] cast MsgPreVote for 2 [logterm: 2, index: 424185] at term 2 Apr 20 18:10:37 bahamaamooz dgraph[28766]: I0420 18:10:37.047095 28766 log.go:34] 1 [logterm: 2, index: 424185, vote: 1] cast MsgPreVote for 3 [logterm: 2, index: 424185] at term 2 Apr 20 18:10:37 bahamaamooz dgraph[28766]: W0420 18:10:37.292070 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 3.404s. Breakdown: [{sync 3.402s} {disk 1ms} {proposals 0s} Apr 20 18:11:04 bahamaamooz dgraph[28766]: W0420 18:11:04.180522 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 692ms. Breakdown: [{sync 692ms} {disk 0s} {proposals 0s} {ad Apr 20 18:11:34 bahamaamooz dgraph[28766]: W0420 18:11:34.772951 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 255ms. Breakdown: [{sync 255ms} {disk 0s} {proposals 0s} {ad Apr 20 18:11:35 bahamaamooz dgraph[28766]: W0420 18:11:35.627772 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 234ms. Breakdown: [{sync 234ms} {disk 0s} {proposals 0s} {ad Apr 20 18:11:36 bahamaamooz dgraph[28766]: W0420 18:11:35.953056 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 296ms. Breakdown: [{sync 295ms} {disk 0s} {proposals 0s} {ad Apr 20 18:12:41 bahamaamooz dgraph[28766]: W0420 18:12:41.595576 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 326ms. Breakdown: [{sync 325ms} {disk 1ms} {proposals 0s} {a Apr 20 18:12:46 bahamaamooz dgraph[28766]: W0420 18:12:46.524123 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 501ms. Breakdown: [{sync 500ms} {disk 0s} {proposals 0s} {ad Apr 20 18:12:47 bahamaamooz dgraph[28766]: W0420 18:12:47.428564 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 446ms. Breakdown: [{sync 445ms} {disk 0s} {proposals 0s} {ad Apr 20 18:13:11 bahamaamooz dgraph[28766]: W0420 18:13:11.139660 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 345ms. Breakdown: [{sync 343ms} {disk 1ms} {advance 1ms} {pr Apr 20 18:13:31 bahamaamooz dgraph[28766]: W0420 18:13:31.942161 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 283ms. Breakdown: [{sync 283ms} {disk 0s} {proposals 0s} {ad Apr 20 18:13:40 bahamaamooz dgraph[28766]: W0420 18:13:40.733562 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 297ms. Breakdown: [{sync 297ms} {disk 0s} {proposals 0s} {ad Apr 20 18:13:41 bahamaamooz dgraph[28766]: W0420 18:13:41.924924 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 207ms. Breakdown: [{sync 207ms} {disk 0s} {proposals 0s} {ad Apr 20 18:13:42 bahamaamooz dgraph[28766]: W0420 18:13:42.538701 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 243ms. Breakdown: [{sync 243ms} {disk 0s} {proposals 0s} {ad Apr 20 18:14:08 bahamaamooz dgraph[28766]: W0420 18:14:08.828918 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 252ms. Breakdown: [{sync 252ms} {disk 0s} {proposals 0s} {ad Apr 20 18:14:11 bahamaamooz dgraph[28766]: W0420 18:14:11.180321 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 215ms. Breakdown: [{sync 215ms} {disk 0s} {proposals 0s} {ad Apr 20 18:15:27 bahamaamooz dgraph[28766]: I0420 18:15:27.767331 28766 draft.go:403] Creating snapshot at index: 434521. ReadTs: 440824. Apr 20 18:15:28 bahamaamooz dgraph[28766]: I0420 18:15:28.650629 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:15:29 bahamaamooz dgraph[28766]: I0420 18:15:29.654595 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 162.221311ms. Apr 20 18:15:29 bahamaamooz dgraph[28766]: I0420 18:15:29.654635 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.3 MB, speed: 2.1 MB/sec Apr 20 18:15:30 bahamaamooz dgraph[28766]: I0420 18:15:30.650663 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.3 MB, speed: 1.4 MB/sec Apr 20 18:15:31 bahamaamooz dgraph[28766]: I0420 18:15:31.708641 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 300.451718ms. Apr 20 18:15:31 bahamaamooz dgraph[28766]: I0420 18:15:31.708699 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.5 MB, speed: 2.1 MB/sec Apr 20 18:15:32 bahamaamooz dgraph[28766]: I0420 18:15:32.182638 28766 log.go:34] Rolling up Created batch of size: 4.7 MB in 9.559719ms. Apr 20 18:15:32 bahamaamooz dgraph[28766]: I0420 18:15:32.650591 28766 log.go:34] Rolling up Time elapsed: 05s, bytes sent: 13 MB, speed: 2.6 MB/sec Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.442282 28766 log.go:34] Rolling up Created batch of size: 1.9 MB in 74.468065ms. Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.442357 28766 log.go:34] Rolling up Sent 37804 keys Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.475653 28766 draft.go:1102] Rolled up 37748 keys. Done Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.475745 28766 draft.go:445] List rollup at Ts 430432: OK. Apr 20 18:17:37 bahamaamooz dgraph[28766]: W0420 18:17:37.018223 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 436ms. Breakdown: [{sync 435ms} {disk 0s} {proposals 0s} {ad Apr 20 18:18:09 bahamaamooz dgraph[28766]: I0420 18:18:09.675547 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 18:18:09 bahamaamooz dgraph[28766]: I0420 18:18:09.675631 28766 log.go:34] Running for level: 0 Apr 20 18:18:10 bahamaamooz dgraph[28766]: I0420 18:18:10.812278 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.136617977s Apr 20 18:18:10 bahamaamooz dgraph[28766]: I0420 18:18:10.812464 28766 log.go:34] Compaction for level: 0 DONE Apr 20 18:18:27 bahamaamooz dgraph[28766]: I0420 18:18:27.793332 28766 draft.go:403] Creating snapshot at index: 445840. ReadTs: 452048. Apr 20 18:18:48 bahamaamooz dgraph[28766]: W0420 18:18:48.796193 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 272ms. Breakdown: [{sync 271ms} {disk 1ms} {proposals 0s} {a Apr 20 18:20:28 bahamaamooz dgraph[28766]: I0420 18:20:28.650235 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:20:29 bahamaamooz dgraph[28766]: I0420 18:20:29.355823 28766 log.go:34] Rolling up Created batch of size: 4.6 MB in 202.849543ms. Apr 20 18:20:29 bahamaamooz dgraph[28766]: I0420 18:20:29.650247 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.6 MB, speed: 2.3 MB/sec Apr 20 18:20:30 bahamaamooz dgraph[28766]: I0420 18:20:30.651730 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.6 MB, speed: 1.5 MB/sec Apr 20 18:20:31 bahamaamooz dgraph[28766]: I0420 18:20:31.253274 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 200.436568ms. Apr 20 18:20:31 bahamaamooz dgraph[28766]: I0420 18:20:31.661658 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.9 MB, speed: 2.2 MB/sec Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.676519 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 91.651172ms. Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.677558 28766 log.go:34] Rolling up Time elapsed: 05s, bytes sent: 13 MB, speed: 2.6 MB/sec Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.782711 28766 log.go:34] Rolling up Created batch of size: 200 kB in 21.368785ms. Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.782745 28766 log.go:34] Rolling up Sent 58750 keys Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.789529 28766 draft.go:1102] Rolled up 58686 keys. Done Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.789978 28766 draft.go:445] List rollup at Ts 452048: OK. Apr 20 18:21:27 bahamaamooz dgraph[28766]: I0420 18:21:27.777269 28766 draft.go:403] Creating snapshot at index: 456975. ReadTs: 463090. Apr 20 18:21:59 bahamaamooz dgraph[28766]: W0420 18:21:59.664278 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 370ms. Breakdown: [{sync 370ms} {disk 0s} {proposals 0s} {ad Apr 20 18:24:23 bahamaamooz dgraph[28766]: I0420 18:24:23.675513 28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]} Apr 20 18:24:23 bahamaamooz dgraph[28766]: I0420 18:24:23.675577 28766 log.go:34] Running for level: 0 Apr 20 18:24:24 bahamaamooz dgraph[28766]: I0420 18:24:24.769537 28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.093931851s Apr 20 18:24:24 bahamaamooz dgraph[28766]: I0420 18:24:24.770522 28766 log.go:34] Compaction for level: 0 DONE Apr 20 18:24:27 bahamaamooz dgraph[28766]: I0420 18:24:27.809462 28766 draft.go:403] Creating snapshot at index: 467284. ReadTs: 473324. Apr 20 18:25:07 bahamaamooz dgraph[28766]: W0420 18:25:07.389408 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 223ms. Breakdown: [{sync 223ms} {disk 0s} {proposals 0s} {ad Apr 20 18:25:09 bahamaamooz dgraph[28766]: W0420 18:25:09.059939 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 446ms. Breakdown: [{sync 445ms} {disk 0s} {proposals 0s} {ad Apr 20 18:25:28 bahamaamooz dgraph[28766]: I0420 18:25:28.650606 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:25:29 bahamaamooz dgraph[28766]: W0420 18:25:29.639356 28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 266ms. Breakdown: [{sync 266ms} {disk 0s} {proposals 0s} {ad Apr 20 18:25:29 bahamaamooz dgraph[28766]: I0420 18:25:29.650630 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:25:30 bahamaamooz dgraph[28766]: I0420 18:25:30.067701 28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 279.379256ms. Apr 20 18:25:30 bahamaamooz dgraph[28766]: I0420 18:25:30.650923 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.3 MB, speed: 1.4 MB/sec Apr 20 18:25:31 bahamaamooz dgraph[28766]: I0420 18:25:31.621366 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 228.702956ms. Apr 20 18:25:31 bahamaamooz dgraph[28766]: I0420 18:25:31.650790 28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.6 MB, speed: 2.1 MB/sec Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.742783 28766 log.go:34] Rolling up Created batch of size: 2.7 MB in 103.084741ms. Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.742835 28766 log.go:34] Rolling up Sent 57821 keys Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.771294 28766 draft.go:1102] Rolled up 57757 keys. Done Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.771354 28766 draft.go:445] List rollup at Ts 473324: OK. Apr 20 18:28:27 bahamaamooz dgraph[28766]: I0420 18:28:27.808939 28766 draft.go:403] Creating snapshot at index: 479238. ReadTs: 485157. Apr 20 18:30:28 bahamaamooz dgraph[28766]: I0420 18:30:28.650650 28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:30:29 bahamaamooz dgraph[28766]: I0420 18:30:29.650450 28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 0 B, speed: 0 B/sec Apr 20 18:30:30 bahamaamooz dgraph[28766]: I0420 18:30:30.225280 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 164.567164ms. Apr 20 18:30:30 bahamaamooz dgraph[28766]: I0420 18:30:30.650342 28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.2 MB, speed: 1.4 MB/sec Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.432855 28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 190.359427ms. Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.443075 28766 log.go:34] Rolling up Created batch of size: 768 kB in 9.980586ms. Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.443121 28766 log.go:34] Rolling up Sent 32654 keys Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.617987 28766 draft.go:1102] Rolled up 32590 keys. Done Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.618044 28766 draft.go:445] List rollup at Ts 485157: OK. Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.393971 28766 export.go:637] Got readonly ts from Zero: 486785 Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.394007 28766 export.go:641] Requesting export for groups: [1] Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.394043 28766 export.go:389] Export requested at 486785. Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.400568 28766 export.go:395] Running export for group 1 at timestamp 486785. Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.400793 28766 export.go:416] Exporting data for group: 1 at /export/dgraph.r486785.u0420.1831/g01.rdf.gz Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.401150 28766 export.go:427] Exporting schema for group: 1 at /export/dgraph.r486785.u0420.1831/g01.schema.gz Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.402635 28766 log.go:34] Export Created batch of size: 1.9 kB in 58.253µs. Apr 20 18:31:42 bahamaamooz dgraph[28766]: I0420 18:31:42.508027 28766 log.go:34] Export Created batch of size: 4.4 MB in 393.898545ms. Apr 20 18:31:42 bahamaamooz dgraph[28766]: I0420 18:31:42.508092 28766 log.go:34] Export Time elapsed: 01s, bytes sent: 4.4 MB, speed: 4.4 MB/sec Apr 20 18:31:43 bahamaamooz dgraph[28766]: I0420 18:31:43.077085 28766 log.go:34] Export Created batch of size: 4.4 MB in 373.492215ms. Apr 20 18:31:43 bahamaamooz dgraph[28766]: I0420 18:31:43.747006 28766 log.go:34] Export Created batch of size: 4.4 MB in 355.184183ms. Apr 20 18:31:43 bahamaamooz dgraph[28766]: I0420 18:31:43.747061 28766 log.go:34] Export Time elapsed: 02s, bytes sent: 13 MB, speed: 6.6 MB/sec Apr 20 18:31:44 bahamaamooz dgraph[28766]: I0420 18:31:44.383806 28766 log.go:34] Export Created batch of size: 4.4 MB in 423.703458ms. Apr 20 18:31:44 bahamaamooz dgraph[28766]: I0420 18:31:44.402056 28766 log.go:34] Export Time elapsed: 03s, bytes sent: 18 MB, speed: 5.8 MB/sec Apr 20 18:31:44 bahamaamooz dgraph[28766]: I0420 18:31:44.961854 28766 log.go:34] Export Created batch of size: 4.4 MB in 417.63053ms. Apr 20 18:31:45 bahamaamooz dgraph[28766]: I0420 18:31:45.646537 28766 log.go:34] Export Created batch of size: 4.4 MB in 478.465043ms. Apr 20 18:31:46 bahamaamooz dgraph[28766]: I0420 18:31:46.121838 28766 log.go:34] Export Created batch of size: 4.2 MB in 472.067314ms. Apr 20 18:31:46 bahamaamooz dgraph[28766]: I0420 18:31:46.121894 28766 log.go:34] Export Time elapsed: 04s, bytes sent: 30 MB, speed: 7.6 MB/sec Apr 20 18:31:47 bahamaamooz dgraph[28766]: I0420 18:31:47.175730 28766 log.go:34] Export Created batch of size: 8.5 MB in 1.048722542s. Apr 20 18:31:49 bahamaamooz dgraph[28766]: I0420 18:31:49.047339 28766 log.go:34] Export Created batch of size: 14 MB in 1.861378622s. Apr 20 18:31:51 bahamaamooz dgraph[28766]: I0420 18:31:51.943622 28766 log.go:34] Export Created batch of size: 17 MB in 2.872388501s. Apr 20 18:31:58 bahamaamooz dgraph[28766]: I0420 18:31:58.180494 28766 log.go:34] Export Created batch of size: 22 MB in 6.208719501s. Apr 20 18:31:58 bahamaamooz dgraph[28766]: I0420 18:31:58.181366 28766 log.go:34] Export Time elapsed: 16s, bytes sent: 92 MB, speed: 5.7 MB/sec Apr 20 18:32:06 bahamaamooz dgraph[28766]: I0420 18:32:06.199603 28766 log.go:34] Export Created batch of size: 62 MB in 7.925219922s. Apr 20 18:32:23 bahamaamooz dgraph[28766]: I0420 18:32:23.975419 28766 log.go:34] Export Created batch of size: 151 MB in 17.775674235s. Apr 20 18:32:40 bahamaamooz dgraph[28766]: I0420 18:32:40.985126 28766 log.go:34] Export Created batch of size: 149 MB in 17.009504471s. Apr 20 18:32:40 bahamaamooz dgraph[28766]: I0420 18:32:40.985176 28766 log.go:34] Export Time elapsed: 59s, bytes sent: 454 MB, speed: 7.7 MB/sec Apr 20 18:32:58 bahamaamooz dgraph[28766]: I0420 18:32:58.388419 28766 log.go:34] Export Created batch of size: 149 MB in 17.403160347s. Apr 20 18:32:58 bahamaamooz dgraph[28766]: I0420 18:32:58.389464 28766 log.go:34] Export Time elapsed: 01m16s, bytes sent: 603 MB, speed: 7.9 MB/sec Apr 20 18:33:15 bahamaamooz dgraph[28766]: I0420 18:33:15.692125 28766 log.go:34] Export Created batch of size: 152 MB in 17.302413526s. Apr 20 18:33:32 bahamaamooz dgraph[28766]: I0420 18:33:32.532131 28766 log.go:34] Export Created batch of size: 148 MB in 16.823994731s. Apr 20 18:33:36 bahamaamooz dgraph[28766]: I0420 18:33:36.257165 28766 log.go:34] Export Created batch of size: 41 MB in 3.659248789s. Apr 20 18:33:36 bahamaamooz dgraph[28766]: I0420 18:33:36.257221 28766 log.go:34] Export Time elapsed: 01m54s, bytes sent: 945 MB, speed: 8.3 MB/sec Apr 20 18:33:36 bahamaamooz dgraph[28766]: I0420 18:33:36.257231 28766 log.go:34] Export Sent 2766963 keys Apr 20 18:33:38 bahamaamooz dgraph[28766]: I0420 18:33:38.347383 28766 export.go:581] Export DONE for group 1 at timestamp 486785. Apr 20 18:33:38 bahamaamooz dgraph[28766]: I0420 18:33:38.347441 28766 export.go:664] Export at readTs 486785 DONE Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.650754 28766 draft.go:1194] Found 1 old transactions. Acting to abort them. Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.668068 28766 draft.go:1155] TryAbort 1 txns with start ts. Error: Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.668117 28766 draft.go:1178] TryAbort selectively proposing only aborted txns: txns: Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.678781 28766 draft.go:1197] Done abortOldTransactions for 1 txns. Error: Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.507683 28766 export.go:637] Got readonly ts from Zero: 486787 Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.511182 28766 export.go:641] Requesting export for groups: [1] Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.511512 28766 export.go:389] Export requested at 486787. Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.518539 28766 export.go:395] Running export for group 1 at timestamp 486787. Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.518988 28766 export.go:416] Exporting data for group: 1 at /export/dgraph.r486787.u0420.2037/g01.rdf.gz Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.519572 28766 export.go:427] Exporting schema for group: 1 at /export/dgraph.r486787.u0420.2037/g01.schema.gz Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.524675 28766 log.go:34] Export Created batch of size: 1.9 kB in 103.882µs. Apr 20 20:37:59 bahamaamooz dgraph[28766]: I0420 20:37:59.469617 28766 log.go:34] Export Created batch of size: 4.4 MB in 370.946991ms. Apr 20 20:37:59 bahamaamooz dgraph[28766]: I0420 20:37:59.520637 28766 log.go:34] Export Time elapsed: 01s, bytes sent: 4.4 MB, speed: 4.4 MB/sec Apr 20 20:37:59 bahamaamooz dgraph[28766]: I0420 20:37:59.883104 28766 log.go:34] Export Created batch of size: 4.4 MB in 297.129212ms. Apr 20 20:38:00 bahamaamooz dgraph[28766]: I0420 20:38:00.480717 28766 log.go:34] Export Created batch of size: 4.4 MB in 301.697796ms. Apr 20 20:38:00 bahamaamooz dgraph[28766]: I0420 20:38:00.520785 28766 log.go:34] Export Time elapsed: 02s, bytes sent: 13 MB, speed: 6.6 MB/sec Apr 20 20:38:01 bahamaamooz dgraph[28766]: I0420 20:38:01.087791 28766 log.go:34] Export Created batch of size: 4.4 MB in 401.533857ms. Apr 20 20:38:01 bahamaamooz dgraph[28766]: I0420 20:38:01.717108 28766 log.go:34] Export Created batch of size: 4.4 MB in 454.724942ms. Apr 20 20:38:01 bahamaamooz dgraph[28766]: I0420 20:38:01.718066 28766 log.go:34] Export Time elapsed: 03s, bytes sent: 22 MB, speed: 7.3 MB/sec Apr 20 20:38:02 bahamaamooz dgraph[28766]: I0420 20:38:02.429775 28766 log.go:34] Export Created batch of size: 4.4 MB in 444.008087ms. Apr 20 20:38:03 bahamaamooz dgraph[28766]: I0420 20:38:03.276731 28766 log.go:34] Export Created batch of size: 8.5 MB in 843.888872ms. Apr 20 20:38:04 bahamaamooz dgraph[28766]: I0420 20:38:04.537976 28766 log.go:34] Export Created batch of size: 9.5 MB in 1.254975667s. Apr 20 20:38:06 bahamaamooz dgraph[28766]: I0420 20:38:06.236652 28766 log.go:34] Export Created batch of size: 13 MB in 1.683339894s. Apr 20 20:38:06 bahamaamooz dgraph[28766]: I0420 20:38:06.236715 28766 log.go:34] Export Time elapsed: 07s, bytes sent: 57 MB, speed: 8.1 MB/sec Apr 20 20:38:08 bahamaamooz dgraph[28766]: I0420 20:38:08.353316 28766 log.go:34] Export Created batch of size: 13 MB in 2.090577021s. Apr 20 20:38:08 bahamaamooz dgraph[28766]: I0420 20:38:08.353361 28766 log.go:34] Export Time elapsed: 09s, bytes sent: 70 MB, speed: 7.8 MB/sec Apr 20 20:38:13 bahamaamooz dgraph[28766]: I0420 20:38:13.832915 28766 log.go:34] Export Created batch of size: 17 MB in 5.464466068s. Apr 20 20:38:13 bahamaamooz dgraph[28766]: I0420 20:38:13.832959 28766 log.go:34] Export Time elapsed: 15s, bytes sent: 87 MB, speed: 5.8 MB/sec Apr 20 20:38:21 bahamaamooz dgraph[28766]: I0420 20:38:21.347762 28766 log.go:34] Export Created batch of size: 71 MB in 7.427539104s. Apr 20 20:38:21 bahamaamooz dgraph[28766]: I0420 20:38:21.347823 28766 log.go:34] Export Time elapsed: 22s, bytes sent: 158 MB, speed: 7.2 MB/sec Apr 20 20:38:37 bahamaamooz dgraph[28766]: I0420 20:38:37.374770 28766 log.go:34] Export Created batch of size: 151 MB in 16.026858075s. Apr 20 20:38:37 bahamaamooz dgraph[28766]: I0420 20:38:37.374853 28766 log.go:34] Export Time elapsed: 38s, bytes sent: 310 MB, speed: 8.1 MB/sec Apr 20 20:38:53 bahamaamooz dgraph[28766]: I0420 20:38:53.193042 28766 log.go:34] Export Created batch of size: 149 MB in 15.818086744s. Apr 20 20:39:08 bahamaamooz dgraph[28766]: I0420 20:39:08.394121 28766 log.go:34] Export Created batch of size: 150 MB in 15.200937645s. Apr 20 20:39:23 bahamaamooz dgraph[28766]: I0420 20:39:23.596759 28766 log.go:34] Export Created batch of size: 152 MB in 15.202505848s. Apr 20 20:39:23 bahamaamooz dgraph[28766]: I0420 20:39:23.596842 28766 log.go:34] Export Time elapsed: 01m25s, bytes sent: 760 MB, speed: 8.9 MB/sec Apr 20 20:39:38 bahamaamooz dgraph[28766]: I0420 20:39:38.889367 28766 log.go:34] Export Created batch of size: 148 MB in 15.266579314s. Apr 20 20:39:41 bahamaamooz dgraph[28766]: I0420 20:39:41.785200 28766 log.go:34] Export Created batch of size: 37 MB in 2.852601845s. Apr 20 20:39:41 bahamaamooz dgraph[28766]: I0420 20:39:41.785265 28766 log.go:34] Export Time elapsed: 01m43s, bytes sent: 945 MB, speed: 9.2 MB/sec Apr 20 20:39:41 bahamaamooz dgraph[28766]: I0420 20:39:41.785288 28766 log.go:34] Export Sent 2766963 keys Apr 20 20:39:42 bahamaamooz dgraph[28766]: I0420 20:39:42.003615 28766 export.go:581] Export DONE for group 1 at timestamp 486787. Apr 20 20:39:42 bahamaamooz dgraph[28766]: I0420 20:39:42.003723 28766 export.go:664] Export at readTs 486787 DONE lines 451-508/508 (END) ```

[minhaj-shakeel]

Github issues have been deprecated. This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.