SysTest - load_test.sh - Test21MillionDataSetSize fails 50% of the time (upsert test exits)

What version of Dgraph are you using?

master

Have you tried reproducing the issue with the latest release?

yes

What is the hardware spec (RAM, OS)?

n1-standard-8 (8 vCPUs, 30 GB memory)
100 GB SSD
Ubuntu Bionic Beaver

Steps to reproduce the issue (command/config used to run Dgraph).

./test.sh -F

The test harness fails when running:

./contrib/scripts/load-test.sh
- with second test running go test -run Test21MillionDataSetSize$ -v -manual=true

Expected behaviour and actual result.

I expected the test to always pass deterministically. However, they will have different types of results.

result 1 (50%) - test passes, but integrity check has many rpc errors with cannot retrieve UIDs from list with key
result 2 (37.5%) - FAIL. upsert test exits with rpc error with Transaction is too old
result 3 (12.5%) - FAIL. upsert test exists with rpc error with cannot retrieve UIDs from list with key

Some will be during upsert test with transaction too old or cannot retreive UIDs. These are reported as failures. The more common result is after the upsert tests, during integrity check, there are several rpc error code with message about cannot retreive UIDs from list iwth key. This was is marked as pass.

Result 1

This are marked as pass, but several rpc errors. This happens 50% of the time (sample of 8 tests)

*  Running account upsert tests
[Decoder]: Using assembly version of decoder
Doing upserts
[Jun 24 06:15:18] Success: 0 Retries: 0
[Jun 24 06:15:18] Success: 0 Retries: 0
[Jun 24 06:15:18] Success: 0 Retries: 0
[Jun 24 06:15:18] Success: 2 Retries: 1
[Jun 24 06:15:18] Success: 4 Retries: 77
[Jun 24 06:15:18] Success: 4 Retries: 78
[Jun 24 06:15:18] Success: 4 Retries: 79
[Jun 24 06:15:18] Success: 4 Retries: 80
[Jun 24 06:15:18] Success: 4 Retries: 81
[Jun 24 06:15:18] Success: 4 Retries: 82
[Jun 24 06:15:18] Success: 4 Retries: 200
[Jun 24 06:15:19] Success: 7 Retries: 207
[Jun 24 06:15:19] Success: 9 Retries: 210
[Jun 24 06:15:19] Success: 11 Retries: 257
[Jun 24 06:15:19] Success: 11 Retries: 257
[Jun 24 06:15:19] Success: 11 Retries: 258
[Jun 24 06:15:19] Success: 12 Retries: 372
[Jun 24 06:15:19] Success: 12 Retries: 379
[Jun 24 06:15:19] Success: 12 Retries: 374
[Jun 24 06:15:19] Success: 12 Retries: 375
[Jun 24 06:15:19] Success: 12 Retries: 376
[Jun 24 06:15:19] Success: 12 Retries: 377
[Jun 24 06:15:19] Success: 12 Retries: 378
[Jun 24 06:15:19] Success: 12 Retries: 373
[Jun 24 06:15:19] Success: 12 Retries: 435
[Jun 24 06:15:19] Success: 16 Retries: 495
[Jun 24 06:15:19] Success: 17 Retries: 496
[Jun 24 06:15:19] Success: 19 Retries: 498
[Jun 24 06:15:20] Success: 22 Retries: 534
[Jun 24 06:15:20] Success: 22 Retries: 535
[Jun 24 06:15:20] Success: 22 Retries: 558
[Jun 24 06:15:20] Success: 22 Retries: 579
[Jun 24 06:15:20] Success: 24 Retries: 613
[Jun 24 06:15:20] Success: 25 Retries: 667
[Jun 24 06:15:20] Success: 25 Retries: 668
[Jun 24 06:15:20] Success: 25 Retries: 703
[Jun 24 06:15:20] Success: 25 Retries: 703
[Jun 24 06:15:20] Success: 26 Retries: 763
[Jun 24 06:15:20] Success: 28 Retries: 768
[Jun 24 06:15:20] Success: 31 Retries: 798
[Jun 24 06:15:20] Success: 31 Retries: 799
[Jun 24 06:15:20] Success: 31 Retries: 845
[Jun 24 06:15:20] Success: 31 Retries: 845
[Jun 24 06:15:21] Success: 35 Retries: 851
[Jun 24 06:15:21] Success: 35 Retries: 902
[Jun 24 06:15:21] Success: 35 Retries: 902
[Jun 24 06:15:21] Success: 35 Retries: 903
[Jun 24 06:15:21] Success: 36 Retries: 946
[Jun 24 06:15:21] Success: 37 Retries: 970
[Jun 24 06:15:21] Success: 37 Retries: 1018
[Jun 24 06:15:21] Success: 43 Retries: 1057
[Jun 24 06:15:21] Success: 43 Retries: 1072
[Jun 24 06:15:21] Success: 45 Retries: 1109
[Jun 24 06:15:22] Success: 47 Retries: 1115
[Jun 24 06:15:22] Success: 47 Retries: 1147
[Jun 24 06:15:22] Success: 48 Retries: 1217
[Jun 24 06:15:22] Success: 51 Retries: 1244
[Jun 24 06:15:22] Success: 51 Retries: 1245
[Jun 24 06:15:22] Success: 51 Retries: 1278
[Jun 24 06:15:22] Success: 52 Retries: 1321
[Jun 24 06:15:22] Success: 61 Retries: 1357
[Jun 24 06:15:22] Success: 62 Retries: 1373
[Jun 24 06:15:22] Success: 62 Retries: 1374
[Jun 24 06:15:22] Success: 62 Retries: 1375
[Jun 24 06:15:23] Success: 64 Retries: 1423
[Jun 24 06:15:23] Success: 66 Retries: 1517
[Jun 24 06:15:23] Success: 67 Retries: 1577
[Jun 24 06:15:23] Success: 72 Retries: 1589
[Jun 24 06:15:23] Success: 74 Retries: 1618
[Jun 24 06:15:23] Success: 80 Retries: 1702
[Jun 24 06:15:23] Success: 80 Retries: 1777
[Jun 24 06:15:23] Success: 80 Retries: 1777
[Jun 24 06:15:23] Success: 81 Retries: 1790
[Jun 24 06:15:23] Success: 81 Retries: 1791
[Jun 24 06:15:23] Success: 81 Retries: 1792
[Jun 24 06:15:23] Success: 81 Retries: 1793
[Jun 24 06:15:23] Success: 81 Retries: 1794
[Jun 24 06:15:23] Success: 81 Retries: 1796
[Jun 24 06:15:23] Success: 81 Retries: 1796
[Jun 24 06:15:23] Success: 81 Retries: 1798
[Jun 24 06:15:23] Success: 81 Retries: 1798
[Jun 24 06:15:23] Success: 81 Retries: 1799
[Jun 24 06:15:23] Success: 81 Retries: 1800
[Jun 24 06:15:23] Success: 81 Retries: 1802
[Jun 24 06:15:23] Success: 81 Retries: 1802
[Jun 24 06:15:23] Success: 81 Retries: 1803
[Jun 24 06:15:23] Success: 81 Retries: 1804
[Jun 24 06:15:23] Success: 81 Retries: 1805
[Jun 24 06:15:23] Success: 81 Retries: 1806
[Jun 24 06:15:23] Success: 81 Retries: 1807
[Jun 24 06:15:23] Success: 84 Retries: 1827
[Jun 24 06:15:24] Success: 87 Retries: 1873
[Jun 24 06:15:24] Success: 88 Retries: 1886
[Jun 24 06:15:24] Success: 89 Retries: 1974
[Jun 24 06:15:24] Success: 89 Retries: 1975
[Jun 24 06:15:24] Success: 94 Retries: 2003
[Jun 24 06:15:24] Success: 94 Retries: 2004
[Jun 24 06:15:24] Success: 96 Retries: 2061
[Jun 24 06:15:24] Success: 100 Retries: 2085
[Jun 24 06:15:24] Success: 102 Retries: 2157
[Jun 24 06:15:24] Success: 102 Retries: 2198
[Jun 24 06:15:24] Success: 102 Retries: 2201
[Jun 24 06:15:25] Success: 107 Retries: 2221
[Jun 24 06:15:25] Success: 112 Retries: 2272
[Jun 24 06:15:25] Success: 113 Retries: 2307
[Jun 24 06:15:25] Success: 113 Retries: 2307
[Jun 24 06:15:25] Success: 114 Retries: 2356
[Jun 24 06:15:25] Success: 117 Retries: 2388
[Jun 24 06:15:25] Success: 118 Retries: 2406
[Jun 24 06:15:25] Success: 122 Retries: 2427
[Jun 24 06:15:25] Success: 125 Retries: 2444
[Jun 24 06:15:25] Success: 125 Retries: 2474
[Jun 24 06:15:26] Success: 128 Retries: 2499
[Jun 24 06:15:26] Success: 128 Retries: 2541
[Jun 24 06:15:26] Success: 130 Retries: 2582
[Jun 24 06:15:26] Success: 135 Retries: 2610
[Jun 24 06:15:26] Success: 135 Retries: 2611
[Jun 24 06:15:26] Success: 135 Retries: 2609
[Jun 24 06:15:26] Success: 136 Retries: 2646
[Jun 24 06:15:26] Success: 138 Retries: 2695
[Jun 24 06:15:26] Success: 142 Retries: 2737
[Jun 24 06:15:26] Success: 144 Retries: 2765
[Jun 24 06:15:27] Success: 145 Retries: 2789
[Jun 24 06:15:27] Success: 149 Retries: 2815
[Jun 24 06:15:27] Success: 151 Retries: 2859
[Jun 24 06:15:27] Success: 151 Retries: 2884
[Jun 24 06:15:27] Success: 154 Retries: 2905
[Jun 24 06:15:27] Success: 155 Retries: 2936
[Jun 24 06:15:27] Success: 156 Retries: 2947
[Jun 24 06:15:27] Success: 163 Retries: 2961
[Jun 24 06:15:27] Success: 163 Retries: 2962
[Jun 24 06:15:27] Success: 165 Retries: 3024
[Jun 24 06:15:28] Success: 165 Retries: 3070
[Jun 24 06:15:28] Success: 165 Retries: 3069
[Jun 24 06:15:28] Success: 174 Retries: 3093
[Jun 24 06:15:28] Success: 176 Retries: 3105
[Jun 24 06:15:28] Success: 176 Retries: 3128
[Jun 24 06:15:28] Success: 177 Retries: 3171
[Jun 24 06:15:28] Success: 178 Retries: 3213
[Jun 24 06:15:28] Success: 185 Retries: 3217
[Jun 24 06:15:28] Success: 187 Retries: 3233
[Jun 24 06:15:29] Success: 188 Retries: 3303
[Jun 24 06:15:29] Success: 190 Retries: 3322
[Jun 24 06:15:29] Success: 194 Retries: 3342
[Jun 24 06:15:29] Success: 197 Retries: 3364
[Jun 24 06:15:29] Success: 197 Retries: 3403
[Jun 24 06:15:29] Success: 200 Retries: 3422
[Jun 24 06:15:29] Success: 205 Retries: 3438
[Jun 24 06:15:29] Success: 208 Retries: 3477
[Jun 24 06:15:29] Success: 211 Retries: 3506
[Jun 24 06:15:30] Success: 215 Retries: 3523
[Jun 24 06:15:30] Success: 218 Retries: 3538
[Jun 24 06:15:30] Success: 218 Retries: 3573
[Jun 24 06:15:30] Success: 218 Retries: 3574
[Jun 24 06:15:30] Success: 225 Retries: 3586
[Jun 24 06:15:30] Success: 228 Retries: 3625
[Jun 24 06:15:30] Success: 231 Retries: 3655
[Jun 24 06:15:30] Success: 235 Retries: 3676
[Jun 24 06:15:30] Success: 238 Retries: 3705
[Jun 24 06:15:30] Success: 238 Retries: 3706
[Jun 24 06:15:30] Success: 238 Retries: 3707
[Jun 24 06:15:30] Success: 239 Retries: 3720
[Jun 24 06:15:30] Success: 248 Retries: 3748
[Jun 24 06:15:31] Success: 254 Retries: 3773
[Jun 24 06:15:31] Success: 259 Retries: 3792
[Jun 24 06:15:31] Success: 264 Retries: 3815
[Jun 24 06:15:31] Success: 270 Retries: 3841
[Jun 24 06:15:31] Success: 278 Retries: 3861
[Jun 24 06:15:31] Success: 284 Retries: 3874
[Jun 24 06:15:31] Success: 290 Retries: 3889
Checking integrity
*  Running sentence swap tests
~/go/src/github.com/dgraph-io/dgraph/contrib/integration/swap ~/go/src/github.com/dgraph-io/dgraph
[Decoder]: Using assembly version of decoder
    accessories:   1
  accommodation:   2
      according:   1
     activities:   1
     additional:   1
 administration:   1
      advertise:   1
    advertising:   1
      agreement:   1
    alternative:   4
    application:   1
   applications:   1
    appropriate:   1
     assessment:   2
     assistance:   4
     associated:   2
     associates:   8
    association:   1
      australia:   1
      authority:   2
   availability:   1
      available:   1
     background:   4
      beautiful:   2
      beginning:   8
     businesses:   4
     california:   1
     categories:   1
      character:   4
      christian:   2
      christmas:   2
     collection:   1
     commercial:   1
     commission:   1
      committee:   1
  communication:   2
 communications:   2
      community:   1
      companies:   1
      completed:   4
     components:   4
      computers:   1
      condition:   1
     conditions:   1
     conference:   1
     connection:   2
     considered:   2
   construction:   1
      copyright:   1
      corporate:   1
    corporation:   1
      countries:   1
      currently:   1
      customers:   1
     department:   1
    description:   1
      developed:   2
    development:   1
      different:   1
      directory:   1
     disclaimer:   4
     discussion:   1
   distribution:   2
      documents:   2
      downloads:   1
      education:   1
    educational:   4
      effective:   1
     electronic:   1
    electronics:   1
      employees:   4
     employment:   1
    engineering:   1
     enterprise:   2
  entertainment:   1
    environment:   1
  environmental:   1
      equipment:   1
     especially:   2
    established:   8
     evaluation:  32
     everything:   1
      excellent:   2
      executive:   1
     experience:   1
     facilities:   1
      financial:   1
      following:   1
     foundation:   2
      functions:   4
      furniture:   2
      galleries:  64
     government:   1
 implementation:   4
      important:   1
      including:   1
      increased:   8
    independent:   2
     individual:   1
     industrial:   2
    information:   1
      institute:   1
   instructions:   8
    instruments:   8
      insurance:   1
     interested:   8
    interesting:  32
      interface:   2
  international:   1
   introduction:   2
     investment:   2
      knowledge:   1
      locations:   2
    maintenance:   4
     management:   1
   manufacturer:   2
      marketing:   1
      materials:   1
     membership:   4
      microsoft:   1
     navigation:   2
      necessary:   1
     newsletter:   1
      operating:   2
      operation:   2
     operations:   2
  opportunities:   1
    opportunity:   2
   organization:   1
  organizations:   4
      otherwise:   4
     particular:   1
    performance:   1
    phentermine:   4
    photography:   4
      political:   1
     population:   2
      potential:   2
      presented: 100
      president:   1
     procedures:  16
     processing:   4
     production:   1
   professional:   1
      programme:   2
    programming:   8
     properties:   2
     protection:   1
      providing:   2
    publication:   4
   publications:   2
      published:   1
      publisher:   2
      questions:   1
    recommended:   4
      reference:   1
     references:  36
      regarding:   8
     registered:   1
   registration:   2
      relations:   4
   relationship:  16
   requirements:   1
     resolution:   4
      resources:   1
 responsibility:   4
    responsible:   2
     restaurant:  16
    restaurants:   2
      selection:   1
      september:   1
    significant:   2
      solutions:   1
      something:   1
      sometimes:  16
      specified:  16
      sponsored:   1
      standards:   1
      statement:   1
     statistics:   2
      structure:   2
      submitted:   4
      subscribe:   1
      technical:   1
   technologies:   2
     technology:   1
      telephone:   2
     television:   8
     themselves:   4
      therefore:   2
     throughout:   8
     trademarks:   4
    traditional:  16
      transport:  32
 transportation:   4
      treatment:   1
     understand:   2
  understanding:   8
     university:   1
     washington:   1
      wednesday:   2
      worldwide:   2
Success:67 Fail:19 Check:2
Success:134 Fail:43 Check:5
Success:198 Fail:64 Check:8
Success:268 Fail:85 Check:11
Success:333 Fail:107 Check:14
Success:400 Fail:128 Check:17
Success:465 Fail:148 Check:20
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e636502016576616c756174696f6e: readTs: 14976 less than minTs: 14999 for key: "\x00\x00\bsentence\x02\x01evaluation"
Success:530 Fail:171 Check:22
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e636502016173736f636961746564: readTs: 15073 less than minTs: 15091 for key: "\x00\x00\bsentence\x02\x01associated"
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e636502016173736f636961746573: readTs: 15103 less than minTs: 15116 for key: "\x00\x00\bsentence\x02\x01associates"
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e636502016173736f63696174696f6e: readTs: 15141 less than minTs: 15156 for key: "\x00\x00\bsentence\x02\x01association"
Success:595 Fail:189 Check:23
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e63650201636f6d70616e696573: readTs: 15165 less than minTs: 15180 for key: "\x00\x00\bsentence\x02\x01companies"
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e6365020174656c6570686f6e65: readTs: 15220 less than minTs: 15238 for key: "\x00\x00\bsentence\x02\x01telephone"
Error while running inv: rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 00000873656e74656e636502016c6f636174696f6e73: readTs: 15316 less than minTs: 15347 for key: "\x00\x00\bsentence\x02\x01locations"
Success:669 Fail:208 Check:24
Success:734 Fail:232 Check:26
~/go/src/github.com/dgraph-io/dgraph
*  Running mutate from #1750.
~/go/src/github.com/dgraph-io/dgraph/contrib/integration/mutates ~/go/src/github.com/dgraph-io/dgraph
Commit OK
Response JSON: "{\"me\":[{\"name\":\"ok 200\"},{\"name\":\"ok 300\"},{\"name\":\"ok 400\"}]}"
~/go/src/github.com/dgraph-io/dgraph

Result 2

This returns that the test fails. This type or result happens 37.5% of the time (sample of 8)

Total aborts 6074
Total success 907
*  Running account upsert tests
[Decoder]: Using assembly version of decoder
Doing upserts
[Jun 24 05:52:55] Success: 0 Retries: 0
[Jun 24 05:52:55] Success: 1 Retries: 1
[Jun 24 05:52:55] Success: 3 Retries: 6
[Jun 24 05:52:55] Success: 4 Retries: 80
[Jun 24 05:52:55] Success: 4 Retries: 173
[Jun 24 05:52:55] Success: 8 Retries: 183
[Jun 24 05:52:55] Success: 11 Retries: 195
[Jun 24 05:52:56] Success: 11 Retries: 212
[Jun 24 05:52:56] Success: 11 Retries: 285
[Jun 24 05:52:56] Success: 12 Retries: 451
[Jun 24 05:52:56] Success: 20 Retries: 481
[Jun 24 05:52:56] Success: 21 Retries: 488
[Jun 24 05:52:56] Success: 23 Retries: 505
[Jun 24 05:52:57] Success: 27 Retries: 557
[Jun 24 05:52:57] Success: 28 Retries: 594
[Jun 24 05:52:57] Success: 29 Retries: 630
[Jun 24 05:52:57] Success: 29 Retries: 671
[Jun 24 05:52:57] Success: 29 Retries: 672
[Jun 24 05:52:57] Success: 29 Retries: 671
[Jun 24 05:52:57] Success: 29 Retries: 715
[Jun 24 05:52:57] Success: 36 Retries: 724
[Jun 24 05:52:57] Success: 36 Retries: 760
[Jun 24 05:52:57] Success: 36 Retries: 761
[Jun 24 05:52:57] Success: 38 Retries: 792
[Jun 24 05:52:57] Success: 40 Retries: 818
[Jun 24 05:52:58] Success: 41 Retries: 821
[Jun 24 05:52:58] Success: 42 Retries: 879
[Jun 24 05:52:58] Success: 42 Retries: 930
[Jun 24 05:52:58] Success: 42 Retries: 965
[Jun 24 05:52:58] Success: 42 Retries: 965
[Jun 24 05:52:58] Success: 45 Retries: 976
[Jun 24 05:52:58] Success: 45 Retries: 977
[Jun 24 05:52:58] Success: 45 Retries: 994
[Jun 24 05:52:58] Success: 47 Retries: 1061
[Jun 24 05:52:59] Success: 52 Retries: 1076
[Jun 24 05:52:59] Success: 54 Retries: 1146
[Jun 24 05:52:59] Success: 54 Retries: 1206
[Jun 24 05:52:59] Success: 55 Retries: 1229
[Jun 24 05:52:59] Success: 59 Retries: 1234
[Jun 24 05:52:59] Success: 61 Retries: 1315
[Jun 24 05:52:59] Success: 64 Retries: 1319
[Jun 24 05:52:59] Success: 64 Retries: 1321
[Jun 24 05:52:59] Success: 65 Retries: 1370
[Jun 24 05:53:00] Success: 66 Retries: 1418
[Jun 24 05:53:00] Success: 66 Retries: 1428
[Jun 24 05:53:00] Success: 72 Retries: 1505
[Jun 24 05:53:00] Success: 74 Retries: 1548
[Jun 24 05:53:00] Success: 76 Retries: 1576
[Jun 24 05:53:00] Success: 76 Retries: 1614
[Jun 24 05:53:00] Success: 77 Retries: 1644
[Jun 24 05:53:00] Success: 82 Retries: 1667
[Jun 24 05:53:01] Success: 82 Retries: 1696
[Jun 24 05:53:01] Success: 82 Retries: 1711
[Jun 24 05:53:01] Success: 85 Retries: 1766
[Jun 24 05:53:01] Success: 88 Retries: 1786
[Jun 24 05:53:01] Success: 88 Retries: 1832
[Jun 24 05:53:01] Success: 93 Retries: 1856
[Jun 24 05:53:01] Success: 93 Retries: 1903
[Jun 24 05:53:01] Success: 93 Retries: 1918
[Jun 24 05:53:01] Success: 97 Retries: 1947
[Jun 24 05:53:01] Success: 97 Retries: 1949
[Jun 24 05:53:01] Success: 97 Retries: 1949
[Jun 24 05:53:02] Success: 98 Retries: 1973
[Jun 24 05:53:02] Success: 103 Retries: 2024
[Jun 24 05:53:02] Success: 104 Retries: 2055
[Jun 24 05:53:02] Success: 105 Retries: 2086
[Jun 24 05:53:02] Success: 106 Retries: 2108
[Jun 24 05:53:02] Success: 106 Retries: 2109
[Jun 24 05:53:02] Success: 106 Retries: 2124
[Jun 24 05:53:02] Success: 111 Retries: 2141
[Jun 24 05:53:02] Success: 112 Retries: 2169
[Jun 24 05:53:02] Success: 112 Retries: 2212
[Jun 24 05:53:03] Success: 114 Retries: 2263
[Jun 24 05:53:03] Success: 117 Retries: 2281
[Jun 24 05:53:03] Success: 117 Retries: 2286
2020/06/24 05:53:03 rpc error: code = Unknown desc = : Transaction is too old

github.com/dgraph-io/dgraph/x.Check
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/x/error.go:42
main.tryUpsert
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/contrib/integration/acctupsert/main.go:152
main.upsert
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/contrib/integration/acctupsert/main.go:124
main.doUpserts.func1
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/contrib/integration/acctupsert/main.go:103
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1357
exit status 1
fail ./contrib/scripts/load-test.sh

Result 4

This returns that the test fails. This type or result happens 12.5% of the time (sample of 8)

*  Running account upsert tests
[Decoder]: Using assembly version of decoder
Doing upserts
[Jun 23 19:32:36] Success: 0 Retries: 0
[Jun 23 19:32:36] Success: 0 Retries: 0
[Jun 23 19:32:36] Success: 3 Retries: 1
[Jun 23 19:32:36] Success: 3 Retries: 18
[Jun 23 19:32:36] Success: 3 Retries: 28
[Jun 23 19:32:36] Success: 3 Retries: 29
[Jun 23 19:32:36] Success: 3 Retries: 30
[Jun 23 19:32:36] Success: 3 Retries: 31
[Jun 23 19:32:36] Success: 3 Retries: 32
[Jun 23 19:32:36] Success: 3 Retries: 33
[Jun 23 19:32:36] Success: 3 Retries: 34
[Jun 23 19:32:36] Success: 3 Retries: 35
[Jun 23 19:32:36] Success: 3 Retries: 36
[Jun 23 19:32:36] Success: 3 Retries: 37
[Jun 23 19:32:36] Success: 3 Retries: 38
[Jun 23 19:32:36] Success: 3 Retries: 39
[Jun 23 19:32:36] Success: 3 Retries: 40
[Jun 23 19:32:36] Success: 3 Retries: 41
[Jun 23 19:32:36] Success: 3 Retries: 42
[Jun 23 19:32:36] Success: 3 Retries: 43
[Jun 23 19:32:36] Success: 3 Retries: 44
[Jun 23 19:32:36] Success: 3 Retries: 45
[Jun 23 19:32:36] Success: 3 Retries: 47
[Jun 23 19:32:36] Success: 3 Retries: 47
[Jun 23 19:32:36] Success: 3 Retries: 48
[Jun 23 19:32:36] Success: 3 Retries: 49
[Jun 23 19:32:36] Success: 3 Retries: 50
[Jun 23 19:32:36] Success: 3 Retries: 52
[Jun 23 19:32:36] Success: 3 Retries: 52
[Jun 23 19:32:36] Success: 3 Retries: 53
[Jun 23 19:32:36] Success: 3 Retries: 54
[Jun 23 19:32:36] Success: 4 Retries: 55
[Jun 23 19:32:36] Success: 4 Retries: 56
[Jun 23 19:32:36] Success: 4 Retries: 58
[Jun 23 19:32:36] Success: 4 Retries: 57
[Jun 23 19:32:36] Success: 4 Retries: 59
[Jun 23 19:32:36] Success: 4 Retries: 60
[Jun 23 19:32:36] Success: 4 Retries: 61
[Jun 23 19:32:36] Success: 4 Retries: 62
[Jun 23 19:32:36] Success: 4 Retries: 63
[Jun 23 19:32:36] Success: 4 Retries: 64
[Jun 23 19:32:36] Success: 4 Retries: 65
[Jun 23 19:32:36] Success: 4 Retries: 66
[Jun 23 19:32:36] Success: 4 Retries: 68
[Jun 23 19:32:36] Success: 4 Retries: 68
[Jun 23 19:32:36] Success: 4 Retries: 69
[Jun 23 19:32:36] Success: 4 Retries: 70
[Jun 23 19:32:36] Success: 4 Retries: 71
[Jun 23 19:32:36] Success: 4 Retries: 72
[Jun 23 19:32:36] Success: 4 Retries: 73
[Jun 23 19:32:36] Success: 4 Retries: 74
[Jun 23 19:32:36] Success: 4 Retries: 108
[Jun 23 19:32:37] Success: 4 Retries: 150
[Jun 23 19:32:37] Success: 4 Retries: 236
[Jun 23 19:32:37] Success: 7 Retries: 265
[Jun 23 19:32:37] Success: 11 Retries: 365
[Jun 23 19:32:37] Success: 14 Retries: 416
[Jun 23 19:32:37] Success: 14 Retries: 417
[Jun 23 19:32:37] Success: 15 Retries: 453
[Jun 23 19:32:37] Success: 16 Retries: 517
[Jun 23 19:32:38] Success: 19 Retries: 521
[Jun 23 19:32:38] Success: 22 Retries: 537
[Jun 23 19:32:38] Success: 23 Retries: 596
[Jun 23 19:32:38] Success: 23 Retries: 597
[Jun 23 19:32:38] Success: 23 Retries: 665
[Jun 23 19:32:38] Success: 23 Retries: 674
[Jun 23 19:32:38] Success: 24 Retries: 725
[Jun 23 19:32:38] Success: 29 Retries: 765
[Jun 23 19:32:38] Success: 29 Retries: 778
[Jun 23 19:32:38] Success: 30 Retries: 798
[Jun 23 19:32:39] Success: 32 Retries: 870
[Jun 23 19:32:39] Success: 32 Retries: 871
[Jun 23 19:32:39] Success: 34 Retries: 905
[Jun 23 19:32:39] Success: 35 Retries: 933
[Jun 23 19:32:39] Success: 37 Retries: 939
[Jun 23 19:32:39] Success: 38 Retries: 993
[Jun 23 19:32:39] Success: 42 Retries: 1056
[Jun 23 19:32:39] Success: 43 Retries: 1060
[Jun 23 19:32:39] Success: 45 Retries: 1151
[Jun 23 19:32:39] Success: 45 Retries: 1153
[Jun 23 19:32:39] Success: 45 Retries: 1152
[Jun 23 19:32:40] Success: 48 Retries: 1196
[Jun 23 19:32:40] Success: 48 Retries: 1214
[Jun 23 19:32:40] Success: 49 Retries: 1291
[Jun 23 19:32:40] Success: 54 Retries: 1311
[Jun 23 19:32:40] Success: 56 Retries: 1377
[Jun 23 19:32:40] Success: 56 Retries: 1425
[Jun 23 19:32:40] Success: 57 Retries: 1449
[Jun 23 19:32:40] Success: 60 Retries: 1458
[Jun 23 19:32:40] Success: 61 Retries: 1550
[Jun 23 19:32:41] Success: 64 Retries: 1553
[Jun 23 19:32:41] Success: 68 Retries: 1615
[Jun 23 19:32:41] Success: 68 Retries: 1618
[Jun 23 19:32:41] Success: 68 Retries: 1619
[Jun 23 19:32:41] Success: 68 Retries: 1617
[Jun 23 19:32:41] Success: 68 Retries: 1617
[Jun 23 19:32:41] Success: 72 Retries: 1689
[Jun 23 19:32:41] Success: 72 Retries: 1700
[Jun 23 19:32:41] Success: 72 Retries: 1701
[Jun 23 19:32:41] Success: 78 Retries: 1782
[Jun 23 19:32:41] Success: 81 Retries: 1828
[Jun 23 19:32:41] Success: 81 Retries: 1903
[Jun 23 19:32:42] Success: 84 Retries: 1910
[Jun 23 19:32:42] Success: 85 Retries: 1943
[Jun 23 19:32:42] Success: 85 Retries: 1944
[Jun 23 19:32:42] Success: 85 Retries: 1977
[Jun 23 19:32:42] Success: 88 Retries: 1997
[Jun 23 19:32:42] Success: 88 Retries: 2006
[Jun 23 19:32:42] Success: 92 Retries: 2025
[Jun 23 19:32:42] Success: 93 Retries: 2052
[Jun 23 19:32:42] Success: 93 Retries: 2060
[Jun 23 19:32:42] Success: 93 Retries: 2061
[Jun 23 19:32:42] Success: 93 Retries: 2062
[Jun 23 19:32:42] Success: 93 Retries: 2105
[Jun 23 19:32:42] Success: 93 Retries: 2105
[Jun 23 19:32:42] Success: 93 Retries: 2106
[Jun 23 19:32:43] Success: 94 Retries: 2142
[Jun 23 19:32:43] Success: 94 Retries: 2143
[Jun 23 19:32:43] Success: 94 Retries: 2182
[Jun 23 19:32:43] Success: 99 Retries: 2203
[Jun 23 19:32:43] Success: 102 Retries: 2206
[Jun 23 19:32:43] Success: 103 Retries: 2242
[Jun 23 19:32:43] Success: 103 Retries: 2243
[Jun 23 19:32:43] Success: 104 Retries: 2275
[Jun 23 19:32:43] Success: 105 Retries: 2311
[Jun 23 19:32:43] Success: 111 Retries: 2320
[Jun 23 19:32:43] Success: 112 Retries: 2359
[Jun 23 19:32:43] Success: 112 Retries: 2360
[Jun 23 19:32:44] Success: 113 Retries: 2401
[Jun 23 19:32:44] Success: 114 Retries: 2429
[Jun 23 19:32:44] Success: 120 Retries: 2496
[Jun 23 19:32:44] Success: 121 Retries: 2508
[Jun 23 19:32:44] Success: 122 Retries: 2518
[Jun 23 19:32:44] Success: 122 Retries: 2553
[Jun 23 19:32:44] Success: 123 Retries: 2554
[Jun 23 19:32:44] Success: 123 Retries: 2586
[Jun 23 19:32:44] Success: 128 Retries: 2605
2020/06/23 19:32:44 rpc error: code = Unknown desc = : cannot retrieve UIDs from list with key 000005666972737402016d617274696e: readTs: 13174 less than minTs: 13211 for key: "\x00\x00\x05first\x02\x01martin"

github.com/dgraph-io/dgraph/x.Check
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/x/error.go:42
main.tryUpsert
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/contrib/integration/acctupsert/main.go:152
main.upsert
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/contrib/integration/acctupsert/main.go:124
main.doUpserts.func1
    /home/pawan0201/go/src/github.com/dgraph-io/dgraph/contrib/integration/acctupsert/main.go:103
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1357
exit status 1

dgraph-io / dgraph