search

elastic / stream2es

Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)
355 stars 62 forks source link

es to json losing 1 event per file? #42

Closed nellicus closed 9 years ago

nellicus commented 9 years ago 
abonuccelli@w530 /tmp/test/tee $ /opt/elk/PROD/scripts/stream2es es --source http://192.168.0.101:9200/logstash-syslog-2015.03.19 --no-indexing --tee /tmp/test/tee/  --log debug  
2015-04-08T16:30:24.140+0200 DEBUG saving json to /tmp/test/tee/
2015-04-08T16:30:24.831+0200 DEBUG 00:00.695 2064.7d/s 1492.2K/s 1435 1435 1061989 0 AUwxHXokwxvhS6xbMjj6
2015-04-08T16:30:25.204+0200 DEBUG 00:01.068 2698.5d/s 1942.0K/s 2882 1447 1061845 0 AUwxMMMzwxvhS6xbMkIR
2015-04-08T16:30:25.441+0200 DEBUG 00:01.306 3340.7d/s 2382.2K/s 4363 1481 1062011 0 AUwxR_iXwxvhS6xbMksH
2015-04-08T16:30:25.779+0200 DEBUG 00:01.643 3557.5d/s 2524.9K/s 5845 1482 1062055 0 AUwxRf5kwxvhS6xbMkph
2015-04-08T16:30:26.032+0200 DEBUG 00:01.896 3861.3d/s 2735.3K/s 7321 1476 1062734 0 AUwxXZSnwxvhS6xbMlQj
2015-04-08T16:30:26.349+0200 DEBUG 00:02.214 3974.3d/s 2810.9K/s 8799 1478 1062110 0 AUwxX43ywxvhS6xbMlTH
2015-04-08T16:30:26.691+0200 DEBUG 00:02.556 4016.8d/s 2840.7K/s 10267 1468 1062387 0 AUwxdoDxwxvhS6xbMl41
2015-04-08T16:30:26.929+0200 DEBUG 00:02.794 4192.2d/s 2969.9K/s 11713 1446 1062023 0 AUwxjCAAwxvhS6xbMmfj
2015-04-08T16:30:27.340+0200 DEBUG 00:03.204 4102.4d/s 2913.5K/s 13144 1431 1061807 0 AUwxiPF1wxvhS6xbMmZK
2015-04-08T16:30:27.567+0200 DEBUG 00:03.432 4242.1d/s 3022.2K/s 14559 1415 1062326 0 AUwxlwiowxvhS6xbMm_i
2015-04-08T16:30:27.900+0200 DEBUG 00:03.765 4242.8d/s 3030.3K/s 15974 1415 1061652 0 AUwxln4UwxvhS6xbMm6q
2015-04-08T16:30:28.145+0200 DEBUG 00:04.010 4334.9d/s 3103.7K/s 17383 1409 1061685 0 AUwxuKtJwxvhS6xbMoDR
2015-04-08T16:30:28.459+0200 DEBUG 00:04.324 4360.8d/s 3118.2K/s 18856 1473 1062018 0 AUwxttbEwxvhS6xbMn-U
2015-04-08T16:30:28.912+0200 DEBUG 00:04.777 4258.7d/s 3039.7K/s 20344 1488 1062266 0 AUwxzLAUwxvhS6xbMoiO
2015-04-08T16:30:29.132+0200 DEBUG 00:04.996 4357.1d/s 3113.9K/s 21768 1424 1061471 0 AUwx4mOuwxvhS6xbMpKk
2015-04-08T16:30:29.462+0200 DEBUG 00:05.326 4350.2d/s 3115.6K/s 23169 1401 1061322 0 AUwx3woTwxvhS6xbMpCl
2015-04-08T16:30:29.761+0200 DEBUG 00:05.625 4371.2d/s 3134.3K/s 24588 1419 1061887 0 AUwx7du0wxvhS6xbMpiC
2015-04-08T16:30:30.116+0200 DEBUG 00:05.981 4326.9d/s 3121.0K/s 25879 1291 1060856 0 AUwx7seNwxvhS6xbMpjo
2015-04-08T16:30:30.340+0200 DEBUG 00:06.204 4372.7d/s 3175.7K/s 27128 1249 1060695 0 AUwyAeRFwxvhS6xbMqXN
2015-04-08T16:30:30.660+0200 DEBUG 00:06.525 4347.1d/s 3178.3K/s 28365 1237 1060739 0 AUwx_JQ-wxvhS6xbMqMz
2015-04-08T16:30:30.901+0200 DEBUG 00:06.765 4375.5d/s 3218.7K/s 29600 1235 1061110 0 AUwyAsLZwxvhS6xbMqyd
2015-04-08T16:30:31.229+0200 DEBUG 00:07.094 4353.8d/s 3215.5K/s 30886 1286 1061021 0 AUwyArp5wxvhS6xbMqvy
2015-04-08T16:30:31.459+0200 DEBUG 00:07.324 4398.3d/s 3256.0K/s 32213 1327 1061079 0 AUwyA0KCwxvhS6xbMrqY
2015-04-08T16:30:31.802+0200 DEBUG 00:07.666 4381.3d/s 3246.0K/s 33587 1374 1061644 0 AUwyApB5wxvhS6xbMqaA
2015-04-08T16:30:32.062+0200 DEBUG 00:07.927 4412.3d/s 3269.9K/s 34976 1389 1061556 0 AUwyA91uwxvhS6xbMsdP
2015-04-08T16:30:32.374+0200 DEBUG 00:08.239 4395.2d/s 3271.7K/s 36212 1236 1060249 0 AUwyBG9XwxvhS6xbMskl
2015-04-08T16:30:32.619+0200 DEBUG 00:08.484 4413.1d/s 3299.3K/s 37441 1229 1060358 0 AUwyBIxWwxvhS6xbMs0D
2015-04-08T16:30:32.962+0200 DEBUG 00:08.827 4382.9d/s 3288.4K/s 38688 1247 1060343 0 AUwyBLWHwxvhS6xbMtJZ
2015-04-08T16:30:33.192+0200 DEBUG 00:09.057 4409.1d/s 3319.2K/s 39933 1245 1060581 0 AUwyBL3mwxvhS6xbMtNs
2015-04-08T16:30:33.556+0200 DEBUG 00:09.420 4384.4d/s 3301.4K/s 41301 1368 1061386 0 AUwyBU0FwxvhS6xbMuFS
2015-04-08T16:30:33.902+0200 DEBUG 00:09.767 4369.2d/s 3290.2K/s 42674 1373 1061000 0 AUwyBSRpwxvhS6xbMt3I
2015-04-08T16:30:34.151+0200 DEBUG 00:10.016 4391.4d/s 3311.8K/s 43984 1310 1061175 0 AUwyBRhHwxvhS6xbMtzK
2015-04-08T16:30:34.498+0200 DEBUG 00:10.363 4368.8d/s 3300.9K/s 45274 1290 1060444 0 AUwyBoV7wxvhS6xbMvP0
2015-04-08T16:30:34.760+0200 DEBUG 00:10.625 4375.0d/s 3316.9K/s 46484 1210 1060558 0 AUwyBlAJwxvhS6xbMu1p
2015-04-08T16:30:35.096+0200 DEBUG 00:10.961 4352.3d/s 3309.8K/s 47706 1222 1060691 0 AUwyBlxTwxvhS6xbMu8P
2015-04-08T16:30:35.345+0200 DEBUG 00:11.210 4374.0d/s 3328.7K/s 49033 1327 1061293 0 AUwyBv-swxvhS6xbMwBn
2015-04-08T16:30:35.710+0200 DEBUG 00:11.575 4353.3d/s 3313.3K/s 50389 1356 1061201 0 AUwyBrsMwxvhS6xbMvpy
2015-04-08T16:30:35.968+0200 DEBUG 00:11.833 4378.9d/s 3328.6K/s 51815 1426 1061576 0 AUwyGKVvwxvhS6xbMwnk
2015-04-08T16:30:36.370+0200 DEBUG 00:12.235 4351.4d/s 3304.0K/s 53239 1424 1061548 0 AUwyE4lUwxvhS6xbMwgC
2015-04-08T16:30:36.583+0200 DEBUG 00:12.448 4388.1d/s 3330.7K/s 54623 1384 1061208 0 AUwyJZxvwxvhS6xbMxFs
2015-04-08T16:30:36.904+0200 DEBUG 00:12.769 4384.9d/s 3328.2K/s 55991 1368 1061414 0 AUwyI9ckwxvhS6xbMw__
2015-04-08T16:30:37.135+0200 DEBUG 00:13.000 4411.4d/s 3348.7K/s 57348 1357 1060862 0 AUwyPESFwxvhS6xbMyBx
2015-04-08T16:30:37.480+0200 DEBUG 00:13.345 4404.9d/s 3339.9K/s 58783 1435 1062040 0 AUwyOWnRwxvhS6xbMx5N
2015-04-08T16:30:37.807+0200 DEBUG 00:13.671 4403.8d/s 3336.1K/s 60205 1422 1062064 0 AUwySo8GwxvhS6xbMybb
2015-04-08T16:30:38.056+0200 DEBUG 00:13.921 4428.4d/s 3350.7K/s 61648 1443 1061974 0 AUwyWuyRwxvhS6xbMy98
2015-04-08T16:30:38.379+0200 DEBUG 00:14.244 4427.4d/s 3347.5K/s 63064 1416 1061758 0 AUwyVuQFwxvhS6xbMy1K
2015-04-08T16:30:38.603+0200 DEBUG 00:14.468 4453.7d/s 3367.3K/s 64436 1372 1061083 0 AUwyaLUXwxvhS6xbMzYN
2015-04-08T16:30:38.951+0200 DEBUG 00:14.816 4445.3d/s 3358.2K/s 65861 1425 1061644 0 AUwydXCXwxvhS6xbMz1s
2015-04-08T16:30:39.180+0200 DEBUG 00:15.045 4469.2d/s 3376.0K/s 67239 1378 1061574 0 AUwyilmiwxvhS6xbM0bm
2015-04-08T16:30:39.554+0200 DEBUG 00:15.419 4452.4d/s 3361.3K/s 68652 1413 1061694 0 AUwyh8XhwxvhS6xbM0XI
2015-04-08T16:30:39.886+0200 DEBUG 00:15.750 4452.1d/s 3356.5K/s 70121 1469 1062295 0 AUwykmEswxvhS6xbM008
2015-04-08T16:30:40.126+0200 DEBUG 00:15.991 4475.5d/s 3370.8K/s 71567 1446 1061774 0 AUwyqdHmwxvhS6xbM1Zp
2015-04-08T16:30:40.455+0200 DEBUG 00:16.320 4472.0d/s 3366.4K/s 72983 1416 1061620 0 AUwyp4kcwxvhS6xbM1UJ
2015-04-08T16:30:40.683+0200 DEBUG 00:16.548 4494.3d/s 3382.6K/s 74371 1388 1061210 0 AUwyy4scwxvhS6xbM2ic
2015-04-08T16:30:41.004+0200 DEBUG 00:16.869 4488.5d/s 3379.7K/s 75716 1345 1060939 0 AUwyxLlUwxvhS6xbM2Rq
2015-04-08T16:30:41.243+0200 DEBUG 00:17.108 4505.1d/s 3393.1K/s 77073 1357 1061839 0 AUwy0LBbwxvhS6xbM20J
2015-04-08T16:30:41.572+0200 DEBUG 00:17.437 4499.1d/s 3388.5K/s 78451 1378 1061968 0 AUwyzdJpwxvhS6xbM2rt
2015-04-08T16:30:41.805+0200 DEBUG 00:17.669 4519.2d/s 3402.7K/s 79849 1398 1061734 0 AUwy7QTnwxvhS6xbM3v1
2015-04-08T16:30:42.128+0200 DEBUG 00:17.993 4519.6d/s 3399.1K/s 81321 1472 1062387 0 AUwy6uqTwxvhS6xbM3sD
2015-04-08T16:30:42.461+0200 DEBUG 00:18.326 4517.0d/s 3393.9K/s 82778 1457 1062131 0 AUwzAWuKwxvhS6xbM4SA
2015-04-08T16:30:42.700+0200 DEBUG 00:18.565 4536.7d/s 3406.1K/s 84223 1445 1062050 0 AUwzFhFswxvhS6xbM40y
2015-04-08T16:30:43.031+0200 DEBUG 00:18.896 4533.0d/s 3401.3K/s 85656 1433 1062391 0 AUwzD-JVwxvhS6xbM4rH
2015-04-08T16:30:43.270+0200 DEBUG 00:19.135 4550.9d/s 3413.1K/s 87081 1425 1062218 0 AUwzJ82VwxvhS6xbM5WN
2015-04-08T16:30:43.584+0200 DEBUG 00:19.448 4550.5d/s 3411.4K/s 88498 1417 1061585 0 AUwzJxiDwxvhS6xbM5T1
2015-04-08T16:30:43.817+0200 DEBUG 00:19.682 4567.7d/s 3423.6K/s 89902 1404 1062141 0 AUwzR78kwxvhS6xbM6c6
2015-04-08T16:30:44.140+0200 DEBUG 00:20.005 4566.7d/s 3420.2K/s 91357 1455 1062507 0 AUwzQz00wxvhS6xbM6UF
2015-04-08T16:30:44.468+0200 DEBUG 00:20.332 4563.1d/s 3416.2K/s 92777 1420 1062113 0 AUwzVzVfwxvhS6xbM65X
2015-04-08T16:30:44.698+0200 DEBUG 00:20.563 4581.0d/s 3428.2K/s 94200 1423 1061822 0 AUwzZEpWwxvhS6xbM7W2
2015-04-08T16:30:45.038+0200 DEBUG 00:20.903 4575.5d/s 3422.1K/s 95642 1442 1062339 0 AUwzYfB5wxvhS6xbM7RB
2015-04-08T16:30:45.268+0200 DEBUG 00:21.133 4594.6d/s 3433.9K/s 97097 1455 1061977 0 AUwzdHObwxvhS6xbM729
2015-04-08T16:30:45.590+0200 DEBUG 00:21.455 4592.4d/s 3430.7K/s 98530 1433 1061814 0 AUwzdCjkwxvhS6xbM72f
2015-04-08T16:30:45.823+0200 DEBUG 00:21.688 4609.0d/s 3441.7K/s 99959 1429 1061865 0 AUwzhOnYwxvhS6xbM8Zg
2015-04-08T16:30:46.142+0200 DEBUG 00:22.007 4606.2d/s 3438.9K/s 101369 1410 1061580 0 AUwzlccuwxvhS6xbM8-M
2015-04-08T16:30:46.471+0200 DEBUG 00:22.336 4601.6d/s 3434.7K/s 102781 1412 1061720 0 AUwzpNluwxvhS6xbM9dl
2015-04-08T16:30:46.701+0200 DEBUG 00:22.566 4617.3d/s 3445.6K/s 104193 1412 1061693 0 AUwztbkTwxvhS6xbM98D
2015-04-08T16:30:47.036+0200 DEBUG 00:22.900 4610.8d/s 3440.6K/s 105587 1394 1061923 0 AUwzryNhwxvhS6xbM9vT
2015-04-08T16:30:47.267+0200 DEBUG 00:23.132 4623.8d/s 3451.0K/s 106957 1370 1061933 0 AUwzw73-wxvhS6xbM-b0
2015-04-08T16:30:47.589+0200 DEBUG 00:23.454 4622.2d/s 3447.8K/s 108409 1452 1061883 0 AUwz0OWwwxvhS6xbM-vg
2015-04-08T16:30:47.823+0200 DEBUG 00:23.688 4638.6d/s 3457.5K/s 109878 1469 1062057 0 AUw0IddnwxvhS6xbM_m7
2015-04-08T16:30:48.062+0200 DEBUG 00:23.926 4655.8d/s 3466.5K/s 111395 1517 1062328 0 AUw0Dn2kwxvhS6xbM_da
2015-04-08T16:30:48.242+0200 DEBUG 00:24.107 4669.7d/s 3473.9K/s 112573 1178 825874 0 AUw0d_ahwxvhS6xbNAYp
2015-04-08T16:30:48.271+0200 INFO  00:24.135 4664.3d/s 3469.9K/s (81.8mb) indexed 112573 streamed 112573 errors 0
2015-04-08T16:30:48.272+0200 INFO  done
abonuccelli@w530 /tmp/test/tee $ ls -alrth
total 48K
drwxr-xr-x  4 abonuccelli abonuccelli 4,0K abr  8 12:28 ..
drwxr-xr-x  6 abonuccelli abonuccelli 4,0K abr  8 16:30 4
drwxr-xr-x 12 abonuccelli abonuccelli 4,0K abr  8 16:30 .
drwxr-xr-x  9 abonuccelli abonuccelli 4,0K abr  8 16:30 5
drwxr-xr-x  7 abonuccelli abonuccelli 4,0K abr  8 16:30 7
drwxr-xr-x  8 abonuccelli abonuccelli 4,0K abr  8 16:30 2
drwxr-xr-x  9 abonuccelli abonuccelli 4,0K abr  8 16:30 1
drwxr-xr-x  7 abonuccelli abonuccelli 4,0K abr  8 16:30 8
drwxr-xr-x  9 abonuccelli abonuccelli 4,0K abr  8 16:30 0
drwxr-xr-x  7 abonuccelli abonuccelli 4,0K abr  8 16:30 9
drwxr-xr-x  7 abonuccelli abonuccelli 4,0K abr  8 16:30 6
drwxr-xr-x  8 abonuccelli abonuccelli 4,0K abr  8 16:30 3
abonuccelli@w530 /tmp/test/tee $ find . -type f | while read x; do gunzip -d $x; done
abonuccelli@w530 /tmp/test/tee $ find . -type f | while read b; do wc -l $b ;done | awk '{sum+=$1} END {print sum}'
112492
abonuccelli@w530 /tmp/test/tee $ curl -XGET "http://192.168.0.101:9200/logstash-syslog-2015.03.19/_search?search_type=count&pretty"
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 112573,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}
abonuccelli@w530 /tmp/test/tee $ find . -type f | wc -l
81
abonuccelli@w530 /tmp/test/tee $ find . -type f | tail -1
./4/8/AUwyh8XhwxvhS6xbM0XI.json

112492 + 81 = 112573 ?
nellicus commented 9 years ago

wc -l counts newlines , last event doesn't have one, closing this