slackfoo
commented
3 years ago Just to add kibana log shows this:
{ statusCode: 403,\n payload:\n { message:\n 'index [.kibana_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] index [.kibana_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];',\n statusCode: 403,\n error: 'Forbidden' },\n headers: {} },\n reformat: [Function],\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/forbidden' }"}
{"type":"log","@timestamp":"2020-10-13T13:26:25Z","tags":["warning","plugins","usageCollection"],"pid":33,"message":"Unable to fetch data from maps-telemetry collector"}
{"type":"log","@timestamp":"2020-10-13T13:26:25Z","tags":["warning","monitoring","kibana-monitoring"],"pid":33,"message":"Error: [export_exception] failed to flush export bulks\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.
Cyb3rWard0g
Describe the problem
Just installed HELK almost a week ago, I noticed yesterday it only shows previous log. Is there limitation to the log ingested
Provide the output of the following commands
NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic
Get disk space, memory, processor cores, and docker storage
NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.1 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal
Docker Space: Filesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-ubuntu--lv 39G 35G 2.3G 94% /
Memory: total used free shared buff/cache available Mem: 11 8 0 0 3 2 Swap: 3 0 3
Cores: 4
Get output of the HELK docker containers:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9589dbf9579 otrf/helk-elastalert:0.4.0 "./elastalert-entryp…" 2 weeks ago Up 9 hours helk-elastalert 4297475fce01 confluentinc/cp-ksql-cli:5.1.3 "/bin/sh" 2 weeks ago Up 3 hours helk-ksql-cli 44fd3d35b860 confluentinc/cp-ksql-server:5.1.3 "/etc/confluent/dock…" 2 weeks ago Up 9 hours 0.0.0.0:8088->8088/tcp helk-ksql-server 23a72c835be1 otrf/helk-kafka-broker:2.4.0 "./kafka-entrypoint.…" 2 weeks ago Up 9 hours 0.0.0.0:9092->9092/tcp helk-kafka-broker f89f2d8b9a1f otrf/helk-zookeeper:2.4.0 "./zookeeper-entrypo…" 2 weeks ago Up 9 hours 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper 159fe862a84b otrf/helk-logstash:7.6.2.1 "/usr/share/logstash…" 2 weeks ago Up 9 hours 0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, 0.0.0.0:8515-8516->8515-8516/udp, 9600/tcp helk-logstash 55297a663cc1 otrf/helk-nginx:0.3.0 "/opt/helk/scripts/n…" 2 weeks ago Up 9 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp helk-nginx 5340e51d5b3b docker.elastic.co/kibana/kibana:7.6.2 "/usr/share/kibana/s…" 2 weeks ago Up 9 hours 5601/tcp helk-kibana 766dd6cc58a9 docker.elastic.co/elasticsearch/elasticsearch:7.6.2 "/usr/share/elastics…" 2 weeks ago Up 9 hours 9200/tcp, 9300/tcp helk-elasticsearch
Provide the HELK installation logs located at /var/log/helk-install.log if you are having install errors
What version of HELK are you using
run the command from within the HELK root directory
cat .git/refs/heads/masterand include what date you cloned the HELK repo
cat .git/refs/heads/master f70c4b8c3fce9c14fe7457b7350797b69ca4deb0
What version of Winlogbeat are you using if you are using Windows/WEF logs
What steps did you take trying to fix the issue
Restart system and the docker container
run packet capture :
sudo tcpdump -Xni ens160 port 9092 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 09:41:21.481028 IP 192.168.24.10.58515 > 192.168.28.99.9092: Flags [P.], seq 3345216621:3345217698, ack 2342406980, win 8207, length 1077 0x0000: 4500 045d 1194 4000 7f06 3049 c0a8 180a E..]..@...0I.... 0x0010: c0a8 1c63 e493 2384 c763 f46d 8b9e 4b44 ...c..#..c.m..KD 0x0020: 5018 200f 11ad 0000 0000 0431 0000 0003 P..........1.... 0x0030: 0000 538a 0005 6265 6174 73ff ff00 0100 ..S...beats..... 0x0040: 0027 1000 0000 0100 0a77 696e 6c6f 6762 .'.......winlogb 0x0050: 6561 7400 0000 0100 0000 0000 0003 fe00 eat............. 0x0060: 0000 0000 0000 0000 0003 f200 0000 0002 ................ 0x0070: fef0 c898 0001 0000 0000 0000 0175 2155 .............u!U 0x0080: e6ea ffff ffff ffff ffff ffff ffff ffff ................ 0x0090: ffff ffff 0000 0000 0000 0001 1f8b 0800 ................ 0x00a0: 0000 0000 00ff ac55 dd6e db36 18cd de44 .......U.n.6...D 0x00b0: fdae 5a40 d444 fd8b 572d ba75 f545 dba0 ..Z@.D..W-.u.E.. 0x00c0: 7630 b410 1050 2215 1391 4881 a293 1681 v0...P"...H..... 0x00d0: 5f68 4fb0 c71b 4829 b61c a7c5 304c 7722 _hO...H)....0Lw" 0x00e0: 3ff2 1c1e 9eef f0ef 1717 1717 bffc f5e2 ?............... 0x00f0: 015e 1bd1 f3d1 d07e 0002 5118 8508 8708 .^.....~..Q..... 0x0100: c79b b024 0926 b80c 221c 7e05 1f5e f7dc ...$.&..".~..^.. 0x0110: 5046 0d05 f200 35a7 0608 dc0b d9a9 1bf7 PF....5......... 0x0120: e383 f93e 7020 70cd 5403 3edc 713d 0a25 ...>p.p.T.>.q=.% 0x0130: 8140 1e94 4104 7b7f aeb6 cb35 6f94 66d7 .@..A.{....5o.f. 0x0140: 8201 89b3 a2c8 131f 06ad ee04 e3fa fa66 ...............f 0x0150: 6787 e121 cdf3 302e d216 3551 4451 12f3 g..!..0...5QDQ..
How could we replicate the issue
Any additionally code or log context you would like to provide
Any additional context or input you have