Closed saleem-unifycare closed 2 years ago
AHarmlessPyro
commented
2 years ago Hi Saleem, From the provided logs, I can see the issue could be that you're trying to hit port 8080 instead of 8081 where the collector is located. Try modifying the METLO_ADDR environment var and see if that works.
Alternatively, check your METLO_KEY value too to make sure that it's valid (or create a new one following this : https://docs.metlo.com/docs/kubernetes).
saleem-unifycare
commented
2 years ago I have updated you the wrong port and I corrected the ticket. It's working on 8000 itself.
On Thu, Oct 27, 2022, 10:15 Ninad Sinha @.***> wrote:
Hi Saleem, From the provided logs, I can see the issue could be that you're trying to hit port 8080 instead of 8081 where the collector is located. Alternatively, check your METLO_KEY value too to make sure that it's valid (or create a new one following this : https://docs.metlo.com/docs/kubernetes).
— Reply to this email directly, view it on GitHub https://github.com/metlo-labs/metlo/issues/52#issuecomment-1292989327, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2OAD72OOTMSIGMVOMDYUWLWFICGBANCNFSM6AAAAAARPT6CIU . You are receiving this because you authored the thread.Message ID: @.***>
AHarmlessPyro
commented
2 years ago If it works then it's all good. Let us know if there's anything else we can help you with 😄
saleem-unifycare
commented
2 years ago Application is working fine but data is not getting exposed to application
On Thu, Oct 27, 2022, 10:52 Ninad Sinha @.***> wrote:
If it works then it's all good. Let us know if there's anything else we can help you with 😄
— Reply to this email directly, view it on GitHub https://github.com/metlo-labs/metlo/issues/52#issuecomment-1293011752, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2OAD724DXYMVXIBW2KQS73WFIGSBANCNFSM6AAAAAARPT6CIU . You are receiving this because you authored the thread.Message ID: @.***>
AHarmlessPyro
commented
2 years ago Hi Saleem.
It seems that your url for METLO_ADDR points to port 8000. It needs to point to port 8081 as that's where the collector is located.
siarhei-itech
commented
2 years ago Hi Team. I have the same issue in AWS Metlo (us-east-1) and daemonset AWS EKS. Ports, metlo_key are apen and correct.
My metlo.yaml:
Part of the logs from the pod:
27/10/2022 -- 08:52:47 -
saleem-unifycare
commented
2 years ago please find additional logs attached:
Node.js v18.12.0
27/10/2022 -- 09:47:33 -
On Thu, Oct 27, 2022 at 2:42 PM siarhei-itech @.***> wrote:
Hi Team. I have the same issue in AWS Metlo (us-east-1) and daemonset AWS EKS. Ports, metlo_key are apen and correct. My metlo.yaml: [image: image] https://user-images.githubusercontent.com/116798891/198243008-f550c515-7ae1-418a-990e-9df5ec5fc212.png
Part of the logs from the pod: 27/10/2022 -- 08:52:47 - - Running in live mode, activating unix socket 27/10/2022 -- 08:52:47 - - 1 rule files processed. 1 rules successfully loaded, 0 rules failed 27/10/2022 -- 08:52:47 - - Threshold config parsed: 0 rule(s) found 27/10/2022 -- 08:52:47 - - 1 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 1 inspect application layer, 0 are decoder event only 27/10/2022 -- 08:52:47 - - Going to use 2 thread(s) 27/10/2022 -- 08:52:47 - - Running in live mode, activating unix socket 27/10/2022 -- 08:52:47 - - Using unix socket file '/var/run/suricata/suricata-command.socket' 27/10/2022 -- 08:52:47 - - all 2 packet processing threads, 2 management threads initialized, engine started. 27/10/2022 -- 08:52:47 - - All AFP capture threads are running. 27/10/2022 -- 08:52:47 - - This is Suricata version 6.0.5 RELEASE running in SYSTEM mode 27/10/2022 -- 08:52:47 - - CPUs/cores online: 2 27/10/2022 -- 08:52:47 - - Found an MTU of 9001 for 'eth0' 27/10/2022 -- 08:52:47 - - Found an MTU of 9001 for 'eth0' 27/10/2022 -- 08:52:47 - - Setting logging socket of non-blocking in live mode. 27/10/2022 -- 08:52:47 - - eve-log output device (unix_stream) initialized: /etc/suricata-logs/eve.sock 27/10/2022 -- 08:52:47 - - JsonRdpLog logger not enabled: protocol rdp is disabled 27/10/2022 -- 08:52:47 - - JsonIKEv2Log logger not enabled: protocol ikev2 is disabled 27/10/2022 -- 08:52:47 - - JsonKRB5Log logger not enabled: protocol krb5 is disabled 27/10/2022 -- 08:52:47 - - JsonSNMPLog logger not enabled: protocol snmp is disabled 27/10/2022 -- 08:52:47 - - JsonRFBLog logger not enabled: protocol rfb is disabled 27/10/2022 -- 08:52:47 - - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.5/configuration/suricata-yaml.html#stats 27/10/2022 -- 08:52:47 - - Running in live mode, activating unix socket 27/10/2022 -- 08:52:47 - - 1 rule files processed. 1 rules successfully loaded, 0 rules failed 27/10/2022 -- 08:52:47 - - Threshold config parsed: 0 rule(s) found 27/10/2022 -- 08:52:47 - - 1 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 1 inspect application layer, 0 are decoder event only 27/10/2022 -- 08:52:47 - - Going to use 2 thread(s) 27/10/2022 -- 08:52:47 - - Running in live mode, activating unix socket 27/10/2022 -- 08:52:47 - - Using unix socket file '/var/run/suricata/suricata-command.socket' 27/10/2022 -- 08:52:47 - - all 2 packet processing threads, 2 management threads initialized, engine started. 27/10/2022 -- 08:52:47 - - All AFP capture threads are running.
— Reply to this email directly, view it on GitHub https://github.com/metlo-labs/metlo/issues/52#issuecomment-1293225757, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2OAD73OMP62GWGCUPJ3EG3WFJBOHANCNFSM6AAAAAARPT6CIU . You are receiving this because you authored the thread.Message ID: @.***>
AHarmlessPyro
commented
2 years ago Hi @siarhei-itech and @saleem-unifycare, sorry for the issue. We're investigating this and will be back to you asap with a solution
AHarmlessPyro
commented
2 years ago Hey guys :slightly_smiling_face: So sorry for the late response! We’ve updated the metlo/suricata-daemon image so it shouldn’t be throwing any more errors. We also have changed some configuration in the example daemonset file.
Depending on your setup, you may also want to try out metlo as a sidecar as described here. Let me know if these work for you :slightly_smiling_face:
Daemonset : https://github.com/metlo-labs/metlo/blob/develop/ingestors/kubernetes/metlo-daemonset.yaml Sidecar : https://github.com/metlo-labs/metlo/blob/develop/ingestors/kubernetes/metlo-sidecar.yaml
fadhilthomas
commented
2 years ago hi, after I try the latest update for the sidecar ingestor, the log still did not come up in metlo. here's the log
STARTING
starting suricata
starting metlo
30/10/2022 -- 20:43:44 - <Notice> - This is Suricata version 6.0.1 RELEASE running in SYSTEM mode
30/10/2022 -- 20:43:44 - <Warning> - [ERRCODE: SC_ERR_SOCKET(200)] - Error connecting to socket "/tmp/eve.sock": No such file or directory (will keep trying)
30/10/2022 -- 20:43:44 - <Notice> - JsonRdpLog logger not enabled: protocol rdp is disabled
30/10/2022 -- 20:43:44 - <Notice> - JsonIKEv2Log logger not enabled: protocol ikev2 is disabled
30/10/2022 -- 20:43:44 - <Notice> - JsonKRB5Log logger not enabled: protocol krb5 is disabled
30/10/2022 -- 20:43:44 - <Notice> - JsonSNMPLog logger not enabled: protocol snmp is disabled
30/10/2022 -- 20:43:44 - <Notice> - JsonRFBLog logger not enabled: protocol rfb is disabled
30/10/2022 -- 20:43:44 - <Error> - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.1/configuration/suricata-yaml.html#stats
Socket: /tmp/eve.sock
Process: 9
Checking for leftover socket.
No leftover socket found.
Creating server.
30/10/2022 -- 20:43:44 - <Notice> - all 4 packet processing threads, 2 management threads initialized, engine started.
akshay288
commented
2 years ago Hey @fadhilthomas ! Can you share your yaml file?
siarhei-itech
commented
2 years ago Hey guys 🙂 So sorry for the late response! We’ve updated the metlo/suricata-daemon image so it shouldn’t be throwing any more errors. We also have changed some configuration in the example daemonset file.
Depending on your setup, you may also want to try out metlo as a sidecar as described here. Let me know if these work for you 🙂
Daemonset : https://github.com/metlo-labs/metlo/blob/develop/ingestors/kubernetes/metlo-daemonset.yaml Sidecar : https://github.com/metlo-labs/metlo/blob/develop/ingestors/kubernetes/metlo-sidecar.yaml
Hi @AHarmlessPyro. After updating the daemonset works for me. Thanks
fadhilthomas
commented
2 years ago hi @akshay288, I use,
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: metlo-app
spec:
selector:
matchLabels:
name: metlo-app
template:
metadata:
labels:
name: metlo-app
spec:
hostNetwork: true
tolerations:
# this toleration is to have the daemonset runnable on master nodes
# remove it if your masters can't run pods
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: suricata-daemon
image: metlo/suricata-daemon
resources:
requests:
ephemeral-storage: "32Mi"
limits:
ephemeral-storage: "64Mi"
securityContext:
privileged: true
volumeMounts:
- mountPath: /tmp
name: socket-volume
env:
- name: METLO_ADDR
value: http://metlo-ingestor:8081
- name: METLO_KEY
value: metlo.----------------
# nodeSelector:
# app: your-app
volumes:
- name: socket-volume
emptyDir: {}and
kind: Pod
apiVersion: v1
metadata:
name: test-app
labels:
app: test-app
spec:
containers:
- name: test-app
image: hashicorp/http-echo:0.2.3
args:
- "-text=Hello World! This is a Metlo Kubernetes with kind App"
# Metlo Sidecar
- name: metlo-sidecar
image: metlo/suricata-daemon
resources:
requests:
ephemeral-storage: "32Mi"
limits:
ephemeral-storage: "64Mi"
securityContext:
privileged: true
volumeMounts:
- mountPath: /tmp
name: socket-volume
env:
- name: METLO_ADDR
value: http://metlo-ingestor:8081
- name: METLO_KEY
value: metlo.-------------
volumes:
- name: socket-volume
emptyDir: {}here's the log from daemonset
STARTING
starting suricata
starting metlo
31/10/2022 -- 09:12:10 - <Notice> - This is Suricata version 6.0.1 RELEASE running in SYSTEM mode
31/10/2022 -- 09:12:11 - <Warning> - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get MTU via ioctl for 'eth0': No such device (19)
31/10/2022 -- 09:12:11 - <Warning> - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get MTU via ioctl for 'eth0': No such device (19)
31/10/2022 -- 09:12:11 - <Warning> - [ERRCODE: SC_ERR_SOCKET(200)] - Error connecting to socket "/tmp/eve.sock": Connection refused (will keep trying)
31/10/2022 -- 09:12:11 - <Notice> - JsonRdpLog logger not enabled: protocol rdp is disabled
31/10/2022 -- 09:12:11 - <Notice> - JsonIKEv2Log logger not enabled: protocol ikev2 is disabled
31/10/2022 -- 09:12:11 - <Notice> - JsonKRB5Log logger not enabled: protocol krb5 is disabled
31/10/2022 -- 09:12:11 - <Notice> - JsonSNMPLog logger not enabled: protocol snmp is disabled
31/10/2022 -- 09:12:11 - <Notice> - JsonRFBLog logger not enabled: protocol rfb is disabled
31/10/2022 -- 09:12:11 - <Error> - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.1/configuration/suricata-yaml.html#stats
31/10/2022 -- 09:12:11 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find type for iface "eth0": No such device
Socket: /tmp/eve.sock
Process: 9
Checking for leftover socket.
Removing leftover socket.
Creating server.
31/10/2022 -- 09:12:11 - <Notice> - all 4 packet processing threads, 2 management threads initialized, engine started.
31/10/2022 -- 09:12:11 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find iface eth0: No such device
31/10/2022 -- 09:12:11 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
31/10/2022 -- 09:12:11 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - thread W#01-eth0 failedLooks like you have everything setup correctly :) Let's chat on discord so we can debug a bit quicker!
shrisukhani
commented
2 years ago Resolved over DMs! 😃
HI Team,
We have configured metlo in GCP and daemonset in GKE. Data is not getting exported to application. KIndly help us on this. Here are the attached log of one of pod
21/10/2022 -- 06:56:59 - - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.5/configuration/suricata-yaml.html#stats 21/10/2022 -- 06:56:59 - - Running in live mode, activating unix socket 21/10/2022 -- 06:56:59 - - 1 rule files processed. 1 rules successfully loaded, 0 rules failed 21/10/2022 -- 06:56:59 - - Threshold config parsed: 0 rule(s) found 21/10/2022 -- 06:56:59 - - 1 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 1 inspect application layer, 0 are decoder event only 21/10/2022 -- 06:56:59 - - Going to use 1 thread(s) 21/10/2022 -- 06:56:59 - - Running in live mode, activating unix socket 21/10/2022 -- 06:56:59 - - Using unix socket file '/var/run/suricata/suricata-command.socket' 21/10/2022 -- 06:56:59 - - all 1 packet processing threads, 2 management threads initialized, engine started. 21/10/2022 -- 06:56:59 - - All AFP capture threads are running. 21/10/2022 -- 06:56:58 - - This is Suricata version 6.0.5 RELEASE running in SYSTEM mode 21/10/2022 -- 06:56:58 - - CPUs/cores online: 1 21/10/2022 -- 06:56:59 - - Found an MTU of 1460 for 'eth0' 21/10/2022 -- 06:56:59 - - Found an MTU of 1460 for 'eth0' 21/10/2022 -- 06:56:59 - - Setting logging socket of non-blocking in live mode. 21/10/2022 -- 06:56:59 - - eve-log output device (unix_stream) initialized: /etc/suricata-logs/eve.sock 21/10/2022 -- 06:56:59 - - JsonRdpLog logger not enabled: protocol rdp is disabled 21/10/2022 -- 06:56:59 - - JsonIKEv2Log logger not enabled: protocol ikev2 is disabled 21/10/2022 -- 06:56:59 - - JsonKRB5Log logger not enabled: protocol krb5 is disabled 21/10/2022 -- 06:56:59 - - JsonSNMPLog logger not enabled: protocol snmp is disabled 21/10/2022 -- 06:56:59 - - JsonRFBLog logger not enabled: protocol rfb is disabled 21/10/2022 -- 06:56:59 - - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.5/configuration/suricata-yaml.html#stats
============================================================================================
metlo.yaml:
apiVersion: apps/v1 kind: DaemonSet metadata: name: metlo-app spec: selector: matchLabels: name: metlo-app template: metadata: labels: name: metlo-app spec: hostNetwork: true tolerations:
this toleration is to have the daemonset runnable on master nodes