search

sysflow-telemetry / sysflow

SysFlow documentation and issues tracker
Other
45 stars 10 forks source link

sf-exporter on docker to syslog and not S3 #29

Closed ghost closed 4 years ago

ghost commented 4 years ago

Indicate project sf-exporter

Describe the bug I am running sf-collected and sf-exporter in docker in vmware and can see data collected in /mnt/data folder but get and issue with the sf-exporter as below It looks like sf-exported is trying to connect to S3 COS, I am not using S3 but rather syslog to a remote SIEM.

To reproduce Steps to reproduce the behavior:

  1. I have a script to invoke 
    
full-server$ cat ../sysflow-run.sh
docker run -d --privileged --name sf-collector \
     -v /var/run/docker.sock:/host/var/run/docker.sock \
     -v /dev:/host/dev -v /proc:/host/proc:ro \
     -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro \
         -v /usr:/host/usr:ro -v /mnt/data:/mnt/data \
         -e INTERVAL=60 \
         -e EXPORTER_ID=$WebServerTarget \
         -e OUTPUT=/mnt/data/    \
         -e FILTER="container.name!=sf-collector and container.name!=sf-exporter and container.type!=host" \
         --rm sysflowtelemetry/sf-collector

docker service create --name sf-exporter \ -e SYSLOG_HOST=172.16.232.131 \ -e SYSLOG_PORT=514 \ -e NODE_IP=172.16.232.128 \ -e INTERVAL=15 \ -e DIR=/mnt/data \ --mount type=bind,source=/mnt/data,destination=/mnt/data \ sysflowtelemetry/sf-exporter:latest

2. Script and containers invokes fine

full-server$ ../sysflow-run.sh 5f200d6bb635e6a5d1ed8ab023fa0df8950bf3c59154d6a82dbb3063969a37b5 z31mb2rqoimrpzvdbimwom4ph overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged

3. docker ps output looks fine

full-server$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0a03f5942086 sysflowtelemetry/sf-exporter:latest "/bin/sh -c 'python …" 14 seconds ago Up 11 seconds sf-exporter.1.yp1zlxk8p1hb2gu8ah4ry8xkq 5f200d6bb635 sysflowtelemetry/sf-collector "/docker-entrypoint.…" 17 seconds ago Up 15 seconds sf-collector

4. I can see errors in docker service sf-exporter

sf-exporter.1.il1yf0fnd5ok@ubuntu | Traceback (most recent call last): sf-exporter.1.il1yf0fnd5ok@ubuntu | File "./exporter.py", line 62, in get_secret sf-exporter.1.il1yf0fnd5ok@ubuntu | with open('%s/%s' % (secrets_dir, secret_name), 'r') as secret_file: sf-exporter.1.il1yf0fnd5ok@ubuntu | FileNotFoundError: [Errno 2] No such file or directory: '/run/secrets/s3_access_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:18,215 [ERROR] Caught exception while reading secret 's3_secret_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | Traceback (most recent call last): sf-exporter.1.il1yf0fnd5ok@ubuntu | File "./exporter.py", line 62, in get_secret sf-exporter.1.il1yf0fnd5ok@ubuntu | with open('%s/%s' % (secrets_dir, secret_name), 'r') as secret_file: sf-exporter.1.il1yf0fnd5ok@ubuntu | FileNotFoundError: [Errno 2] No such file or directory: '/run/secrets/s3_secret_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:18,216 [WARNING] Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350bd3340>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:18,618 [WARNING] Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350bd3490>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:19,419 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350f44bb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:21,022 [WARNING] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350f44a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:24,227 [WARNING] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350c366d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:24,229 [ERROR] Connection timeout! Removing traces older than 60 minutes sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:24,230 [WARNING] Trace '/mnt/data/1582800794' removed before being uploaded to object store sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:39,247 [ERROR] Caught exception while reading secret 's3_access_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | Traceback (most recent call last): sf-exporter.1.il1yf0fnd5ok@ubuntu | File "./exporter.py", line 62, in get_secret sf-exporter.1.il1yf0fnd5ok@ubuntu | with open('%s/%s' % (secrets_dir, secret_name), 'r') as secret_file: sf-exporter.1.il1yf0fnd5ok@ubuntu | FileNotFoundError: [Errno 2] No such file or directory: '/run/secrets/s3_access_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:39,248 [ERROR] Caught exception while reading secret 's3_secret_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | Traceback (most recent call last): sf-exporter.1.il1yf0fnd5ok@ubuntu | File "./exporter.py", line 62, in get_secret sf-exporter.1.il1yf0fnd5ok@ubuntu | with open('%s/%s' % (secrets_dir, secret_name), 'r') as secret_file: sf-exporter.1.il1yf0fnd5ok@ubuntu | FileNotFoundError: [Errno 2] No such file or directory: '/run/secrets/s3_secret_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:39,251 [WARNING] Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350c405e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:39,653 [WARNING] Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350c40370>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:40,456 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350c40820>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:42,059 [WARNING] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350c40b50>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:45,265 [WARNING] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4350c406a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /sf-monitoring/ sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:54:45,267 [ERROR] Connection timeout! Removing traces older than 60 minutes sf-exporter.1.il1yf0fnd5ok@ubuntu | 2020-02-27 11:55:00,285 [ERROR] Caught exception while reading secret 's3_access_key' sf-exporter.1.il1yf0fnd5ok@ubuntu | Traceback (most recent call last):



**Expected behavior**
Expected output for syslog
expected syslog on destination ip on port 514
I used tcpdump and can not see any traffic.
I can not see services on syslog port (514) using netstat -an
I used splunk and log server but can not see any events

**Screenshots**
n/a

**Environment (please complete the following information):**
- OS: [e.g. ubuntu 16.04, 4.15.0-8-generic] (Docker image)
- SysFlow version: [sf-collector:latest, sf-exporter:latest (26Feb2020)]
- Command line arguments and filters (if applicable): as above
- Configurations (if applicable): as above
- Container orchestration framework and version (if applicable): Docker v 18.09.7 build2d0083d running on VMware Workstation Player 15 on a Ubuntu 16.04 v 4.15.0-88 host
- Container runtime (if applicable): Docker

**Additional context**
Add any other context about the problem here.
n/a

**Files**
Attach any additional log files, config files, scripts or filters, sample sysflow, etc. that will help diagnose the problem

[docker-service-error.txt](https://github.com/sysflow-telemetry/sf-docs/files/4261823/docker-service-error.txt)
mutazalsallal commented 4 years ago

I had it before its related to the exportfield when its not defined it will break it. https://github.com/sysflow-telemetry/sf-exporter/pull/3

I think its already solved in the sf-exporter:edge version