Socket source isn't working

[TanFFFF]

• Recently I've tried Vector to read data from unix socket and send to Kafka.

• First, I tested Vector by using socket source and console sink. But there is nothing ouput.

• My /etc/vector/vector.toml is as below:

data_dir = "/var/lib/vector"
# Input data
[sources.sock]
  type = "socket"
  mode = "unix"
  path = "/root/named/etc/dnstap.sock"
# Output data
[sinks.out]
  inputs   = ["sock"]
  type     = "console"
  encoding = "text"

• And output when run Vector is:

root@ubuntu:~# /usr/bin/vector --config /etc/vector/vector.toml
Feb 29 20:49:50.488  INFO vector: Log level "info" is enabled.
Feb 29 20:49:50.489  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Feb 29 20:49:50.494  INFO vector: Vector is starting. version="0.8.0" git_version="v0.8.0" released="Tue, 25 Feb 2020 17:03:25 +0000" arch="x86_64"
Feb 29 20:49:50.495  INFO vector::topology: Running healthchecks.
Feb 29 20:49:50.495  INFO vector::topology: Starting source "sock"
Feb 29 20:49:50.495  INFO vector::topology: Starting sink "out"
Feb 29 20:49:50.500  INFO vector::topology::builder: Healthcheck: Passed.
Feb 29 20:49:50.501  INFO source{name=sock type=socket}: vector::sources::util::unix: listening. path="/root/named/etc/dnstap.sock" type="unix"
.
.
.
.
<nothing appear>
.
...

However, if I change source to file source, and console sink, it works as expect.

Does anyone has problems with socket source? Please let me know if I wrong or something.

[ghost]

I think the issue is that Vector listens for connections on the given socket path, but in order to read logs data from dnstap it is necessary to connect to the socket created by the DNS server. It seems to be a more general issue, so I've created https://github.com/timberio/vector/issues/2042 about it.

[bill-bateman]

I've been looking into getting dnstap working in Vector, and ran into this too.

What I've found is:

• dnstap expects socket to exist, and attempts to connect to it. So vector only having listen mode is fine (for this)
• dnstap uses FrameStream, and data frames are length delimited, so need to add an option to use LengthDelimitedCodec (instead of LinesCodec at current) -- I've been working on this, and hope to have a PR today or tomorrow
• Framestream has control frames (to match up encoding type) as well as data frames, and the control frames expect responses. I haven't been any to find any documentation on this, but here is the relevant framestream source code (https://github.com/farsightsec/fstrm/blob/master/src/fstrm_capture.c#L813) - there's a golang version too but I haven't found the relevant bits

Currently, I have the LengthDelimitedCodec working, and if I hook it up to the unix socket coming from dnstap I get 2 events: one empty (signifying the next is a control frame), and one saying what the content type is (protobuf:dnstap.Dnstap).

I'm gonna hack around trying to get framestream to work. So far (for the LengthDelimitedCodec) I've just been adding to the socket source (unix mode), but this is quite a bit extra so I think it would make more sense as a separate framestream source.

[bill-bateman]

Thought discussion of dnstap / framestream made more sense in a new issue #2516

[AlbertoArdu]

Hello, I agree would be useful to have another option to make vector connect to an existing unix socket instead of open a new one. My use case is use vector to forward in real-time the logs produced by auditd and dispatched by audisp via af_unix plugin. Audisp at startup opens the socket and vector tries to do the same, resulting with error:

thread 'tokio-runtime-worker' panicked at 'failed to bind to listener socket: Os { code: 98, kind: AddrInUse, message: "Address already in use" }', src/sources/util/unix.rs:38:24
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Nov 26 15:32:09.106 ERROR source{name=audit type=socket}: vector::topology: An error occurred that vector couldn't handle.

In particular, the two following sources should be equivalent:

[sources.audit_sock]
  type = "socket"
  mode = "unix"
  path = "/var/run/audispd_events"

[sources.audit_file]
  type = "file"
  include = ["/var/log/audit/audit.log"]
  start_at_beginning = false

The objective is to avoid storing on the server the logs.

[jszwedko]

I believe this is issue is covered by:

• Connecting to a unix socket instead of creating one: https://github.com/timberio/vector/issues/2042
• dnsstap support: https://github.com/timberio/vector/issues/2516

I'll close it out in-lieu of those two.

[jszwedko]

@AlbertoArdu I believe #2042 covers your issue, but let me if not.