New `cratedb` sink

[MarkusH]

CrateDB is a distributed SQL database developed and maintained by @crate. I would like to add a CrateDB sink to Vector.

I have an initial working version that is based on the existing HttpService using CrateDB's HTTP endpoint for bulk inserts.

Since I'm quite new to working with @rust-lang I would love advice and support on what to change/do better, if the integration is accepted.

There are a few issues that I don't know yet how to solve:

• CrateDB supports arrays and objects as column values. The existing json_parser transform "flattens" nested structures like this

  {
  "message": {
    "key1": {
      "k1": "v1",
      "k2": "v2"
    },
    "key2": ["v3", "v4"]
  }
  }

  to

  {
  "key1.k1": "v1",
  "key1.k2": "v2",
  "key2[0]": "v3",
  "key2[1]": "v4"
  }

  . However, I'd like the the data sent to CrateDB to be

  {
  "key1": {
    "k1": "v1",
    "k2": "v2"
  },
  "key2": ["v3", "v4"]
  }

• I'm not sure how to hook into and evaluate the response that comes back when sending requests using HttpService. CrateDB provides error codes for bulk inserts that I'd like to use for logging warnings/info messages in Vector.

[binarylogic]

Thanks @MarkusH this is great. Regarding your points:

1. @LucioFranco is working on https://github.com/timberio/vector/issues/354 which addresses your data structure issues. My hope is that this will be a low level change that will be very easy to implement across all sinks.

2. I'll let @LucioFranco chime in on the best way to handle this.

Once we get these straight, I'd recommend opening a PR for discussion so we can get it merged 😄

[MarkusH]

Thank you @binarylogic. I'm looking forward to the changes :smile:

[binarylogic]

Hey @MarkusH, we just merged #678 which should be what you need. Let me know if that helps, we'd love to review a createdb PR. 😄

[MarkusH]

Sweeet :slightly_smiling_face: Thank you @binarylogic I'll put it on my plate the next days.

[LucioFranco]

@MarkusH I totally forgot to answer this a while back, sorry! Your integration actually looks pretty good! I would say if you want to submit a PR for it and I can provide some comments. I don't expect anything major to change as it already looks pretty good! (couldn't even tell you were new to rust 😄 )