Request a connector to materialize to Tinybird Events API

[sdairs]

System Name

Tinybird Events API

Type

Materialize

Details

The Tinybird Events API is an HTTP API that accepts POST requests with NDJSON formatted payload.

https://www.tinybird.co/docs/guides/high-frequency-ingestion.html

To configure, it would need to expose the following options:

1. Tinybird API - Dropdown Select giving the options of [EU, US] which would configure the connector to use either [https://api.tinybird.co/, https://api.us-east.tinybird.co/] respectively
2. Data Source Name - a String for the table name that the data will arrive in
3. Auth Token - sensitive/password field, for the JWT with permissions to append data, used as a Bearer token in the auth header
4. Batch Size - optional, an integer to control batch sizing if the user wishes to batch events before sending

There are no client libraries/SDKs needed, here's some basic examples of sending an event to the Events API.

Rust:

use reqwest::blocking::Client;
use serde_json::json;

fn main() {
    let client = Client::new();
    let res = client.post("https://api.tinybird.co/v0/events?name=events_example")
        .header("Authorization", "Bearer  <AUTH TOKEN>")
        .json(&json!({
  "timestamp": "2022-10-27T11:43:02.099Z",
  "transaction_id": "8d1e1533-6071-4b10-9cda-b8429c1c7a67",
  "name": "Bobby Drake",
  "email": "bobby.drake@pressure.io",
  "age": 42,
  "passport_number": 3847665,
  "fligh_from": "Barcelona",
  "flight_to": "London",
  "extra_bags": 1,
  "flight_class": "economy",
  "priority_boarding": false,
  "meal_choice": "vegetarian",
  "seat_number": "15D",
  "airline": "Red Balloon"
}))
        .send()
        .unwrap();

    let body = res.text().unwrap();
    println!("{}", body);
}

Go:

package main

import (
    "bytes"
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

func main() {
    url := "https://api.tinybird.co/v0/events?name=events_example"
    fmt.Println("URL:>", url)

    var jsonStr = []byte(`{"timestamp":"2022-10-27T11:43:02.099Z","transaction_id":"8d1e1533-6071-4b10-9cda-b8429c1c7a67","name":"Bobby Drake","email":"bobby.drake@pressure.io","age":42,"passport_number":3847665,"fligh_from":"Barcelona","flight_to":"London","extra_bags":1,"flight_class":"economy","priority_boarding":false,"meal_choice":"vegetarian","seat_number":"15D","airline":"Red Balloon"}`)
    req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
    req.Header.Set("Authorization", "Bearer <AUTH TOKEN>")
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{Timeout: time.Second * 10}
    resp, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, _ := ioutil.ReadAll(resp.Body)
    fmt.Println(string(body))
}

[mdibaiee]

@sdairs thanks for the detailed request, I will bring this up to our team and we will get back to you.

[mdibaiee]

@sdairs do you mind elaborating more on your use case? What kind of data are you looking to capture and materialise to Tinybird? If you want we can discuss more on our slack (you can find it on our website https://www.estuary.dev/) or continue here.

[sdairs]

@mdibaiee I work at Tinybird, so we'd love to enable our customers to ingest data with Estuary (all sorts of use cases) :)

[mdibaiee]

@sdairs I see, we are at the moment pushing for an open beta of our product by end of this month, and we can come back to you with an update after that. 👍🏽