ConduitIO / conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
https://conduit.io
Apache License 2.0
401 stars 50 forks source link

Epic: Better onboarding experience #1920

Open raulb opened 1 month ago

raulb commented 1 month ago

This issue encompasses:

After this, we should be ready to reach out to users and verify that the onboarding experience makes sense, based on the feedback we can continuously improve it.

Tasks:

lovromazgon commented 1 month ago

Draft idea for the quickstart guide:

  1. Install Conduit
  2. Run it
    • Explain that running Conduit now results in an empty instance.
    • Only useful if you want to interact with Conduit via the API.
  3. Build your first pipeline
    • First pipeline should use connectors that don't require external resources.
    • generator -> file is a good candidate.
      • The log destination is another candidate, but file is better, since the records land outside of Conduit and get persisted (as opposed to the log), so it's easier to understand what Conduit is doing.
    • Ideally, we should have a conduit init command in this step.
    • Run Conduit and see data flowing at this stage.
  4. Add your first processor
    • Mask a field or similar. We should identify a simple use case, but one that actually reflects some common processing use case.
    • Restart Conduit and see the updated data flow again.
  5. Refer to next steps
    • Discover connectors
    • Discover advanced features (e.g. DLQ, batching, multiple collections, WASM processors)
hariso commented 3 weeks ago

FTR, Lovro's suggestion has been split into two pages: one for the getting started guide and one for how to add a processor to a pipeline.