DataCater / datacater

The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
https://datacater.io
Other
82 stars 4 forks source link

Feat(pipeline): create initial pipeline implementation #9

Closed ChrisRousey closed 1 year ago

ChrisRousey commented 2 years ago

This PR is for the implementation of the Datacater Pipeline Concept.

The defined test will currently not work, as there is a bug in the currently deployed python-runner on the /batch endpoint. I fixed the endpoint in the PR.

To test manually, the python runner and be run locally:

  1. pip3 install -r requirements.txt
  2. python3 -m uvicorn runner:app

And by then setting environment variables:

HknLof commented 1 year ago

Thanks for the feature :)