pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
2.84k stars 98 forks source link

Document how to run Airbyte connectors (via Airbyte Serverless) with Pathway inside Docker #14

Open dxtrous opened 3 months ago

dxtrous commented 3 months ago

Is your feature request related to a problem? Please describe. Airbyte Serverless which is used inside https://pathway.com/developers/api-docs/pathway-io/airbyte relies on Docker. Dockerizing a Docker app causes Docker in Docker ("DinD") needs a choice of specific setup flags: https://medium.com/@shivam77kushwah/docker-inside-docker-e0483c51cc2c These are not explained in Pathway documentation, causing questions / confusion.

Describe the solution you'd like Choose the preferred resolution for DinD. Mention that dockerization needs special attention (inside pw.io.airbyte docs). Link to tutorial with this resolution - e.g. include in guide or add to https://pathway.com/developers/showcases/etl-python-airbyte as subsection.

Describe alternatives you've considered n/a

Additional context This might also help workaround situations like #13.