search

pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
2.84k stars 98 forks source link

[QUESTION]: Pathway support for custom AirByte connector? #24

Closed vikassinghvi2007 closed 3 months ago

vikassinghvi2007 commented 3 months ago

Question: If we were to build a custom AirByte source connector, would it get automatically supported on Pathway via AirByte Serverless? If not, what would be the turn around time for support via Pathway? Also, would there be additional work on our side towards AirByte Serverless/ PyAirByte or to use Pathway?

dxtrous commented 3 months ago

Hi @vikassinghvi2007 Yes, if you build and package your own Airbyte connector in the package format used by Airbyte Serverless, it will be ready for use by Pathway as-is, following the provided installation instructions. So e.g. forking one of the existing connectors and modifying it will work.

If your only objective is to get the connector running in Pathway, a lighter alternative is to implement the few required connector methods directly in Python - here are two approaches and examples, for Twitter API and for Polygon API: https://pathway.com/developers/user-guide/connectors/custom-python-connectors/ https://pathway.com/developers/user-guide/connectors/websockets-connectors For an alternative form of connectivity, you can also use Pathway to expose a REST endpoint (rather than a client): https://pathway.com/developers/api-docs/pathway-io/http/#pathway.io.http.rest_connector

vikassinghvi2007 commented 3 months ago

Thank you Adrian. When we build a custom one, we might reach out if we need any help. For now, this answers my question, so we can close this issue.

On Wed, 20 Mar 2024 at 1:27 PM, Adrian Kosowski @.***> wrote:

Hi @vikassinghvi2007 https://github.com/vikassinghvi2007 Yes, if you build and package your own Airbyte connector in the package format used by Airbyte Serverless, it will be ready for use by Pathway as-is, following the provided installation instructions. So e.g. forking one of the existing connectors and modifying it will work.

If your only objective is to get the connector running in Pathway, a lighter alternative is to implement the few required connectors methods directly in Python - here are two approaches and example, for Twitter API and for Polygon:

https://pathway.com/developers/user-guide/connectors/custom-python-connectors/ https://pathway.com/developers/user-guide/connectors/websockets-connectors You can also use Pathway to expose a REST endpoint (not client):

https://pathway.com/developers/api-docs/pathway-io/http/#pathway.io.http.rest_connector

— Reply to this email directly, view it on GitHub https://github.com/pathwaycom/pathway/issues/24#issuecomment-2008978612, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNJK32MPFBEUXPBB2ESASTYZE6OVAVCNFSM6AAAAABE64D222VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYHE3TQNRRGI . You are receiving this because you were mentioned.Message ID: @.***>

dxtrous commented 3 months ago

@vikassinghvi2007 by all means, and happy to help out if needed!

vikassinghvi2007 commented 3 months ago

Awesome, thank you so much!

On Wed, 20 Mar 2024 at 8:10 PM, Adrian Kosowski @.***> wrote:

@vikassinghvi2007 https://github.com/vikassinghvi2007 by all means, and happy to help out if needed!

— Reply to this email directly, view it on GitHub https://github.com/pathwaycom/pathway/issues/24#issuecomment-2009733993, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNJK36XGKFVX65OFQULM7LYZGNXDAVCNFSM6AAAAABE64D222VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBZG4ZTGOJZGM . You are receiving this because you were mentioned.Message ID: @.***>