PrefectHQ / prefect-airbyte

https://PrefectHQ.github.io/prefect-airbyte/
Apache License 2.0
40 stars 6 forks source link

add `AirbyteConnection`, `AirbyteSync` and `run_connection_sync` #44

Closed zzstoatzz closed 1 year ago

zzstoatzz commented 1 year ago

Adds a AirbyteConnection(JobBlock) and AirbyteSync(JobRun) enabling the new flow run_connection_sync.

Adds a new client method get_job_info to allow inclusion of records_synced in the AirbyteSyncResult.

Closes #22

Example

from prefect import flow
from prefect_airbyte.server import AirbyteServer
from prefect_airbyte.connections import AirbyteConnection
from prefect_airbyte.flows import run_connection_sync

server = AirbyteServer()

connection = AirbyteConnection(
    airbyte_server=server,
    connection_id="7e78757d-fd7f-421b-80a5-f329dbdadf1b",
    status_updates=True,
)

@flow
def airbyte_syncs():
    # do some setup

    sync_result = run_connection_sync(
        airbyte_connection=connection,
    )

    # do some other things, like trigger DBT based on number of records synced
    print(f'{sync_result.records_synced}')

Screenshots

image image

Checklist