pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
2.84k stars 98 forks source link

how to change sync mode from incremental to full refresh #37

Closed xHishamSaeedx closed 2 months ago

xHishamSaeedx commented 2 months ago

this is the code i am trying to execute

import pathway as pw

issues_table = pw.io.airbyte.read(
    "./connections/github.yaml",
    streams=["tags"],
)

pw.io.jsonlines.write(issues_table, "issues.jsonlines")

pw.run()

and this is the error i am receiving

  Traceback (most recent call last):
    File "/home/hisham/Documents/GitHub/github-service/github.py", line 3, in <module>
      issues_table = pw.io.airbyte.read(
    File "/usr/local/lib/python3.10/dist-packages/pathway/io/airbyte/__init__.py", line 280, in read
      raise ValueError(f"Stream {name} doesn't support 'incremental' sync mode")
  ValueError: Stream tags doesn't support 'incremental' sync mode

how do i change the sync mode ??

KamilPiechowiak commented 2 months ago

Hey @xHishamSaeedx, Currently, the airbyte connector supports only incremental mode (see the docs). The full refresh mode is planned but not yet available.

xHishamSaeedx commented 2 months ago

ok thanks!