pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
2.84k stars 98 forks source link

encountering a ValueError when executing simple pw.io.airbyte.write #36

Closed xHishamSaeedx closed 2 months ago

xHishamSaeedx commented 2 months ago

this is the code i am trying to run

  import pathway as pw

  issues_table = pw.io.airbyte.read(
      "./connections/github.yaml",
      streams=["issues"],
  )

  pw.io.jsonlines.write(issues_table, "issues.jsonlines")
  pw.run()

this is whats in the ./connections/github.yaml

    source:
      docker_image: "airbyte/source-github" 
      config: 
        credentials:

          option_title: "PAT Credentials" 
          personal_access_token:  
        repositories: 
          - pathwaycom/pathway 
        api_url: "https://api.github.com/" 
      streams: issues 

the error i am getting for executing this is :

ValueError: Could not read json file /mnt/temp/catalog.json: Expecting value: line 1 column 1 (char 0). Please ensure that it is a valid JSON.\n", "failure_type": "system_error", "stream_descriptor": null}

KamilPiechowiak commented 2 months ago

Hey @xHishamSaeedx, I was not able to reproduce the issue. The code you provided works with the yaml config file you provided (I only used my personal access token). Please check if ./connections/github.yaml really points to the config file you pasted. Please paste the full stack trace.

xHishamSaeedx commented 2 months ago

Hey @xHishamSaeedx, I was not able to reproduce the issue. The code you provided works with the yaml config file you provided (I only used my personal access token). Please check if ./connections/github.yaml really points to the config file you pasted. Please paste the full stack trace.

Oh never mind, i ran it in sudo in Linux and it started working properly