faros-ai / faros-community-edition

BI, API and Automation layer for your Engineering Operations data
https://community.faros.ai
Apache License 2.0
423 stars 58 forks source link

Error during sync from Gitlab connector: Found multiple possible origins from catalog #228

Closed batjko closed 2 years ago

batjko commented 2 years ago

Describe the bug Unable to sync from Gitlab to Faros destination.

After setting up Faros CE locally, starting it all up using ./start.sh, I set up a Source (Gitlab) and Destination (Faros, which I assume is the safest to start with, just to try things out). This appeared to have been successful.

But when I try to sync in Airbyte, it keeps throwing me this error: Encountered an error while writing to destination: VError: Found multiple possible origins from catalog

Log entry:

2022-10-18 20:15:41 destination > Encountered an error while writing to destination: VError: Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users - {"jse_shortmsg":"Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users","jse_info":{},"message":"Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users"}
2022-10-18 20:15:41 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):415 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@35819db0[type=TRACE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,state=<null>,trace=io.airbyte.protocol.models.AirbyteTraceMessage@3b466570[type=ERROR,emittedAt=1.666124141733E12,error=io.airbyte.protocol.models.AirbyteErrorTraceMessage@3f79f6c0[message=Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users,internalMessage=VError,stackTrace=VError: Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users
    at FarosDestination.getOrigin (/home/node/airbyte/destinations/airbyte-faros-destination/lib/destination.js:269:19)
    at FarosDestination.write (/home/node/airbyte/destinations/airbyte-faros-destination/lib/destination.js:277:29)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Command.<anonymous> (/home/node/airbyte/faros-airbyte-cdk/lib/destinations/destination-runner.js:67:34)
    at async Command.parseAsync (/home/node/airbyte/node_modules/commander/lib/command.js:917:5),failureType=system_error,additionalProperties={jse_shortmsg=Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users, jse_info={}}],additionalProperties={}],additionalProperties={}]
2022-10-18 20:15:41 ERROR i.a.w.g.DefaultReplicationWorker(run):180 - Sync worker failed.

In the destination's replication settings, all streams are all enabled (was by default, I didn't change anything):

image

To Reproduce Steps to reproduce the behavior:

  1. As per Quickstart, I cloned the repo and ran ./start.sh
  2. Set up Gitlab source (self-hosted, v15.4.1) in Airbyte
  3. Select Faros Destination for the connection.
  4. Navigate to Connection and try to sync.
  5. See error and be sad.

image

Expected behavior Successful sync, so I can see my data in a Metabase dashboard.

Deployment

thomas-gerber commented 2 years ago

@batjko you are missing the table prefix setting.

Note that Faros CE should already have a canned connections at startup between GitLab and Faros. Take a look, it is already configured properly: link.

batjko commented 2 years ago

you are missing the table prefix setting.

Ah interesting, that was not obvious in the UI, as the field is optional. I'll try that.

Note that Faros CE should already have a canned connections at startup between GitLab and Faros. Take a look, it is already configured properly: link.

I did not see this. I kinda didn't learn how Airbyte works, and the UI was very confusing, I have to say. So I just followed the quickstart.

Thanks for your help, @thomas-gerber .