search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.88k stars 4.07k forks source link

Airbyte doesn't compile locally #39378

Closed kikoncuo closed 4 months ago

kikoncuo commented 4 months ago

Topic

Following the documentation for local development doesn't work

Relevant information

I tried to follow the docs from https://github.com/airbytehq/airbyte/blob/eeafea7388c6b53dd4fc848d32edb07175209e58/docs/contributing-to-airbyte/resources/developing-locally.md

First off, you need to use python version 3.10 not 3.9 or running SUB_BUILD=PLATFORM ./gradlew build crashes asking you to use 3.10 Once that's updated, and I run the command again, tons of tests fail, so I ran it without tests since I'm just testing a destination SUB_BUILD=PLATFORM ./gradlew build -x test

My current problem is that at some point airbyte-ci connectors is run with the parameter --name=connectors which leads to the error

root: Setting working directory to /home/ubuntu/airbyte
pipelines: airbyte-ci is up to date. Installed version: 4.17.0. Latest version: 4.17.0
pipelines: Called with dagger run: False
pipelines.cli.dagger_run: Running command: ['/home/ubuntu/bin/dagger', 'run', '/home/ubuntu/.cache/pypoetry/virtualenvs/pipelines-CF7hAtMM-py3.10/bin/airbyte-ci', 'connectors', '--name=connectors', '--disable-report-auto-open', 'build', '--use-host-gradle-dist-tar']
1: connect
1: > in init
1: starting engine
1: starting engine [0.38s]ING [20s]
Connected to engine f96e385e1da4 (version v0.9.6)
1: starting session
1: [1.51s] OK!
1: starting session [0.45s]
1: connect DONE

3: /home/ubuntu/.cache/pypoetry/virtualenvs/pipelines-CF7hAtMM-py3.10/bin/airbyte-ci connectors --name=connectors --disable-report-auto-open build --use-host-gradle-dist-tar
root: Setting working directory to /home/ubuntu/airbyte
pipelines: airbyte-ci is up to date. Installed version: 4.17.0. Latest version: 4.17.0
pipelines: Called with dagger run: True
Usage: airbyte-ci connectors [OPTIONS] COMMAND [ARGS]...
Try 'airbyte-ci connectors --help' for help.

Error: Invalid value for '--name': 'connectors' is not one of 'source-zapier-supported-storage' ...
.
3: /home/ubuntu/.cache/pypoetry/virtualenvs/pipelines-CF7hAtMM-py3.10/bin/airbyte-ci connectors --name=connectors --disable-report-auto-open build --use-host-gradle-dist-tar ERROR: exit status 2
exit status 2

> Task :airbyte-integrations:connectors:buildConnectorImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:buildConnectorImage'.
> Process 'command '/home/ubuntu/airbyte/airbyte-integrations/connectors/build/.venv/bin/python'' finished with non-zero exit value 1

I can't find anywhere in the code where airbyte-ci connectors --name=connectors is called.

This is from a clean clone from master.

marcosmarxm commented 4 months ago

@kikoncuo for your change you only need to build the postgres connector using the airbyte-ci: Example: airbyte-ci connectors --name destination-postgres build

kikoncuo commented 4 months ago

That worked!

TCeason commented 2 days ago

@kikoncuo for your change you only need to build the postgres connector using the airbyte-ci: Example: airbyte-ci connectors --name destination-postgres build

Hi I try to exec this cmd , but return err failed to do request: Head "https://registry-1.docker.io/v2/library/docker/manifests/24-dind": dial tcp 162.125.2.3:443: i/o timeout

more info in https://github.com/airbytehq/airbyte/issues/46886