airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.4k stars 3.97k forks source link

airbyte-cron and air byte-worker loop upon installation #29650

Closed MFIHRI closed 3 weeks ago

MFIHRI commented 1 year ago

Discussed in https://github.com/airbytehq/airbyte/discussions/29649

Originally posted by **MFIHRI** August 20, 2023 Airbyte Version: v0.50.19. Docker-version(Windows 10): **24**.0.5, build ced0996 Docker-version(Mac): **20**.10.8, build 3967b7d Running either docker-compose up or ./run-ab-platform.sh gives me hours long loop affecting airbyte-cron and airbyte-worker. Errors I noticed in the Windows log [Windows-10_Installation-log.docx](https://github.com/airbytehq/airbyte/files/12422136/Windows-10_Installation-log.docx) : 1) main ERROR Error processing element **Log4j2Appender** ([Route: null]): CLASS_NOT_FOUND (see line 1308 on Windows log) Stackflow has a mention of that error: https://stackoverflow.com/questions/36978115/log4j2-error-processing-element-appender 2) Could not find **airbyte.yml** file (see line 1457 in Windows log - error repeats downwards). 3) "Debug" loop affecting airbyte-cron and airbyte-worker starts at line **2824**. Noticeable errors on Mac [Mac-installation-log.docx](https://github.com/airbytehq/airbyte/files/12422123/Mac-installation-log.docx) : 1) Unable to connect to SQL database. (see line 119 in the Mac log attached) 2) Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx". (*App).constructCustomLogger.func2 (see lines 257-276 re building the logger (repeats at 433 and db connects at 466 - in the Mac log attached) 3) Could not find airbyte.yml file (Line 1926 - Mac log file) 4) Debug loop starts at line **1971** - attached Mac log file) ![airbyte-on-docker](https://github.com/airbytehq/airbyte/assets/14931737/f93f59f9-af1f-45fa-b729-ae48329c9aaf)
MFIHRI commented 1 year ago

It seems docker-compose up mechanism doesn't handle all exceptions well. Some file gets created on the fly, container stops/starts, writing lost and attempting the process will only lead to a loop. Plus, startup dependencies between containers is now broken. You can only retry if you had set up volumes, just a thought .....

octavia-squidington-iii commented 1 month ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

octavia-squidington-iii commented 3 weeks ago

This issue was closed because it has been inactive for 20 days since being marked as stale.