search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.54k stars 4k forks source link

Include connection ID in worker sync logs #10670

Open hbd opened 2 years ago

hbd commented 2 years ago

Tell us about the problem you're trying to solve

Today, I have a hard time finding logs for a single connection/sync job. There isn't a way to grep logs for the sync itself. The best approach, as far as I can tell, is to tail all logs and parse them manually.

Describe the solution you’d like

It would be great if logs for sync jobs included the connection ID (or any other relevant metadata, like source and destination IDs). This way, one could simply grep the logs, e.g. docker-compose logs -f | grep '[connection-id]'

Describe the alternative you’ve considered or used

I manually/visually parse logs today.

Are you willing to submit a PR?

Yes

marcosmarxm commented 2 years ago

What type of logs are you trying to fetch? API requests or those generated by the sync?

hbd commented 2 years ago

Ideally both. In a debug scenario where a sync is failing, I'd like to get all of the logs related to an individual connection, source, and/or destination.