search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.15k stars 4.13k forks source link

Random "A stream status (null.<stream-name>) has been detected for a stream not present in the catalog" causes multiple retries and finally succeeds #45157

Open Januznl opened 2 months ago

Januznl commented 2 months ago

Topic

No response

Relevant information

After we upgraded from 0.58.0 to 0.63.18 we got randomly the following error "A stream status (null.) has been detected for a stream not present in the catalog".

After a few attempts the job succeeds, this mainly triggers on bigger jobs. Jobs take now 6 hours instead of 2 hours or less.

It doesn't matter which source we use, only thing in common is the destination (Bigquery) we use version 2.8.8.

We have tried upgrading to 0.63.20, but no difference, still all bigger jobs are retrying...

amazon ads -> bigquery A stream status (null.sponsored_display_report_stream) has been detected for a stream not present in the catalog

Custom source -> bigquery A stream status (null.reporting) has been detected for a stream not present in the catalog

Custom source → BigQuery A stream status (null.registered_hours) has been detected for a stream not present in the catalog

Microsoft SQL Server <> BigQuery A stream status (dbo.Messages) has been detected for a stream not present in the catalog

It feels like an issue with airbyte platform itself as multiple jobs fail but eventually succeed.

evantahler commented 2 months ago

cc @benmoriceau

Januznl commented 2 months ago

Based on comments in ticket #40555 we have lowered our destination-bigquery connector to version 2.6.2 and now its working again.

We are running on GKE based on the docker hub images and self-made manifests.

evantahler commented 2 months ago

Can you share your logs from failing sync please?

evantahler commented 2 months ago

@Januznl can you also please confirm that you are running airbyte via abctl or helm? Docker Compose is depreciated.

Januznl commented 2 months ago

We are running on a custom Kubernetes setup based on the helm charts. So technically we are running on helm.

jpgianfaldoni commented 2 months ago

Im having the same problem with MySQL connector v3.7.1, running airbyte on EC2

jdpgrailsdev commented 1 month ago

@Januznl We have committed a fix that should address the issue with the "missing stream". This fix is in release v0.64.5. Please update to that release or any later version to see if it addresses the issue for you. Thank you for your patience!

Januznl commented 1 month ago

@jdpgrailsdev Thanks! We will try to upgrade next week, and i will let you know!

Januznl commented 1 month ago

@jdpgrailsdev We got a little delayed, we keep you posted once we have upgraded