Weird issue where files are not pushed to s3

jgournet commented 6 months ago

Platform Version

0.58.1

What step the error happened?

During the Sync

Relevant information

Setup:

zendesk connection to dump files to s3

What happened:

We did a sync with Zendesk.
We realized that some fields were set as "Full refresh | Overwrite", so we moved them to "Incremental"
Airbyte asked "do you want to clear specific stream data ?" => yes
We launched a sync again: Sync was marked as successful in 1min, with 400MB downloaded (ie: airbyte used its cache I guess ?), BUT files were not sent to S3 ! Even though the logs clearly showed the "s3 upload" lines.
We launched a sync again, in doubt: Sync was marked as successful in 4s, with 1.6G downloaded - but still no files in S3.

That's the gist of the issue A few more fun things:

we nuked the airbyte+temporal databases, re-setup destination,source,connection, and sync'd again: same results ! To recover fully, we finally had to kill minio PVC+PV+EBS Volume (aws disk), nuke the database again and launch again.

Relevant log output

No response

marcosmarxm commented 6 months ago

After nuke the deployment did the sync started to work again?

jgournet commented 6 months ago

Nuking the deployment did not do enough as minio pod and its PVC+PV were somehow kept.

So the fix was: we deleted the deployment, then deleted minio pod, deleted the PVC, deleted the PV, deleted the EBS, and then started again a new deployment - then it worked

Today, I had to the same again ... and minio was properly killed

marcosmarxm commented 6 months ago

Ok, thanks for the info @jgournet I added this to the deployment team backlog.

octavia-squidington-iii commented 19 hours ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

airbytehq / airbyte