search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.07k stars 4.11k forks source link

[platform] airbyte-minio-0 pod restarts because of storage space is full #44556

Open venkateshkelevo opened 2 months ago

venkateshkelevo commented 2 months ago

Topic

Is there way to cleanup minio storage space automatically to avoid the space issues

Relevant information

airbyte-minio-0 pod restarts because of pvc storage space is full. The job logs are going to /storage/airbyte-storage/job-logging. Is there a way to cleanup automatically to avoid the space issues?

Thanks.

marcosmarxm commented 2 months ago

Thanks @venkateshkelevo are you deploying using abctl or helm?

venkateshkelevo commented 2 months ago

I am using helm for deployment

venkateshkelevo commented 2 months ago

@marcosmarxm, Could you confirm is there any way to achieve auto cleanup of logs?

bgroff commented 1 month ago

You should be able to kubectl exec -it -n airbyte-abctl airbyte-minio-0 -- /bin/bash into the pod and remove those files.