search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.83k stars 4.06k forks source link

Minio StatefulSet is deleted and recreated on each `helm upgrade` #29613

Open don-code opened 1 year ago

don-code commented 1 year ago

What method are you using to run Airbyte?

Kubernetes

Platform Version or Helm Chart Version

0.45.35

What step the error happened?

On deploy

Revelant information

We are using Airbyte with the built-in Minio for logging, as we cannot yet use IRSA roles to log directly to Se (see #10570).

The Minio StatefulSet's template in the Helm chart is defined as a pre-upgrade hook: https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/templates/minio.yaml#L8. As a result of defining it this way, it is not simply reapplied on every upgrade - it is instead deleted and recreated, as per the expected behavior of chart hooks.

The end result is that Minio becomes unavailable for several seconds during a deploy, causing some logging issues for any in-progress jobs. Logs are not lost, since the PersistentVolume underlying it is not automatically deleted.

Ideally, Helm should use its normal behavior of patching the StatefulSet in place, rather than destroying and recreating it.

Relevant log output

To prove out that this was happening, I ran kubectl get pods --watch while running a helm upgrade. This shows Minio being destroyed and recreated, which leads to downtime.

airbyte-minio-0                                    1/1     Terminating   0          91m
airbyte-airbyte-bootloader                         0/1     Terminating   0          91m
airbyte-airbyte-bootloader                         0/1     Terminating   0          91m
airbyte-airbyte-bootloader                         0/1     Pending       0          0s
airbyte-airbyte-bootloader                         0/1     Pending       0          0s
airbyte-minio-0                                    0/1     Terminating   0          91m
airbyte-minio-0                                    0/1     Terminating   0          91m
airbyte-minio-0                                    0/1     Terminating   0          91m
airbyte-airbyte-bootloader                         0/1     ContainerCreating   0          0s
airbyte-minio-0                                    0/1     Pending             0          0s
airbyte-minio-0                                    0/1     Pending             0          0s
airbyte-minio-0                                    0/1     ContainerCreating   0          1s
airbyte-airbyte-bootloader                         1/1     Running             0          2s
airbyte-minio-0                                    1/1     Running             0          9s
airbyte-airbyte-bootloader                         0/1     Completed           0          17s
airbyte-airbyte-bootloader                         0/1     Completed           0          19s

The same behavior is shown with the StatefulSet itself.

don-code commented 1 year ago

This is especially bad due to #27750. Not only does Minio restart, but the job logs are all lost on restart. (My earlier comment that logs are not lost was incorrect. The logs still show up in the UI, but when attempting to download them, they are all zero bytes.)