[helm] Minio create bucket : permission denied

abusutil commented 3 months ago

Helm Chart Version

0.87.4

What step the error happened?

Upgrading the Platform or Helm Chart

Relevant information

Hello everyone,

I'm trying to upgrade Airbyte from 0.54.132 to 0.87.4 (also tried 0.83.6) and I have an issue with the minio-create-bucket pod. I trying by remove the PersistentVolumeClaim for my external PSQL and the Minio service, and same message.

I didn't have this with the 0.54.132 chart.

Do you know which can cause this issue ?

Relevant log output

airbyte-minio-create-bucket :

...waiting...
/usr/bin/mc: line 1: can't create �: Permission denied
/usr/bin/mc: line 1: �: not found
/usr/bin/mc: line 1: �: not found
/usr/bin/mc: line 1: ���PpQ�td@,�: not found
/usr/bin/mc: line 2: can't open �: no such file
/usr/bin/mc: line 3: syntax error: unexpected word (expecting ")")
...waiting...
/usr/bin/mc: line 1: can't create �: Permission denied
/usr/bin/mc: line 1: �: not found
/usr/bin/mc: line 1: �: not found
/usr/bin/mc: line 1: ���PpQ�td@,�: not found
/usr/bin/mc: line 2: can't open �: no such file
/usr/bin/mc: line 3: syntax error: unexpected word (expecting ")")

airbyte-minio :

MinIO Object Storage Server
Copyright: 2015-2024 MinIO, Inc.
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Version: RELEASE.2024-05-10T01-41-38Z (go1.22.3 linux/arm64)
API: http://10.1.144.241:9000  http://127.0.0.1:9000
WebUI: http://10.1.144.241:33471 http://127.0.0.1:33471
Docs: https://min.io/docs/minio/linux/index.html
Status:         1 Online, 0 Offline.
STARTUP WARNINGS:
- The standard parity is set to 0. This can lead to data loss.

marcosmarxm commented 3 months ago

What command are you running @abusutil ?

abusutil commented 3 months ago

@marcosmarxm Anything, I use the Airbyte helm chart. Everything is deployed correctly (Minio StatefulSet/Service included), but there is an issue with the specific pod at the end of the deployment chart.

I tried to perform commands directly inside the pod with kubectl exec -it -n airbyte airbyte airbyte-minio-create-bucket -- sh and just run the mc bin and same issue.

user¾MacProM2- github % kubectl exec -it -n airbyte airbyte-minio-create-bucket -- sh 
/ $ mc
/usr/bin/mc: line 1: can't create ?: Permission denied
/usr/bin/mc: line 1: ?: not found
/usr/bin/mc: line 1: ?: not found
/usr/bin/mc: line 1: ???PpQ?t¾,?: not found
/usr/bin/mc: line 2: can't open ?: no such file
/usr/bin/mc: line 3: syntax error: unexpected word (expecting ")")

airbyte-minio-0 logs:

Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
MinIO Object Storage Server
Copyright: 2015-2024 MinIO, Inc.
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Version: RELEASE.2024-05-10T01-41-38Z (go1.22.3 linux/arm64)
API: http://10.1.145.113:9000  http://127.0.0.1:9000
WebUI: http://10.1.145.113:39619 http://127.0.0.1:39619
Docs: https://min.io/docs/minio/linux/index.html
Status:         1 Online, 0 Offline.
STARTUP WARNINGS:
- The standard parity is set to 0. This can lead to data loss.
 You are running an older version of MinIO released 2 weeks before the latest release
 Update: Run `mc admin update ALIAS`

abusutil commented 3 months ago

@marcosmarxm Also, we using a ARM Base AWS EC2 as nodes on this cluster. I have another cluster with exactly same configuration (ArgoCD ApplicationSet), and I don't have any Minio issues. But, it's using x86_64 nodes.

I found a opened PR to let this minio-create-bucket pod follow minio svc settings.

Could be the root cause ?

Edit : I confirm is the root cause. I changed my node to X86_64 and the airbyte-minio-create-bucket ran good without any error.

jprpai commented 3 months ago

@marcosmarxm any updates on this issue? merging the PR would be very helpful. In my case I am unable to run the minio bucket pod as well as we can't set tolerations on the pod.

whanata commented 1 month ago

I'm encountering the same issue when trying to run Airbyte locally using kind on a Arm 64 EC2 Box.

justinmiller61 commented 1 month ago

I'm running into this too. Also on Arm64 (apple silicone).

AdarshKSharma commented 1 month ago

modify file docker\airbyte-mc\Dockerfile as per following: ARG ALPINE_IMAGE_VERSION=3.18 FROM alpine:${ALPINE_IMAGE_VERSION}

RUN adduser -u 1000 -s /bin/sh -D airbyte

RUN wget https://dl.min.io/client/mc/release/linux-arm64/mc && \ mv mc /usr/bin/mc && \ chmod +x /usr/bin/mc

USER airbyte:airbyte

and build local image use it in chart

whanata commented 1 month ago

I just solved this problem by using an external logging bucket, that seems to work.

chulkilee commented 1 month ago

Where is the Dockerfile of airbyte/mc image? The image is multi-arch image but arm64 image has amd64 binary.

sha256sum /usr/bin/mc 
e8e7732b119eadf6515af8ac814e944587352b5aeaa6e64c21f126ef763129e3  /usr/bin/mc

Do we ever need to use custom mc? The minio image does contain mc, so why don't we just use it with command option?

--

Edit: I found the Dockerfile - created a PR to add arm64 support

https://github.com/airbytehq/airbyte-platform/pull/345

jonseymour commented 1 day ago

abctl should not/cannot be used with AWS t4 images until this issue is fixed in the released version.

airbytehq / airbyte