[abctl] Storage not getting created on windows

Sivivatu commented 3 days ago

What happened?

When Installing abctl on Windows (with WSL) the installer fails with the error:

  ERROR   Failed to install airbyte/airbyte Helm Chart
  ERROR   Unable to install Airbyte locally
  ERROR   could not install airbyte chart: could not install helm: failed pre-install: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed

this is being run from in the linux subsystem (WSL2) with docker installed.

full Error output:

```bash  fatpunk   abctl-v0.6.0-linux-amd64   36ms⠀   $ sudo ./abctl local install   bash  79  20:48:03  INFO Using Kubernetes provider: Provider: kind Kubeconfig: /root/.airbyte/abctl/abctl.kubeconfig Context: kind-airbyte-abctl SUCCESS Found Docker installation: version 26.1.4 SUCCESS Port 8000 appears to be available INFO No existing cluster found, cluster 'airbyte-abctl' will be created SUCCESS Cluster 'airbyte-abctl' created INFO Namespace 'airbyte-abctl' created INFO Persistent volume 'airbyte-minio-pv' created INFO Persistent volume 'airbyte-volume-db' created INFO Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created INFO Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created WARNING Encountered an issue deploying Airbyte: Pod: airbyte-db-0.17dd42dfa14c67f5 Reason: BackOff Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25) Count: 6 Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied WARNING Encountered an issue deploying Airbyte: Pod: airbyte-minio-0.17dd42de3a833a83 Reason: BackOff Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b) Count: 6 Logs: API: SYSTEM() Time: 19:49:57 UTC 06/28/2024 Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/fe0cf3cd-8cc7-4e6c-b4fc-c497812698d1) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() API: SYSTEM() Time: 19:49:57 UTC 06/28/2024 Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified WARNING Encountered an issue deploying Airbyte: Pod: airbyte-db-0.17dd42dfa14c67f5 Reason: BackOff Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25) Count: 7 Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied WARNING Encountered an issue deploying Airbyte: Pod: airbyte-minio-0.17dd42de3a833a83 Reason: BackOff Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b) Count: 7 Logs: API: SYSTEM() Time: 19:50:41 UTC 06/28/2024 Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/4f8ec462-1f1f-489f-8aa7-9b774547e91c) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() API: SYSTEM() Time: 19:50:41 UTC 06/28/2024 Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified WARNING Encountered an issue deploying Airbyte: Pod: airbyte-db-0.17dd42dfa14c67f5 Reason: BackOff Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25) Count: 8 Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied WARNING Encountered an issue deploying Airbyte: Pod: airbyte-db-0.17dd42dfa14c67f5 Reason: BackOff Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25) Count: 9 Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied WARNING Encountered an issue deploying Airbyte: Pod: airbyte-minio-0.17dd42de3a833a83 Reason: BackOff Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b) Count: 8 Logs: API: SYSTEM() Time: 19:50:41 UTC 06/28/2024 Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/4f8ec462-1f1f-489f-8aa7-9b774547e91c) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() API: SYSTEM() Time: 19:50:41 UTC 06/28/2024 Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified WARNING Encountered an issue deploying Airbyte: Pod: airbyte-minio-0.17dd42de3a833a83 Reason: BackOff Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b) Count: 9 Logs: API: SYSTEM() Time: 19:50:41 UTC 06/28/2024 Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/4f8ec462-1f1f-489f-8aa7-9b774547e91c) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() API: SYSTEM() Time: 19:50:41 UTC 06/28/2024 Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified WARNING Encountered an issue deploying Airbyte: Pod: airbyte-minio-0.17dd42de3a833a83 Reason: BackOff Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b) Count: 21 Logs: API: SYSTEM() Time: 20:27:50 UTC 06/28/2024 Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/e8d4fe46-d2b3-4685-ba9e-ee93afd548a7) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() API: SYSTEM() Time: 20:27:50 UTC 06/28/2024 Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError) 6: internal/logger/logger.go:258:logger.LogIf() 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp() 4: cmd/xl-storage.go:263:cmd.newXLStorage() 3: cmd/object-api-common.go:63:cmd.newStorageAPI() 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1() 1: github.com/minio/pkg/v2@v2.0.3-0.20231107172951-8a60b89ec9b4/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified WARNING Encountered an issue deploying Airbyte: Pod: airbyte-db-0.17dd42dfa14c67f5 Reason: BackOff Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25) Count: 23 Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied ERROR Failed to install airbyte/airbyte Helm Chart ERROR Unable to install Airbyte locally ERROR could not install airbyte chart: could not install helm: failed pre-install: 1 error occurred: * pod airbyte-abctl-airbyte-bootloader failed ```

What did you expect to happen?

Expected airbyte to install and be accessable via localhost:8000

Abctl Version

attempting to install abctl-v0.6.0-linux-amd64 ```console $ abctl version # paste output here ```

Docker Version

```console $ docker version Client: Version: 24.0.7 API version: 1.43 Go version: go1.21.1 Git commit: 24.0.7-0ubuntu2~23.10.1 Built: Wed Mar 13 20:05:50 2024 OS/Arch: linux/amd64 Context: default Server: Docker Desktop Engine: Version: 26.1.4 API version: 1.45 (minimum version 1.24) Go version: go1.21.11 Git commit: de5c9cf Built: Wed Jun 5 11:29:22 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.33 GitCommit: d2d58213f83a351ca8f528a95fbd145f5654e957 runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0 ```

OS Version

```console # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here $ wmic os get Caption, Version, BuildNumber, OSArchitecture   pwsh   79  20:25:05  BuildNumber Caption OSArchitecture Version 22631 Microsoft Windows 11 Pro 64-bit 10.0.22631 $ wsl --version   pwsh   79  09:25:54  WSL version: 2.1.5.0 Kernel version: 5.15.146.1-2 WSLg version: 1.0.60 MSRDC version: 1.2.5105 Direct3D version: 1.611.1-81528511 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.22631.3810 ``` # On Linux: $ cat /etc/os-release # paste output here $ cat /etc/os-release   bash  79  09:26:12 PRETTY_NAME="Ubuntu 23.10" NAME="Ubuntu" VERSION_ID="23.10" VERSION="23.10 (Mantic Minotaur)" VERSION_CODENAME=mantic ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=mantic LOGO=ubuntu-logo

flobr commented 3 days ago

I have exactly the same problem with Windows 10 Pro & Docker Desktop 4.31.1. I tested it with Powershell and cygwin. Both show the same errors.

duncan4b commented 2 days ago

same here, same error in windows

flobr commented 2 days ago

v0.5.0 of airbyte works (if you ignore the "Message: Readiness probe failed:" messages)

marcosmarxm commented 1 day ago

Hello everyone, 👋 thank you for reporting the issue and providing more information. @bgroff can you check this?

burningion commented 15 hours ago

Just here to make a note, had the same error experience on Ubuntu linux.

Same problem creating the pgdata directory (I had to make it manually and then chown it to my user), followed by the same error above:

Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)

airbytehq / airbyte