search

runfinch / finch

The Finch CLI is an open source client for container development
https://www.runfinch.com
Apache License 2.0
3.47k stars 87 forks source link

fix: Allow to use the COMPOSE_FILE variable in finch compose #994

Closed haytok closed 1 day ago

haytok commented 5 days ago

nerdctl allows us to launch containers using a Docker Compose file specified in the COMPOSE_FILE environment variable.

For example, suppose we have created the following Docker Compose file.

> cat a.yaml
services:
  test:
    image: amazonlinux:2023

By specifying this Docker Compose file in the COMPOSE_FILE environment variable and running nerdctl compose up, we can start the container.

> sudo COMPOSE_FILE=a.yaml _output/nerdctl compose up
INFO[0000] Ensuring image amazonlinux:2023
INFO[0000] Re-creating container nerdctl-test-1
INFO[0000] Attaching to logs
INFO[0000] Container "nerdctl-test-1" exited
INFO[0000] All the containers have exited
INFO[0000] Stopping containers (forcibly)
INFO[0000] Stopping container nerdctl-test-1

However, since the COMPOSE_FILE environment variable is not passed in finch compose, the following error occurs.

> COMPOSE_FILE=a.yaml finch compose up
FATA[0000] no configuration file provided: not found
FATA[0000] exit status 1

And this bug is reported in the following issue.

Therefore, this fix allows the finch compose command to use the Docker Compose file specified in the COMPOSE_FILE environment variable.

Issue #, if available: #347

Description of changes: The details are described in this commit message.

Testing done: Yes

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

haytok commented 5 days ago

A test of the finch compose command using the COMPOSE_FILE environment variable will be added to the common-tests repository.

haytok commented 5 days ago

Added an e2e test for finch compose command with Docker Compose file specified in COMPOSE_FILE.

haytok commented 5 days ago

CI in windows-e2e-tests is failing.

The error messages are as follows.

Details

``` ------------------------------ Additional disk Retains container user data after the VM is deleted C:/actions-runner/_work/finch/finch/e2e/vm/additional_disk_test.go:25 time="2024-06-24T04:00:16Z" level=debug msg="Creating limactl command: ARGUMENTS: [stop --force finch], LIMA_HOME: C:\\actions-runner\\_work\\finch\\finch\\_output\\lima\\data" time="2024-06-24T04:00:16Z" level=info msg="Forcibly stopping Finch virtual machine..." time="2024-06-24T04:00:16Z" level=debug msg="running detach cmd: C:\\Program Files\\WSL\\wsl.exe --unmount \\\\?\\C:\\Users\\Administrator\\AppData\\Local\\.finch\\.disks\\1bdf7778552c21f8.vhdx" time="2024-06-24T04:00:16Z" level=info msg="Finch virtual machine stopped successfully" time="2024-06-24T04:00:17Z" level=debug msg="running detach cmd: C:\\Program Files\\WSL\\wsl.exe --unmount \\\\?\\C:\\Users\\Administrator\\AppData\\Local\\.finch\\.disks\\1bdf7778552c21f8.vhdx" time="2024-06-24T04:00:17Z" level=debug msg="Creating limactl command: ARGUMENTS: [remove --force finch], LIMA_HOME: C:\\actions-runner\\_work\\finch\\finch\\_output\\lima\\data" time="2024-06-24T04:00:17Z" level=info msg="Forcibly removing Finch virtual machine..." time="2024-06-24T04:00:18Z" level=info msg="Finch virtual machine removed successfully" panic: test timed out after 2h0m0s running tests: TestVM (2h0m0s) goroutine 85 [running]: testing.(*M).startAlarm.func1() C:/Program Files/Go/src/testing/testing.go:2366 +0x385 created by time.goFunc C:/Program Files/Go/src/time/sleep.go:177 +0x2d goroutine 1 [chan receive, 119 minutes]: testing.(*T).Run(0xc00018dba0, {0x11c9727?, 0xc01b468244?}, 0x1213a70) C:/Program Files/Go/src/testing/testing.go:1750 +0x3ab testing.runTests.func1(0xc00018dba0) C:/Program Files/Go/src/testing/testing.go:2161 +0x37 testing.tRunner(0xc00018dba0, 0xc0000f3c70) C:/Program Files/Go/src/testing/testing.go:1689 +0xfb testing.runTests(0xc0000085a0, {0x169db40, 0x1, 0x1}, {0x1?, 0xced493?, 0x16d4680?}) C:/Program Files/Go/src/testing/testing.go:2159 +0x445 testing.(*M).Run(0xc000256640) C:/Program Files/Go/src/testing/testing.go:2027 +0x68b main.main() _testmain.go:47 +0x16c goroutine 6 [runnable]: context.WithDeadlineCause({0x12babd0, 0x175cf20}, {0x66790b3b?, 0x2057cbbc?, 0x16d4680?}, {0x0, 0x0}) C:/Program Files/Go/src/context/context.go:642 +0x268 context.WithDeadline(...) C:/Program Files/Go/src/context/context.go:612 golang.org/x/net/context.WithDeadline({0x12babd0?, 0x175cf20?}, {0x16d4680?, 0x12a05f200?, 0x16d4680?}) C:/Users/Administrator/go/pkg/mod/golang.org/x/net@v0.26.0/context/go17.go:47 +0x29 golang.org/x/net/context.WithTimeout({0x12babd0, 0x175cf20}, 0x12a05f200) C:/Users/Administrator/go/pkg/mod/golang.org/x/net@v0.26.0/context/go17.go:62 +0x47 github.com/onsi/ginkgo/v2/internal.(*Suite).generateProgressReport(_, _) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:373 +0x15d github.com/onsi/ginkgo/v2/internal.(*Suite).runNode(_, {0x16, 0x4, {0x11e9df4, 0x33}, 0xc000270880, {{0x136d407, 0x42}, 0x19, {0x0, ...}, ...}, ...}, ...) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:959 +0x119f github.com/onsi/ginkgo/v2/internal.(*group).attemptSpec(0xc0002c9130, 0x1, {{0xc00029a008?, 0xc00049a000?, 0x2?}, 0x0?}) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/group.go:199 +0xc33 github.com/onsi/ginkgo/v2/internal.(*group).run(0xc0002c9130, {0xc00040c060?, 0x0?, 0x0?}) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/group.go:349 +0x925 github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc0000d1508, {0x11dc329, 0x1f}, {0x175cf20, 0x0, 0x0}, {0xc00001f530, 0x2a}, 0x0, {0xc0002ae008, ...}) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:489 +0xac5 github.com/onsi/ginkgo/v2/internal.(*Suite).Run(_, {_, _}, {_, _, _}, {_, _}, _, {0x12be4d0, ...}, ...) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:130 +0x405 github.com/onsi/ginkgo/v2.RunSpecs({0x12b6600, 0xc00018dd40}, {0x11dc329, 0x1f}, {0x0, 0x0, 0x0}) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/core_dsl.go:300 +0x8b3 github.com/runfinch/finch/e2e/vm.TestVM(0xc00018dd40) C:/actions-runner/_work/finch/finch/e2e/vm/vm_windows_test.go:55 +0x1ca testing.tRunner(0xc00018dd40, 0x1213a70) C:/Program Files/Go/src/testing/testing.go:1689 +0xfb created by testing.(*T).Run in goroutine 1 C:/Program Files/Go/src/testing/testing.go:1742 +0x390 goroutine 7 [syscall, 119 minutes]: os/signal.signal_recv() C:/Program Files/Go/src/runtime/sigqueue.go:152 +0x29 os/signal.loop() C:/Program Files/Go/src/os/signal/signal_unix.go:23 +0x13 created by os/signal.Notify.func1.1 in goroutine 6 C:/Program Files/Go/src/os/signal/signal.go:151 +0x1f goroutine 8 [select, 119 minutes]: github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0x0?) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/interrupt_handler/interrupt_handler.go:131 +0x7d created by github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 6 C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/interrupt_handler/interrupt_handler.go:128 +0x165 goroutine 9 [select, 119 minutes]: github.com/onsi/ginkgo/v2/internal.RegisterForProgressSignal.func1() C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/progress_report.go:32 +0x8b created by github.com/onsi/ginkgo/v2/internal.RegisterForProgressSignal in goroutine 6 C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/progress_report.go:30 +0xd9 goroutine 98 [syscall, 119 minutes, locked to thread]: syscall.SyscallN(0x7ff98b573bf0?, {0xc000423a48?, 0x3?, 0x0?}) C:/Program Files/Go/src/runtime/syscall_windows.go:544 +0x107 syscall.Syscall(0x3?, 0x3?, 0xc00040ea60?, 0x0?, 0x0?) C:/Program Files/Go/src/runtime/syscall_windows.go:482 +0x35 syscall.WaitForSingleObject(0x304, 0xffffffff) C:/Program Files/Go/src/syscall/zsyscall_windows.go:1142 +0x5d os.(*Process).wait(0xc000339980) C:/Program Files/Go/src/os/exec_windows.go:18 +0x50 os.(*Process).Wait(...) C:/Program Files/Go/src/os/exec.go:134 os/exec.(*Cmd).Wait(0xc0001e6420) C:/Program Files/Go/src/os/exec/exec.go:897 +0x45 os/exec.(*Cmd).Run(0xc0001e6420) C:/Program Files/Go/src/os/exec/exec.go:607 +0x2d github.com/runfinch/finch/e2e/vm.init.func6(0xc00024d1d0) C:/actions-runner/_work/finch/finch/e2e/vm/vm_test.go:32 +0x2f9 github.com/runfinch/finch/e2e/vm.init.func8.1.1() C:/actions-runner/_work/finch/finch/e2e/vm/additional_disk_test.go:26 +0x3a github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x0?, 0x0?}) C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/node.go:472 +0x13 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3() C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:894 +0x8d created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 6 C:/Users/Administrator/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:881 +0xdb0 FAIL github.com/runfinch/finch/e2e/vm 7200.073s FAIL make: *** [test-e2e-vm-serial] Error 1 Error: Process completed with exit code 1. ##[debug]Finishing: Run e2e tests ```

CI is failing due to the following logic not completing successfully, possibly Flaky Test.

This may be related to the draft that austinvazquez is creating.

However, the root cause of this CI failure is not clear at this time.

Ask/Question

I would appreciate any guidance from other maintainers regarding appropriate approaches to addressing this matter.

pendo324 commented 4 days ago

@haytok Can you bump common-tests to v0.7.23 in this PR?

haytok commented 4 days ago

Sure, i'll update go.mod and go.sum.

haytok commented 4 days ago

Thanks for checking!!!, @pendo324

I bumped common-tests to v0.7.23.

pendo324 commented 4 days ago

Thanks, the change looks good to merge after the new test passes

pendo324 commented 4 days ago

Looks like the new test failed on Windows

haytok commented 4 days ago

I'll check the root cause of the failure.

haytok commented 4 days ago

I will demonstrate the results of the operational verification conducted in the Windows environment.

Details

Set COMPOSE_FILE env var ```bash ~ $env:COMPOSE_FILE = "a.yaml" ``` Check env var ```bash ~ Get-ChildItem env: Name Value ---- ----- ... COMPOSE_FILE a.yaml ... ``` Execute finch compose up ```bash ~ .\_output\bin\finch.exe compose up INFO[0000] Creating network finch_default INFO[0000] Ensuring image busybox ... INFO[0000] Creating container finch-echo_env-1 INFO[0000] Attaching to logs echo_env-1 |PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin echo_env-1 |DEFAULT=1 echo_env-1 |HOME=/root INFO[0000] Container "finch-echo_env-1" exited INFO[0000] All the containers have exited INFO[0000] Stopping containers (forcibly) INFO[0000] Stopping container finch-echo_env-1 ```

It was possible to execute the "finch compose up" command using the Docker Compose file (a.yaml) specified with the COMPOSE_FILE variable.

Therefore, I will proceed to investigate the tests in the Windows environment.

haytok commented 4 days ago

There seemed to be an error because the process to convert the file path specified by the environment variable in the e2e test to a path in WSL was not performed.

I'll think about how to resolve it.

haytok commented 2 days ago

I found out why the test is failing. The path to the file specified in COMPOSE_FILE in Windows environment needs to be converted to a WSL path. So, the following patch is needed.

Details

```bash > git diff .\cmd\finch\nerdctl.go diff --git a/cmd/finch/nerdctl.go b/cmd/finch/nerdctl.go index f363171..820ef70 100644 --- a/cmd/finch/nerdctl.go +++ b/cmd/finch/nerdctl.go @@ -234,13 +234,17 @@ func (nc *nerdctlCommand) run(cmdName string, args []string) error { passedEnvs := []string{ "COSIGN_PASSWORD", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", - "AWS_SESSION_TOKEN", + "AWS_SESSION_TOKEN", "COMPOSE_FILE", } var passedEnvArgs []string for _, e := range passedEnvs { v, b := nc.systemDeps.LookupEnv(e) if b { + if e == "COMPOSE_FILE" { + path, _ := convertToWSLPath(nc.systemDeps, v) + v = path + } passedEnvArgs = append(passedEnvArgs, fmt.Sprintf("%s=%s", e, v)) ```

Test locally on windows and darwin and then create the commit again.

pendo324 commented 2 days ago

I found out why the test is failing. The path to the file specified in COMPOSE_FILE in Windows environment needs to be converted to a WSL path. So, the following patch is needed. Details

Test locally on windows and darwin and then create the commit again.

Oh yeah, makes sense. Nice find!

austinvazquez commented 2 days ago

Hi @haytok, I merged #993 to workaround the Windows CI WSL hanging issue. If you get a chance to rebase your branch that should resolve the Windows test-e2e-vm-serial issue for you.

haytok commented 2 days ago

@austinvazquez Thank you for fixing the CI and shareing the Info!!!

I will rebase and make an additional commit!

haytok commented 2 days ago

Hi, @pendo324

Fix the code, rebase and push. All tests passed, so could please review when you have time?

pendo324 commented 1 day ago

Hi, @pendo324

Fix the code, rebase and push. All tests passed, so could please review when you have time?

LGTM, thanks!

haytok commented 1 day ago

Thanks for approving!!!

I have hecked PR again and found no problem, so merge.