search

vmware-tanzu / tanzu-framework

Tanzu Framework provides a set of building blocks to build atop of the Tanzu platform and leverages Carvel packaging and plugins to provide users with a much stronger, more integrated experience than the loose coupling and stand-alone commands of the previous generation of tools.
Apache License 2.0
196 stars 193 forks source link

Cluster-API-* unit tests failing with ytt not found in $PATH #2436

Open codegold79 opened 2 years ago

codegold79 commented 2 years ago

Bug description

Release stage tests are failing. It seems ytt is not in the runners $PATH environment variable.

See test logs here: https://github.com/vmware-tanzu/tanzu-framework/runs/6535358353?check_suite_focus=true

In short, the errors look like,

"ytt": executable file not found in $PATH

Here is the full log of the Cluster-API-* unit tests:

Verifying package tests...
find ./packages/ -name "test" -type d -exec sh -c "cd *** && go test -coverprofile coverage2.txt -v -timeout 120s  ./..." \;
go: downloading golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e
go: downloading github.com/fsnotify/fsnotify v1.4.9
=== RUN   TestUnit
Running Suite: Cluster API Provider Docker Unit Suite
=====================================================
Random Seed: 1653118852
Will run 1 of 1 specs

• Failure [0.001 seconds]
Cluster-API Provider Docker Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-docker/test/unit/overlay_test.go:20
  enables ClusterTopology feature gates on capd-controller-manager Deployment [It]
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-docker/test/unit/overlay_test.go:29

  Unexpected error:
      <exec.CodeExitError>: ***
          Err: <*errors.errorString | 0xc00002a060>***
              s: "error running ytt --ignore-unknown-comments -f ../../bundle/config:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
          ***,
          Code: -1,
      ***
      error running ytt --ignore-unknown-comments -f ../../bundle/config:
      Command stdout:

      stderr:

      error:
      exec: "ytt": executable file not found in $PATH
  occurred

  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-docker/test/unit/overlay_test.go:31
------------------------------

Summarizing 1 Failure:

[Fail] Cluster-API Provider Docker Ytt Templating [It] enables ClusterTopology feature gates on capd-controller-manager Deployment 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-docker/test/unit/overlay_test.go:31

Ran 1 of 1 Specs in 0.002 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestUnit (0.00s)
FAIL
coverage: [no statements]
FAIL    github.com/vmware-tanzu/tanzu-framework/packages/cluster-api-provider-docker/test/unit  0.008s
FAIL
=== RUN   TestUnit
Running Suite: Cluster API Provider AWS Unit Suite
==================================================
Random Seed: 1653118853
Will run 3 of 3 specs

• Failure [0.000 seconds]
Cluster-API Provider AWS Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:21
  when the httpProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:31
    adds HTTP_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:38

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc000093cd0>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:40
------------------------------
• Failure [0.000 seconds]
Cluster-API Provider AWS Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:21
  when the httpsProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:60
    adds HTTPS_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:67

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc0001dc500>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:69
------------------------------
• Failure [0.000 seconds]
Cluster-API Provider AWS Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:21
  when the noProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:89
    adds NO_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:96

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc0001dcd30>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:98
------------------------------

Summarizing 3 Failures:

[Fail] Cluster-API Provider AWS Ytt Templating when the httpProxy data value is provided [It] adds HTTP_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:40

[Fail] Cluster-API Provider AWS Ytt Templating when the httpsProxy data value is provided [It] adds HTTPS_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:69

[Fail] Cluster-API Provider AWS Ytt Templating when the noProxy data value is provided [It] adds NO_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-aws/test/unit/overlay_test.go:98

Ran 3 of 3 Specs in 0.003 seconds
FAIL! -- 0 Passed | 3 Failed | 0 Pending | 0 Skipped
--- FAIL: TestUnit (0.00s)
FAIL
coverage: [no statements]
FAIL    github.com/vmware-tanzu/tanzu-framework/packages/cluster-api-provider-aws/test/unit 0.010s
FAIL
=== RUN   TestUnit
Running Suite: Cluster API Provider Azure Unit Suite
====================================================
Random Seed: 1653118853
Will run 3 of 3 specs

• Failure [0.001 seconds]
Cluster-API Provider Azure Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:21
  when the httpProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:31
    adds HTTP_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:38

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc0000a6050>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:40
------------------------------
• Failure [0.000 seconds]
Cluster-API Provider Azure Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:21
  when the httpsProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:60
    adds HTTPS_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:67

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc0000a6880>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:69
------------------------------
• Failure [0.000 seconds]
Cluster-API Provider Azure Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:21
  when the noProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:89
    adds NO_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:96

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc0000a70b0>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:98
------------------------------

Summarizing 3 Failures:

[Fail] Cluster-API Provider Azure Ytt Templating when the httpProxy data value is provided [It] adds HTTP_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:40

[Fail] Cluster-API Provider Azure Ytt Templating when the httpsProxy data value is provided [It] adds HTTPS_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:69

[Fail] Cluster-API Provider Azure Ytt Templating when the noProxy data value is provided [It] adds NO_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-azure/test/unit/overlay_test.go:98

Ran 3 of 3 Specs in 0.002 seconds
FAIL! -- 0 Passed | 3 Failed | 0 Pending | 0 Skipped
--- FAIL: TestUnit (0.00s)
FAIL
coverage: [no statements]
FAIL    github.com/vmware-tanzu/tanzu-framework/packages/cluster-api-provider-azure/test/unit   0.010s
FAIL
=== RUN   TestUnit
Running Suite: Cluster API Provider vSphere Unit Suite
======================================================
Random Seed: 1653118854
Will run 3 of 3 specs

• Failure [0.001 seconds]
Cluster-API Provider vSphere Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:21
  when the httpProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:31
    adds HTTP_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:38

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc00002a070>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:40
------------------------------
• Failure [0.000 seconds]
Cluster-API Provider vSphere Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:21
  when the httpsProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:58
    adds HTTPS_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:65

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc00002a8a0>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:67
------------------------------
• Failure [0.000 seconds]
Cluster-API Provider vSphere Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:21
  when the noProxy data value is provided
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:85
    adds NO_PROXY environment variable to the deployment container [It]
    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:92

    Unexpected error:
        <exec.CodeExitError>: ***
            Err: <*errors.errorString | 0xc00002b0d0>***
                s: "error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
            ***,
            Code: -1,
        ***
        error running ytt --ignore-unknown-comments -f ../../bundle/config -f -:
        Command stdout:

        stderr:

        error:
        exec: "ytt": executable file not found in $PATH
    occurred

    /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:94
------------------------------

Summarizing 3 Failures:

[Fail] Cluster-API Provider vSphere Ytt Templating when the httpProxy data value is provided [It] adds HTTP_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:40

[Fail] Cluster-API Provider vSphere Ytt Templating when the httpsProxy data value is provided [It] adds HTTPS_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:67

[Fail] Cluster-API Provider vSphere Ytt Templating when the noProxy data value is provided [It] adds NO_PROXY environment variable to the deployment container 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit/overlay_test.go:94

Ran 3 of 3 Specs in 0.003 seconds
FAIL! -- 0 Passed | 3 Failed | 0 Pending | 0 Skipped
--- FAIL: TestUnit (0.00s)
FAIL
coverage: [no statements]
FAIL    github.com/vmware-tanzu/tanzu-framework/packages/cluster-api-provider-vsphere/test/unit 0.012s
FAIL
=== RUN   TestUnit
Running Suite: Cluster API Bootstrap Kubeadm Unit Suite
=======================================================
Random Seed: 1653118855
Will run 1 of 1 specs

• Failure [0.000 seconds]
Cluster-API-bootstrap-kubeadm Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-bootstrap-kubeadm/test/unit/overlay_test.go:20
  sets the bootstrap token ttl flag to 15m [It]
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-bootstrap-kubeadm/test/unit/overlay_test.go:28

  Unexpected error:
      <exec.CodeExitError>: ***
          Err: <*errors.errorString | 0xc000098040>***
              s: "error running ytt --ignore-unknown-comments -f ../../bundle/config:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
          ***,
          Code: -1,
      ***
      error running ytt --ignore-unknown-comments -f ../../bundle/config:
      Command stdout:

      stderr:

      error:
      exec: "ytt": executable file not found in $PATH
  occurred

  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-bootstrap-kubeadm/test/unit/overlay_test.go:30
------------------------------

Summarizing 1 Failure:

[Fail] Cluster-API-bootstrap-kubeadm Ytt Templating [It] sets the bootstrap token ttl flag to 15m 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api-bootstrap-kubeadm/test/unit/overlay_test.go:30

Ran 1 of 1 Specs in 0.001 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestUnit (0.00s)
FAIL
coverage: [no statements]
FAIL    github.com/vmware-tanzu/tanzu-framework/packages/cluster-api-bootstrap-kubeadm/test/unit    0.007s
FAIL
=== RUN   TestUnit
Running Suite: Cluster API Unit Suite
=====================================
Random Seed: 1653118855
Will run 1 of 1 specs

• Failure [0.001 seconds]
Cluster-API Ytt Templating
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api/test/unit/overlay_test.go:20
  enables ClusterResourceSet and ClusterTopology feature gates on capi-controller-manager Deployment [It]
  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api/test/unit/overlay_test.go:28

  Unexpected error:
      <exec.CodeExitError>: ***
          Err: <*errors.errorString | 0xc00002d5f0>***
              s: "error running ytt --ignore-unknown-comments -f ../../bundle/config:\nCommand stdout:\n\nstderr:\n\nerror:\nexec: \"ytt\": executable file not found in $PATH",
          ***,
          Code: -1,
      ***
      error running ytt --ignore-unknown-comments -f ../../bundle/config:
      Command stdout:

      stderr:

      error:
      exec: "ytt": executable file not found in $PATH
  occurred

  /home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api/test/unit/overlay_test.go:30
------------------------------

Summarizing 1 Failure:

[Fail] Cluster-API Ytt Templating [It] enables ClusterResourceSet and ClusterTopology feature gates on capi-controller-manager Deployment 
/home/runner/work/tanzu-framework/tanzu-framework/packages/cluster-api/test/unit/overlay_test.go:30

Ran 1 of 1 Specs in 0.002 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestUnit (0.00s)
FAIL
coverage: [no statements]
FAIL    github.com/vmware-tanzu/tanzu-framework/packages/cluster-api/test/unit  0.011s
FAIL
echo "... package tests complete!"
... package tests complete!

Affected product area (please put an X in all that apply)

Expected behavior

All unit tests should pass.

Steps to reproduce the bug

Not sure how to verify on local because when I do go test in the /packages/cluster-api-provider-docker/test/unit/ directories, the unit tests tests pass. It's probably because my computer knows where ytt is as it's correctly set in my $PATH. The failure only showed up after I merged in PR 1906. It didn't show as failure in my PR and it allowed me to merge:

image

Version (include the SHA if the version is not obvious)

Environment where the bug was observed (cloud, OS, etc)

Relevant Debug Output (Logs, manifests, etc)

codegold79 commented 2 years ago

@tylerschultz, do you think the error is happening because the runner running the unit tests doesn't have ytt installed? Does that mean we should install ytt somehow before running these tests?

rajathagasthya commented 2 years ago

make tools should install all the tooling binaries required in ./bin/. It's possible the test isn't using binaries from that location.