Helm Chart Issue - wrong version of .Capabilities.KubeVersion.GitVersion passed?

Phil1972 commented 1 year ago

What happened?

It seems an issue related to the fact that the version of k8s passed to helm might be wrong or invalid. I have the latest nuget and when we run our Azure pipeline, it seems that the capability passed is not right since it resolves to the 'else'.

This is in istiod 1.17.1 chart.

{{- if .Values.global.defaultPodDisruptionBudget.enabled }} {{- if (semverCompare ">=1.21-0" .Capabilities.KubeVersion.GitVersion) }} apiVersion: policy/v1 {{- else }} apiVersion: policy/v1beta1 {{- end }} kind: PodDisruptionBudget

is there a way to know what version is passed in .Capabilities.KubeVersion.GitVersion ?

Thanks,

Expected Behavior

"apiVersion: policy/v1" should been resolved in the chart

Steps to reproduce

private const string IstioRepoUrl = "https://istio-release.storage.googleapis.com/charts";

...
            istiod = new Chart(chartName, new ChartArgs
            {
                Chart = "istiod",
                Version = config.Version,
                FetchOptions = new ChartFetchArgs
                {
                    Repo = IstioRepoUrl
                },
                Namespace = istioNamespace.Name,
                ResourcePrefix = resourcePrefix
            }, new ComponentResourceOptions
            {
                DependsOn = baseIstio,
                Parent = parent
            });
...

Output of `pulumi about`

CLI          
Version      3.60.1
Go Version   go1.20.2
Go Compiler  gc

Plugins
NAME          VERSION
azure         5.38.0
azure-native  1.98.1
azuread       5.36.0
azuredevops   2.7.0
confluent     0.2.2
dotnet        unknown
kafka         3.4.0
kubernetes    3.24.2

Host     
OS       ubuntu
Version  20.04
Arch     x86_64

This project is written in dotnet: executable='/opt/hostedtoolcache/dotnet/dotnet' version='6.0.407'

Backend        
Name           ****
URL            azblob://state
User           ****
Organizations  

Dependencies:
NAME                                            VERSION
Azure.Identity                                  1.8.2
Azure.Security.KeyVault.Secrets                 4.5.0
KubernetesClient                                10.1.4
Microsoft.Extensions.Localization.Abstractions  7.0.4
Pulumi                                          3.54.1
Pulumi.Azure                                    5.38.0
Pulumi.AzureAD                                  5.36.0
Pulumi.AzureNative                              1.98.1
Pulumi.Confluent                                0.2.2
Pulumi.Kafka                                    3.4.0
Pulumi.Kubernetes                               3.24.2
System.IO.Abstractions                          19.2.4

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

dixler commented 1 year ago

Thanks for filing. I believe this is a Kubernetes/Helm question so I'll transfer your issue to that repository and they should be better able to help you.

danielrbradley commented 1 year ago

Hello, from a brief look at this it looks related to the fact that charts are executed locally.

Using Release instead of Chart might address this issue as the Release will execute the template in the context of the cluster. The downside of the Release is that Pulumi won't be aware of any resources created by the template.

Phil1972 commented 1 year ago

Hi @danielrbradley, pulumi runs into a azure pipeline and all my CLI are up to date in there. I specifically dump my versions into the pipeline to make sure I am using the right ones:

Helm: version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}

kubectl is at: Client Version: v1.26.3 Kustomize Version: v4.5.7

and of course I use the latest pulumi kubernetes nuget.

So why doesn't it resolve the version properly while templating the chart?

Phil1972 commented 1 year ago

Also, maybe this might be related but I get something similar when trying to render a keda helm chart > 2.8.1.. There is a condition that tests the version and I get this error during the preview:

stderr: error: Running program '/home/vsts/work/1/s/src/---/bin/Debug/net6.0/---.dll' failed with an unhandled exception: Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to create chart from template: chart requires kubeVersion: >=v1.24.0-0 which is incompatible with Kubernetes v1.20.0")

Phil1972 commented 1 year ago

Just to make sure we understand each other (I am not always clear ;)) the way to make it work would be that pulumi passes a '--kube-version' to the helm command line.

For example, in the case of the latest keda chart (2.10.2) the default behavior of helm template keda ./ is to pass the latest kube version supported by the helm cli whereas it seems that pulumi uses the oldest supported version instead. If I put as a command line helm template keda ./ --kube-version "1.20.0" then it fails with the error above-mentionned. (Error: chart requires kubeVersion: >=v1.24.0-0 which is incompatible with Kubernetes v1.20.0)

So maybe the Chart class would need an argument in its ChartArgs class so that we could specify which kube-version to use?

BTW I found this issue but it seems that would not cover the case here. (https://github.com/pulumi/pulumi-kubernetes/pull/2155)

thanks,

Phil1972 commented 1 year ago

@yann-soubeyrand is it possible that there is something wrong with your fix (https://github.com/pulumi/pulumi-kubernetes/pull/2155) when it runs into the context of an Azure Pipeline? Would it be possible to add more logs to know if what you added grabs and puts the proper version into the capabilities?

thanks,

yann-soubeyrand commented 1 year ago

Hello, If there’s an error discovering Kubernetes version, you should get an error message I guess: https://github.com/pulumi/pulumi-kubernetes/blob/58b8c44ae79eb5e19f96641396e1c7826262b2a8/provider/pkg/provider/invoke_helm_template.go#L246-L278 Can you confirm that you don’t set APIVersions in your chart args? Could it be that Pulumi doesn’t have access to your cluster at the time the chart resource is evaluated?

Phil1972 commented 1 year ago

Thanks for the quick response.

I do not specify APIVersions
I do not see any errors when it runs except the error from helm saying that my k8s version is not up to date (as I mentionned above). It thinks that the kube-version is 1.20.0

Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to create chart from template: chart requires kubeVersion: >=v1.23.0-0 which is incompatible with Kubernetes v1.20.0") at async Task<InvokeResponse> Pulumi.GrpcMonitor.InvokeAsync(ResourceInvokeRequest request) at async Task<SerializationResult> Pulumi.Deployment.InvokeRawAsync(string token, SerializationResult argsSerializationResult, InvokeOptions options) x 2 at async Task<T> Pulumi.Deployment.InvokeAsync<T>(string token, InvokeArgs args, InvokeOptions options, bool convertResult) at async Task<OutputData<T>> Pulumi.Output<T>+<>c__DisplayClass12_0.<Create>g__GetData|0(?)+GetData(?) at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) x 3 at async Task<OutputData<object>> Pulumi.Output<T>.Pulumi.IOutput.GetDataAsync() at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources, bool keepOutputValues) at async Task<RawSerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesRawAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources, bool keepOutputValues) at async Task<SerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources, bool keepOutputValues) at async Task<Struct> Pulumi.Deployment.SerializeAllPropertiesAsync(string label, IDictionary<string, object> args, bool keepResources, bool keepOutputValues) at async Task Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output<IDictionary<string, object>> outputs)

yann-soubeyrand commented 1 year ago

Can you run kubectl version and kubectl get apiservices from inside the pipeline?

Phil1972 commented 1 year ago

See this post https://github.com/pulumi/pulumi-kubernetes/issues/2345#issuecomment-1508685520

I did not do it for kubectl get apiservices though, let me modify the pipeline to do it

yann-soubeyrand commented 1 year ago

OK. And apart from these Helm related errors, does your Kubernetes provider work as expected? I mean, can you create Kubernetes ressources using Pulumi?

Phil1972 commented 1 year ago

Yes, everything else works. There is only those issues with the Chart object and there is so little verbose logs that it is hard to troubleshoot.

yann-soubeyrand commented 1 year ago

Sorry to ask this question, but is your Chart resource using your configured Kubernetes provider and not using the default (maybe not working) provider?

Phil1972 commented 1 year ago

Here is what I have for what you asked:

kubectl get apiservices

E0417 15:34:32.018227    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.020874    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.021687    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.023893    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.024970    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Phil1972 commented 1 year ago

Sorry to ask this question, but is your Chart resource using your configured Kubernetes provider and not using the default (maybe not working) provider?

Using the proper provider with the kubeconfig returned from the cluster. As I said, everything else works except the Chart wich seems to be missing the proper version passed to it.

yann-soubeyrand commented 1 year ago

Here is what I have for what you asked:

kubectl get apiservices

E0417 15:34:32.018227    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.020874    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.021687    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.023893    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.024970    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Sorry, I meant running this command using the same kubeconfig used by the Pulumi Kubernetes provider. Also, can you provide a copy of this kubeconfig?

Phil1972 commented 1 year ago

Here is what I have for what you asked: kubectl get apiservices

E0417 15:34:32.018227    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.020874    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.021687    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.023893    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
E0417 15:34:32.024970    2925 memcache.go:265] couldn't get current server API group list: Get "[http://localhost:8080/api?timeout=32s"](http://localhost:8080/api?timeout=32s%22): dial tcp [::1]:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Sorry, I meant running this command using the same kubeconfig used by the Pulumi Kubernetes provider. Also, can you provide a copy of this kubeconfig?

Would you have a code sample for what you need me do to (C#)? I am not that familiar with that functionality sorry

yann-soubeyrand commented 1 year ago

Sorry, I’m not familiar at all with C#. Here’s the documentation to output your kubeconfig: https://www.pulumi.com/learn/building-with-pulumi/stack-outputs/. Also, I’m not a Pulumi developer (I just contributed the PR you linked), so maybe you’ll have more luck finding someone who can help you troubleshooting your issue on Slack.

Phil1972 commented 1 year ago

Ok, will look into that. But as mentioned everything else works properly (even the use of ConfigFile to install Itsio/prometheus). Is it possible to add more verbose logs into that part of the code to indicate what is passed in the KubeVersion as well as any info retrieved from the DiscoveryClient ?

Phil1972 commented 1 year ago

OK, I think I am getting closer to the issue. I think the problem is related to a Parent/Child provider issue.

Our AKS ManagedCluster gets created
We then create a Kubernetes.Provider using the KubeConfig from the newly created AKS cluster.
We then create a ComponentResource (using the Provider created previously passed into the ComponentResourceOptions) containing all the resources under the cluster (Istio, Keda, CertManager, etc...).
Child resources should in theory inherit the parent's provider(?)

I am not jumping straight to conclusions, but I tried passing directly the Provider to the Chart object and it seems now to behave properly. Maybe there is a bug in there which does not pass down the proper provider for child resources?

yann-soubeyrand commented 1 year ago

Seems you’re getting closer indeed :wink: I’ve just check in our code though, and we do not pass the provider directly, our charts are children of component resources which receive a provider map as follow:

{
  "kubernetes": kubernetesProvider
}

But I guess you have the same on your side, right?

Phil1972 commented 1 year ago

There are two ways to instantiate a ComponentResource. One way passing a list of providers (key, value) and one way using a single provider. We use the later so we only use one provider down the chain which is the Kubernetes one. It seems it is not passed down properly and does not match your "kubernetes" one?

hampusohlsson commented 1 year ago

Hi, I am currently debugging this exact issue for installing the k8s cert-manager. Same steps

[OK] Create cluster (v1.24.10-xxx)
[OK] Generate kubeconfig for new cluster
[OK] Create k8s provider from kubeconfig
[FAIL] Create Helm chart

failed to create chart from template: chart requires kubeVersion: >= 1.21.0-0 which is incompatible with Kubernetes v1.20.0

Here's my code if anything obvious could be problematic

// create cluster etc...

k8sProvider, err := kubernetes.NewProvider(ctx, "k8s-provider", &kubernetes.ProviderArgs{
    Kubeconfig: clusterKubeconfig,
})
if err != nil {
    return err
}

_, err = helm.NewChart(ctx, "k8s-cert-manager", helm.ChartArgs{
    Chart:     pulumi.String("cert-manager"),
    Version:   pulumi.String("1.11.1"),
    Namespace: pulumi.String("cert-manager"),
    FetchArgs: helm.FetchArgs{
        Repo: pulumi.String("https://charts.jetstack.io"),
    },
    Values: pulumi.Map{
        "installCRDs": pulumi.Bool(true),
    },
},
    pulumi.Provider(k8sProvider),
)
if err != nil {
    return err
}

yann-soubeyrand commented 1 year ago

@hampusohlsson are you sure this is the same issue? IIUC, @Phil1972 said that passing the provider directly to the chart resource fixed the issue for him.

@Phil1972 you’re right that these two way of passing the providers should end with the same result, but could you try passing the providers using a map? Who knows, maybe we’ll uncover an interesting bug!

hampusohlsson commented 1 year ago

are you sure this is the same issue? IIUC, @Phil1972 said that passing the provider directly to the chart resource fixed the issue for him.

Perhaps not, but it definitely seems related to a wrong version getting passed to Helm, as using helm cli works without any issues. Not sure what is meant by passing the provider directly to the chart resource? I was under the impression I am doing that already

yann-soubeyrand commented 1 year ago

I was under the impression I am doing that already

Yes, that’s what makes me think it’s not the same issue :wink: Which version of the Kubernetes provider are you using?

Phil1972 commented 1 year ago

@hampusohlsson are you sure this is the same issue? IIUC, @Phil1972 said that passing the provider directly to the chart resource fixed the issue for him.

@Phil1972 you’re right that these two way of passing the providers should end with the same result, but could you try passing the providers using a map? Who knows, maybe we’ll uncover an interesting bug!

Using the 'Providers' (with an S) allows to pass a list of providers. (Not sure how I can pass a map). I tried using that one passing a list of my sole provider but it did not work either.

As for hampusohlsson, I am not sure it is the same issue indeed since he is not using composite objects. So it might be a different issue but gives him the same result as me ;)

Phil1972 commented 1 year ago

Here is what I sort of have in pseudo code:

        var cluster = new ManagedCluster();

        var kubeConfig = ListManagedClusterUserCredentials.Invoke(new ListManagedClusterUserCredentialsInvokeArgs
            {
                ResourceGroupName = _resourceGroup.Name,
                ResourceName = cluster.Name
            }).Apply(credentials =>
            {
                var encoded = credentials.Kubeconfigs[0].Value;
                var data = Convert.FromBase64String(encoded);
                return Encoding.UTF8.GetString(data);
            });

        var provider = new Pulumi.Kubernetes.Provider($"{nameof(ResourceKey.KubernetesProvider)}_{cluster.Id}",
            new Pulumi.Kubernetes.ProviderArgs
            {
                KubeConfig = kubeConfig
            });

        var setup = new ClusterSetup(cluster.Id, 
            new ComponentResourceOptions
            {
                Provider = provider
            });

    private class ClusterSetup : ComponentResource
    {
        public ClusterSetup(string clusterId, ComponentResourceOptions options)
            : base(typeof(ClusterSetup).Namespace!.Replace(".", ":"), $"{nameof(ClusterSetup)}_{clusterId}", options)
        {
            // Creates a chart and some other resources
        var baseIstio = new Chart(nameGenerator.GetName(ResourceKey.Istio, new Dictionary<string, string>
        {
            { "clusterId", clusterId }
        }), new ChartArgs
        {
            Chart = "base",
            Version = config.Version,
            FetchOptions = new ChartFetchArgs
            {
                Repo = IstioRepoUrl,
            },
            Namespace = istioNamespace.Name,
            ResourcePrefix = resourcePrefix
        }, new ComponentResourceOptions
        {
            Parent = this
        });

        var keda = new Chart(nameGenerator.GetName(ResourceKey.Keda, new Dictionary<string, string>
        {
            { "clusterId", clusterId }
        }), new ChartArgs
        {
            Chart = "keda",
            Version = config.Version,
            FetchOptions = new ChartFetchArgs
            {
                Repo = KedaRepoUrl,
            },
            Namespace = kedaNamespace.Name,
            ResourcePrefix = resourcePrefix
        }, new ComponentResourceOptions
        {
            DependsOn = kedaPodIdentity,
            Parent = this,
            Protect = false
        });

            var certManager = ...
        }
    }

Note that I am properly setting the parent and passing the provider in the argument and in the base constructor.

yann-soubeyrand commented 1 year ago

Using the 'Providers' (with an S) allows to pass a list of providers. (Not sure how I can pass a map). I tried using that one passing a list of my sole provider but it did not work either.

In Go, there’s a ProviderMap option (https://pkg.go.dev/github.com/pulumi/pulumi/sdk/v3/go/pulumi#ProviderMap), but that’s the same than Providers behind the scene (in fact, I think I should use the latest).

Again, I’m not familiar with C#, but your code seems really similar to what we do in Go, which is working for us. I’m wondering if this line isn’t missing the Parent option: https://github.com/pulumi/pulumi-kubernetes/blob/58b8c44ae79eb5e19f96641396e1c7826262b2a8/sdk/dotnet/Helm/V3/Chart.cs#L363 In the Go SDK, all the resource options which are also invoke options (which is the case of the Parent option) are passed: https://github.com/pulumi/pulumi-kubernetes/blob/58b8c44ae79eb5e19f96641396e1c7826262b2a8/sdk/go/kubernetes/helm/v3/chart.go#L247-L268 Maybe @lblackstone or another maintainer can confirm or inform what I’m supposing?

yann-soubeyrand commented 1 year ago

TypeScript SDK seems to set Parent option as well: https://github.com/pulumi/pulumi-kubernetes/blob/58b8c44ae79eb5e19f96641396e1c7826262b2a8/sdk/nodejs/helm/v3/helm.ts#L243-L258

yann-soubeyrand commented 1 year ago

OK, unless I’m wrong, I think we got a lead :wink: The PR for Dotnet (https://github.com/pulumi/pulumi-kubernetes/pull/2005) has been done in a second time after the other SDKs (https://github.com/pulumi/pulumi-kubernetes/pull/1919), and it seems to me that the implementation is not exactly the same.

Phil1972 commented 1 year ago

If anyone can verify/confirm this, it would be a big one for us using c# ;)

hampusohlsson commented 1 year ago

I've solved my previous issue, and like you all suspected it's a different issue - turns out it was bad spacing in my generated cluster kubeconfig.yaml 🤦‍♂️ I guess it was deemed invalid and Helm provider set the version to some default of 1.20.0

Phil1972 commented 1 year ago

@lblackstone when is this supposed to be merge into the latest release? I do not see any info on the resolution PR/release version

Thanks,

lblackstone commented 1 year ago

@Phil1972 Sorry, I reread the entire thread and it looks like I closed this in error while I was triaging issues the other day. I saw https://github.com/pulumi/pulumi-kubernetes/issues/2345#issuecomment-1513860912 and assumed it was fixed.

Phil1972 commented 1 year ago

Perfect because I confirm that the issue is still present as of the latest release.

Thanks,

D3-LucaPiombino commented 1 year ago

Hi @Phil1972, did you perhaps find any workaround for this? This is a big blocker for us as we are using a very similar pattern.

EronWright commented 9 months ago

To recap the discussion, a bug (in some SDKs) with provider option propagation causes a confusing error message. In some cases, a possible workaround is to set provider rather than providers. Another workaround may be to use a transformation to set the provider. A fix is coming associated with this issue: https://github.com/pulumi/pulumi-kubernetes/issues/2254

pulumi / pulumi-kubernetes