Manifest rendering error with kube-thanos (Jsonnet)

[Unb0rn]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug I'm trying to deploy kube-thanos project with ArgoCD. I am able to render those manifests locally with something like jsonnet -J vendor -m manifests 0-thanos-service.jsonnet but argocd fails with ComparisonError

To Reproduce I've created the application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: thanos-test
  namespace: argocd
spec:
  destination:
    server: 'https://kubernetes.default.svc'
  source:
    path: thanos
    repoURL: 'https://gitlab.com/path/to/gitops/repo.git'
    targetRevision: target-branch
    directory:
      jsonnet:
        libs:
        - thanos/vendor
  project: default
  syncPolicy:
    automated: null
    syncOptions:
      - PruneLast=true
      - ApplyOutOfSyncOnly=true

And jsonnet file describing my customizations:

local t = import 'kube-thanos/thanos.libsonnet';

local commonConfig = {
  local cfg = self,
  namespace: 'thanos-test',
  version: 'v0.25.2',
  image: 'quay.io/thanos/thanos:' + cfg.version,
  replicaLabels: ['prometheus_replica', 'rule_replica'],
  objectStorageConfig: {
    name: 'thanos-objectstorage',
    key: 'thanos.yaml',
    tlsSecretName: '',
    tlsSecretMountPath: '',
  },
  resources: {
    requests: { cpu: '100m', memory: '128Mi' },
    limits: { cpu: '200m', memory: '420Mi' },
  },
  volumeClaimTemplate: {
    spec: {
      accessModes: ['ReadWriteOnce'],
      resources: {
        requests: {
          storage: '25Gi',
        },
      },
    },
  },
  // This enables jaeger tracing for all components, as commonConfig is shared
  tracing+: {
    type: 'JAEGER',
    config+: {
      sampler_type: 'ratelimiting',
      sampler_param: 2,
    },
  },
};

local q = t.query(commonConfig {
  name: 'thanos-query',
  replicas: 1,
  externalPrefix: '',
  resources: {},
  queryTimeout: '5m',
  autoDownsampling: true,
  lookbackDelta: '15m',
  ports: {
    grpc: 10901,
    http: 9090,
  },
  serviceMonitor: true,
  logLevel: 'debug',
});

local s = t.store(commonConfig {
  replicas: 1,
  serviceMonitor: true,
});

local finalQ = t.query(q.config {
  stores: [
    'dnssrv+_grpc._tcp.%s.%s.svc.cluster.local' % [service.metadata.name, service.metadata.namespace]
    for service in [s.service]
  ],
});

{ ['thanos-query-' + name]: finalQ[name] for name in std.objectFields(finalQ) if finalQ[name] != null } +
{ ['thanos-store-' + name]: s[name] for name in std.objectFields(s) if s[name] != null }

Expected behavior

ArgoCD successfully renders manifests from jsonnet project and applies them to the cluster.

Screenshots

Version

argocd-server: v2.2.5+8f981cc

Logs kubectl describe application

Status:
  Conditions:
    Last Transition Time:  2022-03-30T12:38:38Z
    Message:               rpc error: code = Unknown desc = Manifest generation error (cached): rpc error: code = FailedPrecondition desc = Failed to unmarshal generated json "0-thanos-service.jsonnet": Object 'Kind' is missing in '{
   "thanos-query-deployment": {
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "query-layer",
            "app.kubernetes.io/instance": "thanos-query",
            "app.kubernetes.io/name": "thanos-query",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-query",
         "namespace": "thanos-test"
      },
      "spec": {
         "replicas": 1,
         "selector": {
            "matchLabels": {
               "app.kubernetes.io/component": "query-layer",
               "app.kubernetes.io/instance": "thanos-query",
               "app.kubernetes.io/name": "thanos-query"
            }
         },
         "template": {
            "metadata": {
               "labels": {
                  "app.kubernetes.io/component": "query-layer",
                  "app.kubernetes.io/instance": "thanos-query",
                  "app.kubernetes.io/name": "thanos-query",
                  "app.kubernetes.io/version": "v0.25.2"
               }
            },
            "spec": {
               "affinity": {
                  "podAntiAffinity": {
                     "preferredDuringSchedulingIgnoredDuringExecution": [
                        {
                           "podAffinityTerm": {
                              "labelSelector": {
                                 "matchExpressions": [
                                    {
                                       "key": "app.kubernetes.io/name",
                                       "operator": "In",
                                       "values": [
                                          "thanos-query"
                                       ]
                                    }
                                 ]
                              },
                              "namespaces": [
                                 "thanos-test"
                              ],
                              "topologyKey": "kubernetes.io/hostname"
                           },
                           "weight": 100
                        }
                     ]
                  }
               },
               "containers": [
                  {
                     "args": [
                        "query",
                        "--grpc-address=0.0.0.0:10901",
                        "--http-address=0.0.0.0:9090",
                        "--log.level=debug",
                        "--log.format=logfmt",
                        "--query.replica-label=prometheus_replica",
                        "--query.replica-label=rule_replica",
                        "--store=dnssrv+_grpc._tcp.thanos-store.thanos-test.svc.cluster.local",
                        "--query.timeout=5m",
                        "--query.lookback-delta=15m",
                        "--tracing.config=\"config\":\n  \"sampler_param\": 2\n  \"sampler_type\": \"ratelimiting\"\n  \"service_name\": \"thanos-query\"\n\"type\": \"JAEGER\"",
                        "--query.auto-downsampling"
                     ],
                     "env": [
                        {
                           "name": "HOST_IP_ADDRESS",
                           "valueFrom": {
                              "fieldRef": {
                                 "fieldPath": "status.hostIP"
                              }
                           }
                        }
                     ],
                     "image": "quay.io/thanos/thanos:v0.25.2",
                     "imagePullPolicy": "IfNotPresent",
                     "livenessProbe": {
                        "failureThreshold": 4,
                        "httpGet": {
                           "path": "/-/healthy",
                           "port": 9090,
                           "scheme": "HTTP"
                        },
                        "periodSeconds": 30
                     },
                     "name": "thanos-query",
                     "ports": [
                        {
                           "containerPort": 10901,
                           "name": "grpc"
                        },
                        {
                           "containerPort": 9090,
                           "name": "http"
                        }
                     ],
                     "readinessProbe": {
                        "failureThreshold": 20,
                        "httpGet": {
                           "path": "/-/ready",
                           "port": 9090,
                           "scheme": "HTTP"
                        },
                        "periodSeconds": 5
                     },
                     "resources": { },
                     "terminationMessagePolicy": "FallbackToLogsOnError"
                  }
               ],
               "nodeSelector": {
                  "kubernetes.io/os": "linux"
               },
               "securityContext": {
                  "fsGroup": 65534,
                  "runAsUser": 65534
               },
               "serviceAccountName": "thanos-query",
               "terminationGracePeriodSeconds": 120
            }
         }
      }
   },
   "thanos-query-service": {
      "apiVersion": "v1",
      "kind": "Service",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "query-layer",
            "app.kubernetes.io/instance": "thanos-query",
            "app.kubernetes.io/name": "thanos-query",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-query",
         "namespace": "thanos-test"
      },
      "spec": {
         "ports": [
            {
               "name": "grpc",
               "port": 10901,
               "targetPort": 10901
            },
            {
               "name": "http",
               "port": 9090,
               "targetPort": 9090
            }
         ],
         "selector": {
            "app.kubernetes.io/component": "query-layer",
            "app.kubernetes.io/instance": "thanos-query",
            "app.kubernetes.io/name": "thanos-query"
         }
      }
   },
   "thanos-query-serviceAccount": {
      "apiVersion": "v1",
      "kind": "ServiceAccount",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "query-layer",
            "app.kubernetes.io/instance": "thanos-query",
            "app.kubernetes.io/name": "thanos-query",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-query",
         "namespace": "thanos-test"
      }
   },
   "thanos-query-serviceMonitor": {
      "apiVersion": "monitoring.coreos.com/v1",
      "kind": "ServiceMonitor",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "query-layer",
            "app.kubernetes.io/instance": "thanos-query",
            "app.kubernetes.io/name": "thanos-query",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-query",
         "namespace": "thanos-test"
      },
      "spec": {
         "endpoints": [
            {
               "port": "http",
               "relabelings": [
                  {
                     "separator": "/",
                     "sourceLabels": [
                        "namespace",
                        "pod"
                     ],
                     "targetLabel": "instance"
                  }
               ]
            }
         ],
         "selector": {
            "matchLabels": {
               "app.kubernetes.io/component": "query-layer",
               "app.kubernetes.io/instance": "thanos-query",
               "app.kubernetes.io/name": "thanos-query"
            }
         }
      }
   },
   "thanos-store-service": {
      "apiVersion": "v1",
      "kind": "Service",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "object-store-gateway",
            "app.kubernetes.io/instance": "thanos-store",
            "app.kubernetes.io/name": "thanos-store",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-store",
         "namespace": "thanos-test"
      },
      "spec": {
         "clusterIP": "None",
         "ports": [
            {
               "name": "grpc",
               "port": 10901,
               "targetPort": 10901
            },
            {
               "name": "http",
               "port": 10902,
               "targetPort": 10902
            }
         ],
         "selector": {
            "app.kubernetes.io/component": "object-store-gateway",
            "app.kubernetes.io/instance": "thanos-store",
            "app.kubernetes.io/name": "thanos-store"
         }
      }
   },
   "thanos-store-serviceAccount": {
      "apiVersion": "v1",
      "kind": "ServiceAccount",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "object-store-gateway",
            "app.kubernetes.io/instance": "thanos-store",
            "app.kubernetes.io/name": "thanos-store",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-store",
         "namespace": "thanos-test"
      }
   },
   "thanos-store-serviceMonitor": {
      "apiVersion": "monitoring.coreos.com/v1",
      "kind": "ServiceMonitor",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "object-store-gateway",
            "app.kubernetes.io/instance": "thanos-store",
            "app.kubernetes.io/name": "thanos-store",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-store",
         "namespace": "thanos-test"
      },
      "spec": {
         "endpoints": [
            {
               "port": "http",
               "relabelings": [
                  {
                     "separator": "/",
                     "sourceLabels": [
                        "namespace",
                        "pod"
                     ],
                     "targetLabel": "instance"
                  }
               ]
            }
         ],
         "selector": {
            "matchLabels": {
               "app.kubernetes.io/component": "object-store-gateway",
               "app.kubernetes.io/instance": "thanos-store",
               "app.kubernetes.io/name": "thanos-store"
            }
         }
      }
   },
   "thanos-store-statefulSet": {
      "apiVersion": "apps/v1",
      "kind": "StatefulSet",
      "metadata": {
         "labels": {
            "app.kubernetes.io/component": "object-store-gateway",
            "app.kubernetes.io/instance": "thanos-store",
            "app.kubernetes.io/name": "thanos-store",
            "app.kubernetes.io/version": "v0.25.2"
         },
         "name": "thanos-store",
         "namespace": "thanos-test"
      },
      "spec": {
         "replicas": 1,
         "selector": {
            "matchLabels": {
               "app.kubernetes.io/component": "object-store-gateway",
               "app.kubernetes.io/instance": "thanos-store",
               "app.kubernetes.io/name": "thanos-store"
            }
         },
         "serviceName": "thanos-store",
         "template": {
            "metadata": {
               "labels": {
                  "app.kubernetes.io/component": "object-store-gateway",
                  "app.kubernetes.io/instance": "thanos-store",
                  "app.kubernetes.io/name": "thanos-store",
                  "app.kubernetes.io/version": "v0.25.2"
               }
            },
            "spec": {
               "affinity": {
                  "podAntiAffinity": {
                     "preferredDuringSchedulingIgnoredDuringExecution": [
                        {
                           "podAffinityTerm": {
                              "labelSelector": {
                                 "matchExpressions": [
                                    {
                                       "key": "app.kubernetes.io/name",
                                       "operator": "In",
                                       "values": [
                                          "thanos-store"
                                       ]
                                    },
                                    {
                                       "key": "app.kubernetes.io/instance",
                                       "operator": "In",
                                       "values": [
                                          "thanos-store"
                                       ]
                                    }
                                 ]
                              },
                              "namespaces": [
                                 "thanos-test"
                              ],
                              "topologyKey": "kubernetes.io/hostname"
                           },
                           "weight": 100
                        }
                     ]
                  }
               },
               "containers": [
                  {
                     "args": [
                        "store",
                        "--log.level=info",
                        "--log.format=logfmt",
                        "--data-dir=/var/thanos/store",
                        "--grpc-address=0.0.0.0:10901",
                        "--http-address=0.0.0.0:10902",
                        "--objstore.config=$(OBJSTORE_CONFIG)",
                        "--ignore-deletion-marks-delay=24h",
                        "--tracing.config=\"config\":\n  \"sampler_param\": 2\n  \"sampler_type\": \"ratelimiting\"\n  \"service_name\": \"thanos-store\"\n\"type\": \"JAEGER\""
                     ],
                     "env": [
                        {
                           "name": "OBJSTORE_CONFIG",
                           "valueFrom": {
                              "secretKeyRef": {
                                 "key": "thanos.yaml",
                                 "name": "thanos-objectstorage"
                              }
                           }
                        },
                        {
                           "name": "HOST_IP_ADDRESS",
                           "valueFrom": {
                              "fieldRef": {
                                 "fieldPath": "status.hostIP"
                              }
                           }
                        }
                     ],
                     "image": "quay.io/thanos/thanos:v0.25.2",
                     "imagePullPolicy": "IfNotPresent",
                     "livenessProbe": {
                        "failureThreshold": 8,
                        "httpGet": {
                           "path": "/-/healthy",
                           "port": 10902,
                           "scheme": "HTTP"
                        },
                        "periodSeconds": 30
                     },
                     "name": "thanos-store",
                     "ports": [
                        {
                           "containerPort": 10901,
                           "name": "grpc"
                        },
                        {
                           "containerPort": 10902,
                           "name": "http"
                        }
                     ],
                     "readinessProbe": {
                        "failureThreshold": 20,
                        "httpGet": {
                           "path": "/-/ready",
                           "port": 10902,
                           "scheme": "HTTP"
                        },
                        "periodSeconds": 5
                     },
                     "resources": {
                        "limits": {
                           "cpu": "200m",
                           "memory": "420Mi"
                        },
                        "requests": {
                           "cpu": "100m",
                           "memory": "128Mi"
                        }
                     },
                     "terminationMessagePolicy": "FallbackToLogsOnError",
                     "volumeMounts": [
                        {
                           "mountPath": "/var/thanos/store",
                           "name": "data",
                           "readOnly": false
                        }
                     ]
                  }
               ],
               "nodeSelector": {
                  "kubernetes.io/os": "linux"
               },
               "securityContext": {
                  "fsGroup": 65534,
                  "runAsUser": 65534
               },
               "serviceAccountName": "thanos-store",
               "terminationGracePeriodSeconds": 120,
               "volumes": [ ]
            }
         },
         "volumeClaimTemplates": [
            {
               "metadata": {
                  "labels": {
                     "app.kubernetes.io/component": "object-store-gateway",
                     "app.kubernetes.io/instance": "thanos-store",
                     "app.kubernetes.io/name": "thanos-store"
                  },
                  "name": "data"
               },
               "spec": {
                  "accessModes": [
                     "ReadWriteOnce"
                  ],
                  "resources": {
                     "requests": {
                        "storage": "25Gi"
                     }
                  }
               }
            }
         ]
      }
   }
}'

It looks like valid JSON...

[gorshunovr]

@Unb0rn Your JSON has structure:

{
   "thanos-query-deployment": {
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      ...
  },
   "thanos-store-statefulSet": {
      "apiVersion": "apps/v1",
      "kind": "StatefulSet",
      ...
  },
  ...
}

ArgoCD expects array with plain Kubernetes resource definitions in JSON, like the following output for this jsonnet example:

[
   {
      "apiVersion": "v1",
      "kind": "Service",
      ...
   },
   {
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      ...
   }
]

Instead of dumping named objects with your

{ ['thanos-query-' + name]: finalQ[name] for name in std.objectFields(finalQ) if finalQ[name] != null } +
{ ['thanos-store-' + name]: s[name] for name in std.objectFields(s) if s[name] != null }

, try dumping object fields directly, e.g.:

std.objectFields(finalQ) +
std.objectFields(s)

To get rid of ArgoCD trying to parse jsonnetfile.json and jsonnetfile.lock.json, add those files to exclude list in ArgoCD Application definition under directory (doc link):

...
    directory:
      jsonnet:
        libs:
          - my-thanos/vendor
      exclude: '{jsonnetfile.json,jsonnetfile.lock.json}'

Hope it helps.