search

NVIDIA / gpu-operator

NVIDIA GPU Operator creates, configures, and manages GPUs in Kubernetes
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
Apache License 2.0
1.85k stars 298 forks source link

error creating csv gpu-operator-certified #68

Closed deathemperor closed 3 years ago

deathemperor commented 4 years ago

I'm using OCP4.2 (There's an ongoing issue with OCP 4.5 https://github.com/NVIDIA/gpu-operator/issues/64) installing NVIDIA GPU Operator using this instructions https://docs.nvidia.com/datacenter/kubernetes/openshift-on-gpu-install-guide/index.html, the "4.1. Installing via OpenShift OperatorHub" method.

After pressing install, the operator failed immediately Screen Shot 2020-07-20 at 4 40 08 PM

My installed Operators Screen Shot 2020-07-20 at 4 47 09 PM

My nodes set up:

This is the YAML file of the install pod:

apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  generateName: install-
  selfLink: >-
    /apis/operators.coreos.com/v1alpha1/namespaces/gpu-operator-resources/installplans/install-kksfg
  resourceVersion: '1592906'
  name: install-kksfg
  uid: 43454749-ca6b-11ea-96bc-525400d51c39
  creationTimestamp: '2020-07-20T09:27:42Z'
  generation: 1
  namespace: gpu-operator-resources
  ownerReferences:
    - apiVersion: operators.coreos.com/v1alpha1
      blockOwnerDeletion: false
      controller: false
      kind: Subscription
      name: gpu-operator-certified
      uid: 434160a6-ca6b-11ea-92dc-525400e79d67
spec:
  approval: Automatic
  approved: true
  clusterServiceVersionNames:
    - gpu-operator-certified.v1.1.7-r2
    - nfd.4.2.36-202006230600.p0
status:
  catalogSources:
    - certified-operators
    - redhat-operators
  conditions:
    - lastTransitionTime: '2020-07-20T09:27:42Z'
      lastUpdateTime: '2020-07-20T09:27:42Z'
      message: >-
        error creating csv gpu-operator-certified.v1.1.7-r2:
        ClusterServiceVersion.operators.coreos.com
        "gpu-operator-certified.v1.1.7-r2" is invalid: []: Invalid value:
        map[string]interface {}{"apiVersion":"operators.coreos.com/v1alpha1",
        "kind":"ClusterServiceVersion", "metadata":map[string]interface
        {}{"annotations":map[string]interface
        {}{"alm-examples":"[{\"apiVersion\":\"nvidia.com/v1\",\"kind\":\"ClusterPolicy\",\"metadata\":{\"name\":\"cluster-policy\"},\"spec\":{\"dcgmExporter\":{\"image\":\"dcgm-exporter\",\"repository\":\"nvidia\",\"version\":\"1.7.2-2.0.0-rc.9-ubi8\"},\"devicePlugin\":{\"image\":\"k8s-device-plugin\",\"repository\":\"nvidia\",\"version\":\"1.0.0-beta6-ubi8\"},\"driver\":{\"image\":\"driver\",\"repository\":\"nvidia\",\"version\":\"440.64.00\"},\"operator\":{\"defaultRuntime\":\"crio\"},\"toolkit\":{\"image\":\"container-toolkit\",\"repository\":\"nvidia\",\"version\":\"1.0.2-ubi8\"}}}]",
        "capabilities":"Basic Install", "categories":"AI/Machine Learning,
        OpenShift Optional", "certified":"false",
        "containerImage":"nvidia/gpu-operator:1.1.7",
        "createdAt":"2019-02-27T00:21:25Z", "description":"Automate the
        management and monitoring of NVIDIA GPUs.", "provider":"NVIDIA",
        "repository":"http://github.com/NVIDIA/gpu-operator",
        "support":"NVIDIA"}, "creationTimestamp":"2020-07-20T09:27:44Z",
        "generation":1, "name":"gpu-operator-certified.v1.1.7-r2",
        "namespace":"gpu-operator-resources",
        "uid":"44352747-ca6b-11ea-96bc-525400d51c39"},
        "spec":map[string]interface
        {}{"apiservicedefinitions":map[string]interface {}{},
        "customresourcedefinitions":map[string]interface {}{"owned":[]interface
        {}{map[string]interface {}{"description":"Cluster policy resource",
        "displayName":"ClusterPolicy", "kind":"ClusterPolicy",
        "name":"clusterpolicies.nvidia.com", "version":"v1"}},
        "required":[]interface {}{map[string]interface
        {}{"description":"Discovers special resources and labels the nodes with
        the PCI vendor ID.", "displayName":"Node Feature Discovery",
        "kind":"NodeFeatureDiscovery",
        "name":"nodefeaturediscoveries.nfd.openshift.io",
        "version":"v1alpha1"}}}, "description":"Kubernetes provides access to
        special hardware resources such as NVIDIA GPUs, NICs, Infiniband
        adapters and other devices through the [device plugin
        framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/).
        However, configuring and managing nodes with these hardware resources
        requires configuration of multiple software components such as drivers,
        container runtimes or other libraries which are difficult and prone to
        errors. The NVIDIA GPU Operator uses the [operator
        framework](https://coreos.com/blog/introducing-operator-framework)
        within Kubernetes to automate the management of all NVIDIA software
        components needed to provision and monitor GPUs. These components
        include the NVIDIA drivers (to enable CUDA), Kubernetes device plugin
        for GPUs, the NVIDIA Container Runtime, automatic node labelling and
        NVIDIA DCGM exporter. Visit the official site of the [GPU
        Operator](https://github.com/NVIDIA/gpu-operator) for more information.
        For getting started with using the GPU Operator with OpenShift, see the
        instructions
        [here](https://docs.nvidia.com/datacenter/kubernetes/openshift-on-gpu-install-guide/index.html).\n",
        "displayName":"NVIDIA GPU Operator", "icon":[]interface
        {}{map[string]interface
        {}{"base64data":"iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAMAAACdt4HsAAAB2lBMVEUAAAD///8EBAN3uQACAgIAAAQJDQUCAgB1tgAHCQf+/v5Ufg5Hagxxqwt+xgJ3uAB9wwB4vQBRUVEeLA3e3t5nZ2coKCgODg4FBwZ9wwR6wAJ4vADz8/MbGxt5tw1vpw1/wgoOFwkLDwh9xQH5+fny8vLw8PDFxcWysrKFhYVvb282NjYyMjIqKioXFxdikxRYgxNCYxJQdhFqoQ9xrg16ugxyqgyAxQkEBQj7+/v29vbIyMhjY2NbW1tHR0cvLy8kJCQdHR0ZGRlKbxJ8uhFNcxFVgBAxSBBgkg93tQ50sA4qPg4XIg18vwsbKQsSGgsLCwsMEwqCyQeByQFztADPz8+/v7+6urqWlpZra2tKSkogICASEhJmmRE8XBA5VRA2UBBonA9biA9GaQ4sQg4jMw4mOQ0aJw2GzgsUHgttpAqJ0Ql/wQWG0AJ8vwF0uQCtra2jo6OQkJB9fX1VVVVCQkI9PT0iIiIUFBRSfBNgjhA7WRBGZw+GywmFzgaAyASBxQN2twDb29u2traenp6Kiop+fn53d3dzc3NyqRV4sxM/YBNAXRElNhBjlQ+IzA00TQ16vgxJbgp6vAl4tgJ3vgDs7Ozn5+fa2trS0tJCXRY6VBV6thSL1gf4nFdFAAAD80lEQVRYw+zSOXPaQBgG4He0LJJmbGRGDUIzuvgBQiAEPfcdwC33DTbUtmOwSyc+4iRucvzXRImLFJmRShc8xXbfu+9+szg4OHjjAsH/iFD49q7rqM6xc/wPtWyBhS8sC94ObWRCZDksh1+RzmcEfI0DoPrjylEkSTgViMs9udjYTwMG4Gf51Z1BM81ioRwit+QvgYsdUQZeKFr3ladyKXvVr+pAM5uKcmRLXFzoCIxn+0i/8lSaBMHnfi7qowfQuZnm3PuFPwGs13zD3NlViozY/z4YD6/TCQORbPr2q78GLB0ou5IO40pd5AxQZnJ83m2y9Ju2JYKfgEhWC18aEIfrZLURHwQC0B87ySZwHxX8BNDWB1KfQfyxT2TA24uPQMt8yTWA3obz8wQGlhTN06Z900MkuJLrYu3u5LkK9LTtGRF8NEDLeSnXYLUdHUFVlpPqTa4IamlhJZ464biY1w4CKGrROOW7uwLlV+Q02lanCF6cbSoPVLzUfPwDll5I9T6WyXWhZre1yjiI6VCSzCWY3+FKaAwGHngzpEygx6+V6Uzk6TJR7yhWxJ1bFgTPJ7gMc58aUCq+n+qNT6Pn8y/xOcCiZZVjnJ+AAPhEuj0SKZ9bL9ZpNS9SgM6z9p5w3jt43cMvecfWBhm7dtfEpfhYMDBYpFd7mDZIAxPCFKgBhB0hkWbE2wVMyqycfhOMEiebSzFz5IMTEjw7E87UFj4GVR7GXqaSkoIcISEc/I38/PwhOTUMRBrADgwK09zgYGUBqbwcARiQyp3Eyk6kC4BloqtbJTcaSHIHShALWFmBSRuCWBGC+AtDMAAGIpAAc9mBiB0sCLSXHUSygxSxEIoE7IKEgbhopKgogC96x04QCMMw/H0cG6f0cEmBHaLc7FFQzApoTLwtQgWUWo26glx2mzGkyoHM1PPMO/NrnSH8e2QAiRsZ8S3ZuJoW5Udg5moGoMRLN2gAnkcUctueJ1gADsdtlZ2AgmSYoaDZBXwRctcwy6HN3XX/wfnTnA7Q5x0S0Gku4wHpe7Ql8Mbtu4TqC3qcADGtUl4O3eK0AkZdKH1mU/a6MFQGA7pQGoAVoAuuPYZlLJF2BawVLLjwac6Q8wUax61/CpKQAT6ZX3hFqoqqAFvuf4AzM+NgsoBS/wcSOD7SFzyf6CE9UQK9II1MRvIJm8QSgsLiBZuypsAWKyARElgx5FcLv1N4nFLbB45Sh6+TzsQRtn7bz/B3fS9GQ12bgUE2PKycQbwgXD0SWLwVhpZFq4eHhWloOjLoqGvoRYRGAR2vp2EtpNUaTUpiRAizMAEhKNXpYZNnAUlBCSgFYTIxQTlMMJNGwSgYBdQHAFsKs+/bUkeyAAAAAElFTkSuQmCC",
        "mediatype":"image/png"}}, "install":map[string]interface
        {}{"spec":map[string]interface {}{"clusterPermissions":[]interface
        {}{map[string]interface {}{"rules":[]interface {}{map[string]interface
        {}{"apiGroups":[]interface {}{"security.openshift.io"},
        "resources":[]interface {}{"securitycontextconstraints",
        "securitycontextconstraints/finalizers"}, "verbs":[]interface {}{"*"}},
        map[string]interface {}{"apiGroups":[]interface {}{""},
        "resources":[]interface {}{"nodes"}, "verbs":[]interface {}{"*"}},
        map[string]interface {}{"apiGroups":[]interface {}{"nvidia.com"},
        "resources":[]interface {}{"*"}, "verbs":[]interface {}{"*"}},
        map[string]interface {}{"apiGroups":[]interface {}{""},
        "resources":[]interface {}{"secrets"}, "verbs":[]interface {}{"get",
        "watch", "list"}}, map[string]interface {}{"apiGroups":[]interface
        {}{"config.openshift.io"}, "resources":[]interface
        {}{"clusterversions"}, "verbs":[]interface {}{"get", "watch", "list"}}},
        "serviceAccountName":"gpu-operator"}}, "deployments":[]interface
        {}{map[string]interface {}{"name":"gpu-operator",
        "spec":map[string]interface {}{"replicas":1,
        "selector":map[string]interface {}{"matchLabels":map[string]interface
        {}{"name":"gpu-operator"}}, "strategy":map[string]interface {}{},
        "template":map[string]interface {}{"metadata":map[string]interface
        {}{"creationTimestamp":interface {}(nil), "labels":map[string]interface
        {}{"name":"gpu-operator"}}, "spec":map[string]interface
        {}{"containers":[]interface {}{map[string]interface
        {}{"command":[]interface {}{"gpu-operator"}, "env":[]interface
        {}{map[string]interface {}{"name":"WATCH_NAMESPACE",
        "valueFrom":map[string]interface {}{"fieldRef":map[string]interface
        {}{"fieldPath":"metadata.annotations['olm.targetNamespaces']"}}},
        map[string]interface {}{"name":"POD_NAME",
        "valueFrom":map[string]interface {}{"fieldRef":map[string]interface
        {}{"fieldPath":"metadata.name"}}}, map[string]interface
        {}{"name":"OPERATOR_NAME", "value":"gpu-operator"}, map[string]interface
        {}{"name":"NVIDIA_DCGM_EXPORTER",
        "value":"nvidia/dcgm-exporter:1.7.2-2.0.0-rc.9-ubi8"},
        map[string]interface {}{"name":"NVIDIA_DEVICE_PLUGIN",
        "value":"nvidia/k8s-device-plugin:1.0.0-beta6-ubi8"},
        map[string]interface {}{"name":"NVIDIA_DRIVER",
        "value":"nvidia/driver:440.64.00"}, map[string]interface
        {}{"name":"NVIDIA_TOOLKIT",
        "value":"nvidia/container-toolkit:1.0.2-ubi8"}, map[string]interface
        {}{"name":"NVIDIA_TOOLKIT_DEFAULT_RUNTIME", "value":"crio"},
        map[string]interface {}{"name":"KUBE_UNSUPPORTED_FLAVOR", "value":""}},
        "image":"nvidia/gpu-operator:1.1.7", "imagePullPolicy":"Always",
        "name":"gpu-operator", "resources":map[string]interface {}{}}},
        "serviceAccountName":"gpu-operator"}}}}}, "permissions":[]interface
        {}{map[string]interface {}{"rules":[]interface {}{map[string]interface
        {}{"apiGroups":[]interface {}{"rbac.authorization.k8s.io"},
        "resources":[]interface {}{"roles", "rolebindings"}, "verbs":[]interface
        {}{"*"}}, map[string]interface {}{"apiGroups":[]interface {}{""},
        "resources":[]interface {}{"pods", "services", "endpoints",
        "persistentvolumeclaims", "events", "configmaps", "configmaplists",
        "secrets", "serviceaccounts", "nodes"}, "verbs":[]interface {}{"*"}},
        map[string]interface {}{"apiGroups":[]interface {}{""},
        "resources":[]interface {}{"namespaces"}, "verbs":[]interface
        {}{"get"}}, map[string]interface {}{"apiGroups":[]interface {}{"apps"},
        "resources":[]interface {}{"deployments", "daemonsets", "replicasets",
        "statefulsets"}, "verbs":[]interface {}{"*"}}, map[string]interface
        {}{"apiGroups":[]interface {}{"monitoring.coreos.com"},
        "resources":[]interface {}{"servicemonitors"}, "verbs":[]interface
        {}{"get", "list", "create", "watch", "update"}}, map[string]interface
        {}{"apiGroups":[]interface {}{"nvidia.com"}, "resources":[]interface
        {}{"*"}, "verbs":[]interface {}{"*"}}, map[string]interface
        {}{"apiGroups":[]interface {}{"scheduling.k8s.io"},
        "resources":[]interface {}{"priorityclasses"}, "verbs":[]interface
        {}{"get", "list", "watch", "create"}}, map[string]interface
        {}{"apiGroups":[]interface {}{"batch"}, "resources":[]interface
        {}{"jobs"}, "verbs":[]interface {}{"get", "list", "watch", "create"}},
        map[string]interface {}{"apiGroups":[]interface
        {}{"route.openshift.io"}, "resources":[]interface {}{"routes"},
        "verbs":[]interface {}{"get", "list", "watch", "create", "update"}}},
        "serviceAccountName":"gpu-operator"}}}, "strategy":"deployment"},
        "installModes":[]interface {}{map[string]interface {}{"supported":true,
        "type":"OwnNamespace"}, map[string]interface {}{"supported":true,
        "type":"SingleNamespace"}, map[string]interface {}{"supported":false,
        "type":"MultiNamespace"}, map[string]interface {}{"supported":false,
        "type":"AllNamespaces"}}, "keywords":[]interface {}{"gpu", "cuda",
        "compute", "operator", "deep learning", "monitoring", "tesla"},
        "links":[]interface {}{map[string]interface {}{"name":"GPU Operator
        home", "url":"https://github.com/NVIDIA/gpu-operator"},
        map[string]interface {}{"name":"Support",
        "url":"https://github.com/NVIDIA/gpu-operator/issues/new"},
        map[string]interface {}{"name":"OpenShift on GPU Install Guide",
        "url":"https://docs.nvidia.com/datacenter/kubernetes/openshift-on-gpu-install-guide/index.html"},
        map[string]interface {}{"name":"Docker Repository",
        "url":"https://hub.docker.com/u/nvidia"}}, "maintainers":[]interface
        {}{map[string]interface {}{"name":"NVIDIA"}}, "maturity":"alpha",
        "provider":map[string]interface {}{"name":"NVIDIA Corporation"},
        "version":"1.1.7-r2"}}: validation failure list:

        spec.maintainers.email in body is required
      reason: InstallComponentFailed
      status: 'False'
      type: Installed
  phase: Failed
  plan:
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: operators.coreos.com
        kind: ClusterServiceVersion
        manifest: >
          {"kind":"ClusterServiceVersion","apiVersion":"operators.coreos.com/v1alpha1","metadata":{"name":"gpu-operator-certified.v1.1.7-r2","namespace":"gpu-operator-resources","selfLink":"/apis/operators.coreos.com/v1alpha1/namespaces/gpu-operator-resources/clusterserviceversions/gpu-operator-certified.v1.1.7-r2","generation":1,"creationTimestamp":null,"annotations":{"alm-examples":"[{\"apiVersion\":\"nvidia.com/v1\",\"kind\":\"ClusterPolicy\",\"metadata\":{\"name\":\"cluster-policy\"},\"spec\":{\"dcgmExporter\":{\"image\":\"dcgm-exporter\",\"repository\":\"nvidia\",\"version\":\"1.7.2-2.0.0-rc.9-ubi8\"},\"devicePlugin\":{\"image\":\"k8s-device-plugin\",\"repository\":\"nvidia\",\"version\":\"1.0.0-beta6-ubi8\"},\"driver\":{\"image\":\"driver\",\"repository\":\"nvidia\",\"version\":\"440.64.00\"},\"operator\":{\"defaultRuntime\":\"crio\"},\"toolkit\":{\"image\":\"container-toolkit\",\"repository\":\"nvidia\",\"version\":\"1.0.2-ubi8\"}}}]","capabilities":"Basic
          Install","categories":"AI/Machine Learning, OpenShift
          Optional","certified":"false","containerImage":"nvidia/gpu-operator:1.1.7","createdAt":"2019-02-27T00:21:25Z","description":"Automate
          the management and monitoring of NVIDIA
          GPUs.","provider":"NVIDIA","repository":"http://github.com/NVIDIA/gpu-operator","support":"NVIDIA"}},"spec":{"install":{"strategy":"deployment","spec":{"clusterPermissions":[{"rules":[{"apiGroups":["security.openshift.io"],"resources":["securitycontextconstraints","securitycontextconstraints/finalizers"],"verbs":["*"]},{"apiGroups":[""],"resources":["nodes"],"verbs":["*"]},{"apiGroups":["nvidia.com"],"resources":["*"],"verbs":["*"]},{"apiGroups":[""],"resources":["secrets"],"verbs":["get","watch","list"]},{"apiGroups":["config.openshift.io"],"resources":["clusterversions"],"verbs":["get","watch","list"]}],"serviceAccountName":"gpu-operator"}],"deployments":[{"name":"gpu-operator","spec":{"replicas":1,"selector":{"matchLabels":{"name":"gpu-operator"}},"strategy":{},"template":{"metadata":{"creationTimestamp":null,"labels":{"name":"gpu-operator"}},"spec":{"containers":[{"command":["gpu-operator"],"env":[{"name":"WATCH_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.annotations['olm.targetNamespaces']"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"OPERATOR_NAME","value":"gpu-operator"},{"name":"NVIDIA_DCGM_EXPORTER","value":"nvidia/dcgm-exporter:1.7.2-2.0.0-rc.9-ubi8"},{"name":"NVIDIA_DEVICE_PLUGIN","value":"nvidia/k8s-device-plugin:1.0.0-beta6-ubi8"},{"name":"NVIDIA_DRIVER","value":"nvidia/driver:440.64.00"},{"name":"NVIDIA_TOOLKIT","value":"nvidia/container-toolkit:1.0.2-ubi8"},{"name":"NVIDIA_TOOLKIT_DEFAULT_RUNTIME","value":"crio"},{"name":"KUBE_UNSUPPORTED_FLAVOR","value":""}],"image":"nvidia/gpu-operator:1.1.7","imagePullPolicy":"Always","name":"gpu-operator","resources":{}}],"serviceAccountName":"gpu-operator"}}}}],"permissions":[{"rules":[{"apiGroups":["rbac.authorization.k8s.io"],"resources":["roles","rolebindings"],"verbs":["*"]},{"apiGroups":[""],"resources":["pods","services","endpoints","persistentvolumeclaims","events","configmaps","configmaplists","secrets","serviceaccounts","nodes"],"verbs":["*"]},{"apiGroups":[""],"resources":["namespaces"],"verbs":["get"]},{"apiGroups":["apps"],"resources":["deployments","daemonsets","replicasets","statefulsets"],"verbs":["*"]},{"apiGroups":["monitoring.coreos.com"],"resources":["servicemonitors"],"verbs":["get","list","create","watch","update"]},{"apiGroups":["nvidia.com"],"resources":["*"],"verbs":["*"]},{"apiGroups":["scheduling.k8s.io"],"resources":["priorityclasses"],"verbs":["get","list","watch","create"]},{"apiGroups":["batch"],"resources":["jobs"],"verbs":["get","list","watch","create"]},{"apiGroups":["route.openshift.io"],"resources":["routes"],"verbs":["get","list","watch","create","update"]}],"serviceAccountName":"gpu-operator"}]}},"version":"1.1.7-r2","maturity":"alpha","customresourcedefinitions":{"owned":[{"name":"clusterpolicies.nvidia.com","version":"v1","kind":"ClusterPolicy","displayName":"ClusterPolicy","description":"Cluster
          policy
          resource"}],"required":[{"name":"nodefeaturediscoveries.nfd.openshift.io","version":"v1alpha1","kind":"NodeFeatureDiscovery","displayName":"Node
          Feature Discovery","description":"Discovers special resources and
          labels the nodes with the PCI vendor
          ID."}]},"apiservicedefinitions":{},"displayName":"NVIDIA GPU
          Operator","description":"Kubernetes provides access to special
          hardware resources such as NVIDIA GPUs, NICs, Infiniband adapters and
          other devices through the [device plugin
          framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/).
          However, configuring and managing nodes with these hardware resources
          requires configuration of multiple software components such as
          drivers, container runtimes or other libraries which are difficult and
          prone to errors. The NVIDIA GPU Operator uses the [operator
          framework](https://coreos.com/blog/introducing-operator-framework)
          within Kubernetes to automate the management of all NVIDIA software
          components needed to provision and monitor GPUs. These components
          include the NVIDIA drivers (to enable CUDA), Kubernetes device plugin
          for GPUs, the NVIDIA Container Runtime, automatic node labelling and
          NVIDIA DCGM exporter. Visit the official site of the [GPU
          Operator](https://github.com/NVIDIA/gpu-operator) for more
          information. For getting started with using the GPU Operator with
          OpenShift, see the instructions
          [here](https://docs.nvidia.com/datacenter/kubernetes/openshift-on-gpu-install-guide/index.html).\n","keywords":["gpu","cuda","compute","operator","deep
          learning","monitoring","tesla"],"maintainers":[{"name":"NVIDIA"}],"provider":{"name":"NVIDIA
          Corporation"},"links":[{"name":"GPU Operator
          home","url":"https://github.com/NVIDIA/gpu-operator"},{"name":"Support","url":"https://github.com/NVIDIA/gpu-operator/issues/new"},{"name":"OpenShift
          on GPU Install
          Guide","url":"https://docs.nvidia.com/datacenter/kubernetes/openshift-on-gpu-install-guide/index.html"},{"name":"Docker
          Repository","url":"https://hub.docker.com/u/nvidia"}],"icon":[{"base64data":"iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAMAAACdt4HsAAAB2lBMVEUAAAD///8EBAN3uQACAgIAAAQJDQUCAgB1tgAHCQf+/v5Ufg5Hagxxqwt+xgJ3uAB9wwB4vQBRUVEeLA3e3t5nZ2coKCgODg4FBwZ9wwR6wAJ4vADz8/MbGxt5tw1vpw1/wgoOFwkLDwh9xQH5+fny8vLw8PDFxcWysrKFhYVvb282NjYyMjIqKioXFxdikxRYgxNCYxJQdhFqoQ9xrg16ugxyqgyAxQkEBQj7+/v29vbIyMhjY2NbW1tHR0cvLy8kJCQdHR0ZGRlKbxJ8uhFNcxFVgBAxSBBgkg93tQ50sA4qPg4XIg18vwsbKQsSGgsLCwsMEwqCyQeByQFztADPz8+/v7+6urqWlpZra2tKSkogICASEhJmmRE8XBA5VRA2UBBonA9biA9GaQ4sQg4jMw4mOQ0aJw2GzgsUHgttpAqJ0Ql/wQWG0AJ8vwF0uQCtra2jo6OQkJB9fX1VVVVCQkI9PT0iIiIUFBRSfBNgjhA7WRBGZw+GywmFzgaAyASBxQN2twDb29u2traenp6Kiop+fn53d3dzc3NyqRV4sxM/YBNAXRElNhBjlQ+IzA00TQ16vgxJbgp6vAl4tgJ3vgDs7Ozn5+fa2trS0tJCXRY6VBV6thSL1gf4nFdFAAAD80lEQVRYw+zSOXPaQBgG4He0LJJmbGRGDUIzuvgBQiAEPfcdwC33DTbUtmOwSyc+4iRucvzXRImLFJmRShc8xXbfu+9+szg4OHjjAsH/iFD49q7rqM6xc/wPtWyBhS8sC94ObWRCZDksh1+RzmcEfI0DoPrjylEkSTgViMs9udjYTwMG4Gf51Z1BM81ioRwit+QvgYsdUQZeKFr3ladyKXvVr+pAM5uKcmRLXFzoCIxn+0i/8lSaBMHnfi7qowfQuZnm3PuFPwGs13zD3NlViozY/z4YD6/TCQORbPr2q78GLB0ou5IO40pd5AxQZnJ83m2y9Ju2JYKfgEhWC18aEIfrZLURHwQC0B87ySZwHxX8BNDWB1KfQfyxT2TA24uPQMt8yTWA3obz8wQGlhTN06Z900MkuJLrYu3u5LkK9LTtGRF8NEDLeSnXYLUdHUFVlpPqTa4IamlhJZ464biY1w4CKGrROOW7uwLlV+Q02lanCF6cbSoPVLzUfPwDll5I9T6WyXWhZre1yjiI6VCSzCWY3+FKaAwGHngzpEygx6+V6Uzk6TJR7yhWxJ1bFgTPJ7gMc58aUCq+n+qNT6Pn8y/xOcCiZZVjnJ+AAPhEuj0SKZ9bL9ZpNS9SgM6z9p5w3jt43cMvecfWBhm7dtfEpfhYMDBYpFd7mDZIAxPCFKgBhB0hkWbE2wVMyqycfhOMEiebSzFz5IMTEjw7E87UFj4GVR7GXqaSkoIcISEc/I38/PwhOTUMRBrADgwK09zgYGUBqbwcARiQyp3Eyk6kC4BloqtbJTcaSHIHShALWFmBSRuCWBGC+AtDMAAGIpAAc9mBiB0sCLSXHUSygxSxEIoE7IKEgbhopKgogC96x04QCMMw/H0cG6f0cEmBHaLc7FFQzApoTLwtQgWUWo26glx2mzGkyoHM1PPMO/NrnSH8e2QAiRsZ8S3ZuJoW5Udg5moGoMRLN2gAnkcUctueJ1gADsdtlZ2AgmSYoaDZBXwRctcwy6HN3XX/wfnTnA7Q5x0S0Gku4wHpe7Ql8Mbtu4TqC3qcADGtUl4O3eK0AkZdKH1mU/a6MFQGA7pQGoAVoAuuPYZlLJF2BawVLLjwac6Q8wUax61/CpKQAT6ZX3hFqoqqAFvuf4AzM+NgsoBS/wcSOD7SFzyf6CE9UQK9II1MRvIJm8QSgsLiBZuypsAWKyARElgx5FcLv1N4nFLbB45Sh6+TzsQRtn7bz/B3fS9GQ12bgUE2PKycQbwgXD0SWLwVhpZFq4eHhWloOjLoqGvoRYRGAR2vp2EtpNUaTUpiRAizMAEhKNXpYZNnAUlBCSgFYTIxQTlMMJNGwSgYBdQHAFsKs+/bUkeyAAAAAElFTkSuQmCC","mediatype":"image/png"}],"installModes":[{"type":"OwnNamespace","supported":true},{"type":"SingleNamespace","supported":true},{"type":"MultiNamespace","supported":false},{"type":"AllNamespaces","supported":false}]},"status":{"lastUpdateTime":null,"lastTransitionTime":null,"certsLastUpdated":null,"certsRotateAt":null}}
        name: gpu-operator-certified.v1.1.7-r2
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1alpha1
      status: Unknown
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: apiextensions.k8s.io
        kind: CustomResourceDefinition
        manifest: >
          {"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"creationTimestamp":null,"name":"clusterpolicies.nvidia.com"},"spec":{"group":"nvidia.com","names":{"kind":"ClusterPolicy","listKind":"ClusterPolicyList","plural":"clusterpolicies","singular":"clusterpolicy"},"scope":"Cluster","subresources":{"status":{}},"validation":{"openAPIV3Schema":{"description":"ClusterPolicy
          allows you to configure the GPU
          Operator","properties":{"apiVersion":{"description":"APIVersion
          defines the versioned schema of this representation of an object.
          Servers should convert recognized schemas to the latest internal
          value, and may reject unrecognized values. More info:
          https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind
          is a string value representing the REST resource this object
          represents. Servers may infer this from the endpoint the client
          submits requests to. Cannot be updated. In CamelCase. More info:
          https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"ClusterPolicySpec
          defines the desired state of
          ClusterPolicy","properties":{"dcgmExporter":{"description":"ComponentSpec
          defines the path to the container
          image","properties":{"image":{"pattern":"[a-zA-Z0-9\\-]+","type":"string"},"repository":{"pattern":"[a-zA-Z0-9\\.\\-\\/]+","type":"string"},"version":{"pattern":"[a-zA-Z0-9\\.-]+","type":"string"}},"required":["image","repository","version"],"type":"object"},"devicePlugin":{"description":"ComponentSpec
          defines the path to the container
          image","properties":{"image":{"pattern":"[a-zA-Z0-9\\-]+","type":"string"},"repository":{"pattern":"[a-zA-Z0-9\\.\\-\\/]+","type":"string"},"version":{"pattern":"[a-zA-Z0-9\\.-]+","type":"string"}},"required":["image","repository","version"],"type":"object"},"driver":{"description":"ComponentSpec
          defines the path to the container
          image","properties":{"image":{"pattern":"[a-zA-Z0-9\\-]+","type":"string"},"repository":{"pattern":"[a-zA-Z0-9\\.\\-\\/]+","type":"string"},"version":{"pattern":"[a-zA-Z0-9\\.-]+","type":"string"}},"required":["image","repository","version"],"type":"object"},"operator":{"description":"OperatorSpec
          describes configuration options for the
          operator","properties":{"defaultRuntime":{"enum":["docker","crio"],"type":"string"}},"required":["defaultRuntime"],"type":"object"},"toolkit":{"description":"ComponentSpec
          defines the path to the container
          image","properties":{"image":{"pattern":"[a-zA-Z0-9\\-]+","type":"string"},"repository":{"pattern":"[a-zA-Z0-9\\.\\-\\/]+","type":"string"},"version":{"pattern":"[a-zA-Z0-9\\.-]+","type":"string"}},"required":["image","repository","version"],"type":"object"}},"required":["dcgmExporter","devicePlugin","driver","operator","toolkit"],"type":"object"},"status":{"description":"ClusterPolicyStatus
          defines the observed state of
          ClusterPolicy","properties":{"state":{"enum":["ignored","ready","notReady"],"type":"string"}},"required":["state"],"type":"object"}},"type":"object"}},"version":"v1","versions":[{"name":"v1","served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":null,"storedVersions":null}}
        name: clusterpolicies.nvidia.com
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1beta1
      status: Unknown
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: ''
        kind: ServiceAccount
        manifest: >
          {"metadata":{"name":"gpu-operator","namespace":"gpu-operator-resources","creationTimestamp":null,"ownerReferences":[{"apiVersion":"operators.coreos.com/v1alpha1","kind":"ClusterServiceVersion","name":"gpu-operator-certified.v1.1.7-r2","uid":"","controller":false,"blockOwnerDeletion":false}]}}
        name: gpu-operator
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: rbac.authorization.k8s.io
        kind: Role
        manifest: >
          {"metadata":{"name":"gpu-operator-certified.v1.1.7-r2-whzdb","namespace":"gpu-operator-resources","creationTimestamp":null,"labels":{"olm.owner":"gpu-operator-certified.v1.1.7-r2","olm.owner.kind":"ClusterServiceVersion","olm.owner.namespace":"gpu-operator-resources"},"ownerReferences":[{"apiVersion":"operators.coreos.com/v1alpha1","kind":"ClusterServiceVersion","name":"gpu-operator-certified.v1.1.7-r2","uid":"","controller":false,"blockOwnerDeletion":false}]},"rules":[{"verbs":["*"],"apiGroups":["rbac.authorization.k8s.io"],"resources":["roles","rolebindings"]},{"verbs":["*"],"apiGroups":[""],"resources":["pods","services","endpoints","persistentvolumeclaims","events","configmaps","configmaplists","secrets","serviceaccounts","nodes"]},{"verbs":["get"],"apiGroups":[""],"resources":["namespaces"]},{"verbs":["*"],"apiGroups":["apps"],"resources":["deployments","daemonsets","replicasets","statefulsets"]},{"verbs":["get","list","create","watch","update"],"apiGroups":["monitoring.coreos.com"],"resources":["servicemonitors"]},{"verbs":["*"],"apiGroups":["nvidia.com"],"resources":["*"]},{"verbs":["get","list","watch","create"],"apiGroups":["scheduling.k8s.io"],"resources":["priorityclasses"]},{"verbs":["get","list","watch","create"],"apiGroups":["batch"],"resources":["jobs"]},{"verbs":["get","list","watch","create","update"],"apiGroups":["route.openshift.io"],"resources":["routes"]}]}
        name: gpu-operator-certified.v1.1.7-r2-whzdb
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: rbac.authorization.k8s.io
        kind: RoleBinding
        manifest: >
          {"metadata":{"name":"gpu-operator-certified.v1.1.7-r2-whzdb-gpu-operator-mvp9j","namespace":"gpu-operator-resources","creationTimestamp":null,"labels":{"olm.owner":"gpu-operator-certified.v1.1.7-r2","olm.owner.kind":"ClusterServiceVersion","olm.owner.namespace":"gpu-operator-resources"},"ownerReferences":[{"apiVersion":"operators.coreos.com/v1alpha1","kind":"ClusterServiceVersion","name":"gpu-operator-certified.v1.1.7-r2","uid":"","controller":false,"blockOwnerDeletion":false}]},"subjects":[{"kind":"ServiceAccount","name":"gpu-operator","namespace":"gpu-operator-resources"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"gpu-operator-certified.v1.1.7-r2-whzdb"}}
        name: gpu-operator-certified.v1.1.7-r2-whzdb-gpu-operator-mvp9j
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: rbac.authorization.k8s.io
        kind: ClusterRole
        manifest: >
          {"metadata":{"name":"gpu-operator-certified.v1.1.7-r2-btpwk","creationTimestamp":null,"labels":{"olm.owner":"gpu-operator-certified.v1.1.7-r2","olm.owner.kind":"ClusterServiceVersion","olm.owner.namespace":"gpu-operator-resources"}},"rules":[{"verbs":["*"],"apiGroups":["security.openshift.io"],"resources":["securitycontextconstraints","securitycontextconstraints/finalizers"]},{"verbs":["*"],"apiGroups":[""],"resources":["nodes"]},{"verbs":["*"],"apiGroups":["nvidia.com"],"resources":["*"]},{"verbs":["get","watch","list"],"apiGroups":[""],"resources":["secrets"]},{"verbs":["get","watch","list"],"apiGroups":["config.openshift.io"],"resources":["clusterversions"]}]}
        name: gpu-operator-certified.v1.1.7-r2-btpwk
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: gpu-operator-certified.v1.1.7-r2
      resource:
        group: rbac.authorization.k8s.io
        kind: ClusterRoleBinding
        manifest: >
          {"metadata":{"name":"gpu-operator-certified.v1.1.7-r2-btpwk-gpu-operator-55hcg","namespace":"gpu-operator-resources","creationTimestamp":null,"labels":{"olm.owner":"gpu-operator-certified.v1.1.7-r2","olm.owner.kind":"ClusterServiceVersion","olm.owner.namespace":"gpu-operator-resources"}},"subjects":[{"kind":"ServiceAccount","name":"gpu-operator","namespace":"gpu-operator-resources"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"gpu-operator-certified.v1.1.7-r2-btpwk"}}
        name: gpu-operator-certified.v1.1.7-r2-btpwk-gpu-operator-55hcg
        sourceName: certified-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: nfd.4.2.36-202006230600.p0
      resource:
        group: operators.coreos.com
        kind: ClusterServiceVersion
        manifest: >
          {"kind":"ClusterServiceVersion","apiVersion":"operators.coreos.com/v1alpha1","metadata":{"name":"nfd.4.2.36-202006230600.p0","namespace":"gpu-operator-resources","creationTimestamp":null,"annotations":{"alm-examples":"[\n 
          {\n    \"apiVersion\": \"nfd.openshift.io/v1alpha1\",\n    \"kind\":
          \"NodeFeatureDiscovery\",\n    \"metadata\": {\n      \"name\":
          \"nfd-master-server\"\n    },\n    \"spec\": {\n      \"namespace\":
          \"openshift-nfd\"\n    }\n  }\n]","capabilities":"Basic
          Install","categories":"Database","certified":"false","containerImage":"","createdAt":"2019-05-30T00:00:00Z","description":"This
          software enables node feature discovery for Kubernetes. It detects
          hardware features available on each node in a Kubernetes cluster, and
          advertises those features using node labels.","provider":"Red
          Hat","repository":"https://github.com/openshift/cluster-nfd-operator","support":"Red
          Hat"}},"spec":{"install":{"strategy":"deployment","spec":{"clusterPermissions":[{"rules":[{"apiGroups":["rbac.authorization.k8s.io"],"resources":["roles","rolebindings","clusterroles","clusterrolebindings"],"verbs":["*"]},{"apiGroups":[""],"resources":["pods","services","endpoints","persistentvolumeclaims","events","configmaps","secrets","serviceaccounts","nodes"],"verbs":["*"]},{"apiGroups":[""],"resources":["namespaces"],"verbs":["*"]},{"apiGroups":["apps"],"resources":["deployments","daemonsets","replicasets","statefulsets"],"verbs":["*"]},{"apiGroups":["monitoring.coreos.com"],"resources":["servicemonitors"],"verbs":["get","create"]},{"apiGroups":["nfd.openshift.io"],"resources":["*"],"verbs":["*"]},{"apiGroups":["policy"],"resourceNames":["nfd-worker"],"resources":["podsecuritypolicies"],"verbs":["use"]},{"apiGroups":["security.openshift.io"],"resources":["securitycontextconstraints"],"verbs":["*"]}],"serviceAccountName":"nfd-operator"}],"deployments":[{"name":"nfd-operator","spec":{"replicas":1,"selector":{"matchLabels":{"name":"nfd-operator"}},"strategy":{},"template":{"metadata":{"labels":{"name":"nfd-operator"}},"spec":{"containers":[{"command":["cluster-nfd-operator"],"env":[{"name":"WATCH_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.annotations['olm.targetNamespaces']"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"OPERATOR_NAME","value":"cluster-nfd-operator"},{"name":"NODE_FEATURE_DISCOVERY_IMAGE","value":"registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:98184c2459a3c6489968eab361094c5651381902e026e07dd64eded8e01315c7"}],"image":"registry.redhat.io/openshift4/ose-cluster-nfd-operator@sha256:34c5bce0197f60f5e925d3a60666e3e0921336b8f973250af027f058eafc39f6","imagePullPolicy":"Always","name":"nfd-operator","ports":[{"containerPort":60000,"name":"metrics"}],"readinessProbe":{"exec":{"command":["stat","/tmp/operator-sdk-ready"]},"failureThreshold":1,"initialDelaySeconds":4,"periodSeconds":10},"resources":{},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/tmp","name":"tmp"}]}],"nodeSelector":{"node-role.kubernetes.io/master":""},"serviceAccountName":"nfd-operator","tolerations":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Equal"}],"volumes":[{"emptyDir":{},"name":"tmp"}]}}}}]}},"version":"4.2.0","maturity":"alpha","customresourcedefinitions":{"owned":[{"name":"nodefeaturediscoveries.nfd.openshift.io","version":"v1alpha1","kind":"NodeFeatureDiscovery","displayName":"Node
          Feature Discovery","description":"The NFD operator creates and
          maintains the Node Feature Discovery (NFD) on Kubernetes. \nIt detects
          hardware features available on each node in a Kubernetes cluster, and
          advertises those features using node
          labels.\n"}]},"apiservicedefinitions":{},"displayName":"Node Feature
          Discovery","description":"The NFD operator creates and maintiains the
          Node Feature Discovery (NFD) on Kubernetes. \nIt detects hardware
          features available on each node in a Kubernetes cluster, and
          advertises those features using node
          labels.\n","maintainers":[{"name":"Red
          Hat","email":"openshift-psap@redhat.com"}],"provider":{"name":"Red
          Hat"},"links":[{"name":"Documentation
          operator","url":"https://github.com/openshift/cluster-nfd-operator"},{"name":"Documentation
          operand","url":"https://github.com/openshift/node-feature-discovery"}],"icon":[{"base64data":"iVBORw0KGgoAAAANSUhEUgAAAYAAAAGGCAYAAABynlaiAAAACXBIWXMAABYlAAAWJQFJUiTwAAAgAElEQVR4nO3df4id15nY8ec4imuLbUZXiJXBdUYOlOyyUTQp2ZZQbzRi26Vk16tJWSi7xNWI0tIu2XoMS/tPG422/aewkFG6sKVL0cjJH6WUemSTkOIUzyTpP2lYz1QpoV0aj+x1iVOhq3GD5c06fsvRfV7pnfu+9973x3nfe857vh+4SJo7mpn3nXvPc85zznmOSZJEgNgMjTklIiv6ODvh8ndEZF8f24Mk2c59BhAwAgCiMjTmmIhsiMiFmte9Z4NB+hgkyZ3cZwCBIAAgGkNjVrXxX3B4zWlA2GKEgNAQACKnqZBTY3dhf5Ak+325Mw56/WUdpMFAAwKjA3iNABARbQhtzntZH4szrj74dIdes/35z+SebN9OJhj0JqCiPwgAERgaYxv7VQc94OvamG3mnvHQ0JglbYBnBbou2GC6STCATwgAPabpnc0pq1zqOtCvu+FrY6aN/7bjfL8rBAN4gQDQU0Nj1kRkvYMG8JoGgt3cM3PieeM/bk/nJ5gzQOcIAD3T4YTnOJvvXp/3SpiWVvp0JagUG8JHAOiROU94puYWCLTxv5p7IjwHOnexydJStIkA0BOeNP5ZnQaCoTGbcxj1dOGmzhdsMl8A1wgAPTE0Ztejxj9rT+cIWklr6ET3lqfX7tqOBgJSRHCCANADgfR+DzQ376QnqyOeNX2EmO9vIk0ReTX5jvAQAAIXaN679jJIbfhXteH3YX3/vLGKCLURAAKm6Y/dwHvAN3XuYlcfd7K92kypCruZzS7vPJ/7ChBGBaiDABCwoTHbLWzyQvgYFaAUAkCgerTkEe3xfsc25osAECDNg+9HOPmJ+nY0EGxxD5F6iDsRpBhXvqAZmyp8YWjMvi0Top0IRI4RQGDo/cORdNJ4nfRQvBgBhIfeP1xY0L0jrw2N2dKS4YgMI4CA0PtHy1rdtQ3/MAIIC71/tMmW07jKPEE8GAEExL4x2f2KDqXlOzbYT9BPBIBAsO4fc0Qg6CkCQCDY9QsPEAh6hgAQAK2H81rs9wHeIBD0BJPAYViL/QbAK3YhwiW7Im1ozDqTxeFiBBAAJn/hOUYEgWIE4LmhMSs0/vBcdkSwyi8rHAQA/63EfgMQjIXMPgJ2FgeAFJDnhsbcYfMXAmUrkK5Sa8hfjAA8pukfGn+E6qzWGtpgothPBAC/kU9FHzxrj/vUDg08QgrIU9pjGsZ+H9A71zUtxGohDzAC8Be9JfTReV0txCSxBwgA/iIAoK/svNYrdm6A3/B8kQLyEKUfEBF7BsEyKaH5YATgJ3r/iMUZTQkt8RvvHgHAT6z+QUxsSmibINA9UkCeIf2DiB1oOmiXF0E3GAH4h94/YsVIoGOMADxD5U+AkUBXGAF4RHs+NP6InR0JbFI+on0EAL+Q/gFGzugZA2gRKSCPkP4Bcs4NkmQ791E4wQjAExz8AhTaLPog3CAA+IPNX0De4tAYzsRuCSkgT3DwCzDRzUGSnJr0JOpjBOABDn4BplrkrOF2EAD8wIsbmI40UAtIAc0ZB78ApT3J+cJuMQKYPyZ/gXIYBThGAJg/AgBQDu8Vx0gBzRGVP4HKSAM5xAhgvujRANXwnnGIADBfrP4BquEweYdIAc0J6R+gloNBklAl1BFGAPPDUBaoboEDY9whAMwPS9qAekgDOUIAmAMOfgEaIQA4QgCYDyZ/gfoIAI4QAOaD/D9Q34IuokBDBICOkf4BnGAU4AABoHtM/gLNEQAcIAB0j/QP0BxLQR0gAHSIg18AZ85oKXU0QADoFr1/wB3SQA0RALpFAADcIQ3UEAGgI3qmKekfwB1GAA0RALpD7x9w6yz3sxkCQAd0sup87y8U6NjQGEYBDRAAukHvH2gH8wANEAC6QQAA2sEIoAEOhGmZpn+Gvb5IYH44IKYBRgDto/cPtIfCcA0QANpH7R+gXaSBaiIAtEh7Jmd6e4GAHwgANREA2kX6B2gfK4FqYhK4RUNjdhkBAJ0YDJLkDre6GkYALSH9A3SKUUANBID2kP4BusM8QA0EgPZw8DvQHQJADQSAFvyGMR8n/QN0isJwNRAAWvCwyBd6d1GA54bGMA9QEQGgBY8wHAXmgfddRQQAx4wxqx8V+XGvLgoIAwGgIgKAe6vv3RsEAOgYKaCKCAAOmdHa/7NfFznZm4sCwrFIYbhqCABu3Sv89nqPLggIDGmgCggAbt1b+28DwJsit3p0XUAoSANVQABwxE7+2trk6Vf7dyIfCP2agAAxAqiAYnCOmLHCbzYS/E+Ruw+LPBr6tQGBoTBcSYwAHDCjDSiHdv4eiMgfivwk1GsCAkYaqCQCgBuFp35tiCz8mci7uScAtIk0UEkEgIbM6ND3wsqfdhTwD0Tezz0BoE0EgJIIAM2tZCd/x70kcvR7rAgCukRhuJIIAM0Vpn+ynhY5QSoI6A6F4cohADRQNPlbhFQQ0DnSQCUQAJqZ2ftP2VTQV0gFAV0hAJTAPoCadPJ3f1r+v8g3RW59TOREwVMA3Lk5SBLqAs1AABijucNj+tFT+pDxf78qcvSXRX4x9wVmsNHi2yK3HicIAG17cpAk+9zlyY5MfKZntEpg9nEss2HE/nuxyhV/c5Tar8z+p98SOfEyu4SBttk00CZ3ebJeBYBMI7+caeArN+5lXK2Y+sm6ISJ/U+TRb4i880GRo7lPAOACK4FmCDYAaKom++hs7a+t9Pl6wxSODQK/K2Ku5J4B4AgTwTMEEQCGownXZX102tgXcVXp88ujFNDdK6SCgDacsW0HheEm8zIAjDX4y2XW2nfpv4gMXH07ggDQKtth3OYWF/MmAGj+fkUb/PO5T/DET0Tu3nDcWBMEgNYsEwAmm2sA0J7+qj686uVPclPktog8PuHp2ggCQCuYB5hiLgFgaMyKNvre9vQneUXkvQlPNUYQAJyjMNwUnZWCsL39oTHrQ2PsxowXQmz8rf8h8rO5Dzpkg8BZTTV1fW1AH1EYbrLWA4DN7Q+N2dSyCZfaWJPfpdc66J2n+wQIAoATpIEmaC0AZBr+10TkQtWaOb660dHPZb/PXxV59I3RnAOA+hgBTOC8FpBO7K7poxeNftbx3EfatTAqO3H7ie6/NdAXFIabwOkIQCd3dzXV07vGfx4ORsujjr/ESACoa1GXmWOMkwCgE7xbOrkbdI7fVxdEjn+BU8WAukgDFWgcAIbGLOsEb5CrekLyByKP/DorhIA6mAgu0CgA2GWdo6XxpHu68m0mh4E6CAAFak0C60Tvhq7uiYovM7E24n5J5PbTTA4DpQySxHCnDqs8AtDGfzvGxt96KveR+TjQeYFnRe7+ucg7nvxYgLc0XY2MSgEg0/gHUbenDU96ln+3O4f/hsjRNzlwHpiFADCmdACg8R/5lIe59xujkckJlooCUxEAxlQZAUTf+MvoFeTlGQppSuiCyDusEgIKsRR0TKkAoCUdom/8rcdETvq85OklkaN2ldD3SAkB4xYoDHfYzAAwNGY11gnfSX7d8x726yLyaZETduMYE8TAA98Q+R1uxwNTA4BGy43cE5F7JpDetd049osiR9kzAIw8JvJRbsUDUwOANv5s8hrzSZEnQrkpr2stoSsiB4wGELvjIidivwdZEwPA0Jg1TtOZ7KLIcOKTHrossmBHA8wNIGbvijxiKAx3X+FOYF3yuU/vf7I7IgcfCfT+fF7k3X8mkjzM0ZOI0HGRi0mS2IUt0Zs0AuhlLX+XjoksPBPocks7N/BRkUe/zWgAcWI/gMqNAOj9l/d9kbf+usjJUH7eIra0xfMiB8f4fSMSx0VuJhwQc0/RCGCVxr+cnxc5+XTgE6u2uqhNZTFJjIgsmlFHN3pFAWAt9xFMdEXkzyc9F5J0kpi0ECIRfRpIxgOArvvnRK8KbOrk8z05qev10Sa3E/bQGYrLoY/+r8j/0cuKPgBIwQhgNfcZmMmuqOlTzsymhU6LnLClpv+MYyjRI//vwYg9+gAgBQFgJfcZmMkup/xyD3vMttT0z4k8wvwA+uL2gzYv+tpmkg0Aemo+6Z+abDnm0CeEixxk5ge+JvJWwacAwfhjkffTn9VwQMyhEUD0N6OpfyvyUF+XT9n5gc+JnFwapYiYH0CQvnW4kxt9m0cAcOgviDzSx1RQVnai+PuMCBCYG4d/3OjbvGwAoE62AzYV1JdVQdPYiWK7CY5AgFC8LTJ8/fDPGn2ts2wAYFLEkd8TeeR0L65kNgIBQvGiyE/Hf9TY5wHuBYAh1fGcuy5yENN2agIBfPdHxaWgo858pCMAAoBjdoPYSxFOlqaBwI6tWTUEX9jqvTeKfxZGAGjHx0ROXIv0NK4bmVVDNhBwUD3m6dK97Tp5z4j8ytCYrWGkqaA0ADAB3JKnRY6HWjbahXT5qC0/bTeUEQjQNbub/cUJZ1/sjj5+XkReGRqzrWegRyMNAFTGa9EVkUdjDgKS2VD2mMijtsQEtYbQlX8ikhxM+F5jaSGbubw6NGY/lkBACqgjNgg8FcWVzmZLTJzWvQQvi7zh+8+LcNmOxpcn9P5Tf3Lv0w5ZjCUQEAA69B9F7sayPLQMO2H8d0SeWBoFyIO3AztnGf77reKVP4e8IvJe7oMjvQ8EBIAO2aJxLxMEcl7X9NApkcEzIvJdRgVw4Asi705Y+XPIt2bXQMsGgl4VzCQAdIwgMN1XReRXGBWgoZdEbtuzr8t8lTJBQtlA8IJOFvdi1VAaAPZzz6A1BIHZsqOCdK6AFUQo4w2R2xfuHf1bjn2tVXxtndVVQ5uhb6IlAMwJQaC8dK7go7qCiBQRJrGN/6crNP6pm/X261ywK0mHxqznngkEKaA5SoNAH88RaMOBriCyKaInRzWXhiwnRcqWKbeN/6Qln9NMmQiexVZ8uaTzA8GlhUySJPf+MjQmyT2Lztie7azlaij2YRH52yLDvyfy08dLrPpA/3xF5NY/bvC7/9VR58KFayKyNkiSOyHc5GwA2OdEsPmyqxbKTlyhWBoMflPknb8s8njhJ6E37FGlvytimnaebCp2J/fR2uwgZHWQJFu+3+dsANimPvb8vVRxAguTLYwOr7n7jMitj4uceJgRVq98T+TW50ROvO7ooloo2nVdA4G3o4HsHMB27ll0ztYOelHkVkylpNuSnTN4bFSOQ/69yJssLQ2bre1jU6afdtj4Wz+4NxfslK0x5PXeAQKAh+ypYt8Uuf3h2G+EY3aPwW+LPG6Xli7pvMt3WV4aFLs35OdGR686H83973YWxSzo3oGNoTHe1VzLpoDsD0fPyCO2YfoNkUe/HfuN6IDNAf+yyPAzIj8mXeQX+z74zyJ3/7nIcZc9/nH/UuSt3xY5mXvCnT1NCe36coPvBwAZBYFdjob0D5PD3SMgzJ9N1V0VeWhDZKHO0s6qnhodG9m2A10ltOnDPR4PAGsi8sXcZ2Hu7BrnZ0ROdPFGQN6HRysk7n5K5PayyJHH2u0pRs3u+n5e5ImvdnwTHK8EmsWL5aLjAcCmRl/NfRa8YDc92eqGFWqXoEW2x/hJkeGnRH58WuRhgkI9NsXz30dlm0/Yg1vm2cnp+Pg+mxJaGSTJ3CoxHAoAwn4A77la94x22KDwpMjdXxD50TmRIydFjn5IZMDtPuyHIm99Q+QD/0HkhE9zXHM4v9XGu+V5zQsUBYD10RGa8BkpobBkA8NfEXnouMj7H4mko2V7+LbWzssiR74ucvKGtno+muMB3hfnMS9QFABsdbvXcp8J79wROfi7IgusEgrXguae7Z+/pOvQ7cjhAyLv/SWRnw1p8tlO2t4Sefs7IkduB9DYF5ljALAuD5Kk08JyuQAgoyCwpZsYEAC7NvryqA1BT6XHiWYDhZWOJrJX7XJkYXvvfyryo/Tfdq38a/r97EEqB1pOuc3lmV2acwCwrg2SpLPTxyYFALtz7YXcE/AWE8RAM3allycL9DtbIVS4802LGLneFo0W2SqYO6PJG6YFgBo82nlvzxnY7mLncGEAUMEechCzZ0UWbojc4qAZoBrPSq+c6SIITAwAOiNNbzJA2dEAEwNAOR/3L+vRehCYGADURu4jCIYdDbwqcvAUvzJgJrv6ysO71GoQKJwETuk33WVjWPjsvoHPOy6fC/TJD0Xuerzsdk83jDmdGJ46AtBvxlxAD9gS0/9N5B0miYG803pGd+4Jf5xpIyMzNQDIg7kAVgT1wAdFjtq00A9ICwGH/H2RWwHckQtDY5zuFi57AEJnGxPQvmMiC7bs7TdFbnHoDCDya/f6R0G4oOV6nCgVAAZJsq3nW6JHPiZywk7wfIkjKBGx09opCugOXBoa46RTPnUSOEtrBO0KJQd6yW75/0ORn1BSArH5ishbnwmvlLeTKqKlA4BwYEwUbIG5SyIPU24aMVgIu/KlnZtdarIyqNIhyIMk2ej20Bx0zQ6Fr4g8aieKn+GwdPTcWtir4uzy/K3cRyuoNAIQUkHRoeQ0+sqj4m9N1S4jXWkEIKNRwD57A+KRrhj6ryJvsXQUffIHYSz9LMNOCi/X+Y+VRwApzgyIE3ME6APbmXmxX7/JWvMBlUcAGasUi4sPcwQInc1dP9+/tmuxTmam9ghARqMAO+x4JfcEomFHBF8Sef+qyIDeAELwosgtWxqlp7+sc7pvq5RGAUA4RB4q3UdwVWSBgnPw1edF3v09kUd6/AuqlApqHACE+QCMeVnkjX8t8gQrh+CTp0XeuSZyNIJfSulVQa4CgC0bva0V64B7vi/y1r8R+RATxpi306OOic/lnl17UldsTuUkAMgoCCxpEGB/AA4hPYR5irDxt3YGSTJzaaizACBMCqOE74q8cUXkia9ys9CBSBv/1MwJYacBQEZBwC4PvZp7Ash4W2T4oshPf59TytASu0z590WSD8aR9y+yN0iSpYKP3+c8AMgoCGyMjqQFZkvnCl4UeZSlpHAhgtU+ZV3UQ70KtRIAZBQE7De9kHsCmMKuIHqeFBFqshOQX+73Ov+qbg6S5NSk/9NaABCWh6IBO3H8DZG3/5XIyRvcSJRwenRq1UFgh7t0YeJcQNsBgOWhaCydL/gjkRMEAxS5JHLwLA3/JBNXBLUaAIQgAMdsMPiayDtfFXmcNBGe0ro+9PpnKtwX0HoAEIIAWmLTRDsit57XXcdMIMfjw1rOmVx/adcGSZI7R7iTACAEAXTA7jH4msjP/CeRAUtL+8l28/+FyK3P0fBXdTBIkmPj/6ezACAEAXQomypidBA+2+O/KHLwj0QejnRTlwu5JaGdBgAhCGBO/kTkzZdFjnxd5CRF6sJhV/b8U5G3PiNyMvZ74cD1QZKsZL9M5wFACALwAAHBb3YX7z8UefvnafhdG2RLRc8lAAhBAJ75gcjN74gcsSkju9SUOYTu2RU9vyPyxlmRE6R5WvPZQZJspV98bgFAHgSBTTaLwTd2DuF/ifz4OyIPf103ozGP4N6vjh5vfkbk6IdEBn27Pg8dWg001wCQomwEQjAeFF5npFCZncw9K3L3GZFbH6enPw+HSkN4EQCEIICApemj2zqnQGB4IG3wf03k1l8T+Rl6+V64vynMmwAglJJGz9jAcFvkoT8Wef+HIh/6ru5P6GtwOK0N/i+J3DwncmRR5Dg9fC/dXw7qVQCQB0Fgg5PF0Gd2F/OfivzopyJHXhF5z17qt0QW03kGX1cmndY35idFho+JvG0b+r8ocuQxVuuE5MogSdbExwAgHC8JHGJHEum/0xFF9vl0dNH0rv0tkbeOiLyb/ts27h/Q4PQRkcXcf0Co7heH8zIAyCgI2ImKLZaJAoBT98tCPOTrfdVJiuVRiW8AgCP3MyveBgAZBYE7unX5cu5JAEAtQ2PupYC8DgCpQZKsj2pBsRcHAFwJIgDIKAhsakroZu5JAEAV9zaDBRMAZBQEdkVkiXkBAGgkvAAgzAsAgDPBBYCUzgucY14AAOoJNgDIKAhs61BmJ/ckAGCqoAOAPEgJLZMSAoBqgg8AqUxKiFVCAFBCbwKAPEgJsUoIAKa7Vw66VwFADq8SYuMYABTrZwBI6caxJSaIAaBYbwOAaEE5JogBIOeO+FwO2jU9Y2CT8tIAYjdIEiN9HwFk2TISgyRZYjQAIHL350ajCQApXS76CRHZyz0JAP23m15hdAFAGA0AiNt+evVRBoBUZjTASiEAsSAApHQ0YFcKPce+AQAR2E4vMZpVQGUMjTmmK4XO+//TAkAtA7thVggAxfS8TBsIFgs/AQDCdHOQJKfSnzz6FFCRTE0hJokB9Ml29loIABNoTSE7Sfwkk8QAeuJQACAFVNLQGFtgboO0EICA3c//CyOA8gZJspVJC7FaCEBo9rKNvxAAqsmkhWwguBbSzw4gepvjN4AUUAO6WsgGhLPBXgSAWDxpKyRnr5UA4MDQmFUNBMwPAPDRnpa/OYQUkAOZw2eYHwDgo1z6RxgBuKe7ie1o4Nm+XRuAYA3GJ4CFEYB7OlG8pvsHmCgGMG/Xihp/IQC0R4+jtHMD59hIBmCOCtM/QgqoO6wYAjAHO1rtuBAjgI7Y+kL6izjHaWQAOjKx9y+MAOaHpaMAWnao8mcRRgBzYpeO6i/nov1FRXkTALRpbdbXZgTgCUYEAByamvtPEQA8o4FglcliAA2c03NNpiIAeIpVQwBquqZL0GciAHiOQACgAluK5tSkjV/jmAT2XGb56CfYWQxghtWyjb8wAgjP0JhTOiKwJ5QtxH4/ANx3fZAkK1VuBwEgUFp0bk0fBAIgbpVSPykCQA+whBSIXqlVP+OYA+iBzKYyW2bieuz3A4jM5TqNvzAC6CfmCYBoVM77ZxEAekznCVZ1noD0ENAvtqjkctW8fxYBIBK6n8AGgvOx3wugB+yk79L4Ie9VEQAio+mhtNwEowIgPAfa899t+pMTACJG3SEgOM4afyEAQB6MCtZ00phRAeAnp42/EAAwbmjMio4KmCsA/OG88RcCACbRUcEKK4iAuWul8RcCAMoYGrOUSRGxrwDojj0tcKWNxl8IAKhKJ45XSBEBrWu8zn8WAgBqyWwys48z3EXAqdKHujRBAEBjzBcATl209b26uKUEADil8wWrLCkFKtvTA11ayfcXIQCgNQQDoLQrtoBjm/n+IgQAdIJgABQ60F7/VtGTbSMAoHMaDFb0wQQyYjWXXn8WAQBzlZlAXqEmESJhc/1rdQ9xcYkAAG/o0tI0GCyz6Qw9c6A9/g1fLosAAG9pXaJl5g0QONvw20Z/Y57pniIEAAQhkypaZhcyAnJNe/2NDm5pCwEAQdITztKAwEQyfON1w58iACB4mbmDZX2QLsI8eJvqmYQAgN7RdNFyZv6AyWS0yVbsXBeRrVAa/hQBAL2n+w6WGCHAMZvm2fRhOWddBABEZ2yEsMQcAirY0zRPcL39IgQARE/nEJbGggJpI6Rso7+pjb7Xk7pVEQCAApm0Ufpgl3JcetvoZxEAgJJ06WkaEE4RFHrFruCxBdm2+5LeKYMAADSQGSmcIn0UFNvgb6ePLmvw+4QAADiWmVNIA8MSgWHubmpjvxtzgz+OAAB0JBMYTmUCwzFSSc7Z/P1+2tjbP2NJ6VRFAAA8UBAcsg/2LRTbEZE72tDbxz49+2oIAEAAMgFCdK5BMgFCejqK2NE/dzMN/R169O4QAICe0dVKqTTNlFouuNq2g8fO2L/Txjy1rw+hce8WAQAAIvUQv3gAiBMBAAAiRQAAgEgRAAAgUgQAAIgUAQAAIkUAAIBIEQAAIFIEAACIFAEAACJFAACASBEAACBSBAAAiBQBAAAiRQAAgEgRAAAgUgQAAIgUAQAAIkUAAIBIEQAAIFIEAACIFAEAACJFAACASBEAACBSBAAAiBQBAAAiRQAAgEgRAAAgUgQAAIgUAQAAIkUAAIBIHeEXjz4yxhwTkSURWdaH/ftukiTL/MKBEQIAesEYs5Rp6O3jDL9ZYDoCAIJjjDmV6d3bP8/yWwSqaz0AGGO2PXqD7uif+zYdoCmB7dxnwVvGGNvov+LTz2eMWReRS7knyrucJMk6P2d/6Ig020lZ87GtiW0EcDbz5wUZ/aIORMT+YjaTJNnK/Q8AmELnm5bHGvyFyf/DH6SARr+o8/ZhjLlpA4GIbCRJcif3mQCip6PQbGO/GOo9IQActqhD5FVjzBojAgApY8xmmjnoC/YBFLOB4AVjzJYO7wDgVN/uAAFgOpsa2tYJHQDoFQLAbGcIAgD6iABQzoIGAdJBAHqDAFDegi4XBYBeIABUc0Y30wBA8AgA1V3SUgQAEDQCQD2MAgAEjwBQzwVGAQBCF8JO4D0RaVqWoY3aHGv6AIAghRAAnFTR03X8qyLybO7Jela6CgBae8TaT5JkP/cJQI/oe7VoyfWdJEl2cx9FbdHUAtIXzpoxZkNEthwcGLJo00CuGuRMRcHsoSa5UYsxJv3rjpa1tsFx2+fAUFAtUQqu70BLdN/RP7e1XDdF+XooUy75lL4mjpV5T469/nfT10qT17+mc8ukdIuCUllLmZ+9rNYDXnTF4OwLRXvUuw6q+C1pI1ybMWZVRxPnK36Ns2Nlrfe0kummD42mNvqr+igTbBcy5brPp3XrjTHXbcBOkmQz9z8QDG1kVzKdnFznpqKz2XNG9PW/oa+Vqq//1YbnJJTxxRr/ZyfTYWpFlJPA+gJZzT1RXe3yELbhN8bY4HG1RuNf5Iy+yGyAW5/XrmX7RteqiUP9eZqOtOy9uWqMuaN7MNiNHQh9LWzo6/w1fT2cb6lW/hl9L+2zV6e8aFcB6bzCzdwT1VReCWSMWck0/G3UEV/Q3ox9I6zknm2JDTiaXnutpZK56XW9kHsG3tDXge3c7Opr4dmO6+Uv6F6dfep3zRb7MtCm9f5LBwB9Y2xpA9bFG2JBS1pvtj0a0ECz73CCHYHJjPzSzs28D+W377FXNcWKCWI/EKaTXLn2RLbmdHLQBZ2AWmljolh7/TT82Pb0ZCybPhTmkIrFPg+w+C4AAAkSSURBVAJonTb+835z2N7YrsshcWZEQ+MP8Xx3/NUu06EhiT0ANN3NO7VHnWn8fTgg2llJa/0a244mr9ED2sM+8PhKNtm9nxdtANBGrGmvYGIA8KzxT7kKApse5Hjhnw2PfycLnv98cxHzHMCag8a5cJOGNrBbnjX+qTP6Rqg1OaY5f3r+bi13tHSx1TXl+rpy8b5qy3m7B8hFZYG+iDIA6MoAFxs/CgOA9pB9nBBL2WJ2dsNMpVVQmkcl5+/eoU1NobL7a7SD0PamqiZWOdjpgehSQNrTupp7orq9olU12kiG0EPeqJIK0s9lJQVm2fB8LoDJ4IzejwAyhaVW9OGqZz6p9+wqz3g9rYcz9vFlvY7GtYx0uF429bDe4tB+J/P34HvCMas5CriZqesjmR56+t5dclQ+wlrQJdFN9wD1QggB4JUaRZS6kOsNG2PWHASY61oBNTe6UPbNsa71jDYaBoJ7xfFm1U7R1RMuUz9p3aKtouscK4y36nFOGcVmzQUcaAdqSwu5TXr9HUrVaOp23VENr2wA2CyZFmryfnuuoDM3y6T74k6SJK0+9MYmPXtsFt0zXRXU5FpXi77utIe+eIu+VtnH2rSv7+h7pA/7Blgu+h7THhoEmt7bRBubqd+rzkMbpaLv17fHeoXf2fg9uaOvo5Wiz6/wdY/p66jz10HDtqzy676LR+z7AOo4KEqbaO+kSc/kYp3dikmS2O97LfdEeVPPNHC0XNa6kiTJUp0VGHpflhpeJ7qVzgXY9N5Fu+fGvlabpl50tLDsoI5X9ISNYLWsF6UtGjaSV5psVdcgsJN7opzFGTuEV6YM5cuywW1qoClxjXf0Op/LPQnvaENtA77t+TotUa5fK9cJq4B5JkUAqOZ6kiS5SV7tJddd+VM4oqhhtcHqi2nBa9pzZTQKbgU4ESoQEzpKrrCU0wECQHl7UzZPNdlgM3MStgx9s9VtaKc18k2WtO417fkDRTga1Y3Yq4GWtaeTOJMa6mkplFlOOdwFWrfEQ+HKhsxZxHXR+AMeIwDMtqMrFyY1/tJwBNDG4SmVTdgi3ySw7RV8PSB7Bm/6vjk24bWWvn7Sc6L36fm7RQCY7nKSJGV6532oMlh0DU2KxrFrGPdkVpItV1xUkJus1T1BO8wFuUEAKLaj6+PLvsh8rvtTVlEAaDKy4Q0aOe3prztaSZbVi9pJPiAAHHag6Z4YUxdOj40k/RMv7fGvUzjQfwSAw2IuOVCUg0U3rnWUMltte85J95RwXkQgCAB5GzSG6Nh+FyMmB6u6pvL0ECRMEcI+gHNJkpiyDwdbxM9oWQc04OLoSYQjc0wojX9A+rgRzEXj7fMB16FgFBUXX0/AwxS9CwA6lK5bFye1WHFzVl8LUzVJS0zbXYwe0UOQWJUToL6WgnDRg1+rkMbo6+aUJtdFAIgHO74D1ctJYDsKMMbsNOyVLGggKfPi3m7wvXY8STkV7XRuspbfjqJWHReCg2d0rX+T99nNzOEwd7J7b7QDlqYSlzM7hhltONLnVUC2UX0l99FqntUTs2b1hLcbHIR91tct7vbNaIw5aJDb3dDD54uCC/qh7sqiA91sObGDoK+bNA05fjpYkvsPqKy31UAdzQVImd65fq8mB2FPfBN4oMk8wMKUs5PRD0U7yMtYmdb4oxt9LwftIrVyoeT66SYN3VljjK9vhqYNuLNr05QAS3T9UmcEcD3CneKt7sGoq9cBQF9kLo4RLBNImjZyF1w0lHYzjjFm36auck/WoL20JqMb0Wvb1nxxLZlNRl5UT0UjMdaJIgDMiYtRwNlZowANNnu5J6qxDeXujCMaJzLGrGkjuajzF65GFS6CiZ3rsNe2XmWTmA0aeh2vUl6gN/pQPbeqsz5uMO19ANDJVRejgDKNqYtgYxu5V22jVyb1ZBtT+8KyvX4R+eLYhG3a8266K9fFKED0Z7OT5ft6favjwU4b/GUNFLan+Bq9/t5ZiXSn+FV9XU/s4OkIflXfHxM/z5VYagGtO2hEZi5rTJJky8Hy09QFbcAPdMg8njM9pUviZvWK7c+yrQe+1FqNY4OoppTqrnQat5Benzyo8Y4w1VkCvVD3NZkpMT0Puw7e2/Y9dElf83uZ5ddLBavtJrY1rkQRALQBu+YgCJRZ1rim6QpXFhzUPz+j6ZeVCmccjNvQCdg+nH0Ad+ouX05fk7Yxn/qe0kY/PUymyRnVTbleqj33lGZMh8K76DUszNoYpg3sc7kn5m9Re121hpX6BmUFDsY1WSVmX5NXRWSoc1/29bmlj2193NE04NU5N/5SMAoPXjQBwOFcwMwSEUmS2N7y9dwT85cOvWs15DrRfTn3BKKlHQMXr/UzOso9r4901OtNgTnt3PWq7ldMIwDR3nvTycyFkqtiVh2sCmrDgk5G1Q0C644CKfrDyZLjQPRq81pUAUB7Ky5erBdmrWnX77XsaRCQhkHA1+CGOXC46z4ErlbEeSG2EYBoAHDxC5zZE/A8COw0zN8uMxJAxmqfGsZJNJXcmxFPdAHA4Shg5uaw9PslSbLkWWN5OUmS2stC5cF12Tf9ldyTiI42jFGUANc0aC9GPDGOAMThKKD0yiJtLD87516SHYl8Ql/ATiRJsubBdcEDmgq6GMlrYaUPadAoA4DjUUDpPLrdKKYbuLruNds35HN2JNJgH8BEHV7XjjYw8JRulFzuYLXMXBvfTHo36DRorCMAmccoQB6kTmyv+ckOXjw3ddnmKV2a2pqWr8s2/Ods2qrHp6/1hnYylvS153o0YF/TFzWtOleZNOhnQ10eGm0A0AjuIhVS9fzg9Pvv64tnoBvHXPVoDrQB/mySJLbhX+/yQJaC66qbK02v4xM6XxFb+eCgaeO4riPDyw4ayOva8J+aVo5lHuwI2P5cGgiuOwh6e/rab72zY5Kk3YN1dOfp1I1TM+y22YCVrPU/yx0XqRXdYLasj6UJ9UHG7Wg9kXv1gnxsKDNH+y1rgzBpCe2uvui3J93PsWMCq3Lyeyr4maZdUxmdnAg3759T24LlzPGOk8qbpDVytjOv61wb0OS92/b7RH+2pUzNrkn2s4+u37+tBwAAgJ9ingMAgHiJyP8HG/WoYMpqf/kAAAAASUVORK5CYII=","mediatype":"image/png"}],"installModes":[{"type":"OwnNamespace","supported":true},{"type":"SingleNamespace","supported":true},{"type":"MultiNamespace","supported":false},{"type":"AllNamespaces","supported":true}]},"status":{"lastUpdateTime":null,"lastTransitionTime":null,"certsLastUpdated":null,"certsRotateAt":null}}
        name: nfd.4.2.36-202006230600.p0
        sourceName: redhat-operators
        sourceNamespace: openshift-marketplace
        version: v1alpha1
      status: Unknown
    - resolving: nfd.4.2.36-202006230600.p0
      resource:
        group: apiextensions.k8s.io
        kind: CustomResourceDefinition
        manifest: >
          {"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"name":"nodefeaturediscoveries.nfd.openshift.io"},"spec":{"group":"nfd.openshift.io","names":{"kind":"NodeFeatureDiscovery","listKind":"NodeFeatureDiscoveryList","plural":"nodefeaturediscoveries","singular":"nodefeaturediscovery"},"scope":"Namespaced","subresources":{"status":{}},"version":"v1alpha1"}}
        name: nodefeaturediscoveries.nfd.openshift.io
        sourceName: redhat-operators
        sourceNamespace: openshift-marketplace
        version: v1beta1
      status: Unknown
    - resolving: nfd.4.2.36-202006230600.p0
      resource:
        group: ''
        kind: ServiceAccount
        manifest: >
          {"metadata":{"name":"nfd-operator","creationTimestamp":null,"ownerReferences":[{"apiVersion":"operators.coreos.com/v1alpha1","kind":"ClusterServiceVersion","name":"nfd.4.2.36-202006230600.p0","uid":"","controller":false,"blockOwnerDeletion":false}]}}
        name: nfd-operator
        sourceName: redhat-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: nfd.4.2.36-202006230600.p0
      resource:
        group: rbac.authorization.k8s.io
        kind: ClusterRole
        manifest: >
          {"metadata":{"name":"nfd.4.2.36-202006230600.p0-tjttw","creationTimestamp":null,"labels":{"olm.owner":"nfd.4.2.36-202006230600.p0","olm.owner.kind":"ClusterServiceVersion","olm.owner.namespace":"gpu-operator-resources"}},"rules":[{"verbs":["*"],"apiGroups":["rbac.authorization.k8s.io"],"resources":["roles","rolebindings","clusterroles","clusterrolebindings"]},{"verbs":["*"],"apiGroups":[""],"resources":["pods","services","endpoints","persistentvolumeclaims","events","configmaps","secrets","serviceaccounts","nodes"]},{"verbs":["*"],"apiGroups":[""],"resources":["namespaces"]},{"verbs":["*"],"apiGroups":["apps"],"resources":["deployments","daemonsets","replicasets","statefulsets"]},{"verbs":["get","create"],"apiGroups":["monitoring.coreos.com"],"resources":["servicemonitors"]},{"verbs":["*"],"apiGroups":["nfd.openshift.io"],"resources":["*"]},{"verbs":["use"],"apiGroups":["policy"],"resources":["podsecuritypolicies"],"resourceNames":["nfd-worker"]},{"verbs":["*"],"apiGroups":["security.openshift.io"],"resources":["securitycontextconstraints"]}]}
        name: nfd.4.2.36-202006230600.p0-tjttw
        sourceName: redhat-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: nfd.4.2.36-202006230600.p0
      resource:
        group: rbac.authorization.k8s.io
        kind: ClusterRoleBinding
        manifest: >
          {"metadata":{"name":"nfd.4.2.36-202006230600.p0-tjttw-nfd-operator-5dhq5","namespace":"gpu-operator-resources","creationTimestamp":null,"labels":{"olm.owner":"nfd.4.2.36-202006230600.p0","olm.owner.kind":"ClusterServiceVersion","olm.owner.namespace":"gpu-operator-resources"}},"subjects":[{"kind":"ServiceAccount","name":"nfd-operator","namespace":"gpu-operator-resources"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"nfd.4.2.36-202006230600.p0-tjttw"}}
        name: nfd.4.2.36-202006230600.p0-tjttw-nfd-operator-5dhq5
        sourceName: redhat-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Unknown
    - resolving: nfd.4.2.36-202006230600.p0
      resource:
        group: operators.coreos.com
        kind: Subscription
        manifest: >
          {"metadata":{"name":"nfd-4.2-redhat-operators-openshift-marketplace","namespace":"gpu-operator-resources","creationTimestamp":null},"spec":{"source":"redhat-operators","sourceNamespace":"openshift-marketplace","name":"nfd","channel":"4.2","startingCSV":"nfd.4.2.36-202006230600.p0","installPlanApproval":"Automatic","config":{"resources":{}}},"status":{"lastUpdated":null}}
        name: nfd-4.2-redhat-operators-openshift-marketplace
        sourceName: redhat-operators
        sourceNamespace: openshift-marketplace
        version: v1alpha1
      status: Unknown

I'm not sure what error creating csv gpu-operator-certified.v1.1.7-r2 really means.

I've attached my pods and ds info. pods.txt ds.txt

Warren-Xia commented 4 years ago

I have same problem, and my openshift is 4.2.21. Do you have solved it?

deathemperor commented 4 years ago

I have same problem, and my openshift is 4.2.21. Do you have solved it?

@Warren-Xia no I haven't. still trying to find a fix myself.

adamburrows commented 4 years ago

Also experiencing this issue on OCP 4.2

deathemperor commented 4 years ago

I gave up on this and OCP4. I didn't quit software for woodwork but I did switch to VMWare for cluster GPU.

shivamerla commented 3 years ago

Please try with latest operator available on RH catalog. 1.3.1.